v12.1.4 Luminous (RC) released

v12.1.4 Luminous (RC) released

[Abhishek]

This is the fifth release candidate for Luminous, the next long term
stable release. We’ve had to do this release as there was a bug in
the previous RC, which affected upgrades to Luminous.[1]

Please note that this is still a *release candidate* and
not the final release, we're expecting the final Luminous release in
a week's time, meanwhile, testing and feedback is very much welcom.

Ceph Luminous (v12.2.0) will be the foundation for the next long-term
stable release series. There have been major changes since Kraken
(v11.2.z) and Jewel (v10.2.z), and the upgrade process is non-trivial.
Please read these release notes carefully. Full details and changelog at
http://ceph.com/releases/v12-1-4-luminous-rc-released/

Notable Changes from 12.1.3
---------------------------
* core: Wip 20985 divergent handling luminous (issue#20985, pr#17001, 
Greg Farnum)
* qa/tasks/thrashosds-health.yaml: ignore MON_DOWN (issue#20910, 
pr#17003, Sage Weil)
* crush, mon: fix weight set vs crush device classes (issue#20939, Sage 
Weil)


Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-12.1.4.tar.gz
* For packages, see 
http://docs.ceph.com/docs/master/install/get-packages/
* For ceph-deploy, see 
http://docs.ceph.com/docs/master/install/install-ceph-deploy
* Release sha1: a5f84b37668fc8e03165aaf5cbb380c78e4deba4

[1]: http://tracker.ceph.com/issues/20985


Best Regards
Abhishek

Re: [ceph-users] v12.1.4 Luminous (RC) released

[Gregory Farnum]

On Tue, Aug 15, 2017 at 2:05 PM, Abhishek <abhishek@suse.com> wrote:
> This is the fifth release candidate for Luminous, the next long term
> stable release. We’ve had to do this release as there was a bug in
> the previous RC, which affected upgrades to Luminous.[1]

In particular, this will fix things for those of you who upgraded from
Jewel or a previous RC and saw OSDs crash instantly on boot. We had an
oversight in dealing with another bug. (Standard disclaimer: this was
a logic error that resulted in no data changes. There were no
durability implications — not that that helps much when you can't read
your data out again.)

Sorry guys!
-Greg

>
> Please note that this is still a *release candidate* and
> not the final release, we're expecting the final Luminous release in
> a week's time, meanwhile, testing and feedback is very much welcom.
>
> Ceph Luminous (v12.2.0) will be the foundation for the next long-term
> stable release series. There have been major changes since Kraken
> (v11.2.z) and Jewel (v10.2.z), and the upgrade process is non-trivial.
> Please read these release notes carefully. Full details and changelog at
> http://ceph.com/releases/v12-1-4-luminous-rc-released/
>
> Notable Changes from 12.1.3
> ---------------------------
> * core: Wip 20985 divergent handling luminous (issue#20985, pr#17001, Greg
> Farnum)
> * qa/tasks/thrashosds-health.yaml: ignore MON_DOWN (issue#20910, pr#17003,
> Sage Weil)
> * crush, mon: fix weight set vs crush device classes (issue#20939, Sage
> Weil)
>
>
> Getting Ceph
> ------------
> * Git at git://github.com/ceph/ceph.git
> * Tarball at http://download.ceph.com/tarballs/ceph-12.1.4.tar.gz
> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/
> * For ceph-deploy, see
> http://docs.ceph.com/docs/master/install/install-ceph-deploy
> * Release sha1: a5f84b37668fc8e03165aaf5cbb380c78e4deba4
>
> [1]: http://tracker.ceph.com/issues/20985
>
>
> Best Regards
> Abhishek
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v12.1.4 Luminous (RC) released

[Matt Benjamin]

I think we need a v12.1.5 including #17040

Matt

On Tue, Aug 15, 2017 at 5:16 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
> On Tue, Aug 15, 2017 at 2:05 PM, Abhishek <abhishek@suse.com> wrote:
>> This is the fifth release candidate for Luminous, the next long term
>> stable release. We’ve had to do this release as there was a bug in
>> the previous RC, which affected upgrades to Luminous.[1]
>
> In particular, this will fix things for those of you who upgraded from
> Jewel or a previous RC and saw OSDs crash instantly on boot. We had an
> oversight in dealing with another bug. (Standard disclaimer: this was
> a logic error that resulted in no data changes. There were no
> durability implications — not that that helps much when you can't read
> your data out again.)
>
> Sorry guys!
> -Greg
>
>>
>> Please note that this is still a *release candidate* and
>> not the final release, we're expecting the final Luminous release in
>> a week's time, meanwhile, testing and feedback is very much welcom.
>>
>> Ceph Luminous (v12.2.0) will be the foundation for the next long-term
>> stable release series. There have been major changes since Kraken
>> (v11.2.z) and Jewel (v10.2.z), and the upgrade process is non-trivial.
>> Please read these release notes carefully. Full details and changelog at
>> http://ceph.com/releases/v12-1-4-luminous-rc-released/
>>
>> Notable Changes from 12.1.3
>> ---------------------------
>> * core: Wip 20985 divergent handling luminous (issue#20985, pr#17001, Greg
>> Farnum)
>> * qa/tasks/thrashosds-health.yaml: ignore MON_DOWN (issue#20910, pr#17003,
>> Sage Weil)
>> * crush, mon: fix weight set vs crush device classes (issue#20939, Sage
>> Weil)
>>
>>
>> Getting Ceph
>> ------------
>> * Git at git://github.com/ceph/ceph.git
>> * Tarball at http://download.ceph.com/tarballs/ceph-12.1.4.tar.gz
>> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/
>> * For ceph-deploy, see
>> http://docs.ceph.com/docs/master/install/install-ceph-deploy
>> * Release sha1: a5f84b37668fc8e03165aaf5cbb380c78e4deba4
>>
>> [1]: http://tracker.ceph.com/issues/20985
>>
>>
>> Best Regards
>> Abhishek
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: v12.1.4 Luminous (RC) released

[Alfredo Deza]

On Tue, Aug 15, 2017 at 10:35 PM, Matt Benjamin <mbenjami@redhat.com> wrote:
> I think we need a v12.1.5 including #17040

*I* think that this is getting to a point where we should just have
nightly development releases.

What is the benefit of waiting for each RC every two weeks (or so) otherwise?

On one side we are treating the RC releases somewhat like normal
releases, with proper announcements, waiting for
QA suites to complete, and have leads "ack" when their components are
good enough. But on the other side of things
we then try to cut releases to include fixes as immediate as possible
(and as often as that means)

We've had 3 releases already in August and this would mean discussing
a *fourth*.

>
> Matt
>
> On Tue, Aug 15, 2017 at 5:16 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>> On Tue, Aug 15, 2017 at 2:05 PM, Abhishek <abhishek@suse.com> wrote:
>>> This is the fifth release candidate for Luminous, the next long term
>>> stable release. We’ve had to do this release as there was a bug in
>>> the previous RC, which affected upgrades to Luminous.[1]
>>
>> In particular, this will fix things for those of you who upgraded from
>> Jewel or a previous RC and saw OSDs crash instantly on boot. We had an
>> oversight in dealing with another bug. (Standard disclaimer: this was
>> a logic error that resulted in no data changes. There were no
>> durability implications — not that that helps much when you can't read
>> your data out again.)
>>
>> Sorry guys!
>> -Greg
>>
>>>
>>> Please note that this is still a *release candidate* and
>>> not the final release, we're expecting the final Luminous release in
>>> a week's time, meanwhile, testing and feedback is very much welcom.
>>>
>>> Ceph Luminous (v12.2.0) will be the foundation for the next long-term
>>> stable release series. There have been major changes since Kraken
>>> (v11.2.z) and Jewel (v10.2.z), and the upgrade process is non-trivial.
>>> Please read these release notes carefully. Full details and changelog at
>>> http://ceph.com/releases/v12-1-4-luminous-rc-released/
>>>
>>> Notable Changes from 12.1.3
>>> ---------------------------
>>> * core: Wip 20985 divergent handling luminous (issue#20985, pr#17001, Greg
>>> Farnum)
>>> * qa/tasks/thrashosds-health.yaml: ignore MON_DOWN (issue#20910, pr#17003,
>>> Sage Weil)
>>> * crush, mon: fix weight set vs crush device classes (issue#20939, Sage
>>> Weil)
>>>
>>>
>>> Getting Ceph
>>> ------------
>>> * Git at git://github.com/ceph/ceph.git
>>> * Tarball at http://download.ceph.com/tarballs/ceph-12.1.4.tar.gz
>>> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/
>>> * For ceph-deploy, see
>>> http://docs.ceph.com/docs/master/install/install-ceph-deploy
>>> * Release sha1: a5f84b37668fc8e03165aaf5cbb380c78e4deba4
>>>
>>> [1]: http://tracker.ceph.com/issues/20985
>>>
>>>
>>> Best Regards
>>> Abhishek
>>>
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v12.1.4 Luminous (RC) released

[Abhishek Lekshmanan]

Alfredo Deza <adeza@redhat.com> writes:

> On Tue, Aug 15, 2017 at 10:35 PM, Matt Benjamin <mbenjami@redhat.com> wrote:
>> I think we need a v12.1.5 including #17040
We discussed this in the RGW Standups today & may not need one more RC
for the bug above, and should be fine as far as the fix is in 12.2.0

>
> *I* think that this is getting to a point where we should just have
> nightly development releases.
>
> What is the benefit of waiting for each RC every two weeks (or so) otherwise?

We could consider something of this sort for M maybe?

Abhishek
> On one side we are treating the RC releases somewhat like normal
> releases, with proper announcements, waiting for
> QA suites to complete, and have leads "ack" when their components are
> good enough. But on the other side of things
> we then try to cut releases to include fixes as immediate as possible
> (and as often as that means)
>
> We've had 3 releases already in August and this would mean discussing
> a *fourth*. 
>
>>
>> Matt
>>
>> On Tue, Aug 15, 2017 at 5:16 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>>> On Tue, Aug 15, 2017 at 2:05 PM, Abhishek <abhishek@suse.com> wrote:
>>>> This is the fifth release candidate for Luminous, the next long term
>>>> stable release. We’ve had to do this release as there was a bug in
>>>> the previous RC, which affected upgrades to Luminous.[1]
>>>
>>> In particular, this will fix things for those of you who upgraded from
>>> Jewel or a previous RC and saw OSDs crash instantly on boot. We had an
>>> oversight in dealing with another bug. (Standard disclaimer: this was
>>> a logic error that resulted in no data changes. There were no
>>> durability implications — not that that helps much when you can't read
>>> your data out again.)
>>>
>>> Sorry guys!
>>> -Greg
>>>
>>>>
>>>> Please note that this is still a *release candidate* and
>>>> not the final release, we're expecting the final Luminous release in
>>>> a week's time, meanwhile, testing and feedback is very much welcom.
>>>>
>>>> Ceph Luminous (v12.2.0) will be the foundation for the next long-term
>>>> stable release series. There have been major changes since Kraken
>>>> (v11.2.z) and Jewel (v10.2.z), and the upgrade process is non-trivial.
>>>> Please read these release notes carefully. Full details and changelog at
>>>> http://ceph.com/releases/v12-1-4-luminous-rc-released/
>>>>
>>>> Notable Changes from 12.1.3
>>>> ---------------------------
>>>> * core: Wip 20985 divergent handling luminous (issue#20985, pr#17001, Greg
>>>> Farnum)
>>>> * qa/tasks/thrashosds-health.yaml: ignore MON_DOWN (issue#20910, pr#17003,
>>>> Sage Weil)
>>>> * crush, mon: fix weight set vs crush device classes (issue#20939, Sage
>>>> Weil)
>>>>
>>>>
>>>> Getting Ceph
>>>> ------------
>>>> * Git at git://github.com/ceph/ceph.git
>>>> * Tarball at http://download.ceph.com/tarballs/ceph-12.1.4.tar.gz
>>>> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/
>>>> * For ceph-deploy, see
>>>> http://docs.ceph.com/docs/master/install/install-ceph-deploy
>>>> * Release sha1: a5f84b37668fc8e03165aaf5cbb380c78e4deba4
>>>>
>>>> [1]: http://tracker.ceph.com/issues/20985
>>>>
>>>>
>>>> Best Regards
>>>> Abhishek
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] v12.1.4 Luminous (RC) released

[Gregory Farnum]

<Reducing this to ceph-devel/ceph-maintainers, this discussion doesn't
need to blast all over>

On Wed, Aug 16, 2017 at 5:13 AM, Alfredo Deza <adeza@redhat.com> wrote:
> On Tue, Aug 15, 2017 at 10:35 PM, Matt Benjamin <mbenjami@redhat.com> wrote:
>> I think we need a v12.1.5 including #17040
>
> *I* think that this is getting to a point where we should just have
> nightly development releases.
>
> What is the benefit of waiting for each RC every two weeks (or so) otherwise?

RC releases are supposed to be release candidates, as in, things that
we think could be the release. As with much else in the Luminous cycle
that broke down a bit, and that's frequently not really the case
because we know it's missing features we want to make available, but
in general it's not too far off — we believe the code is stable; it's
passing our nightlies; we want people to test and use it a lot more
rigorously than dev checkpoints. Every RC is practice for the real
thing and making them available the same way as real releases is part
of that.

>
> On one side we are treating the RC releases somewhat like normal
> releases, with proper announcements, waiting for
> QA suites to complete, and have leads "ack" when their components are
> good enough. But on the other side of things
> we then try to cut releases to include fixes as immediate as possible
> (and as often as that means)
>
> We've had 3 releases already in August and this would mean discussing
> a *fourth*.

We had 12.1.2 on August 2 and 12.1.3 on August 9. This last 12.1.4 was
an emergency because we broke things, and that'll happen sometimes no
matter what dev process you're using.

Let's not jump overboard just because we had one emergency and another
idle suggestion that it'd be nice to fix bug X. If your concern is
that these builds take a lot of time, I'm sure we'd all love a
smoother build infrastructure. But build time is not the dominating
issue when deciding how we do releases when we want users to practice
with them. :)
-Greg

>
>>
>> Matt
>>
>> On Tue, Aug 15, 2017 at 5:16 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>>> On Tue, Aug 15, 2017 at 2:05 PM, Abhishek <abhishek@suse.com> wrote:
>>>> This is the fifth release candidate for Luminous, the next long term
>>>> stable release. We’ve had to do this release as there was a bug in
>>>> the previous RC, which affected upgrades to Luminous.[1]
>>>
>>> In particular, this will fix things for those of you who upgraded from
>>> Jewel or a previous RC and saw OSDs crash instantly on boot. We had an
>>> oversight in dealing with another bug. (Standard disclaimer: this was
>>> a logic error that resulted in no data changes. There were no
>>> durability implications — not that that helps much when you can't read
>>> your data out again.)
>>>
>>> Sorry guys!
>>> -Greg
>>>
>>>>
>>>> Please note that this is still a *release candidate* and
>>>> not the final release, we're expecting the final Luminous release in
>>>> a week's time, meanwhile, testing and feedback is very much welcom.
>>>>
>>>> Ceph Luminous (v12.2.0) will be the foundation for the next long-term
>>>> stable release series. There have been major changes since Kraken
>>>> (v11.2.z) and Jewel (v10.2.z), and the upgrade process is non-trivial.
>>>> Please read these release notes carefully. Full details and changelog at
>>>> http://ceph.com/releases/v12-1-4-luminous-rc-released/
>>>>
>>>> Notable Changes from 12.1.3
>>>> ---------------------------
>>>> * core: Wip 20985 divergent handling luminous (issue#20985, pr#17001, Greg
>>>> Farnum)
>>>> * qa/tasks/thrashosds-health.yaml: ignore MON_DOWN (issue#20910, pr#17003,
>>>> Sage Weil)
>>>> * crush, mon: fix weight set vs crush device classes (issue#20939, Sage
>>>> Weil)
>>>>
>>>>
>>>> Getting Ceph
>>>> ------------
>>>> * Git at git://github.com/ceph/ceph.git
>>>> * Tarball at http://download.ceph.com/tarballs/ceph-12.1.4.tar.gz
>>>> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/
>>>> * For ceph-deploy, see
>>>> http://docs.ceph.com/docs/master/install/install-ceph-deploy
>>>> * Release sha1: a5f84b37668fc8e03165aaf5cbb380c78e4deba4
>>>>
>>>> [1]: http://tracker.ceph.com/issues/20985
>>>>
>>>>
>>>> Best Regards
>>>> Abhishek
>>>>
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] v12.1.4 Luminous (RC) released

[Alfredo Deza]

On Wed, Aug 16, 2017 at 11:42 AM, Gregory Farnum <gfarnum@redhat.com> wrote:
> <Reducing this to ceph-devel/ceph-maintainers, this discussion doesn't
> need to blast all over>
>
> On Wed, Aug 16, 2017 at 5:13 AM, Alfredo Deza <adeza@redhat.com> wrote:
>> On Tue, Aug 15, 2017 at 10:35 PM, Matt Benjamin <mbenjami@redhat.com> wrote:
>>> I think we need a v12.1.5 including #17040
>>
>> *I* think that this is getting to a point where we should just have
>> nightly development releases.
>>
>> What is the benefit of waiting for each RC every two weeks (or so) otherwise?
>
> RC releases are supposed to be release candidates, as in, things that
> we think could be the release. As with much else in the Luminous cycle
> that broke down a bit, and that's frequently not really the case
> because we know it's missing features we want to make available, but
> in general it's not too far off — we believe the code is stable; it's
> passing our nightlies; we want people to test and use it a lot more
> rigorously than dev checkpoints. Every RC is practice for the real
> thing and making them available the same way as real releases is part
> of that.
>
>>
>> On one side we are treating the RC releases somewhat like normal
>> releases, with proper announcements, waiting for
>> QA suites to complete, and have leads "ack" when their components are
>> good enough. But on the other side of things
>> we then try to cut releases to include fixes as immediate as possible
>> (and as often as that means)
>>
>> We've had 3 releases already in August and this would mean discussing
>> a *fourth*.
>
> We had 12.1.2 on August 2 and 12.1.3 on August 9. This last 12.1.4 was
> an emergency because we broke things, and that'll happen sometimes no
> matter what dev process you're using.
>

And that is understandable, I agree that even for stable releases that
can happen.

> Let's not jump overboard just because we had one emergency and another
> idle suggestion that it'd be nice to fix bug X. If your concern is
> that these builds take a lot of time, I'm sure we'd all love a
> smoother build infrastructure. But build time is not the dominating
> issue when deciding how we do releases when we want users to practice
> with them. :)

I wouldn't raise this if it was infrequent.

A build time is a concern if it can't be automated and it takes a long
time. Currently releases take about a day if all goes well. If it took
a whole week
but doesn't need any intervention it wouldn't be a problem, but we
can't get there if we want to treat these releases like stable ones,
where other things
need to happen in order to get them out the door.

The dev builds take about 1 hour, and they don't require any
babysitting, and that is what I would envision for these types of
releases.

How would a daily release impact negatively the goals you are
describing for an RC?

> -Greg
>
>>
>>>
>>> Matt
>>>
>>> On Tue, Aug 15, 2017 at 5:16 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
>>>> On Tue, Aug 15, 2017 at 2:05 PM, Abhishek <abhishek@suse.com> wrote:
>>>>> This is the fifth release candidate for Luminous, the next long term
>>>>> stable release. We’ve had to do this release as there was a bug in
>>>>> the previous RC, which affected upgrades to Luminous.[1]
>>>>
>>>> In particular, this will fix things for those of you who upgraded from
>>>> Jewel or a previous RC and saw OSDs crash instantly on boot. We had an
>>>> oversight in dealing with another bug. (Standard disclaimer: this was
>>>> a logic error that resulted in no data changes. There were no
>>>> durability implications — not that that helps much when you can't read
>>>> your data out again.)
>>>>
>>>> Sorry guys!
>>>> -Greg
>>>>
>>>>>
>>>>> Please note that this is still a *release candidate* and
>>>>> not the final release, we're expecting the final Luminous release in
>>>>> a week's time, meanwhile, testing and feedback is very much welcom.
>>>>>
>>>>> Ceph Luminous (v12.2.0) will be the foundation for the next long-term
>>>>> stable release series. There have been major changes since Kraken
>>>>> (v11.2.z) and Jewel (v10.2.z), and the upgrade process is non-trivial.
>>>>> Please read these release notes carefully. Full details and changelog at
>>>>> http://ceph.com/releases/v12-1-4-luminous-rc-released/
>>>>>
>>>>> Notable Changes from 12.1.3
>>>>> ---------------------------
>>>>> * core: Wip 20985 divergent handling luminous (issue#20985, pr#17001, Greg
>>>>> Farnum)
>>>>> * qa/tasks/thrashosds-health.yaml: ignore MON_DOWN (issue#20910, pr#17003,
>>>>> Sage Weil)
>>>>> * crush, mon: fix weight set vs crush device classes (issue#20939, Sage
>>>>> Weil)
>>>>>
>>>>>
>>>>> Getting Ceph
>>>>> ------------
>>>>> * Git at git://github.com/ceph/ceph.git
>>>>> * Tarball at http://download.ceph.com/tarballs/ceph-12.1.4.tar.gz
>>>>> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/
>>>>> * For ceph-deploy, see
>>>>> http://docs.ceph.com/docs/master/install/install-ceph-deploy
>>>>> * Release sha1: a5f84b37668fc8e03165aaf5cbb380c78e4deba4
>>>>>
>>>>> [1]: http://tracker.ceph.com/issues/20985
>>>>>
>>>>>
>>>>> Best Regards
>>>>> Abhishek
>>>>>
>>>>> _______________________________________________
>>>>> ceph-users mailing list
>>>>> ceph-users@lists.ceph.com
>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>> _______________________________________________
>>>> ceph-users mailing list
>>>> ceph-users@lists.ceph.com
>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

After upgrading all the ceph nodes to v12.1.4 I see that OSD versions are still shown as older version.
# ceph versions
{
    "mon": {
        "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
    },
    "mgr": {
        "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
    },
    "osd": {
        "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50
    },
    "mds": {
        "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
    },
    "overall": {
        "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50,
        "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 3
    }
}


# ceph-osd --version
ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)

Why is it showing old 12.1.0 version for the 50 osds, while all the nodes show ceph osd version as 12.1.4?

Thanks,
Nitin




On 8/16/17, 11:26 AM, "ceph-devel-owner@vger.kernel.org on behalf of Alfredo Deza" <ceph-devel-owner@vger.kernel.org on behalf of adeza@redhat.com> wrote:

    On Wed, Aug 16, 2017 at 11:42 AM, Gregory Farnum <gfarnum@redhat.com> wrote:
    > <Reducing this to ceph-devel/ceph-maintainers, this discussion doesn't
    > need to blast all over>
    >
    > On Wed, Aug 16, 2017 at 5:13 AM, Alfredo Deza <adeza@redhat.com> wrote:
    >> On Tue, Aug 15, 2017 at 10:35 PM, Matt Benjamin <mbenjami@redhat.com> wrote:
    >>> I think we need a v12.1.5 including #17040
    >>
    >> *I* think that this is getting to a point where we should just have
    >> nightly development releases.
    >>
    >> What is the benefit of waiting for each RC every two weeks (or so) otherwise?
    >
    > RC releases are supposed to be release candidates, as in, things that
    > we think could be the release. As with much else in the Luminous cycle
    > that broke down a bit, and that's frequently not really the case
    > because we know it's missing features we want to make available, but
    > in general it's not too far off — we believe the code is stable; it's
    > passing our nightlies; we want people to test and use it a lot more
    > rigorously than dev checkpoints. Every RC is practice for the real
    > thing and making them available the same way as real releases is part
    > of that.
    >
    >>
    >> On one side we are treating the RC releases somewhat like normal
    >> releases, with proper announcements, waiting for
    >> QA suites to complete, and have leads "ack" when their components are
    >> good enough. But on the other side of things
    >> we then try to cut releases to include fixes as immediate as possible
    >> (and as often as that means)
    >>
    >> We've had 3 releases already in August and this would mean discussing
    >> a *fourth*.
    >
    > We had 12.1.2 on August 2 and 12.1.3 on August 9. This last 12.1.4 was
    > an emergency because we broke things, and that'll happen sometimes no
    > matter what dev process you're using.
    >
    
    And that is understandable, I agree that even for stable releases that
    can happen.
    
    > Let's not jump overboard just because we had one emergency and another
    > idle suggestion that it'd be nice to fix bug X. If your concern is
    > that these builds take a lot of time, I'm sure we'd all love a
    > smoother build infrastructure. But build time is not the dominating
    > issue when deciding how we do releases when we want users to practice
    > with them. :)
    
    I wouldn't raise this if it was infrequent.
    
    A build time is a concern if it can't be automated and it takes a long
    time. Currently releases take about a day if all goes well. If it took
    a whole week
    but doesn't need any intervention it wouldn't be a problem, but we
    can't get there if we want to treat these releases like stable ones,
    where other things
    need to happen in order to get them out the door.
    
    The dev builds take about 1 hour, and they don't require any
    babysitting, and that is what I would envision for these types of
    releases.
    
    How would a daily release impact negatively the goals you are
    describing for an RC?
    
    > -Greg
    >
    >>
    >>>
    >>> Matt
    >>>
    >>> On Tue, Aug 15, 2017 at 5:16 PM, Gregory Farnum <gfarnum@redhat.com> wrote:
    >>>> On Tue, Aug 15, 2017 at 2:05 PM, Abhishek <abhishek@suse.com> wrote:
    >>>>> This is the fifth release candidate for Luminous, the next long term
    >>>>> stable release. We’ve had to do this release as there was a bug in
    >>>>> the previous RC, which affected upgrades to Luminous.[1]
    >>>>
    >>>> In particular, this will fix things for those of you who upgraded from
    >>>> Jewel or a previous RC and saw OSDs crash instantly on boot. We had an
    >>>> oversight in dealing with another bug. (Standard disclaimer: this was
    >>>> a logic error that resulted in no data changes. There were no
    >>>> durability implications — not that that helps much when you can't read
    >>>> your data out again.)
    >>>>
    >>>> Sorry guys!
    >>>> -Greg
    >>>>
    >>>>>
    >>>>> Please note that this is still a *release candidate* and
    >>>>> not the final release, we're expecting the final Luminous release in
    >>>>> a week's time, meanwhile, testing and feedback is very much welcom.
    >>>>>
    >>>>> Ceph Luminous (v12.2.0) will be the foundation for the next long-term
    >>>>> stable release series. There have been major changes since Kraken
    >>>>> (v11.2.z) and Jewel (v10.2.z), and the upgrade process is non-trivial.
    >>>>> Please read these release notes carefully. Full details and changelog at
    >>>>> http://ceph.com/releases/v12-1-4-luminous-rc-released/
    >>>>>
    >>>>> Notable Changes from 12.1.3
    >>>>> ---------------------------
    >>>>> * core: Wip 20985 divergent handling luminous (issue#20985, pr#17001, Greg
    >>>>> Farnum)
    >>>>> * qa/tasks/thrashosds-health.yaml: ignore MON_DOWN (issue#20910, pr#17003,
    >>>>> Sage Weil)
    >>>>> * crush, mon: fix weight set vs crush device classes (issue#20939, Sage
    >>>>> Weil)
    >>>>>
    >>>>>
    >>>>> Getting Ceph
    >>>>> ------------
    >>>>> * Git at git://github.com/ceph/ceph.git
    >>>>> * Tarball at http://download.ceph.com/tarballs/ceph-12.1.4.tar.gz
    >>>>> * For packages, see http://docs.ceph.com/docs/master/install/get-packages/
    >>>>> * For ceph-deploy, see
    >>>>> http://docs.ceph.com/docs/master/install/install-ceph-deploy
    >>>>> * Release sha1: a5f84b37668fc8e03165aaf5cbb380c78e4deba4
    >>>>>
    >>>>> [1]: http://tracker.ceph.com/issues/20985
    >>>>>
    >>>>>
    >>>>> Best Regards
    >>>>> Abhishek
    >>>>>
    >>>>> _______________________________________________
    >>>>> ceph-users mailing list
    >>>>> ceph-users@lists.ceph.com
    >>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
    >>>> _______________________________________________
    >>>> ceph-users mailing list
    >>>> ceph-users@lists.ceph.com
    >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
    >>> --
    >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
    >>> the body of a message to majordomo@vger.kernel.org
    >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
    --
    To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at  http://vger.kernel.org/majordomo-info.html
    

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Dan Mick]

On 08/16/2017 04:01 PM, Kamble, Nitin A wrote:
> After upgrading all the ceph nodes to v12.1.4 I see that OSD versions are still shown as older version.
> # ceph versions
> {
>     "mon": {
>         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
>     },
>     "mgr": {
>         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
>     },
>     "osd": {
>         "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50
>     },
>     "mds": {
>         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
>     },
>     "overall": {
>         "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50,
>         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 3
>     }
> }
> 
> 
> # ceph-osd --version
> ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
> 
> Why is it showing old 12.1.0 version for the 50 osds, while all the nodes show ceph osd version as 12.1.4?

You didn't say you restarted the osds.  Did you?

-- 
Dan Mick
Red Hat, Inc.
Ceph docs: http://ceph.com/docs

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

On 8/16/17, 4:18 PM, "ceph-devel-owner@vger.kernel.org on behalf of Dan Mick" <ceph-devel-owner@vger.kernel.org on behalf of dmick@redhat.com> wrote:

    On 08/16/2017 04:01 PM, Kamble, Nitin A wrote:
    > After upgrading all the ceph nodes to v12.1.4 I see that OSD versions are still shown as older version.
    > # ceph versions
    > {
    >     "mon": {
    >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
    >     },
    >     "mgr": {
    >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
    >     },
    >     "osd": {
    >         "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50
    >     },
    >     "mds": {
    >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
    >     },
    >     "overall": {
    >         "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50,
    >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 3
    >     }
    > }
    > 
    > 
    > # ceph-osd --version
    > ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
    > 
    > Why is it showing old 12.1.0 version for the 50 osds, while all the nodes show ceph osd version as 12.1.4?
    
    You didn't say you restarted the osds.  Did you?

Yes, I did restart them, and that too more than once.

    -- 
    Dan Mick
    Red Hat, Inc.
    Ceph docs: http://ceph.com/docs
    --
    To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
    the body of a message to majordomo@vger.kernel.org
    More majordomo info at  http://vger.kernel.org/majordomo-info.html
    

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Dan Mick]

>     >     "overall": {
>     >         "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50,
>     >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 3
>     >     }
>     > }
>     > 
>     > 
>     > # ceph-osd --version
>     > ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
>     > 
>     > Why is it showing old 12.1.0 version for the 50 osds, while all the nodes show ceph osd version as 12.1.4?
>     
>     You didn't say you restarted the osds.  Did you?
> 
> Yes, I did restart them, and that too more than once.

Dunno, then.  That seems impossible.

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

On 8/16/17, 6:42 PM, "Dan Mick" <dmick@redhat.com> wrote:

    
    >     >     "overall": {
    >     >         "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50,
    >     >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 3
    >     >     }
    >     > }
    >     > 
    >     > 
    >     > # ceph-osd --version
    >     > ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
    >     > 
    >     > Why is it showing old 12.1.0 version for the 50 osds, while all the nodes show ceph osd version as 12.1.4?
    >     
    >     You didn't say you restarted the osds.  Did you?
    > 
    > Yes, I did restart them, and that too more than once.
    
    Dunno, then.  That seems impossible.
    
It is showing the version of OSD before the upgrade.
Is it possible the old version gets cached somewhere, and that is not getting updated after upgrade?

Nitin

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Vasu Kulkarni]

On Thu, Aug 17, 2017 at 9:18 AM, Kamble, Nitin A
<Nitin.Kamble@teradata.com> wrote:
>
>
> On 8/16/17, 6:42 PM, "Dan Mick" <dmick@redhat.com> wrote:
>
>
>     >     >     "overall": {
>     >     >         "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50,
>     >     >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 3
>     >     >     }
>     >     > }
>     >     >
>     >     >
>     >     > # ceph-osd --version
>     >     > ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
>     >     >
>     >     > Why is it showing old 12.1.0 version for the 50 osds, while all the nodes show ceph osd version as 12.1.4?
>     >
>     >     You didn't say you restarted the osds.  Did you?
>     >
>     > Yes, I did restart them, and that too more than once.
>
>     Dunno, then.  That seems impossible.
>
> It is showing the version of OSD before the upgrade.
> Is it possible the old version gets cached somewhere, and that is not getting updated after upgrade?
I wonder if its a systemd issue, Does the systemd restart actually
change the process id? (output of ps before/after would be good),
Hopefully all the pkgs are properly updated when you query them and
you dont see any old traces left behind.

>
> Nitin
>

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

On 8/17/17, 9:42 AM, "Vasu Kulkarni" <vakulkar@redhat.com> wrote:

    On Thu, Aug 17, 2017 at 9:18 AM, Kamble, Nitin A
    <Nitin.Kamble@teradata.com> wrote:
    >
    >
    > On 8/16/17, 6:42 PM, "Dan Mick" <dmick@redhat.com> wrote:
    >
    >
    >     >     >     "overall": {
    >     >     >         "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50,
    >     >     >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 3
    >     >     >     }
    >     >     > }
    >     >     >
    >     >     >
    >     >     > # ceph-osd --version
    >     >     > ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
    >     >     >
    >     >     > Why is it showing old 12.1.0 version for the 50 osds, while all the nodes show ceph osd version as 12.1.4?
    >     >
    >     >     You didn't say you restarted the osds.  Did you?
    >     >
    >     > Yes, I did restart them, and that too more than once.
    >
    >     Dunno, then.  That seems impossible.
    >
    > It is showing the version of OSD before the upgrade.
    > Is it possible the old version gets cached somewhere, and that is not getting updated after upgrade?
    I wonder if its a systemd issue, Does the systemd restart actually
    change the process id? (output of ps before/after would be good),
    Hopefully all the pkgs are properly updated when you query them and
    you dont see any old traces left behind.
    

On the osd nodes I am finding that OSD daemons are restarting with the errors mentioned in last email and shown below. Looks like because the restart time is ~5 minutes, system is not taking it as an restart loop error.

 

#grep " pid " /var/log/ceph/ceph-osd.2.log

2017-08-17 09:26:44.407213 7f2eee6d8cc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 920308

2017-08-17 09:31:42.397509 7f302f72bcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 920824

2017-08-17 09:36:40.153434 7f56ee55bcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 921496

2017-08-17 09:38:01.971158 7fdb7857ecc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 921839

2017-08-17 09:43:00.653569 7f690cedfcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 922381

2017-08-17 09:47:58.645293 7fbc0f5accc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 923127

 

# grep ERROR -C 3 /var/log/ceph/ceph-osd.2.log

2017-08-17 09:31:22.036901 7f2eee6d8cc0  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 277.219788 seconds

2017-08-17 09:31:22.041983 7f2eee6d8cc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors

2017-08-17 09:31:22.041988 7f2eee6d8cc0 -1 osd.2 0 OSD:init: unable to mount object store

2017-08-17 09:31:22.041994 7f2eee6d8cc0 -1  ** ERROR: osd init failed: (5) Input/output error

2017-08-17 09:31:42.397497 7f302f72bcc0  0 set uid:gid to 500:167 (ceph:ceph)

2017-08-17 09:31:42.397509 7f302f72bcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 920824

2017-08-17 09:31:42.402139 7f302f72bcc0  0 pidfile_write: ignore empty --pid-file

--

2017-08-17 09:36:19.832905 7f302f72bcc0  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 277.127693 seconds

2017-08-17 09:36:19.838803 7f302f72bcc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors

2017-08-17 09:36:19.838809 7f302f72bcc0 -1 osd.2 0 OSD:init: unable to mount object store

2017-08-17 09:36:19.838815 7f302f72bcc0 -1  ** ERROR: osd init failed: (5) Input/output error

2017-08-17 09:36:40.153423 7f56ee55bcc0  0 set uid:gid to 500:167 (ceph:ceph)

2017-08-17 09:36:40.153434 7f56ee55bcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 921496

2017-08-17 09:36:40.158052 7f56ee55bcc0  0 pidfile_write: ignore empty --pid-file

--

2017-08-17 09:42:40.428889 7fdb7857ecc0  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 278.111786 seconds

2017-08-17 09:42:40.434401 7fdb7857ecc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors

2017-08-17 09:42:40.434408 7fdb7857ecc0 -1 osd.2 0 OSD:init: unable to mount object store

2017-08-17 09:42:40.434414 7fdb7857ecc0 -1  ** ERROR: osd init failed: (5) Input/output error

2017-08-17 09:43:00.653558 7f690cedfcc0  0 set uid:gid to 500:167 (ceph:ceph)

2017-08-17 09:43:00.653569 7f690cedfcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 922381

2017-08-17 09:43:00.658323 7f690cedfcc0  0 pidfile_write: ignore empty --pid-file

--

2017-08-17 09:47:38.476877 7f690cedfcc0  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 277.499695 seconds

2017-08-17 09:47:38.482565 7f690cedfcc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors

2017-08-17 09:47:38.482572 7f690cedfcc0 -1 osd.2 0 OSD:init: unable to mount object store

2017-08-17 09:47:38.482578 7f690cedfcc0 -1  ** ERROR: osd init failed: (5) Input/output error

2017-08-17 09:47:58.645283 7fbc0f5accc0  0 set uid:gid to 500:167 (ceph:ceph)

2017-08-17 09:47:58.645293 7fbc0f5accc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 923127

2017-08-17 09:47:58.649958 7fbc0f5accc0  0 pidfile_write: ignore empty --pid-file

 

There are no old version ceph packages on any of the ceph nodes. All the ceph packages are of version 12.1.4 now.


    >
    > Nitin
    >
    

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

On 8/17/17, 9:59 AM, "Kamble, Nitin A" <Nitin.Kamble@Teradata.com> wrote:

    
    
    On 8/17/17, 9:42 AM, "Vasu Kulkarni" <vakulkar@redhat.com> wrote:
    
        On Thu, Aug 17, 2017 at 9:18 AM, Kamble, Nitin A
        <Nitin.Kamble@teradata.com> wrote:
        >
        >
        > On 8/16/17, 6:42 PM, "Dan Mick" <dmick@redhat.com> wrote:
        >
        >
        >     >     >     "overall": {
        >     >     >         "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50,
        >     >     >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 3
        >     >     >     }
        >     >     > }
        >     >     >
        >     >     >
        >     >     > # ceph-osd --version
        >     >     > ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
        >     >     >
        >     >     > Why is it showing old 12.1.0 version for the 50 osds, while all the nodes show ceph osd version as 12.1.4?
        >     >
        >     >     You didn't say you restarted the osds.  Did you?
        >     >
        >     > Yes, I did restart them, and that too more than once.
        >
        >     Dunno, then.  That seems impossible.
        >
        > It is showing the version of OSD before the upgrade.
        > Is it possible the old version gets cached somewhere, and that is not getting updated after upgrade?
        I wonder if its a systemd issue, Does the systemd restart actually
        change the process id? (output of ps before/after would be good),
        Hopefully all the pkgs are properly updated when you query them and
        you dont see any old traces left behind.
        
    
    On the osd nodes I am finding that OSD daemons are restarting with the errors mentioned in last email and shown below. Looks like because the restart time is ~5 minutes, system is not taking it as an restart loop error.
    
     
    
    #grep " pid " /var/log/ceph/ceph-osd.2.log
    
    2017-08-17 09:26:44.407213 7f2eee6d8cc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 920308
    
    2017-08-17 09:31:42.397509 7f302f72bcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 920824
    
    2017-08-17 09:36:40.153434 7f56ee55bcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 921496
    
    2017-08-17 09:38:01.971158 7fdb7857ecc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 921839
    
    2017-08-17 09:43:00.653569 7f690cedfcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 922381
    
    2017-08-17 09:47:58.645293 7fbc0f5accc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 923127
    
     
    
    # grep ERROR -C 3 /var/log/ceph/ceph-osd.2.log
    
    2017-08-17 09:31:22.036901 7f2eee6d8cc0  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 277.219788 seconds
    
    2017-08-17 09:31:22.041983 7f2eee6d8cc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors
    
    2017-08-17 09:31:22.041988 7f2eee6d8cc0 -1 osd.2 0 OSD:init: unable to mount object store
    
    2017-08-17 09:31:22.041994 7f2eee6d8cc0 -1  ** ERROR: osd init failed: (5) Input/output error
    
    2017-08-17 09:31:42.397497 7f302f72bcc0  0 set uid:gid to 500:167 (ceph:ceph)
    
    2017-08-17 09:31:42.397509 7f302f72bcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 920824
    
    2017-08-17 09:31:42.402139 7f302f72bcc0  0 pidfile_write: ignore empty --pid-file
    
    --
    
    2017-08-17 09:36:19.832905 7f302f72bcc0  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 277.127693 seconds
    
    2017-08-17 09:36:19.838803 7f302f72bcc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors
    
    2017-08-17 09:36:19.838809 7f302f72bcc0 -1 osd.2 0 OSD:init: unable to mount object store
    
    2017-08-17 09:36:19.838815 7f302f72bcc0 -1  ** ERROR: osd init failed: (5) Input/output error
    
    2017-08-17 09:36:40.153423 7f56ee55bcc0  0 set uid:gid to 500:167 (ceph:ceph)
    
    2017-08-17 09:36:40.153434 7f56ee55bcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 921496
    
    2017-08-17 09:36:40.158052 7f56ee55bcc0  0 pidfile_write: ignore empty --pid-file
    
    --
    
    2017-08-17 09:42:40.428889 7fdb7857ecc0  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 278.111786 seconds
    
    2017-08-17 09:42:40.434401 7fdb7857ecc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors
    
    2017-08-17 09:42:40.434408 7fdb7857ecc0 -1 osd.2 0 OSD:init: unable to mount object store
    
    2017-08-17 09:42:40.434414 7fdb7857ecc0 -1  ** ERROR: osd init failed: (5) Input/output error
    
    2017-08-17 09:43:00.653558 7f690cedfcc0  0 set uid:gid to 500:167 (ceph:ceph)
    
    2017-08-17 09:43:00.653569 7f690cedfcc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 922381
    
    2017-08-17 09:43:00.658323 7f690cedfcc0  0 pidfile_write: ignore empty --pid-file
    
    --
    
    2017-08-17 09:47:38.476877 7f690cedfcc0  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 277.499695 seconds
    
    2017-08-17 09:47:38.482565 7f690cedfcc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _mount fsck found 1 errors
    
    2017-08-17 09:47:38.482572 7f690cedfcc0 -1 osd.2 0 OSD:init: unable to mount object store
    
    2017-08-17 09:47:38.482578 7f690cedfcc0 -1  ** ERROR: osd init failed: (5) Input/output error
    
    2017-08-17 09:47:58.645283 7fbc0f5accc0  0 set uid:gid to 500:167 (ceph:ceph)
    
    2017-08-17 09:47:58.645293 7fbc0f5accc0  0 ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc), process (unknown), pid 923127
    
    2017-08-17 09:47:58.649958 7fbc0f5accc0  0 pidfile_write: ignore empty --pid-file
    
     
    
    There are no old version ceph packages on any of the ceph nodes. All the ceph packages are of version 12.1.4 now.
    
    
        >
        > Nitin
        >
        
BTW I found a workaround which is working and giving the right versions now.

The workaround is mentioned here: https://github.com/ceph/ceph/pull/10825/files

Basically adding this line to global section of ceph.conf resolves the issue.
bluestore fsck on mount = 'false'
    

Thanks,
Nitin

    

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

On 8/17/17, 10:23 AM, "ceph-devel-owner@vger.kernel.org on behalf of Kamble, Nitin A" <ceph-devel-owner@vger.kernel.org on behalf of Nitin.Kamble@Teradata.com> wrote:
            
    BTW I found a workaround which is working and giving the right versions now.
    
    The workaround is mentioned here: https://github.com/ceph/ceph/pull/10825/files
    
    Basically adding this line to global section of ceph.conf resolves the issue.
    bluestore fsck on mount = 'false'
        
    

Still not sure why fsck failed. Can this be the reason? systemd stop/start of osd daemons does not perform umount and remount, and fsck on mounted filesystem fails.


Thanks,
Nitin

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[David Zafman]

I wouldn't run the OSDs without identifying what is going on.  I suggest 
you stop the OSD and run fsck using ceph-objectstore-tool:


ceph-objectstore-tool --data-path .... --op fsck

If you don't get any informative messages add --debug to get detail logging.


David



On 8/17/17 10:37 AM, Kamble, Nitin A wrote:
>
> On 8/17/17, 10:23 AM, "ceph-devel-owner@vger.kernel.org on behalf of Kamble, Nitin A" <ceph-devel-owner@vger.kernel.org on behalf of Nitin.Kamble@Teradata.com> wrote:
>              
>      BTW I found a workaround which is working and giving the right versions now.
>      
>      The workaround is mentioned here: https://github.com/ceph/ceph/pull/10825/files
>      
>      Basically adding this line to global section of ceph.conf resolves the issue.
>      bluestore fsck on mount = 'false'
>          
>      
>
> Still not sure why fsck failed. Can this be the reason? systemd stop/start of osd daemons does not perform umount and remount, and fsck on mounted filesystem fails.
>
>
> Thanks,
> Nitin
>
>
> N�����r��y���b�X��ǧv�^�)޺{.n�+���z�]z���{ay�\x1dʇڙ�,j\a��f���h���z�\x1e�w���\f���j:+v���w�j�m����\a����zZ+�����ݢj"��!tml=

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:

    
    I wouldn't run the OSDs without identifying what is going on.  I suggest 
    you stop the OSD and run fsck using ceph-objectstore-tool:
    
    
    ceph-objectstore-tool --data-path .... --op fsck
    
    If you don't get any informative messages add --debug to get detail logging.
    
    
    David
    
 
Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.

Below is the log of manual fsck.

This is the error line in it:
fsck error: leaked some space;65536 bytes leaked


# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined 
2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0

2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY

2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT

2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY

2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes

2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files: 

2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst 

2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ; 

2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir: 
2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937

2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:

2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
  cache_index_and_filter_blocks: 1
  cache_index_and_filter_blocks_with_high_priority: 1
  pin_l0_filter_and_index_blocks_in_cache: 1
  index_type: 0
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x5601f87cddd8
  block_cache_name: LRUCache
  block_cache_options:
    capacity : 536870912
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  format_version: 2

2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0

2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932

2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940

2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
fsck found 1 errors

    

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[David Zafman]

I don't know what caused the leak, but it doesn't seem to be a big 
deal.  Eventually, it would be nice if you could re-enable fsck with a 
code fix or a repair of your bluestore filesystem.

David

On 8/17/17 2:09 PM, Kamble, Nitin A wrote:
>
> On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:
>
>      
>      I wouldn't run the OSDs without identifying what is going on.  I suggest
>      you stop the OSD and run fsck using ceph-objectstore-tool:
>      
>      
>      ceph-objectstore-tool --data-path .... --op fsck
>      
>      If you don't get any informative messages add --debug to get detail logging.
>      
>      
>      David
>      
>   
> Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
>
> Below is the log of manual fsck.
>
> This is the error line in it:
> fsck error: leaked some space;65536 bytes leaked
>
>
> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
> warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined
> 2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
> 2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
> 2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
> 2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
> 2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
> 2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
> 2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
> 2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
> 2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
> 2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
> 2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
> 2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
> 2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
> 2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
> 2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
> 2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
> 2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
> 2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
> 2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
> 2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
> 2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
> 2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
> 2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
> 2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
> 2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
> 2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
> 2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
> 2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
> 2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
> 2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
> 2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
> 2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
> 2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
> 2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0
>
> 2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
> 2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
> 2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY
>
> 2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT
>
> 2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY
>
> 2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes
>
> 2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files:
>
> 2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst
>
> 2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ;
>
> 2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
> 2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
> 2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
> 2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
> 2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
> 2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
> 2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
> 2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
> 2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
> 2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
> 2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
> 2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
> 2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
> 2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
> 2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
> 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
> 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
> 2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
> 2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
> 2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir:
> 2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
> 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
> 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
> 2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
> 2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
> 2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
> 2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
> 2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
> 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
> 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
> 2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
> 2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
> 2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
> 2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
> 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
> 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
> 2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
> 2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
> 2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
> 2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
> 2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
> 2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
> 2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
> 2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
> 2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
> 2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
> 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
> 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
> 2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
> 2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
> 2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
> 2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
> 2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
> 2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
> 2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
> 2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
> 2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
> 2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
> 2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
> 2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
> 2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
> 2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
> 2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
> 2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
>
> 2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
>
> 2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
> 2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
> 2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
> 2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
> 2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
> 2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
> 2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
>    cache_index_and_filter_blocks: 1
>    cache_index_and_filter_blocks_with_high_priority: 1
>    pin_l0_filter_and_index_blocks_in_cache: 1
>    index_type: 0
>    hash_index_allow_collision: 1
>    checksum: 1
>    no_block_cache: 0
>    block_cache: 0x5601f87cddd8
>    block_cache_name: LRUCache
>    block_cache_options:
>      capacity : 536870912
>      num_shard_bits : 4
>      strict_capacity_limit : 0
>      high_pri_pool_ratio: 0.000
>    block_cache_compressed: (nil)
>    persistent_cache: (nil)
>    block_size: 4096
>    block_size_deviation: 10
>    block_restart_interval: 16
>    index_block_restart_interval: 1
>    filter_policy: rocksdb.BuiltinBloomFilter
>    whole_key_filtering: 1
>    format_version: 2
>
> 2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
> 2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
> 2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
> 2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
> 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
> 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
> 2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
> 2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
> 2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
> 2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
> 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
> 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
> 2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
> 2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
> 2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
> 2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
> 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
> 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
> 2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
> 2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
> 2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
> 2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
> 2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
> 2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
> 2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
> 2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
> 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
> 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
> 2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
> 2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
> 2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
> 2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
> 2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
> 2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
> 2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
> 2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
> 2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
> 2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
> 2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
> 2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
> 2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
> 2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
> 2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
> 2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
> 2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
> 2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
> 2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
> 2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
> 2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
> 2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
> 2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
> 2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
> 2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
> 2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
> 2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
>
> 2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
>
> 2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
> 2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
> 2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
>
> 2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
> 2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
> 2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
> 2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
> 2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
> 2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
> 2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
> 2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
> 2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
> 2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
> 2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
> 2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
> 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
> 2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
> 2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
> 2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
> 2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
> 2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
> 2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
> 2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
> 2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
> 2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
> fsck found 1 errors
>
>      
>
> N�����r��y���b�X��ǧv�^�)޺{.n�+���z�]z���{ay�\x1dʇڙ�,j\a��f���h���z�\x1e�w���\f���j:+v���w�j�m����\a����zZ+�����ݢj"��!tml=

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

On 8/17/17, 2:21 PM, "David Zafman" <dzafman@redhat.com> wrote:

    
    I don't know what caused the leak, but it doesn't seem to be a big 
    deal.  Eventually, it would be nice if you could re-enable fsck with a 
    code fix or a repair of your bluestore filesystem.
    
    David
 
If it is not a big deal, then fsck reporting it as a non-fetal warning would
be more appropriate to avoid the OSD startup issue.

Thanks,
Nitin

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 24916 bytes --]

Hi Nitin,

As David mentioned, it's a non-fatal error, so disabling fsck on mount is 
a fine workaround.  (Out of curiousity, how did fsck get enabled in the 
first place?  It should be off by default.)

If you don't mind, I have a branch that will additionally spit out exactly 
which extent(s) were leaked:

	https://github.com/liewegas/ceph/commit/wip-bluestore-fsck-leaks-luminous

Would you mind running fsck with that branch and send me the log?

 CEPH_ARGS="--log-file c" ceph-objectstore-tool --op fsck --data-path ...

One with --debug-bluestore 20 in the CEPH_ARGS would be ideal if it's not 
too big.  You can ceph-post-file the log.

Thanks!
sage

On Thu, 17 Aug 2017, Kamble, Nitin A wrote:

> 
> 
> On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:
> 
>     
>     I wouldn't run the OSDs without identifying what is going on.  I suggest 
>     you stop the OSD and run fsck using ceph-objectstore-tool:
>     
>     
>     ceph-objectstore-tool --data-path .... --op fsck
>     
>     If you don't get any informative messages add --debug to get detail logging.
>     
>     
>     David
>     
>  
> Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
> 
> Below is the log of manual fsck.
> 
> This is the error line in it:
> fsck error: leaked some space;65536 bytes leaked
> 
> 
> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
> warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined 
> 2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
> 2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
> 2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
> 2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
> 2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
> 2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
> 2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
> 2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
> 2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
> 2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
> 2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
> 2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
> 2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
> 2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
> 2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
> 2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
> 2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
> 2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
> 2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
> 2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
> 2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
> 2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
> 2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
> 2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
> 2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
> 2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
> 2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
> 2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
> 2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
> 2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
> 2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
> 2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
> 2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
> 2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0
> 
> 2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
> 2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
> 2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY
> 
> 2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT
> 
> 2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY
> 
> 2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes
> 
> 2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files: 
> 
> 2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst 
> 
> 2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ; 
> 
> 2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
> 2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
> 2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
> 2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
> 2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
> 2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
> 2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
> 2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
> 2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
> 2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
> 2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
> 2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
> 2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
> 2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
> 2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
> 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
> 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
> 2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
> 2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
> 2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir: 
> 2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
> 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
> 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
> 2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
> 2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
> 2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
> 2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
> 2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
> 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
> 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
> 2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
> 2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
> 2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
> 2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
> 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
> 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
> 2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
> 2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
> 2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
> 2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
> 2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
> 2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
> 2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
> 2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
> 2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
> 2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
> 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
> 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
> 2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
> 2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
> 2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
> 2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
> 2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
> 2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
> 2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
> 2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
> 2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
> 2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
> 2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
> 2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
> 2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
> 2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
> 2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
> 2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
> 
> 2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
> 
> 2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
> 2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
> 2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
> 2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
> 2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
> 2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
> 2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
>   cache_index_and_filter_blocks: 1
>   cache_index_and_filter_blocks_with_high_priority: 1
>   pin_l0_filter_and_index_blocks_in_cache: 1
>   index_type: 0
>   hash_index_allow_collision: 1
>   checksum: 1
>   no_block_cache: 0
>   block_cache: 0x5601f87cddd8
>   block_cache_name: LRUCache
>   block_cache_options:
>     capacity : 536870912
>     num_shard_bits : 4
>     strict_capacity_limit : 0
>     high_pri_pool_ratio: 0.000
>   block_cache_compressed: (nil)
>   persistent_cache: (nil)
>   block_size: 4096
>   block_size_deviation: 10
>   block_restart_interval: 16
>   index_block_restart_interval: 1
>   filter_policy: rocksdb.BuiltinBloomFilter
>   whole_key_filtering: 1
>   format_version: 2
> 
> 2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
> 2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
> 2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
> 2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
> 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
> 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
> 2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
> 2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
> 2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
> 2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
> 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
> 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
> 2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
> 2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
> 2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
> 2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
> 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
> 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
> 2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
> 2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
> 2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
> 2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
> 2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
> 2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
> 2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
> 2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
> 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
> 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
> 2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
> 2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
> 2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
> 2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
> 2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
> 2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
> 2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
> 2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
> 2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
> 2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
> 2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
> 2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
> 2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
> 2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
> 2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
> 2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
> 2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
> 2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
> 2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
> 2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
> 2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
> 2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
> 2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
> 2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
> 2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
> 2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
> 2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
> 
> 2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
> 
> 2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
> 2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
> 2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
> 
> 2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
> 2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
> 2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
> 2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
> 2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
> 2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
> 2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
> 2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
> 2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
> 2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
> 2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
> 2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
> 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
> 2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
> 2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
> 2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
> 2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
> 2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
> 2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
> 2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
> 2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
> 2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
> fsck found 1 errors
> 
>     
> 
> N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay?\x1dʇڙ??j\a??f???h?????\x1e?w???\f???j:+v???w????????\a????zZ+???????j"????i

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

Hi Sage,
My comments inline below.

On 8/23/17, 12:15 PM, "Sage Weil" <sweil@redhat.com> wrote:

    Hi Nitin,
    
    As David mentioned, it's a non-fatal error, so disabling fsck on mount is 
    a fine workaround.  (Out of curiousity, how did fsck get enabled in the 
    first place?  It should be off by default.)
    
I find it is enabled by default.

    If you don't mind, I have a branch that will additionally spit out exactly 
    which extent(s) were leaked:
    
    	https://github.com/liewegas/ceph/commit/wip-bluestore-fsck-leaks-luminous
    
    Would you mind running fsck with that branch and send me the log?
    
Sure, I can try it out. Looks like I will only need ceph-objectstore-tool replaced in my ceph setup, correct?

     CEPH_ARGS="--log-file c" ceph-objectstore-tool --op fsck --data-path ...
    
    One with --debug-bluestore 20 in the CEPH_ARGS would be ideal if it's not 
    too big.  You can ceph-post-file the log.

What is ceph-post-file thingy? I have not used it before.
    
Thanks,
Nitin

    Thanks!
    sage
    
    On Thu, 17 Aug 2017, Kamble, Nitin A wrote:
    
    > 
    > 
    > On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:
    > 
    >     
    >     I wouldn't run the OSDs without identifying what is going on.  I suggest 
    >     you stop the OSD and run fsck using ceph-objectstore-tool:
    >     
    >     
    >     ceph-objectstore-tool --data-path .... --op fsck
    >     
    >     If you don't get any informative messages add --debug to get detail logging.
    >     
    >     
    >     David
    >     
    >  
    > Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
    > 
    > Below is the log of manual fsck.
    > 
    > This is the error line in it:
    > fsck error: leaked some space;65536 bytes leaked
    > 
    > 
    > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
    > warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined 
    > 2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
    > 2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
    > 2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
    > 2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
    > 2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
    > 2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
    > 2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
    > 2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
    > 2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
    > 2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
    > 2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
    > 2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
    > 2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
    > 2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
    > 2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
    > 2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
    > 2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
    > 2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
    > 2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
    > 2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
    > 2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
    > 2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
    > 2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
    > 2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
    > 2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
    > 2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
    > 2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
    > 2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
    > 2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
    > 2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
    > 2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
    > 2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
    > 2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
    > 2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0
    > 
    > 2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
    > 2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
    > 2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY
    > 
    > 2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT
    > 
    > 2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY
    > 
    > 2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes
    > 
    > 2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files: 
    > 
    > 2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst 
    > 
    > 2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ; 
    > 
    > 2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
    > 2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
    > 2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
    > 2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
    > 2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
    > 2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
    > 2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
    > 2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
    > 2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
    > 2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
    > 2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
    > 2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
    > 2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
    > 2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
    > 2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
    > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
    > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
    > 2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
    > 2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
    > 2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir: 
    > 2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
    > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
    > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
    > 2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
    > 2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
    > 2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
    > 2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
    > 2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
    > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
    > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
    > 2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
    > 2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
    > 2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
    > 2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
    > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
    > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
    > 2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
    > 2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
    > 2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
    > 2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
    > 2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
    > 2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
    > 2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
    > 2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
    > 2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
    > 2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
    > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
    > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
    > 2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
    > 2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
    > 2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
    > 2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
    > 2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
    > 2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
    > 2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
    > 2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
    > 2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
    > 2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
    > 2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
    > 2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
    > 2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
    > 2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
    > 2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
    > 2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
    > 
    > 2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
    > 
    > 2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
    > 2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
    > 2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
    > 2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
    > 2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
    > 2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
    > 2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
    >   cache_index_and_filter_blocks: 1
    >   cache_index_and_filter_blocks_with_high_priority: 1
    >   pin_l0_filter_and_index_blocks_in_cache: 1
    >   index_type: 0
    >   hash_index_allow_collision: 1
    >   checksum: 1
    >   no_block_cache: 0
    >   block_cache: 0x5601f87cddd8
    >   block_cache_name: LRUCache
    >   block_cache_options:
    >     capacity : 536870912
    >     num_shard_bits : 4
    >     strict_capacity_limit : 0
    >     high_pri_pool_ratio: 0.000
    >   block_cache_compressed: (nil)
    >   persistent_cache: (nil)
    >   block_size: 4096
    >   block_size_deviation: 10
    >   block_restart_interval: 16
    >   index_block_restart_interval: 1
    >   filter_policy: rocksdb.BuiltinBloomFilter
    >   whole_key_filtering: 1
    >   format_version: 2
    > 
    > 2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
    > 2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
    > 2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
    > 2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
    > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
    > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
    > 2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
    > 2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
    > 2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
    > 2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
    > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
    > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
    > 2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
    > 2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
    > 2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
    > 2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
    > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
    > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
    > 2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
    > 2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
    > 2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
    > 2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
    > 2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
    > 2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
    > 2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
    > 2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
    > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
    > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
    > 2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
    > 2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
    > 2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
    > 2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
    > 2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
    > 2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
    > 2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
    > 2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
    > 2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
    > 2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
    > 2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
    > 2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
    > 2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
    > 2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
    > 2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
    > 2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
    > 2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
    > 2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
    > 2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
    > 2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
    > 2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
    > 2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
    > 2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
    > 2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
    > 2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
    > 2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
    > 2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
    > 
    > 2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
    > 
    > 2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
    > 2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
    > 2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
    > 
    > 2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
    > 2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
    > 2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
    > 2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
    > 2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
    > 2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
    > 2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
    > 2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
    > 2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
    > 2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
    > 2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
    > 2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
    > 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
    > 2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
    > 2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
    > 2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
    > 2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
    > 2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
    > 2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
    > 2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
    > 2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
    > 2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
    > fsck found 1 errors
    > 
    >     
    > 
    > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 27520 bytes --]

On Wed, 23 Aug 2017, Kamble, Nitin A wrote:
> Hi Sage,
> My comments inline below.
> 
> On 8/23/17, 12:15 PM, "Sage Weil" <sweil@redhat.com> wrote:
> 
>     Hi Nitin,
>     
>     As David mentioned, it's a non-fatal error, so disabling fsck on mount is 
>     a fine workaround.  (Out of curiousity, how did fsck get enabled in the 
>     first place?  It should be off by default.)
>     
> I find it is enabled by default.

Which version are you running?  Master branch has

    Option("bluestore_fsck_on_mount", Option::TYPE_BOOL, Option::LEVEL_DEV)
    .set_default(false)

>     If you don't mind, I have a branch that will additionally spit out exactly 
>     which extent(s) were leaked:
>     
>     	https://github.com/liewegas/ceph/commit/wip-bluestore-fsck-leaks-luminous
>     
>     Would you mind running fsck with that branch and send me the log?
>     
> Sure, I can try it out. Looks like I will only need 
> ceph-objectstore-tool replaced in my ceph setup, correct?

Right.  Thanks!

>      CEPH_ARGS="--log-file c" ceph-objectstore-tool --op fsck --data-path ...
>     
>     One with --debug-bluestore 20 in the CEPH_ARGS would be ideal if it's not 
>     too big.  You can ceph-post-file the log.
> 
> What is ceph-post-file thingy? I have not used it before.

It will upload a file or files to a server the developers have access to 
(any developers with access to the sepia lab) and spit out a uuid they can 
use to identify which upload is yours.

sage


>     
> Thanks,
> Nitin
> 
>     Thanks!
>     sage
>     
>     On Thu, 17 Aug 2017, Kamble, Nitin A wrote:
>     
>     > 
>     > 
>     > On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:
>     > 
>     >     
>     >     I wouldn't run the OSDs without identifying what is going on.  I suggest 
>     >     you stop the OSD and run fsck using ceph-objectstore-tool:
>     >     
>     >     
>     >     ceph-objectstore-tool --data-path .... --op fsck
>     >     
>     >     If you don't get any informative messages add --debug to get detail logging.
>     >     
>     >     
>     >     David
>     >     
>     >  
>     > Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
>     > 
>     > Below is the log of manual fsck.
>     > 
>     > This is the error line in it:
>     > fsck error: leaked some space;65536 bytes leaked
>     > 
>     > 
>     > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
>     > warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined 
>     > 2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
>     > 2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
>     > 2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
>     > 2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
>     > 2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
>     > 2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
>     > 2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
>     > 2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
>     > 2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
>     > 2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
>     > 2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
>     > 2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
>     > 2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
>     > 2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
>     > 2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
>     > 2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
>     > 2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
>     > 2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
>     > 2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
>     > 2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
>     > 2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
>     > 2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
>     > 2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
>     > 2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
>     > 2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
>     > 2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
>     > 2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
>     > 2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
>     > 2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
>     > 2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
>     > 2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
>     > 2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
>     > 2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
>     > 2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0
>     > 
>     > 2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
>     > 2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
>     > 2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY
>     > 
>     > 2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT
>     > 
>     > 2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY
>     > 
>     > 2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes
>     > 
>     > 2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files: 
>     > 
>     > 2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst 
>     > 
>     > 2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ; 
>     > 
>     > 2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
>     > 2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
>     > 2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
>     > 2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
>     > 2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
>     > 2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
>     > 2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
>     > 2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
>     > 2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
>     > 2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
>     > 2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
>     > 2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
>     > 2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
>     > 2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
>     > 2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
>     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
>     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
>     > 2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
>     > 2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
>     > 2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir: 
>     > 2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
>     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
>     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
>     > 2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
>     > 2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
>     > 2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
>     > 2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
>     > 2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
>     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
>     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
>     > 2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
>     > 2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
>     > 2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
>     > 2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
>     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
>     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
>     > 2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
>     > 2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
>     > 2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
>     > 2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
>     > 2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
>     > 2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
>     > 2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
>     > 2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
>     > 2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
>     > 2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
>     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
>     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
>     > 2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
>     > 2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
>     > 2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
>     > 2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
>     > 2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
>     > 2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
>     > 2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
>     > 2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
>     > 2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
>     > 2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
>     > 2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
>     > 2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
>     > 2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
>     > 2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
>     > 2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
>     > 2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
>     > 
>     > 2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
>     > 
>     > 2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
>     > 2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
>     > 2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
>     > 2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
>     > 2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
>     > 2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
>     > 2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
>     >   cache_index_and_filter_blocks: 1
>     >   cache_index_and_filter_blocks_with_high_priority: 1
>     >   pin_l0_filter_and_index_blocks_in_cache: 1
>     >   index_type: 0
>     >   hash_index_allow_collision: 1
>     >   checksum: 1
>     >   no_block_cache: 0
>     >   block_cache: 0x5601f87cddd8
>     >   block_cache_name: LRUCache
>     >   block_cache_options:
>     >     capacity : 536870912
>     >     num_shard_bits : 4
>     >     strict_capacity_limit : 0
>     >     high_pri_pool_ratio: 0.000
>     >   block_cache_compressed: (nil)
>     >   persistent_cache: (nil)
>     >   block_size: 4096
>     >   block_size_deviation: 10
>     >   block_restart_interval: 16
>     >   index_block_restart_interval: 1
>     >   filter_policy: rocksdb.BuiltinBloomFilter
>     >   whole_key_filtering: 1
>     >   format_version: 2
>     > 
>     > 2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
>     > 2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
>     > 2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
>     > 2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
>     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
>     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
>     > 2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
>     > 2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
>     > 2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
>     > 2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
>     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
>     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
>     > 2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
>     > 2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
>     > 2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
>     > 2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
>     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
>     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
>     > 2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
>     > 2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
>     > 2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
>     > 2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
>     > 2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
>     > 2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
>     > 2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
>     > 2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
>     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
>     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
>     > 2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
>     > 2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
>     > 2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
>     > 2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
>     > 2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
>     > 2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
>     > 2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
>     > 2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
>     > 2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
>     > 2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
>     > 2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
>     > 2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
>     > 2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
>     > 2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
>     > 2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
>     > 2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
>     > 2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
>     > 2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
>     > 2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
>     > 2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
>     > 2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
>     > 2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
>     > 2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
>     > 2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
>     > 2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
>     > 2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
>     > 2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
>     > 
>     > 2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
>     > 
>     > 2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
>     > 2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
>     > 2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
>     > 
>     > 2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
>     > 2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
>     > 2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
>     > 2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
>     > 2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
>     > 2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
>     > 2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
>     > 2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
>     > 2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
>     > 2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
>     > 2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
>     > 2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
>     > 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
>     > 2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
>     > 2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
>     > 2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
>     > 2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
>     > 2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
>     > 2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
>     > 2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
>     > 2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
>     > 2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
>     > fsck found 1 errors
>     > 
>     >     
>     > 
>     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
> 
> N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay?\x1dʇڙ??j\a??f???h?????\x1e?w???\f???j:+v???w????????\a????zZ+???????j"????i

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

Hi Sage,
  The log file is uploaded. It gave id like this:
  
  ceph-post-file: 48a58181-5272-40ef-938c-2f9712443d10

Thanks,
Nitin


On 8/23/17, 2:33 PM, "Sage Weil" <sweil@redhat.com> wrote:

    On Wed, 23 Aug 2017, Kamble, Nitin A wrote:
    > Hi Sage,
    > My comments inline below.
    > 
    > On 8/23/17, 12:15 PM, "Sage Weil" <sweil@redhat.com> wrote:
    > 
    >     Hi Nitin,
    >     
    >     As David mentioned, it's a non-fatal error, so disabling fsck on mount is 
    >     a fine workaround.  (Out of curiousity, how did fsck get enabled in the 
    >     first place?  It should be off by default.)
    >     
    > I find it is enabled by default.
    
    Which version are you running?  Master branch has
    
        Option("bluestore_fsck_on_mount", Option::TYPE_BOOL, Option::LEVEL_DEV)
        .set_default(false)
    
    >     If you don't mind, I have a branch that will additionally spit out exactly 
    >     which extent(s) were leaked:
    >     
    >     	https://github.com/liewegas/ceph/commit/wip-bluestore-fsck-leaks-luminous
    >     
    >     Would you mind running fsck with that branch and send me the log?
    >     
    > Sure, I can try it out. Looks like I will only need 
    > ceph-objectstore-tool replaced in my ceph setup, correct?
    
    Right.  Thanks!
    
    >      CEPH_ARGS="--log-file c" ceph-objectstore-tool --op fsck --data-path ...
    >     
    >     One with --debug-bluestore 20 in the CEPH_ARGS would be ideal if it's not 
    >     too big.  You can ceph-post-file the log.
    > 
    > What is ceph-post-file thingy? I have not used it before.
    
    It will upload a file or files to a server the developers have access to 
    (any developers with access to the sepia lab) and spit out a uuid they can 
    use to identify which upload is yours.
    
    sage

    >     
    > Thanks,
    > Nitin
    > 
    >     Thanks!
    >     sage
    >     
    >     On Thu, 17 Aug 2017, Kamble, Nitin A wrote:
    >     
    >     > 
    >     > 
    >     > On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:
    >     > 
    >     >     
    >     >     I wouldn't run the OSDs without identifying what is going on.  I suggest 
    >     >     you stop the OSD and run fsck using ceph-objectstore-tool:
    >     >     
    >     >     
    >     >     ceph-objectstore-tool --data-path .... --op fsck
    >     >     
    >     >     If you don't get any informative messages add --debug to get detail logging.
    >     >     
    >     >     
    >     >     David
    >     >     
    >     >  
    >     > Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
    >     > 
    >     > Below is the log of manual fsck.
    >     > 
    >     > This is the error line in it:
    >     > fsck error: leaked some space;65536 bytes leaked
    >     > 
    >     > 
    >     > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
    >     > warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined 
    >     > 2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
    >     > 2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
    >     > 2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
    >     > 2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
    >     > 2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
    >     > 2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
    >     > 2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
    >     > 2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
    >     > 2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
    >     > 2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
    >     > 2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
    >     > 2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
    >     > 2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
    >     > 2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
    >     > 2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
    >     > 2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
    >     > 2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
    >     > 2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
    >     > 2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
    >     > 2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
    >     > 2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
    >     > 2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
    >     > 2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
    >     > 2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
    >     > 2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
    >     > 2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
    >     > 2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
    >     > 2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
    >     > 2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
    >     > 2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
    >     > 2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
    >     > 2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
    >     > 2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
    >     > 2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0
    >     > 
    >     > 2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
    >     > 2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
    >     > 2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY
    >     > 
    >     > 2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT
    >     > 
    >     > 2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY
    >     > 
    >     > 2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes
    >     > 
    >     > 2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files: 
    >     > 
    >     > 2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst 
    >     > 
    >     > 2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ; 
    >     > 
    >     > 2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
    >     > 2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
    >     > 2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
    >     > 2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
    >     > 2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
    >     > 2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
    >     > 2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
    >     > 2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
    >     > 2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
    >     > 2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
    >     > 2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
    >     > 2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
    >     > 2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
    >     > 2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
    >     > 2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
    >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
    >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
    >     > 2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
    >     > 2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
    >     > 2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir: 
    >     > 2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
    >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
    >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
    >     > 2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
    >     > 2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
    >     > 2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
    >     > 2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
    >     > 2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
    >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
    >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
    >     > 2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
    >     > 2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
    >     > 2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
    >     > 2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
    >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
    >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
    >     > 2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
    >     > 2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
    >     > 2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
    >     > 2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
    >     > 2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
    >     > 2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
    >     > 2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
    >     > 2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
    >     > 2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
    >     > 2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
    >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
    >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
    >     > 2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
    >     > 2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
    >     > 2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
    >     > 2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
    >     > 2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
    >     > 2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
    >     > 2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
    >     > 2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
    >     > 2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
    >     > 2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
    >     > 2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
    >     > 2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
    >     > 2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
    >     > 2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
    >     > 2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
    >     > 2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
    >     > 
    >     > 2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
    >     > 
    >     > 2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
    >     > 2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
    >     > 2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
    >     > 2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
    >     > 2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
    >     > 2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
    >     > 2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
    >     >   cache_index_and_filter_blocks: 1
    >     >   cache_index_and_filter_blocks_with_high_priority: 1
    >     >   pin_l0_filter_and_index_blocks_in_cache: 1
    >     >   index_type: 0
    >     >   hash_index_allow_collision: 1
    >     >   checksum: 1
    >     >   no_block_cache: 0
    >     >   block_cache: 0x5601f87cddd8
    >     >   block_cache_name: LRUCache
    >     >   block_cache_options:
    >     >     capacity : 536870912
    >     >     num_shard_bits : 4
    >     >     strict_capacity_limit : 0
    >     >     high_pri_pool_ratio: 0.000
    >     >   block_cache_compressed: (nil)
    >     >   persistent_cache: (nil)
    >     >   block_size: 4096
    >     >   block_size_deviation: 10
    >     >   block_restart_interval: 16
    >     >   index_block_restart_interval: 1
    >     >   filter_policy: rocksdb.BuiltinBloomFilter
    >     >   whole_key_filtering: 1
    >     >   format_version: 2
    >     > 
    >     > 2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
    >     > 2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
    >     > 2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
    >     > 2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
    >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
    >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
    >     > 2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
    >     > 2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
    >     > 2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
    >     > 2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
    >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
    >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
    >     > 2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
    >     > 2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
    >     > 2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
    >     > 2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
    >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
    >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
    >     > 2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
    >     > 2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
    >     > 2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
    >     > 2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
    >     > 2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
    >     > 2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
    >     > 2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
    >     > 2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
    >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
    >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
    >     > 2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
    >     > 2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
    >     > 2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
    >     > 2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
    >     > 2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
    >     > 2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
    >     > 2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
    >     > 2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
    >     > 2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
    >     > 2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
    >     > 2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
    >     > 2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
    >     > 2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
    >     > 2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
    >     > 2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
    >     > 2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
    >     > 2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
    >     > 2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
    >     > 2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
    >     > 2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
    >     > 2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
    >     > 2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
    >     > 2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
    >     > 2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
    >     > 2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
    >     > 2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
    >     > 2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
    >     > 
    >     > 2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
    >     > 
    >     > 2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
    >     > 2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
    >     > 2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
    >     > 
    >     > 2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
    >     > 2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
    >     > 2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
    >     > 2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
    >     > 2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
    >     > 2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
    >     > 2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
    >     > 2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
    >     > 2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
    >     > 2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
    >     > 2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
    >     > 2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
    >     > 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
    >     > 2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
    >     > 2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
    >     > 2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
    >     > 2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
    >     > 2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
    >     > 2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
    >     > 2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
    >     > 2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
    >     > 2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
    >     > fsck found 1 errors
    >     > 
    >     >     
    >     > 
    >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
    > 
    > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 29867 bytes --]

Thanks!  It's right at the end of the device, and I can reproduce it with 
vstart by making the block device not aligned to a 64k (min_alloc_size) 
boundary.

sage

On Thu, 24 Aug 2017, Kamble, Nitin A wrote:

> Hi Sage,
>   The log file is uploaded. It gave id like this:
>   
>   ceph-post-file: 48a58181-5272-40ef-938c-2f9712443d10
> 
> Thanks,
> Nitin
> 
> 
> On 8/23/17, 2:33 PM, "Sage Weil" <sweil@redhat.com> wrote:
> 
>     On Wed, 23 Aug 2017, Kamble, Nitin A wrote:
>     > Hi Sage,
>     > My comments inline below.
>     > 
>     > On 8/23/17, 12:15 PM, "Sage Weil" <sweil@redhat.com> wrote:
>     > 
>     >     Hi Nitin,
>     >     
>     >     As David mentioned, it's a non-fatal error, so disabling fsck on mount is 
>     >     a fine workaround.  (Out of curiousity, how did fsck get enabled in the 
>     >     first place?  It should be off by default.)
>     >     
>     > I find it is enabled by default.
>     
>     Which version are you running?  Master branch has
>     
>         Option("bluestore_fsck_on_mount", Option::TYPE_BOOL, Option::LEVEL_DEV)
>         .set_default(false)
>     
>     >     If you don't mind, I have a branch that will additionally spit out exactly 
>     >     which extent(s) were leaked:
>     >     
>     >     	https://github.com/liewegas/ceph/commit/wip-bluestore-fsck-leaks-luminous
>     >     
>     >     Would you mind running fsck with that branch and send me the log?
>     >     
>     > Sure, I can try it out. Looks like I will only need 
>     > ceph-objectstore-tool replaced in my ceph setup, correct?
>     
>     Right.  Thanks!
>     
>     >      CEPH_ARGS="--log-file c" ceph-objectstore-tool --op fsck --data-path ...
>     >     
>     >     One with --debug-bluestore 20 in the CEPH_ARGS would be ideal if it's not 
>     >     too big.  You can ceph-post-file the log.
>     > 
>     > What is ceph-post-file thingy? I have not used it before.
>     
>     It will upload a file or files to a server the developers have access to 
>     (any developers with access to the sepia lab) and spit out a uuid they can 
>     use to identify which upload is yours.
>     
>     sage
> 
>     >     
>     > Thanks,
>     > Nitin
>     > 
>     >     Thanks!
>     >     sage
>     >     
>     >     On Thu, 17 Aug 2017, Kamble, Nitin A wrote:
>     >     
>     >     > 
>     >     > 
>     >     > On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:
>     >     > 
>     >     >     
>     >     >     I wouldn't run the OSDs without identifying what is going on.  I suggest 
>     >     >     you stop the OSD and run fsck using ceph-objectstore-tool:
>     >     >     
>     >     >     
>     >     >     ceph-objectstore-tool --data-path .... --op fsck
>     >     >     
>     >     >     If you don't get any informative messages add --debug to get detail logging.
>     >     >     
>     >     >     
>     >     >     David
>     >     >     
>     >     >  
>     >     > Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
>     >     > 
>     >     > Below is the log of manual fsck.
>     >     > 
>     >     > This is the error line in it:
>     >     > fsck error: leaked some space;65536 bytes leaked
>     >     > 
>     >     > 
>     >     > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
>     >     > warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined 
>     >     > 2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
>     >     > 2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
>     >     > 2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
>     >     > 2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
>     >     > 2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
>     >     > 2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
>     >     > 2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
>     >     > 2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
>     >     > 2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
>     >     > 2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
>     >     > 2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
>     >     > 2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
>     >     > 2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
>     >     > 2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
>     >     > 2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
>     >     > 2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
>     >     > 2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
>     >     > 2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
>     >     > 2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
>     >     > 2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
>     >     > 2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
>     >     > 2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
>     >     > 2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
>     >     > 2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
>     >     > 2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
>     >     > 2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
>     >     > 2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
>     >     > 2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
>     >     > 2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
>     >     > 2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
>     >     > 2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
>     >     > 2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
>     >     > 2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
>     >     > 2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0
>     >     > 
>     >     > 2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
>     >     > 2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
>     >     > 2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY
>     >     > 
>     >     > 2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT
>     >     > 
>     >     > 2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY
>     >     > 
>     >     > 2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes
>     >     > 
>     >     > 2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files: 
>     >     > 
>     >     > 2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst 
>     >     > 
>     >     > 2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ; 
>     >     > 
>     >     > 2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
>     >     > 2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
>     >     > 2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
>     >     > 2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
>     >     > 2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
>     >     > 2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
>     >     > 2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
>     >     > 2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
>     >     > 2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
>     >     > 2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
>     >     > 2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
>     >     > 2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
>     >     > 2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
>     >     > 2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
>     >     > 2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
>     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
>     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
>     >     > 2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
>     >     > 2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
>     >     > 2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir: 
>     >     > 2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
>     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
>     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
>     >     > 2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
>     >     > 2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
>     >     > 2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
>     >     > 2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
>     >     > 2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
>     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
>     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
>     >     > 2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
>     >     > 2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
>     >     > 2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
>     >     > 2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
>     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
>     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
>     >     > 2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
>     >     > 2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
>     >     > 2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
>     >     > 2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
>     >     > 2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
>     >     > 2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
>     >     > 2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
>     >     > 2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
>     >     > 2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
>     >     > 2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
>     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
>     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
>     >     > 2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
>     >     > 2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
>     >     > 2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
>     >     > 2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
>     >     > 2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
>     >     > 2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
>     >     > 2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
>     >     > 2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
>     >     > 2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
>     >     > 2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
>     >     > 2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
>     >     > 2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
>     >     > 2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
>     >     > 2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
>     >     > 2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
>     >     > 2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
>     >     > 
>     >     > 2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
>     >     > 
>     >     > 2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
>     >     > 2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
>     >     > 2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
>     >     > 2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
>     >     > 2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
>     >     > 2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
>     >     > 2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
>     >     >   cache_index_and_filter_blocks: 1
>     >     >   cache_index_and_filter_blocks_with_high_priority: 1
>     >     >   pin_l0_filter_and_index_blocks_in_cache: 1
>     >     >   index_type: 0
>     >     >   hash_index_allow_collision: 1
>     >     >   checksum: 1
>     >     >   no_block_cache: 0
>     >     >   block_cache: 0x5601f87cddd8
>     >     >   block_cache_name: LRUCache
>     >     >   block_cache_options:
>     >     >     capacity : 536870912
>     >     >     num_shard_bits : 4
>     >     >     strict_capacity_limit : 0
>     >     >     high_pri_pool_ratio: 0.000
>     >     >   block_cache_compressed: (nil)
>     >     >   persistent_cache: (nil)
>     >     >   block_size: 4096
>     >     >   block_size_deviation: 10
>     >     >   block_restart_interval: 16
>     >     >   index_block_restart_interval: 1
>     >     >   filter_policy: rocksdb.BuiltinBloomFilter
>     >     >   whole_key_filtering: 1
>     >     >   format_version: 2
>     >     > 
>     >     > 2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
>     >     > 2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
>     >     > 2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
>     >     > 2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
>     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
>     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
>     >     > 2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
>     >     > 2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
>     >     > 2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
>     >     > 2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
>     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
>     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
>     >     > 2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
>     >     > 2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
>     >     > 2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
>     >     > 2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
>     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
>     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
>     >     > 2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
>     >     > 2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
>     >     > 2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
>     >     > 2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
>     >     > 2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
>     >     > 2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
>     >     > 2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
>     >     > 2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
>     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
>     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
>     >     > 2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
>     >     > 2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
>     >     > 2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
>     >     > 2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
>     >     > 2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
>     >     > 2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
>     >     > 2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
>     >     > 2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
>     >     > 2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
>     >     > 2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
>     >     > 2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
>     >     > 2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
>     >     > 2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
>     >     > 2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
>     >     > 2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
>     >     > 2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
>     >     > 2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
>     >     > 2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
>     >     > 2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
>     >     > 2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
>     >     > 2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
>     >     > 2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
>     >     > 2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
>     >     > 2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
>     >     > 2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
>     >     > 2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
>     >     > 2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
>     >     > 
>     >     > 2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
>     >     > 
>     >     > 2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
>     >     > 2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
>     >     > 2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
>     >     > 
>     >     > 2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
>     >     > 2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
>     >     > 2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
>     >     > 2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
>     >     > 2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
>     >     > 2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
>     >     > 2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
>     >     > 2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
>     >     > 2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
>     >     > 2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
>     >     > 2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
>     >     > 2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
>     >     > 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
>     >     > 2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
>     >     > 2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
>     >     > 2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
>     >     > 2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
>     >     > 2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
>     >     > 2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
>     >     > 2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
>     >     > 2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
>     >     > 2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
>     >     > fsck found 1 errors
>     >     > 
>     >     >     
>     >     > 
>     >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
>     > 
>     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
> 
> 

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 30894 bytes --]

On Thu, 24 Aug 2017, Sage Weil wrote:
> Thanks!  It's right at the end of the device, and I can reproduce it with 
> vstart by making the block device not aligned to a 64k (min_alloc_size) 
> boundary.

I lied.. I can't reproduce it (I just had some debug code to trigger a 
failure that I forgot about).

Can you share what version you used when the OSD was originally 
created?  I'm pretty sure this is just an old bug in that version but I'd 
like to confirm before making it repair itself.

Thanks!
sage

> 
> sage
> 
> On Thu, 24 Aug 2017, Kamble, Nitin A wrote:
> 
> > Hi Sage,
> >   The log file is uploaded. It gave id like this:
> >   
> >   ceph-post-file: 48a58181-5272-40ef-938c-2f9712443d10
> > 
> > Thanks,
> > Nitin
> > 
> > 
> > On 8/23/17, 2:33 PM, "Sage Weil" <sweil@redhat.com> wrote:
> > 
> >     On Wed, 23 Aug 2017, Kamble, Nitin A wrote:
> >     > Hi Sage,
> >     > My comments inline below.
> >     > 
> >     > On 8/23/17, 12:15 PM, "Sage Weil" <sweil@redhat.com> wrote:
> >     > 
> >     >     Hi Nitin,
> >     >     
> >     >     As David mentioned, it's a non-fatal error, so disabling fsck on mount is 
> >     >     a fine workaround.  (Out of curiousity, how did fsck get enabled in the 
> >     >     first place?  It should be off by default.)
> >     >     
> >     > I find it is enabled by default.
> >     
> >     Which version are you running?  Master branch has
> >     
> >         Option("bluestore_fsck_on_mount", Option::TYPE_BOOL, Option::LEVEL_DEV)
> >         .set_default(false)
> >     
> >     >     If you don't mind, I have a branch that will additionally spit out exactly 
> >     >     which extent(s) were leaked:
> >     >     
> >     >     	https://github.com/liewegas/ceph/commit/wip-bluestore-fsck-leaks-luminous
> >     >     
> >     >     Would you mind running fsck with that branch and send me the log?
> >     >     
> >     > Sure, I can try it out. Looks like I will only need 
> >     > ceph-objectstore-tool replaced in my ceph setup, correct?
> >     
> >     Right.  Thanks!
> >     
> >     >      CEPH_ARGS="--log-file c" ceph-objectstore-tool --op fsck --data-path ...
> >     >     
> >     >     One with --debug-bluestore 20 in the CEPH_ARGS would be ideal if it's not 
> >     >     too big.  You can ceph-post-file the log.
> >     > 
> >     > What is ceph-post-file thingy? I have not used it before.
> >     
> >     It will upload a file or files to a server the developers have access to 
> >     (any developers with access to the sepia lab) and spit out a uuid they can 
> >     use to identify which upload is yours.
> >     
> >     sage
> > 
> >     >     
> >     > Thanks,
> >     > Nitin
> >     > 
> >     >     Thanks!
> >     >     sage
> >     >     
> >     >     On Thu, 17 Aug 2017, Kamble, Nitin A wrote:
> >     >     
> >     >     > 
> >     >     > 
> >     >     > On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:
> >     >     > 
> >     >     >     
> >     >     >     I wouldn't run the OSDs without identifying what is going on.  I suggest 
> >     >     >     you stop the OSD and run fsck using ceph-objectstore-tool:
> >     >     >     
> >     >     >     
> >     >     >     ceph-objectstore-tool --data-path .... --op fsck
> >     >     >     
> >     >     >     If you don't get any informative messages add --debug to get detail logging.
> >     >     >     
> >     >     >     
> >     >     >     David
> >     >     >     
> >     >     >  
> >     >     > Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
> >     >     > 
> >     >     > Below is the log of manual fsck.
> >     >     > 
> >     >     > This is the error line in it:
> >     >     > fsck error: leaked some space;65536 bytes leaked
> >     >     > 
> >     >     > 
> >     >     > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
> >     >     > warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined 
> >     >     > 2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
> >     >     > 2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
> >     >     > 2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
> >     >     > 2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
> >     >     > 2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
> >     >     > 2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
> >     >     > 2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
> >     >     > 2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
> >     >     > 2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
> >     >     > 2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
> >     >     > 2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
> >     >     > 2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
> >     >     > 2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
> >     >     > 2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
> >     >     > 2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
> >     >     > 2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
> >     >     > 2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
> >     >     > 2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
> >     >     > 2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
> >     >     > 2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
> >     >     > 2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
> >     >     > 2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
> >     >     > 2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
> >     >     > 2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
> >     >     > 2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
> >     >     > 2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
> >     >     > 2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
> >     >     > 2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
> >     >     > 2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
> >     >     > 2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
> >     >     > 2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
> >     >     > 2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
> >     >     > 2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
> >     >     > 2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0
> >     >     > 
> >     >     > 2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
> >     >     > 2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
> >     >     > 2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY
> >     >     > 
> >     >     > 2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT
> >     >     > 
> >     >     > 2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY
> >     >     > 
> >     >     > 2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes
> >     >     > 
> >     >     > 2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files: 
> >     >     > 
> >     >     > 2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst 
> >     >     > 
> >     >     > 2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ; 
> >     >     > 
> >     >     > 2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
> >     >     > 2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
> >     >     > 2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
> >     >     > 2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
> >     >     > 2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
> >     >     > 2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
> >     >     > 2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
> >     >     > 2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
> >     >     > 2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
> >     >     > 2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
> >     >     > 2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
> >     >     > 2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
> >     >     > 2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
> >     >     > 2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
> >     >     > 2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
> >     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
> >     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
> >     >     > 2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
> >     >     > 2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
> >     >     > 2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir: 
> >     >     > 2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
> >     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
> >     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
> >     >     > 2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
> >     >     > 2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
> >     >     > 2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
> >     >     > 2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
> >     >     > 2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
> >     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
> >     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
> >     >     > 2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
> >     >     > 2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
> >     >     > 2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
> >     >     > 2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
> >     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
> >     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
> >     >     > 2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
> >     >     > 2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
> >     >     > 2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
> >     >     > 2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
> >     >     > 2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
> >     >     > 2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
> >     >     > 2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
> >     >     > 2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
> >     >     > 2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
> >     >     > 2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
> >     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
> >     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
> >     >     > 2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
> >     >     > 2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
> >     >     > 2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
> >     >     > 2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
> >     >     > 2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
> >     >     > 2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
> >     >     > 2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
> >     >     > 2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
> >     >     > 2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
> >     >     > 2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
> >     >     > 2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
> >     >     > 2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
> >     >     > 2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
> >     >     > 2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
> >     >     > 2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
> >     >     > 2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
> >     >     > 
> >     >     > 2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
> >     >     > 
> >     >     > 2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
> >     >     > 2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
> >     >     > 2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
> >     >     > 2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
> >     >     > 2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
> >     >     > 2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
> >     >     > 2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
> >     >     >   cache_index_and_filter_blocks: 1
> >     >     >   cache_index_and_filter_blocks_with_high_priority: 1
> >     >     >   pin_l0_filter_and_index_blocks_in_cache: 1
> >     >     >   index_type: 0
> >     >     >   hash_index_allow_collision: 1
> >     >     >   checksum: 1
> >     >     >   no_block_cache: 0
> >     >     >   block_cache: 0x5601f87cddd8
> >     >     >   block_cache_name: LRUCache
> >     >     >   block_cache_options:
> >     >     >     capacity : 536870912
> >     >     >     num_shard_bits : 4
> >     >     >     strict_capacity_limit : 0
> >     >     >     high_pri_pool_ratio: 0.000
> >     >     >   block_cache_compressed: (nil)
> >     >     >   persistent_cache: (nil)
> >     >     >   block_size: 4096
> >     >     >   block_size_deviation: 10
> >     >     >   block_restart_interval: 16
> >     >     >   index_block_restart_interval: 1
> >     >     >   filter_policy: rocksdb.BuiltinBloomFilter
> >     >     >   whole_key_filtering: 1
> >     >     >   format_version: 2
> >     >     > 
> >     >     > 2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
> >     >     > 2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
> >     >     > 2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
> >     >     > 2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
> >     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
> >     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
> >     >     > 2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
> >     >     > 2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
> >     >     > 2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
> >     >     > 2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
> >     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
> >     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
> >     >     > 2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
> >     >     > 2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
> >     >     > 2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
> >     >     > 2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
> >     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
> >     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
> >     >     > 2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
> >     >     > 2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
> >     >     > 2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
> >     >     > 2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
> >     >     > 2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
> >     >     > 2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
> >     >     > 2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
> >     >     > 2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
> >     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
> >     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
> >     >     > 2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
> >     >     > 2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
> >     >     > 2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
> >     >     > 2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
> >     >     > 2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
> >     >     > 2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
> >     >     > 2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
> >     >     > 2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
> >     >     > 2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
> >     >     > 2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
> >     >     > 2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
> >     >     > 2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
> >     >     > 2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
> >     >     > 2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
> >     >     > 2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
> >     >     > 2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
> >     >     > 2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
> >     >     > 2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
> >     >     > 2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
> >     >     > 2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
> >     >     > 2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
> >     >     > 2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
> >     >     > 2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
> >     >     > 2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
> >     >     > 2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
> >     >     > 2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
> >     >     > 2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
> >     >     > 
> >     >     > 2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
> >     >     > 
> >     >     > 2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
> >     >     > 2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
> >     >     > 2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
> >     >     > 
> >     >     > 2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
> >     >     > 2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
> >     >     > 2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
> >     >     > 2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
> >     >     > 2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
> >     >     > 2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
> >     >     > 2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
> >     >     > 2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
> >     >     > 2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
> >     >     > 2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
> >     >     > 2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
> >     >     > 2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
> >     >     > 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
> >     >     > 2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
> >     >     > 2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
> >     >     > 2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
> >     >     > 2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
> >     >     > 2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
> >     >     > 2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
> >     >     > 2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
> >     >     > 2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
> >     >     > 2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
> >     >     > fsck found 1 errors
> >     >     > 
> >     >     >     
> >     >     > 
> >     >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
> >     > 
> >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
> > 
> > 

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

On 8/23/17, 7:19 PM, "Sage Weil" <sweil@redhat.com> wrote:

    Thanks!  It's right at the end of the device, and I can reproduce it with 
    vstart by making the block device not aligned to a 64k (min_alloc_size) 
    boundary.
    
    sage

Great! 
Does that mean ceph-disk needs to consider the min_alloc_size while creating the block device partitions?


Nitin

    On Thu, 24 Aug 2017, Kamble, Nitin A wrote:
    
    > Hi Sage,
    >   The log file is uploaded. It gave id like this:
    >   
    >   ceph-post-file: 48a58181-5272-40ef-938c-2f9712443d10
    > 
    > Thanks,
    > Nitin
    > 
    > 
    > On 8/23/17, 2:33 PM, "Sage Weil" <sweil@redhat.com> wrote:
    > 
    >     On Wed, 23 Aug 2017, Kamble, Nitin A wrote:
    >     > Hi Sage,
    >     > My comments inline below.
    >     > 
    >     > On 8/23/17, 12:15 PM, "Sage Weil" <sweil@redhat.com> wrote:
    >     > 
    >     >     Hi Nitin,
    >     >     
    >     >     As David mentioned, it's a non-fatal error, so disabling fsck on mount is 
    >     >     a fine workaround.  (Out of curiousity, how did fsck get enabled in the 
    >     >     first place?  It should be off by default.)
    >     >     
    >     > I find it is enabled by default.
    >     
    >     Which version are you running?  Master branch has
    >     
    >         Option("bluestore_fsck_on_mount", Option::TYPE_BOOL, Option::LEVEL_DEV)
    >         .set_default(false)
    >     
    >     >     If you don't mind, I have a branch that will additionally spit out exactly 
    >     >     which extent(s) were leaked:
    >     >     
    >     >     	https://github.com/liewegas/ceph/commit/wip-bluestore-fsck-leaks-luminous
    >     >     
    >     >     Would you mind running fsck with that branch and send me the log?
    >     >     
    >     > Sure, I can try it out. Looks like I will only need 
    >     > ceph-objectstore-tool replaced in my ceph setup, correct?
    >     
    >     Right.  Thanks!
    >     
    >     >      CEPH_ARGS="--log-file c" ceph-objectstore-tool --op fsck --data-path ...
    >     >     
    >     >     One with --debug-bluestore 20 in the CEPH_ARGS would be ideal if it's not 
    >     >     too big.  You can ceph-post-file the log.
    >     > 
    >     > What is ceph-post-file thingy? I have not used it before.
    >     
    >     It will upload a file or files to a server the developers have access to 
    >     (any developers with access to the sepia lab) and spit out a uuid they can 
    >     use to identify which upload is yours.
    >     
    >     sage
    > 
    >     >     
    >     > Thanks,
    >     > Nitin
    >     > 
    >     >     Thanks!
    >     >     sage
    >     >     
    >     >     On Thu, 17 Aug 2017, Kamble, Nitin A wrote:
    >     >     
    >     >     > 
    >     >     > 
    >     >     > On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:
    >     >     > 
    >     >     >     
    >     >     >     I wouldn't run the OSDs without identifying what is going on.  I suggest 
    >     >     >     you stop the OSD and run fsck using ceph-objectstore-tool:
    >     >     >     
    >     >     >     
    >     >     >     ceph-objectstore-tool --data-path .... --op fsck
    >     >     >     
    >     >     >     If you don't get any informative messages add --debug to get detail logging.
    >     >     >     
    >     >     >     
    >     >     >     David
    >     >     >     
    >     >     >  
    >     >     > Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
    >     >     > 
    >     >     > Below is the log of manual fsck.
    >     >     > 
    >     >     > This is the error line in it:
    >     >     > fsck error: leaked some space;65536 bytes leaked
    >     >     > 
    >     >     > 
    >     >     > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
    >     >     > warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined 
    >     >     > 2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
    >     >     > 2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
    >     >     > 2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
    >     >     > 2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
    >     >     > 2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
    >     >     > 2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
    >     >     > 2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
    >     >     > 2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
    >     >     > 2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
    >     >     > 2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
    >     >     > 2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
    >     >     > 2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
    >     >     > 2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
    >     >     > 2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
    >     >     > 2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
    >     >     > 2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
    >     >     > 2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
    >     >     > 2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
    >     >     > 2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
    >     >     > 2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
    >     >     > 2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
    >     >     > 2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
    >     >     > 2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
    >     >     > 2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
    >     >     > 2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
    >     >     > 2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
    >     >     > 2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
    >     >     > 2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
    >     >     > 2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
    >     >     > 2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
    >     >     > 2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
    >     >     > 2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
    >     >     > 2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
    >     >     > 2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0
    >     >     > 
    >     >     > 2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
    >     >     > 2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
    >     >     > 2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY
    >     >     > 
    >     >     > 2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT
    >     >     > 
    >     >     > 2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY
    >     >     > 
    >     >     > 2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes
    >     >     > 
    >     >     > 2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files: 
    >     >     > 
    >     >     > 2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst 
    >     >     > 
    >     >     > 2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ; 
    >     >     > 
    >     >     > 2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
    >     >     > 2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
    >     >     > 2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
    >     >     > 2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
    >     >     > 2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
    >     >     > 2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
    >     >     > 2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
    >     >     > 2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
    >     >     > 2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
    >     >     > 2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
    >     >     > 2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
    >     >     > 2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
    >     >     > 2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
    >     >     > 2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
    >     >     > 2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
    >     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
    >     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
    >     >     > 2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
    >     >     > 2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
    >     >     > 2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir: 
    >     >     > 2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
    >     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
    >     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
    >     >     > 2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
    >     >     > 2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
    >     >     > 2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
    >     >     > 2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
    >     >     > 2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
    >     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
    >     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
    >     >     > 2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
    >     >     > 2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
    >     >     > 2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
    >     >     > 2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
    >     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
    >     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
    >     >     > 2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
    >     >     > 2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
    >     >     > 2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
    >     >     > 2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
    >     >     > 2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
    >     >     > 2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
    >     >     > 2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
    >     >     > 2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
    >     >     > 2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
    >     >     > 2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
    >     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
    >     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
    >     >     > 2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
    >     >     > 2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
    >     >     > 2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
    >     >     > 2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
    >     >     > 2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
    >     >     > 2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
    >     >     > 2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
    >     >     > 2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
    >     >     > 2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
    >     >     > 2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
    >     >     > 2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
    >     >     > 2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
    >     >     > 2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
    >     >     > 2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
    >     >     > 2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
    >     >     > 2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
    >     >     > 
    >     >     > 2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
    >     >     > 
    >     >     > 2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
    >     >     > 2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
    >     >     > 2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
    >     >     > 2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
    >     >     > 2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
    >     >     > 2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
    >     >     > 2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
    >     >     >   cache_index_and_filter_blocks: 1
    >     >     >   cache_index_and_filter_blocks_with_high_priority: 1
    >     >     >   pin_l0_filter_and_index_blocks_in_cache: 1
    >     >     >   index_type: 0
    >     >     >   hash_index_allow_collision: 1
    >     >     >   checksum: 1
    >     >     >   no_block_cache: 0
    >     >     >   block_cache: 0x5601f87cddd8
    >     >     >   block_cache_name: LRUCache
    >     >     >   block_cache_options:
    >     >     >     capacity : 536870912
    >     >     >     num_shard_bits : 4
    >     >     >     strict_capacity_limit : 0
    >     >     >     high_pri_pool_ratio: 0.000
    >     >     >   block_cache_compressed: (nil)
    >     >     >   persistent_cache: (nil)
    >     >     >   block_size: 4096
    >     >     >   block_size_deviation: 10
    >     >     >   block_restart_interval: 16
    >     >     >   index_block_restart_interval: 1
    >     >     >   filter_policy: rocksdb.BuiltinBloomFilter
    >     >     >   whole_key_filtering: 1
    >     >     >   format_version: 2
    >     >     > 
    >     >     > 2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
    >     >     > 2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
    >     >     > 2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
    >     >     > 2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
    >     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
    >     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
    >     >     > 2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
    >     >     > 2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
    >     >     > 2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
    >     >     > 2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
    >     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
    >     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
    >     >     > 2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
    >     >     > 2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
    >     >     > 2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
    >     >     > 2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
    >     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
    >     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
    >     >     > 2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
    >     >     > 2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
    >     >     > 2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
    >     >     > 2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
    >     >     > 2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
    >     >     > 2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
    >     >     > 2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
    >     >     > 2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
    >     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
    >     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
    >     >     > 2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
    >     >     > 2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
    >     >     > 2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
    >     >     > 2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
    >     >     > 2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
    >     >     > 2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
    >     >     > 2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
    >     >     > 2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
    >     >     > 2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
    >     >     > 2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
    >     >     > 2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
    >     >     > 2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
    >     >     > 2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
    >     >     > 2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
    >     >     > 2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
    >     >     > 2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
    >     >     > 2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
    >     >     > 2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
    >     >     > 2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
    >     >     > 2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
    >     >     > 2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
    >     >     > 2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
    >     >     > 2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
    >     >     > 2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
    >     >     > 2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
    >     >     > 2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
    >     >     > 2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
    >     >     > 
    >     >     > 2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
    >     >     > 
    >     >     > 2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
    >     >     > 2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
    >     >     > 2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
    >     >     > 
    >     >     > 2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
    >     >     > 2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
    >     >     > 2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
    >     >     > 2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
    >     >     > 2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
    >     >     > 2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
    >     >     > 2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
    >     >     > 2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
    >     >     > 2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
    >     >     > 2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
    >     >     > 2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
    >     >     > 2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
    >     >     > 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
    >     >     > 2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
    >     >     > 2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
    >     >     > 2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
    >     >     > 2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
    >     >     > 2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
    >     >     > 2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
    >     >     > 2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
    >     >     > 2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
    >     >     > 2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
    >     >     > fsck found 1 errors
    >     >     > 
    >     >     >     
    >     >     > 
    >     >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
    >     > 
    >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
    > 
    > 

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 32392 bytes --]

On Thu, 24 Aug 2017, Kamble, Nitin A wrote:
> On 8/23/17, 7:19 PM, "Sage Weil" <sweil@redhat.com> wrote:
> 
>     Thanks!  It's right at the end of the device, and I can reproduce it with 
>     vstart by making the block device not aligned to a 64k (min_alloc_size) 
>     boundary.
>     
>     sage
> 
> Great! 
> Does that mean ceph-disk needs to consider the min_alloc_size while 
> creating the block device partitions?

BlueStore should be smart enough to handle it. And it appears to be, now.  
Did you see my other email?  Can you share what version you originally 
used when the OSD was created?  I think that's where the bug was.

Thanks!
sage

 > 
> 
> Nitin
> 
>     On Thu, 24 Aug 2017, Kamble, Nitin A wrote:
>     
>     > Hi Sage,
>     >   The log file is uploaded. It gave id like this:
>     >   
>     >   ceph-post-file: 48a58181-5272-40ef-938c-2f9712443d10
>     > 
>     > Thanks,
>     > Nitin
>     > 
>     > 
>     > On 8/23/17, 2:33 PM, "Sage Weil" <sweil@redhat.com> wrote:
>     > 
>     >     On Wed, 23 Aug 2017, Kamble, Nitin A wrote:
>     >     > Hi Sage,
>     >     > My comments inline below.
>     >     > 
>     >     > On 8/23/17, 12:15 PM, "Sage Weil" <sweil@redhat.com> wrote:
>     >     > 
>     >     >     Hi Nitin,
>     >     >     
>     >     >     As David mentioned, it's a non-fatal error, so disabling fsck on mount is 
>     >     >     a fine workaround.  (Out of curiousity, how did fsck get enabled in the 
>     >     >     first place?  It should be off by default.)
>     >     >     
>     >     > I find it is enabled by default.
>     >     
>     >     Which version are you running?  Master branch has
>     >     
>     >         Option("bluestore_fsck_on_mount", Option::TYPE_BOOL, Option::LEVEL_DEV)
>     >         .set_default(false)
>     >     
>     >     >     If you don't mind, I have a branch that will additionally spit out exactly 
>     >     >     which extent(s) were leaked:
>     >     >     
>     >     >     	https://github.com/liewegas/ceph/commit/wip-bluestore-fsck-leaks-luminous
>     >     >     
>     >     >     Would you mind running fsck with that branch and send me the log?
>     >     >     
>     >     > Sure, I can try it out. Looks like I will only need 
>     >     > ceph-objectstore-tool replaced in my ceph setup, correct?
>     >     
>     >     Right.  Thanks!
>     >     
>     >     >      CEPH_ARGS="--log-file c" ceph-objectstore-tool --op fsck --data-path ...
>     >     >     
>     >     >     One with --debug-bluestore 20 in the CEPH_ARGS would be ideal if it's not 
>     >     >     too big.  You can ceph-post-file the log.
>     >     > 
>     >     > What is ceph-post-file thingy? I have not used it before.
>     >     
>     >     It will upload a file or files to a server the developers have access to 
>     >     (any developers with access to the sepia lab) and spit out a uuid they can 
>     >     use to identify which upload is yours.
>     >     
>     >     sage
>     > 
>     >     >     
>     >     > Thanks,
>     >     > Nitin
>     >     > 
>     >     >     Thanks!
>     >     >     sage
>     >     >     
>     >     >     On Thu, 17 Aug 2017, Kamble, Nitin A wrote:
>     >     >     
>     >     >     > 
>     >     >     > 
>     >     >     > On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:
>     >     >     > 
>     >     >     >     
>     >     >     >     I wouldn't run the OSDs without identifying what is going on.  I suggest 
>     >     >     >     you stop the OSD and run fsck using ceph-objectstore-tool:
>     >     >     >     
>     >     >     >     
>     >     >     >     ceph-objectstore-tool --data-path .... --op fsck
>     >     >     >     
>     >     >     >     If you don't get any informative messages add --debug to get detail logging.
>     >     >     >     
>     >     >     >     
>     >     >     >     David
>     >     >     >     
>     >     >     >  
>     >     >     > Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
>     >     >     > 
>     >     >     > Below is the log of manual fsck.
>     >     >     > 
>     >     >     > This is the error line in it:
>     >     >     > fsck error: leaked some space;65536 bytes leaked
>     >     >     > 
>     >     >     > 
>     >     >     > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
>     >     >     > warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined 
>     >     >     > 2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
>     >     >     > 2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
>     >     >     > 2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
>     >     >     > 2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
>     >     >     > 2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
>     >     >     > 2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
>     >     >     > 2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
>     >     >     > 2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
>     >     >     > 2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
>     >     >     > 2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
>     >     >     > 2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
>     >     >     > 2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
>     >     >     > 2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
>     >     >     > 2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
>     >     >     > 2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
>     >     >     > 2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
>     >     >     > 2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
>     >     >     > 2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
>     >     >     > 2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
>     >     >     > 2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
>     >     >     > 2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
>     >     >     > 2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
>     >     >     > 2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
>     >     >     > 2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
>     >     >     > 2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
>     >     >     > 2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
>     >     >     > 2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
>     >     >     > 2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
>     >     >     > 2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
>     >     >     > 2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
>     >     >     > 2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
>     >     >     > 2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
>     >     >     > 2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
>     >     >     > 2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
>     >     >     > 2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
>     >     >     > 2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files: 
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst 
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ; 
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
>     >     >     > 2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
>     >     >     > 2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
>     >     >     > 2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
>     >     >     > 2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
>     >     >     > 2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
>     >     >     > 2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
>     >     >     > 2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
>     >     >     > 2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
>     >     >     > 2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
>     >     >     > 2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
>     >     >     > 2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
>     >     >     > 2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
>     >     >     > 2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
>     >     >     > 2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
>     >     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
>     >     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
>     >     >     > 2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
>     >     >     > 2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
>     >     >     > 2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir: 
>     >     >     > 2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
>     >     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
>     >     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
>     >     >     > 2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
>     >     >     > 2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
>     >     >     > 2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
>     >     >     > 2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
>     >     >     > 2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
>     >     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
>     >     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
>     >     >     > 2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
>     >     >     > 2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
>     >     >     > 2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
>     >     >     > 2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
>     >     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
>     >     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
>     >     >     > 2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
>     >     >     > 2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
>     >     >     > 2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
>     >     >     > 2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
>     >     >     > 2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
>     >     >     > 2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
>     >     >     > 2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
>     >     >     > 2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
>     >     >     > 2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
>     >     >     > 2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
>     >     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
>     >     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
>     >     >     > 2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
>     >     >     > 2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
>     >     >     > 2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
>     >     >     > 2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
>     >     >     > 2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
>     >     >     > 2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
>     >     >     > 2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
>     >     >     > 2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
>     >     >     > 2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
>     >     >     > 2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
>     >     >     > 2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
>     >     >     > 2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
>     >     >     > 2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
>     >     >     > 2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
>     >     >     > 2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
>     >     >     > 2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
>     >     >     > 2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
>     >     >     > 2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
>     >     >     > 2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
>     >     >     > 2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
>     >     >     > 2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
>     >     >     > 2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
>     >     >     >   cache_index_and_filter_blocks: 1
>     >     >     >   cache_index_and_filter_blocks_with_high_priority: 1
>     >     >     >   pin_l0_filter_and_index_blocks_in_cache: 1
>     >     >     >   index_type: 0
>     >     >     >   hash_index_allow_collision: 1
>     >     >     >   checksum: 1
>     >     >     >   no_block_cache: 0
>     >     >     >   block_cache: 0x5601f87cddd8
>     >     >     >   block_cache_name: LRUCache
>     >     >     >   block_cache_options:
>     >     >     >     capacity : 536870912
>     >     >     >     num_shard_bits : 4
>     >     >     >     strict_capacity_limit : 0
>     >     >     >     high_pri_pool_ratio: 0.000
>     >     >     >   block_cache_compressed: (nil)
>     >     >     >   persistent_cache: (nil)
>     >     >     >   block_size: 4096
>     >     >     >   block_size_deviation: 10
>     >     >     >   block_restart_interval: 16
>     >     >     >   index_block_restart_interval: 1
>     >     >     >   filter_policy: rocksdb.BuiltinBloomFilter
>     >     >     >   whole_key_filtering: 1
>     >     >     >   format_version: 2
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
>     >     >     > 2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
>     >     >     > 2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
>     >     >     > 2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
>     >     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
>     >     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
>     >     >     > 2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
>     >     >     > 2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
>     >     >     > 2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
>     >     >     > 2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
>     >     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
>     >     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
>     >     >     > 2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
>     >     >     > 2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
>     >     >     > 2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
>     >     >     > 2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
>     >     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
>     >     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
>     >     >     > 2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
>     >     >     > 2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
>     >     >     > 2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
>     >     >     > 2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
>     >     >     > 2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
>     >     >     > 2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
>     >     >     > 2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
>     >     >     > 2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
>     >     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
>     >     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
>     >     >     > 2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
>     >     >     > 2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
>     >     >     > 2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
>     >     >     > 2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
>     >     >     > 2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
>     >     >     > 2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
>     >     >     > 2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
>     >     >     > 2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
>     >     >     > 2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
>     >     >     > 2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
>     >     >     > 2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
>     >     >     > 2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
>     >     >     > 2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
>     >     >     > 2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
>     >     >     > 2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
>     >     >     > 2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
>     >     >     > 2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
>     >     >     > 2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
>     >     >     > 2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
>     >     >     > 2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
>     >     >     > 2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
>     >     >     > 2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
>     >     >     > 2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
>     >     >     > 2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
>     >     >     > 2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
>     >     >     > 2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
>     >     >     > 2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
>     >     >     > 2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
>     >     >     > 2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
>     >     >     > 
>     >     >     > 2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
>     >     >     > 2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
>     >     >     > 2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
>     >     >     > 2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
>     >     >     > 2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
>     >     >     > 2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
>     >     >     > 2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
>     >     >     > 2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
>     >     >     > 2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
>     >     >     > 2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
>     >     >     > 2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
>     >     >     > 2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
>     >     >     > 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
>     >     >     > 2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
>     >     >     > 2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
>     >     >     > 2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
>     >     >     > 2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
>     >     >     > 2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
>     >     >     > 2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
>     >     >     > 2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
>     >     >     > 2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
>     >     >     > 2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
>     >     >     > fsck found 1 errors
>     >     >     > 
>     >     >     >     
>     >     >     > 
>     >     >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
>     >     > 
>     >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
>     > 
>     > 
> 
> 

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

On 8/24/17, 12:58 PM, "Sage Weil" <sweil@redhat.com> wrote:

    On Thu, 24 Aug 2017, Kamble, Nitin A wrote:
    > On 8/23/17, 7:19 PM, "Sage Weil" <sweil@redhat.com> wrote:
    > 
    >     Thanks!  It's right at the end of the device, and I can reproduce it with 
    >     vstart by making the block device not aligned to a 64k (min_alloc_size) 
    >     boundary.
    >     
    >     sage
    > 
    > Great! 
    > Does that mean ceph-disk needs to consider the min_alloc_size while 
    > creating the block device partitions?
    
    BlueStore should be smart enough to handle it. And it appears to be, now.  
    Did you see my other email?  Can you share what version you originally 
    used when the OSD was created?  I think that's where the bug was.
    
I did not see the other email. The OSD was created on ceph version v12.1.0.

    Thanks!
    sage
    
     > 
    > 
    > Nitin
    > 
    >     On Thu, 24 Aug 2017, Kamble, Nitin A wrote:
    >     
    >     > Hi Sage,
    >     >   The log file is uploaded. It gave id like this:
    >     >   
    >     >   ceph-post-file: 48a58181-5272-40ef-938c-2f9712443d10
    >     > 
    >     > Thanks,
    >     > Nitin
    >     > 
    >     > 
    >     > On 8/23/17, 2:33 PM, "Sage Weil" <sweil@redhat.com> wrote:
    >     > 
    >     >     On Wed, 23 Aug 2017, Kamble, Nitin A wrote:
    >     >     > Hi Sage,
    >     >     > My comments inline below.
    >     >     > 
    >     >     > On 8/23/17, 12:15 PM, "Sage Weil" <sweil@redhat.com> wrote:
    >     >     > 
    >     >     >     Hi Nitin,
    >     >     >     
    >     >     >     As David mentioned, it's a non-fatal error, so disabling fsck on mount is 
    >     >     >     a fine workaround.  (Out of curiousity, how did fsck get enabled in the 
    >     >     >     first place?  It should be off by default.)
    >     >     >     
    >     >     > I find it is enabled by default.
    >     >     
    >     >     Which version are you running?  Master branch has
    >     >     
    >     >         Option("bluestore_fsck_on_mount", Option::TYPE_BOOL, Option::LEVEL_DEV)
    >     >         .set_default(false)
    >     >     
    >     >     >     If you don't mind, I have a branch that will additionally spit out exactly 
    >     >     >     which extent(s) were leaked:
    >     >     >     
    >     >     >     	https://github.com/liewegas/ceph/commit/wip-bluestore-fsck-leaks-luminous
    >     >     >     
    >     >     >     Would you mind running fsck with that branch and send me the log?
    >     >     >     
    >     >     > Sure, I can try it out. Looks like I will only need 
    >     >     > ceph-objectstore-tool replaced in my ceph setup, correct?
    >     >     
    >     >     Right.  Thanks!
    >     >     
    >     >     >      CEPH_ARGS="--log-file c" ceph-objectstore-tool --op fsck --data-path ...
    >     >     >     
    >     >     >     One with --debug-bluestore 20 in the CEPH_ARGS would be ideal if it's not 
    >     >     >     too big.  You can ceph-post-file the log.
    >     >     > 
    >     >     > What is ceph-post-file thingy? I have not used it before.
    >     >     
    >     >     It will upload a file or files to a server the developers have access to 
    >     >     (any developers with access to the sepia lab) and spit out a uuid they can 
    >     >     use to identify which upload is yours.
    >     >     
    >     >     sage
    >     > 
    >     >     >     
    >     >     > Thanks,
    >     >     > Nitin
    >     >     > 
    >     >     >     Thanks!
    >     >     >     sage
    >     >     >     
    >     >     >     On Thu, 17 Aug 2017, Kamble, Nitin A wrote:
    >     >     >     
    >     >     >     > 
    >     >     >     > 
    >     >     >     > On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:
    >     >     >     > 
    >     >     >     >     
    >     >     >     >     I wouldn't run the OSDs without identifying what is going on.  I suggest 
    >     >     >     >     you stop the OSD and run fsck using ceph-objectstore-tool:
    >     >     >     >     
    >     >     >     >     
    >     >     >     >     ceph-objectstore-tool --data-path .... --op fsck
    >     >     >     >     
    >     >     >     >     If you don't get any informative messages add --debug to get detail logging.
    >     >     >     >     
    >     >     >     >     
    >     >     >     >     David
    >     >     >     >     
    >     >     >     >  
    >     >     >     > Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
    >     >     >     > 
    >     >     >     > Below is the log of manual fsck.
    >     >     >     > 
    >     >     >     > This is the error line in it:
    >     >     >     > fsck error: leaked some space;65536 bytes leaked
    >     >     >     > 
    >     >     >     > 
    >     >     >     > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
    >     >     >     > warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined 
    >     >     >     > 2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
    >     >     >     > 2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
    >     >     >     > 2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
    >     >     >     > 2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
    >     >     >     > 2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
    >     >     >     > 2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
    >     >     >     > 2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
    >     >     >     > 2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
    >     >     >     > 2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
    >     >     >     > 2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
    >     >     >     > 2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
    >     >     >     > 2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
    >     >     >     > 2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
    >     >     >     > 2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
    >     >     >     > 2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
    >     >     >     > 2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
    >     >     >     > 2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
    >     >     >     > 2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
    >     >     >     > 2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
    >     >     >     > 2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
    >     >     >     > 2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
    >     >     >     > 2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
    >     >     >     > 2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
    >     >     >     > 2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
    >     >     >     > 2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
    >     >     >     > 2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
    >     >     >     > 2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
    >     >     >     > 2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
    >     >     >     > 2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
    >     >     >     > 2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
    >     >     >     > 2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
    >     >     >     > 2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
    >     >     >     > 2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
    >     >     >     > 2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
    >     >     >     > 2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
    >     >     >     > 2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files: 
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst 
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ; 
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
    >     >     >     > 2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
    >     >     >     > 2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
    >     >     >     > 2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
    >     >     >     > 2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
    >     >     >     > 2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
    >     >     >     > 2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
    >     >     >     > 2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
    >     >     >     > 2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
    >     >     >     > 2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
    >     >     >     > 2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
    >     >     >     > 2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
    >     >     >     > 2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
    >     >     >     > 2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
    >     >     >     > 2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
    >     >     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
    >     >     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
    >     >     >     > 2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
    >     >     >     > 2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
    >     >     >     > 2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir: 
    >     >     >     > 2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
    >     >     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
    >     >     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
    >     >     >     > 2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
    >     >     >     > 2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
    >     >     >     > 2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
    >     >     >     > 2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
    >     >     >     > 2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
    >     >     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
    >     >     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
    >     >     >     > 2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
    >     >     >     > 2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
    >     >     >     > 2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
    >     >     >     > 2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
    >     >     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
    >     >     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
    >     >     >     > 2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
    >     >     >     > 2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
    >     >     >     > 2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
    >     >     >     > 2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
    >     >     >     > 2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
    >     >     >     > 2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
    >     >     >     > 2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
    >     >     >     > 2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
    >     >     >     > 2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
    >     >     >     > 2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
    >     >     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
    >     >     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
    >     >     >     > 2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
    >     >     >     > 2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
    >     >     >     > 2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
    >     >     >     > 2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
    >     >     >     > 2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
    >     >     >     > 2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
    >     >     >     > 2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
    >     >     >     > 2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
    >     >     >     > 2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
    >     >     >     > 2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
    >     >     >     > 2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
    >     >     >     > 2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
    >     >     >     > 2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
    >     >     >     > 2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
    >     >     >     > 2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
    >     >     >     > 2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
    >     >     >     > 2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
    >     >     >     > 2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
    >     >     >     > 2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
    >     >     >     > 2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
    >     >     >     > 2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
    >     >     >     > 2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
    >     >     >     >   cache_index_and_filter_blocks: 1
    >     >     >     >   cache_index_and_filter_blocks_with_high_priority: 1
    >     >     >     >   pin_l0_filter_and_index_blocks_in_cache: 1
    >     >     >     >   index_type: 0
    >     >     >     >   hash_index_allow_collision: 1
    >     >     >     >   checksum: 1
    >     >     >     >   no_block_cache: 0
    >     >     >     >   block_cache: 0x5601f87cddd8
    >     >     >     >   block_cache_name: LRUCache
    >     >     >     >   block_cache_options:
    >     >     >     >     capacity : 536870912
    >     >     >     >     num_shard_bits : 4
    >     >     >     >     strict_capacity_limit : 0
    >     >     >     >     high_pri_pool_ratio: 0.000
    >     >     >     >   block_cache_compressed: (nil)
    >     >     >     >   persistent_cache: (nil)
    >     >     >     >   block_size: 4096
    >     >     >     >   block_size_deviation: 10
    >     >     >     >   block_restart_interval: 16
    >     >     >     >   index_block_restart_interval: 1
    >     >     >     >   filter_policy: rocksdb.BuiltinBloomFilter
    >     >     >     >   whole_key_filtering: 1
    >     >     >     >   format_version: 2
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
    >     >     >     > 2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
    >     >     >     > 2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
    >     >     >     > 2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
    >     >     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
    >     >     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
    >     >     >     > 2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
    >     >     >     > 2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
    >     >     >     > 2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
    >     >     >     > 2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
    >     >     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
    >     >     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
    >     >     >     > 2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
    >     >     >     > 2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
    >     >     >     > 2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
    >     >     >     > 2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
    >     >     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
    >     >     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
    >     >     >     > 2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
    >     >     >     > 2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
    >     >     >     > 2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
    >     >     >     > 2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
    >     >     >     > 2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
    >     >     >     > 2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
    >     >     >     > 2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
    >     >     >     > 2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
    >     >     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
    >     >     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
    >     >     >     > 2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
    >     >     >     > 2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
    >     >     >     > 2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
    >     >     >     > 2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
    >     >     >     > 2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
    >     >     >     > 2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
    >     >     >     > 2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
    >     >     >     > 2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
    >     >     >     > 2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
    >     >     >     > 2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
    >     >     >     > 2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
    >     >     >     > 2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
    >     >     >     > 2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
    >     >     >     > 2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
    >     >     >     > 2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
    >     >     >     > 2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
    >     >     >     > 2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
    >     >     >     > 2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
    >     >     >     > 2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
    >     >     >     > 2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
    >     >     >     > 2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
    >     >     >     > 2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
    >     >     >     > 2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
    >     >     >     > 2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
    >     >     >     > 2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
    >     >     >     > 2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
    >     >     >     > 2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
    >     >     >     > 2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
    >     >     >     > 2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
    >     >     >     > 
    >     >     >     > 2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
    >     >     >     > 2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
    >     >     >     > 2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
    >     >     >     > 2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
    >     >     >     > 2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
    >     >     >     > 2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
    >     >     >     > 2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
    >     >     >     > 2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
    >     >     >     > 2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
    >     >     >     > 2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
    >     >     >     > 2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
    >     >     >     > 2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
    >     >     >     > 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
    >     >     >     > 2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
    >     >     >     > 2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
    >     >     >     > 2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
    >     >     >     > 2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
    >     >     >     > 2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
    >     >     >     > 2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
    >     >     >     > 2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
    >     >     >     > 2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
    >     >     >     > 2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
    >     >     >     > fsck found 1 errors
    >     >     >     > 
    >     >     >     >     
    >     >     >     > 
    >     >     >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
    >     >     > 
    >     >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
    >     > 
    >     > 
    > 
    > 

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 35652 bytes --]

On Thu, 24 Aug 2017, Kamble, Nitin A wrote:
> 
> 
> On 8/24/17, 12:58 PM, "Sage Weil" <sweil@redhat.com> wrote:
> 
>     On Thu, 24 Aug 2017, Kamble, Nitin A wrote:
>     > On 8/23/17, 7:19 PM, "Sage Weil" <sweil@redhat.com> wrote:
>     > 
>     >     Thanks!  It's right at the end of the device, and I can reproduce it with 
>     >     vstart by making the block device not aligned to a 64k (min_alloc_size) 
>     >     boundary.
>     >     
>     >     sage
>     > 
>     > Great! 
>     > Does that mean ceph-disk needs to consider the min_alloc_size while 
>     > creating the block device partitions?
>     
>     BlueStore should be smart enough to handle it. And it appears to be, now.  
>     Did you see my other email?  Can you share what version you originally 
>     used when the OSD was created?  I think that's where the bug was.
>     
> I did not see the other email. The OSD was created on ceph version v12.1.0.

Okay, I have a patch that repairs this.  See wip-21089-luminous, which 
should appear at

	https://shaman.ceph.com/builds/ceph/wip-21089-luminous/

in the next hour or so.  If you run fsck with this build you see a line 
like

2017-08-25 18:01:58.656446 7f9e23825480 -1 bluestore(/home/sage/src/ceph4/build/dev/osd0) fsck warning: fixing leaked block 0x280000000~1000 due to old bug
2017-08-25 18:01:58.656451 7f9e23825480 -1 bluestore(/home/sage/src/ceph4/build/dev/osd0) fsck  freelist block and key count changed, fixing 0x280001000~7f000

go by in the log on the next fsck, and no such message thereafter.  Can 
you give it a try on one of those OSDs to confirm it works?  I tested both 
cases I see locally but want to ensure it works in your case too.

Thanks!
sage



 > >     Thanks!
>     sage
>     
>      > 
>     > 
>     > Nitin
>     > 
>     >     On Thu, 24 Aug 2017, Kamble, Nitin A wrote:
>     >     
>     >     > Hi Sage,
>     >     >   The log file is uploaded. It gave id like this:
>     >     >   
>     >     >   ceph-post-file: 48a58181-5272-40ef-938c-2f9712443d10
>     >     > 
>     >     > Thanks,
>     >     > Nitin
>     >     > 
>     >     > 
>     >     > On 8/23/17, 2:33 PM, "Sage Weil" <sweil@redhat.com> wrote:
>     >     > 
>     >     >     On Wed, 23 Aug 2017, Kamble, Nitin A wrote:
>     >     >     > Hi Sage,
>     >     >     > My comments inline below.
>     >     >     > 
>     >     >     > On 8/23/17, 12:15 PM, "Sage Weil" <sweil@redhat.com> wrote:
>     >     >     > 
>     >     >     >     Hi Nitin,
>     >     >     >     
>     >     >     >     As David mentioned, it's a non-fatal error, so disabling fsck on mount is 
>     >     >     >     a fine workaround.  (Out of curiousity, how did fsck get enabled in the 
>     >     >     >     first place?  It should be off by default.)
>     >     >     >     
>     >     >     > I find it is enabled by default.
>     >     >     
>     >     >     Which version are you running?  Master branch has
>     >     >     
>     >     >         Option("bluestore_fsck_on_mount", Option::TYPE_BOOL, Option::LEVEL_DEV)
>     >     >         .set_default(false)
>     >     >     
>     >     >     >     If you don't mind, I have a branch that will additionally spit out exactly 
>     >     >     >     which extent(s) were leaked:
>     >     >     >     
>     >     >     >     	https://github.com/liewegas/ceph/commit/wip-bluestore-fsck-leaks-luminous
>     >     >     >     
>     >     >     >     Would you mind running fsck with that branch and send me the log?
>     >     >     >     
>     >     >     > Sure, I can try it out. Looks like I will only need 
>     >     >     > ceph-objectstore-tool replaced in my ceph setup, correct?
>     >     >     
>     >     >     Right.  Thanks!
>     >     >     
>     >     >     >      CEPH_ARGS="--log-file c" ceph-objectstore-tool --op fsck --data-path ...
>     >     >     >     
>     >     >     >     One with --debug-bluestore 20 in the CEPH_ARGS would be ideal if it's not 
>     >     >     >     too big.  You can ceph-post-file the log.
>     >     >     > 
>     >     >     > What is ceph-post-file thingy? I have not used it before.
>     >     >     
>     >     >     It will upload a file or files to a server the developers have access to 
>     >     >     (any developers with access to the sepia lab) and spit out a uuid they can 
>     >     >     use to identify which upload is yours.
>     >     >     
>     >     >     sage
>     >     > 
>     >     >     >     
>     >     >     > Thanks,
>     >     >     > Nitin
>     >     >     > 
>     >     >     >     Thanks!
>     >     >     >     sage
>     >     >     >     
>     >     >     >     On Thu, 17 Aug 2017, Kamble, Nitin A wrote:
>     >     >     >     
>     >     >     >     > 
>     >     >     >     > 
>     >     >     >     > On 8/17/17, 1:50 PM, "David Zafman" <dzafman@redhat.com> wrote:
>     >     >     >     > 
>     >     >     >     >     
>     >     >     >     >     I wouldn't run the OSDs without identifying what is going on.  I suggest 
>     >     >     >     >     you stop the OSD and run fsck using ceph-objectstore-tool:
>     >     >     >     >     
>     >     >     >     >     
>     >     >     >     >     ceph-objectstore-tool --data-path .... --op fsck
>     >     >     >     >     
>     >     >     >     >     If you don't get any informative messages add --debug to get detail logging.
>     >     >     >     >     
>     >     >     >     >     
>     >     >     >     >     David
>     >     >     >     >     
>     >     >     >     >  
>     >     >     >     > Thanks, David for your input. It is interesting that all the 50 osds had the same fsck issue immediately after the upgrade.
>     >     >     >     > 
>     >     >     >     > Below is the log of manual fsck.
>     >     >     >     > 
>     >     >     >     > This is the error line in it:
>     >     >     >     > fsck error: leaked some space;65536 bytes leaked
>     >     >     >     > 
>     >     >     >     > 
>     >     >     >     > # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --op fsck --debug
>     >     >     >     > warning: line 36: 'bluestore_fsck_on_mount' in section 'global' redefined 
>     >     >     >     > 2017-08-17 14:03:43.895377 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck (shallow) start
>     >     >     >     > 2017-08-17 14:03:43.895410 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
>     >     >     >     > 2017-08-17 14:03:43.895416 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
>     >     >     >     > 2017-08-17 14:03:43.895684 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
>     >     >     >     > 2017-08-17 14:03:43.895942 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes max 0.5 < ratio 0.99
>     >     >     >     > 2017-08-17 14:03:43.895957 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _set_cache_sizes cache_size 1073741824 meta 0.5 kv 0.5 data 0
>     >     >     >     > 2017-08-17 14:03:43.895990 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.db type kernel
>     >     >     >     > 2017-08-17 14:03:43.895994 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open path /var/lib/ceph/osd/ceph-2/block.db
>     >     >     >     > 2017-08-17 14:03:43.896162 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) open size 1073741824 (0x40000000, 1024 MB) block_size 4096 (4096 B) non-rotational
>     >     >     >     > 2017-08-17 14:03:43.896168 7fcb138f8280  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-2/block.db size 1024 MB
>     >     >     >     > 2017-08-17 14:03:43.896364 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block type kernel
>     >     >     >     > 2017-08-17 14:03:43.896372 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block
>     >     >     >     > 2017-08-17 14:03:43.896529 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) open size 4000681103360 (0x3a37b2d1000, 3725 GB) block_size 4096 (4096 B) rotational
>     >     >     >     > 2017-08-17 14:03:43.896535 7fcb138f8280  1 bluefs add_block_device bdev 2 path /var/lib/ceph/osd/ceph-2/block size 3725 GB
>     >     >     >     > 2017-08-17 14:03:43.896546 7fcb138f8280  1 bdev create path /var/lib/ceph/osd/ceph-2/block.wal type kernel
>     >     >     >     > 2017-08-17 14:03:43.896549 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open path /var/lib/ceph/osd/ceph-2/block.wal
>     >     >     >     > 2017-08-17 14:03:43.896733 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) open size 10737418240 (0x280000000, 10240 MB) block_size 4096 (4096 B) non-rotational
>     >     >     >     > 2017-08-17 14:03:43.896739 7fcb138f8280  1 bluefs add_block_device bdev 0 path /var/lib/ceph/osd/ceph-2/block.wal size 10240 MB
>     >     >     >     > 2017-08-17 14:03:43.896912 7fcb138f8280  1 bluefs mount
>     >     >     >     > 2017-08-17 14:03:43.902514 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
>     >     >     >     > 2017-08-17 14:03:43.902525 7fcb138f8280  0  set rocksdb option compression = kNoCompression
>     >     >     >     > 2017-08-17 14:03:43.902533 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
>     >     >     >     > 2017-08-17 14:03:43.902538 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
>     >     >     >     > 2017-08-17 14:03:43.902541 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
>     >     >     >     > 2017-08-17 14:03:43.902544 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
>     >     >     >     > 2017-08-17 14:03:43.902548 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
>     >     >     >     > 2017-08-17 14:03:43.902572 7fcb138f8280  0  set rocksdb option compaction_readahead_size = 2097152
>     >     >     >     > 2017-08-17 14:03:43.902577 7fcb138f8280  0  set rocksdb option compression = kNoCompression
>     >     >     >     > 2017-08-17 14:03:43.902580 7fcb138f8280  0  set rocksdb option max_write_buffer_number = 4
>     >     >     >     > 2017-08-17 14:03:43.902584 7fcb138f8280  0  set rocksdb option min_write_buffer_number_to_merge = 1
>     >     >     >     > 2017-08-17 14:03:43.902587 7fcb138f8280  0  set rocksdb option recycle_log_file_num = 4
>     >     >     >     > 2017-08-17 14:03:43.902590 7fcb138f8280  0  set rocksdb option writable_file_max_buffer_size = 0
>     >     >     >     > 2017-08-17 14:03:43.902593 7fcb138f8280  0  set rocksdb option write_buffer_size = 268435456
>     >     >     >     > 2017-08-17 14:03:43.902706 7fcb138f8280  4 rocksdb: RocksDB version: 5.4.0
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.902713 7fcb138f8280  4 rocksdb: Git sha rocksdb_build_git_sha:@0@
>     >     >     >     > 2017-08-17 14:03:43.902715 7fcb138f8280  4 rocksdb: Compile date Aug 15 2017
>     >     >     >     > 2017-08-17 14:03:43.902719 7fcb138f8280  4 rocksdb: DB SUMMARY
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.902727 7fcb138f8280  4 rocksdb: CURRENT file:  CURRENT
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.902729 7fcb138f8280  4 rocksdb: IDENTITY file:  IDENTITY
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.902735 7fcb138f8280  4 rocksdb: MANIFEST file:  MANIFEST-000937 size: 1083 Bytes
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.902739 7fcb138f8280  4 rocksdb: SST files in db dir, Total Num: 0, files: 
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.902745 7fcb138f8280  4 rocksdb: SST files in db.slow dir, Total Num: 6, files: 000187.sst 000199.sst 000219.sst 000220.sst 000934.sst 000935.sst 
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.902753 7fcb138f8280  4 rocksdb: Write Ahead Log file in db.wal: 000938.log size: 0 ; 
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.902755 7fcb138f8280  4 rocksdb:                         Options.error_if_exists: 0
>     >     >     >     > 2017-08-17 14:03:43.902756 7fcb138f8280  4 rocksdb:                       Options.create_if_missing: 0
>     >     >     >     > 2017-08-17 14:03:43.902757 7fcb138f8280  4 rocksdb:                         Options.paranoid_checks: 1
>     >     >     >     > 2017-08-17 14:03:43.902758 7fcb138f8280  4 rocksdb:                                     Options.env: 0x5601f8a43e80
>     >     >     >     > 2017-08-17 14:03:43.902759 7fcb138f8280  4 rocksdb:                                Options.info_log: 0x5601f8a443e0
>     >     >     >     > 2017-08-17 14:03:43.902760 7fcb138f8280  4 rocksdb:                          Options.max_open_files: -1
>     >     >     >     > 2017-08-17 14:03:43.902762 7fcb138f8280  4 rocksdb:                Options.max_file_opening_threads: 16
>     >     >     >     > 2017-08-17 14:03:43.902763 7fcb138f8280  4 rocksdb:                               Options.use_fsync: 0
>     >     >     >     > 2017-08-17 14:03:43.902764 7fcb138f8280  4 rocksdb:                       Options.max_log_file_size: 0
>     >     >     >     > 2017-08-17 14:03:43.902765 7fcb138f8280  4 rocksdb:                  Options.max_manifest_file_size: 18446744073709551615
>     >     >     >     > 2017-08-17 14:03:43.902766 7fcb138f8280  4 rocksdb:                   Options.log_file_time_to_roll: 0
>     >     >     >     > 2017-08-17 14:03:43.902767 7fcb138f8280  4 rocksdb:                       Options.keep_log_file_num: 1000
>     >     >     >     > 2017-08-17 14:03:43.902768 7fcb138f8280  4 rocksdb:                    Options.recycle_log_file_num: 4
>     >     >     >     > 2017-08-17 14:03:43.902769 7fcb138f8280  4 rocksdb:                         Options.allow_fallocate: 1
>     >     >     >     > 2017-08-17 14:03:43.902770 7fcb138f8280  4 rocksdb:                        Options.allow_mmap_reads: 0
>     >     >     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                       Options.allow_mmap_writes: 0
>     >     >     >     > 2017-08-17 14:03:43.902771 7fcb138f8280  4 rocksdb:                        Options.use_direct_reads: 0
>     >     >     >     > 2017-08-17 14:03:43.902772 7fcb138f8280  4 rocksdb:                        Options.use_direct_io_for_flush_and_compaction: 0
>     >     >     >     > 2017-08-17 14:03:43.902774 7fcb138f8280  4 rocksdb:          Options.create_missing_column_families: 0
>     >     >     >     > 2017-08-17 14:03:43.902781 7fcb138f8280  4 rocksdb:                              Options.db_log_dir: 
>     >     >     >     > 2017-08-17 14:03:43.902783 7fcb138f8280  4 rocksdb:                                 Options.wal_dir: db.wal
>     >     >     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                Options.table_cache_numshardbits: 6
>     >     >     >     > 2017-08-17 14:03:43.902784 7fcb138f8280  4 rocksdb:                      Options.max_subcompactions: 1
>     >     >     >     > 2017-08-17 14:03:43.902785 7fcb138f8280  4 rocksdb:                  Options.max_background_flushes: 1
>     >     >     >     > 2017-08-17 14:03:43.902786 7fcb138f8280  4 rocksdb:                         Options.WAL_ttl_seconds: 0
>     >     >     >     > 2017-08-17 14:03:43.902787 7fcb138f8280  4 rocksdb:                       Options.WAL_size_limit_MB: 0
>     >     >     >     > 2017-08-17 14:03:43.902788 7fcb138f8280  4 rocksdb:             Options.manifest_preallocation_size: 4194304
>     >     >     >     > 2017-08-17 14:03:43.902789 7fcb138f8280  4 rocksdb:                     Options.is_fd_close_on_exec: 1
>     >     >     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                   Options.advise_random_on_open: 1
>     >     >     >     > 2017-08-17 14:03:43.902790 7fcb138f8280  4 rocksdb:                    Options.db_write_buffer_size: 0
>     >     >     >     > 2017-08-17 14:03:43.902791 7fcb138f8280  4 rocksdb:         Options.access_hint_on_compaction_start: 1
>     >     >     >     > 2017-08-17 14:03:43.902792 7fcb138f8280  4 rocksdb:  Options.new_table_reader_for_compaction_inputs: 1
>     >     >     >     > 2017-08-17 14:03:43.902793 7fcb138f8280  4 rocksdb:               Options.compaction_readahead_size: 2097152
>     >     >     >     > 2017-08-17 14:03:43.902794 7fcb138f8280  4 rocksdb:           Options.random_access_max_buffer_size: 1048576
>     >     >     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:           Options.writable_file_max_buffer_size: 0
>     >     >     >     > 2017-08-17 14:03:43.902795 7fcb138f8280  4 rocksdb:                      Options.use_adaptive_mutex: 0
>     >     >     >     > 2017-08-17 14:03:43.902796 7fcb138f8280  4 rocksdb:                            Options.rate_limiter: (nil)
>     >     >     >     > 2017-08-17 14:03:43.902798 7fcb138f8280  4 rocksdb:     Options.sst_file_manager.rate_bytes_per_sec: 0
>     >     >     >     > 2017-08-17 14:03:43.902799 7fcb138f8280  4 rocksdb:                          Options.bytes_per_sync: 0
>     >     >     >     > 2017-08-17 14:03:43.902801 7fcb138f8280  4 rocksdb:                      Options.wal_bytes_per_sync: 0
>     >     >     >     > 2017-08-17 14:03:43.902802 7fcb138f8280  4 rocksdb:                       Options.wal_recovery_mode: 0
>     >     >     >     > 2017-08-17 14:03:43.902803 7fcb138f8280  4 rocksdb:                  Options.enable_thread_tracking: 0
>     >     >     >     > 2017-08-17 14:03:43.902804 7fcb138f8280  4 rocksdb:         Options.allow_concurrent_memtable_write: 1
>     >     >     >     > 2017-08-17 14:03:43.902805 7fcb138f8280  4 rocksdb:      Options.enable_write_thread_adaptive_yield: 1
>     >     >     >     > 2017-08-17 14:03:43.902806 7fcb138f8280  4 rocksdb:             Options.write_thread_max_yield_usec: 100
>     >     >     >     > 2017-08-17 14:03:43.902807 7fcb138f8280  4 rocksdb:            Options.write_thread_slow_yield_usec: 3
>     >     >     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                               Options.row_cache: None
>     >     >     >     > 2017-08-17 14:03:43.902808 7fcb138f8280  4 rocksdb:                              Options.wal_filter: None
>     >     >     >     > 2017-08-17 14:03:43.902809 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_recovery: 0
>     >     >     >     > 2017-08-17 14:03:43.902810 7fcb138f8280  4 rocksdb:             Options.base_background_compactions: 1
>     >     >     >     > 2017-08-17 14:03:43.902811 7fcb138f8280  4 rocksdb:             Options.max_background_compactions: 1
>     >     >     >     > 2017-08-17 14:03:43.902812 7fcb138f8280  4 rocksdb:             Options.avoid_flush_during_shutdown: 0
>     >     >     >     > 2017-08-17 14:03:43.902813 7fcb138f8280  4 rocksdb:             Options.delayed_write_rate : 16777216
>     >     >     >     > 2017-08-17 14:03:43.902814 7fcb138f8280  4 rocksdb:             Options.max_total_wal_size: 0
>     >     >     >     > 2017-08-17 14:03:43.902816 7fcb138f8280  4 rocksdb:             Options.delete_obsolete_files_period_micros: 21600000000
>     >     >     >     > 2017-08-17 14:03:43.902817 7fcb138f8280  4 rocksdb:                   Options.stats_dump_period_sec: 600
>     >     >     >     > 2017-08-17 14:03:43.902818 7fcb138f8280  4 rocksdb: Compression algorithms supported:
>     >     >     >     > 2017-08-17 14:03:43.902819 7fcb138f8280  4 rocksdb:     Snappy supported: 0
>     >     >     >     > 2017-08-17 14:03:43.902820 7fcb138f8280  4 rocksdb:     Zlib supported: 0
>     >     >     >     > 2017-08-17 14:03:43.902821 7fcb138f8280  4 rocksdb:     Bzip supported: 0
>     >     >     >     > 2017-08-17 14:03:43.902822 7fcb138f8280  4 rocksdb:     LZ4 supported: 0
>     >     >     >     > 2017-08-17 14:03:43.902823 7fcb138f8280  4 rocksdb:     ZSTD supported: 0
>     >     >     >     > 2017-08-17 14:03:43.902824 7fcb138f8280  4 rocksdb: Fast CRC32 supported: 1
>     >     >     >     > 2017-08-17 14:03:43.902920 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2609] Recovering from manifest file: MANIFEST-000937
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.902958 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/column_family.cc:407] --------------- Options for column family [default]:
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.902960 7fcb138f8280  4 rocksdb:               Options.comparator: leveldb.BytewiseComparator
>     >     >     >     > 2017-08-17 14:03:43.902965 7fcb138f8280  4 rocksdb:           Options.merge_operator: .T:int64_array.b:bitwise_xor
>     >     >     >     > 2017-08-17 14:03:43.902966 7fcb138f8280  4 rocksdb:        Options.compaction_filter: None
>     >     >     >     > 2017-08-17 14:03:43.902967 7fcb138f8280  4 rocksdb:        Options.compaction_filter_factory: None
>     >     >     >     > 2017-08-17 14:03:43.902969 7fcb138f8280  4 rocksdb:         Options.memtable_factory: SkipListFactory
>     >     >     >     > 2017-08-17 14:03:43.902970 7fcb138f8280  4 rocksdb:            Options.table_factory: BlockBasedTable
>     >     >     >     > 2017-08-17 14:03:43.902985 7fcb138f8280  4 rocksdb:            table_factory options:   flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601f8782208)
>     >     >     >     >   cache_index_and_filter_blocks: 1
>     >     >     >     >   cache_index_and_filter_blocks_with_high_priority: 1
>     >     >     >     >   pin_l0_filter_and_index_blocks_in_cache: 1
>     >     >     >     >   index_type: 0
>     >     >     >     >   hash_index_allow_collision: 1
>     >     >     >     >   checksum: 1
>     >     >     >     >   no_block_cache: 0
>     >     >     >     >   block_cache: 0x5601f87cddd8
>     >     >     >     >   block_cache_name: LRUCache
>     >     >     >     >   block_cache_options:
>     >     >     >     >     capacity : 536870912
>     >     >     >     >     num_shard_bits : 4
>     >     >     >     >     strict_capacity_limit : 0
>     >     >     >     >     high_pri_pool_ratio: 0.000
>     >     >     >     >   block_cache_compressed: (nil)
>     >     >     >     >   persistent_cache: (nil)
>     >     >     >     >   block_size: 4096
>     >     >     >     >   block_size_deviation: 10
>     >     >     >     >   block_restart_interval: 16
>     >     >     >     >   index_block_restart_interval: 1
>     >     >     >     >   filter_policy: rocksdb.BuiltinBloomFilter
>     >     >     >     >   whole_key_filtering: 1
>     >     >     >     >   format_version: 2
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.902995 7fcb138f8280  4 rocksdb:        Options.write_buffer_size: 268435456
>     >     >     >     > 2017-08-17 14:03:43.902996 7fcb138f8280  4 rocksdb:  Options.max_write_buffer_number: 4
>     >     >     >     > 2017-08-17 14:03:43.902997 7fcb138f8280  4 rocksdb:          Options.compression: NoCompression
>     >     >     >     > 2017-08-17 14:03:43.902999 7fcb138f8280  4 rocksdb:                  Options.bottommost_compression: Disabled
>     >     >     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:       Options.prefix_extractor: nullptr
>     >     >     >     > 2017-08-17 14:03:43.903000 7fcb138f8280  4 rocksdb:   Options.memtable_insert_with_hint_prefix_extractor: nullptr
>     >     >     >     > 2017-08-17 14:03:43.903001 7fcb138f8280  4 rocksdb:             Options.num_levels: 7
>     >     >     >     > 2017-08-17 14:03:43.903002 7fcb138f8280  4 rocksdb:        Options.min_write_buffer_number_to_merge: 1
>     >     >     >     > 2017-08-17 14:03:43.903008 7fcb138f8280  4 rocksdb:     Options.max_write_buffer_number_to_maintain: 0
>     >     >     >     > 2017-08-17 14:03:43.903009 7fcb138f8280  4 rocksdb:            Options.compression_opts.window_bits: -14
>     >     >     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:                  Options.compression_opts.level: -1
>     >     >     >     > 2017-08-17 14:03:43.903010 7fcb138f8280  4 rocksdb:               Options.compression_opts.strategy: 0
>     >     >     >     > 2017-08-17 14:03:43.903011 7fcb138f8280  4 rocksdb:         Options.compression_opts.max_dict_bytes: 0
>     >     >     >     > 2017-08-17 14:03:43.903012 7fcb138f8280  4 rocksdb:      Options.level0_file_num_compaction_trigger: 4
>     >     >     >     > 2017-08-17 14:03:43.903013 7fcb138f8280  4 rocksdb:          Options.level0_slowdown_writes_trigger: 20
>     >     >     >     > 2017-08-17 14:03:43.903014 7fcb138f8280  4 rocksdb:              Options.level0_stop_writes_trigger: 36
>     >     >     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:                   Options.target_file_size_base: 67108864
>     >     >     >     > 2017-08-17 14:03:43.903015 7fcb138f8280  4 rocksdb:             Options.target_file_size_multiplier: 1
>     >     >     >     > 2017-08-17 14:03:43.903016 7fcb138f8280  4 rocksdb:                Options.max_bytes_for_level_base: 268435456
>     >     >     >     > 2017-08-17 14:03:43.903017 7fcb138f8280  4 rocksdb: Options.level_compaction_dynamic_level_bytes: 0
>     >     >     >     > 2017-08-17 14:03:43.903018 7fcb138f8280  4 rocksdb:          Options.max_bytes_for_level_multiplier: 10.000000
>     >     >     >     > 2017-08-17 14:03:43.903021 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1
>     >     >     >     > 2017-08-17 14:03:43.903022 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1
>     >     >     >     > 2017-08-17 14:03:43.903023 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1
>     >     >     >     > 2017-08-17 14:03:43.903024 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1
>     >     >     >     > 2017-08-17 14:03:43.903025 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1
>     >     >     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1
>     >     >     >     > 2017-08-17 14:03:43.903026 7fcb138f8280  4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1
>     >     >     >     > 2017-08-17 14:03:43.903027 7fcb138f8280  4 rocksdb:       Options.max_sequential_skip_in_iterations: 8
>     >     >     >     > 2017-08-17 14:03:43.903028 7fcb138f8280  4 rocksdb:                    Options.max_compaction_bytes: 1677721600
>     >     >     >     > 2017-08-17 14:03:43.903029 7fcb138f8280  4 rocksdb:                        Options.arena_block_size: 33554432
>     >     >     >     > 2017-08-17 14:03:43.903030 7fcb138f8280  4 rocksdb:   Options.soft_pending_compaction_bytes_limit: 68719476736
>     >     >     >     > 2017-08-17 14:03:43.903031 7fcb138f8280  4 rocksdb:   Options.hard_pending_compaction_bytes_limit: 274877906944
>     >     >     >     > 2017-08-17 14:03:43.903032 7fcb138f8280  4 rocksdb:       Options.rate_limit_delay_max_milliseconds: 100
>     >     >     >     > 2017-08-17 14:03:43.903033 7fcb138f8280  4 rocksdb:                Options.disable_auto_compactions: 0
>     >     >     >     > 2017-08-17 14:03:43.903034 7fcb138f8280  4 rocksdb:                         Options.compaction_style: kCompactionStyleLevel
>     >     >     >     > 2017-08-17 14:03:43.903035 7fcb138f8280  4 rocksdb:                           Options.compaction_pri: kByCompensatedSize
>     >     >     >     > 2017-08-17 14:03:43.903036 7fcb138f8280  4 rocksdb:  Options.compaction_options_universal.size_ratio: 1
>     >     >     >     > 2017-08-17 14:03:43.903037 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.min_merge_width: 2
>     >     >     >     > 2017-08-17 14:03:43.903038 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295
>     >     >     >     > 2017-08-17 14:03:43.903039 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200
>     >     >     >     > 2017-08-17 14:03:43.903040 7fcb138f8280  4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1
>     >     >     >     > 2017-08-17 14:03:43.903041 7fcb138f8280  4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824
>     >     >     >     > 2017-08-17 14:03:43.903042 7fcb138f8280  4 rocksdb:                   Options.table_properties_collectors:
>     >     >     >     > 2017-08-17 14:03:43.903043 7fcb138f8280  4 rocksdb:                   Options.inplace_update_support: 0
>     >     >     >     > 2017-08-17 14:03:43.903044 7fcb138f8280  4 rocksdb:                 Options.inplace_update_num_locks: 10000
>     >     >     >     > 2017-08-17 14:03:43.903045 7fcb138f8280  4 rocksdb:               Options.memtable_prefix_bloom_size_ratio: 0.000000
>     >     >     >     > 2017-08-17 14:03:43.903046 7fcb138f8280  4 rocksdb:   Options.memtable_huge_page_size: 0
>     >     >     >     > 2017-08-17 14:03:43.903047 7fcb138f8280  4 rocksdb:                           Options.bloom_locality: 0
>     >     >     >     > 2017-08-17 14:03:43.903048 7fcb138f8280  4 rocksdb:                    Options.max_successive_merges: 0
>     >     >     >     > 2017-08-17 14:03:43.903049 7fcb138f8280  4 rocksdb:                Options.optimize_filters_for_hits: 0
>     >     >     >     > 2017-08-17 14:03:43.903050 7fcb138f8280  4 rocksdb:                Options.paranoid_file_checks: 0
>     >     >     >     > 2017-08-17 14:03:43.903051 7fcb138f8280  4 rocksdb:                Options.force_consistency_checks: 0
>     >     >     >     > 2017-08-17 14:03:43.903053 7fcb138f8280  4 rocksdb:                Options.report_bg_io_stats: 0
>     >     >     >     > 2017-08-17 14:03:43.946483 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2859] Recovered from manifest file:db/MANIFEST-000937 succeeded,manifest_file_number is 937, next_file_number is 939, last_sequence is 64576158, log_number is 0,prev_log_number is 0,max_column_family is 0
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.946492 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2867] Column family [default] (ID 0), log number is 932
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.946530 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823946523, "job": 1, "event": "recovery_started", "log_files": [938]}
>     >     >     >     > 2017-08-17 14:03:43.946533 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:482] Recovering log #938 mode 0
>     >     >     >     > 2017-08-17 14:03:43.946541 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/version_set.cc:2395] Creating manifest 940
>     >     >     >     > 
>     >     >     >     > 2017-08-17 14:03:43.947010 7fcb138f8280  4 rocksdb: EVENT_LOG_v1 {"time_micros": 1503003823947008, "job": 1, "event": "recovery_finished"}
>     >     >     >     > 2017-08-17 14:03:43.947694 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl_open.cc:1063] DB pointer 0x5601f8a8ab00
>     >     >     >     > 2017-08-17 14:03:43.947722 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db opened rocksdb path db options compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152
>     >     >     >     > 2017-08-17 14:03:43.995973 7fcb138f8280  1 freelist init
>     >     >     >     > 2017-08-17 14:03:44.027637 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc opening allocation metadata
>     >     >     >     > 2017-08-17 14:03:45.927394 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) _open_alloc loaded 3701 G in 3919 extents
>     >     >     >     > 2017-08-17 14:03:46.426774 7fcb138f8280  1 bluefs fsck
>     >     >     >     > 2017-08-17 14:03:46.426792 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck walking object keyspace
>     >     >     >     > 2017-08-17 14:03:50.856077 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking shared_blobs
>     >     >     >     > 2017-08-17 14:03:50.856119 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking for stray omap data
>     >     >     >     > 2017-08-17 14:03:52.405816 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking deferred events
>     >     >     >     > 2017-08-17 14:03:52.405862 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck checking freelist vs allocated
>     >     >     >     > 2017-08-17 14:04:06.311748 7fcb138f8280 -1 bluestore(/var/lib/ceph/osd/ceph-2) fsck error: leaked some space;65536 bytes leaked
>     >     >     >     > 2017-08-17 14:04:06.410019 7fcb138f8280  1 freelist shutdown
>     >     >     >     > 2017-08-17 14:04:06.410080 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:217] Shutdown: canceling all background work
>     >     >     >     > 2017-08-17 14:04:06.410464 7fcb138f8280  4 rocksdb: [/build/nitin/nightly_builds/20170815_141538-luminous/ceph.git/rpmbuild/BUILD/ceph-12.1.4-12.1.4/src/rocksdb/db/db_impl.cc:343] Shutdown complete
>     >     >     >     > 2017-08-17 14:04:06.451387 7fcb138f8280  1 bluefs umount
>     >     >     >     > 2017-08-17 14:04:06.452720 7fcb138f8280  1 bdev(0x5601f87b3800 /var/lib/ceph/osd/ceph-2/block.wal) close
>     >     >     >     > 2017-08-17 14:04:06.744745 7fcb138f8280  1 bdev(0x5601f87b3400 /var/lib/ceph/osd/ceph-2/block.db) close
>     >     >     >     > 2017-08-17 14:04:06.988729 7fcb138f8280  1 bdev(0x5601f87b3600 /var/lib/ceph/osd/ceph-2/block) close
>     >     >     >     > 2017-08-17 14:04:07.200717 7fcb138f8280  1 bdev(0x5601f87b3200 /var/lib/ceph/osd/ceph-2/block) close
>     >     >     >     > 2017-08-17 14:04:07.448859 7fcb138f8280  1 bluestore(/var/lib/ceph/osd/ceph-2) fsck finish with 1 errors in 23.553470 seconds
>     >     >     >     > fsck found 1 errors
>     >     >     >     > 
>     >     >     >     >     
>     >     >     >     > 
>     >     >     >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
>     >     >     > 
>     >     >     > N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay??ʇڙ??j???f???h???????w???????j:+v???w?????????????zZ+???????j"????i
>     >     > 
>     >     > 
>     > 
>     > 
> 
> N?????r??y??????X??ǧv???)޺{.n?????z?]z????ay?\x1dʇڙ??j\a??f???h?????\x1e?w???\f???j:+v???w????????\a????zZ+???????j"????i

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

On 8/25/17, 3:11 PM, "Sage Weil" <sweil@redhat.com> wrote:
Okay, I have a patch that repairs this.  See wip-21089-luminous, which 
    should appear at
    
    	https://shaman.ceph.com/builds/ceph/wip-21089-luminous/
    
    in the next hour or so.  If you run fsck with this build you see a line 
    like
    
    2017-08-25 18:01:58.656446 7f9e23825480 -1 bluestore(/home/sage/src/ceph4/build/dev/osd0) fsck warning: fixing leaked block 0x280000000~1000 due to old bug
    2017-08-25 18:01:58.656451 7f9e23825480 -1 bluestore(/home/sage/src/ceph4/build/dev/osd0) fsck  freelist block and key count changed, fixing 0x280001000~7f000
    
    go by in the log on the next fsck, and no such message thereafter.  Can 
    you give it a try on one of those OSDs to confirm it works?  I tested both 
    cases I see locally but want to ensure it works in your case too.
    
    Thanks!
    sage
    
    
Ok, will report my findings with this code.

Thanks,
Nitin
    

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Kamble, Nitin A]

Hi Sage,

I am planning to test in a normal setup. Do I need to set debug level 20 for bluestore to see these messages?

Nitin

On 8/25/17, 3:17 PM, "ceph-devel-owner@vger.kernel.org on behalf of Kamble, Nitin A" <ceph-devel-owner@vger.kernel.org on behalf of Nitin.Kamble@Teradata.com> wrote:

    
    
    On 8/25/17, 3:11 PM, "Sage Weil" <sweil@redhat.com> wrote:
    Okay, I have a patch that repairs this.  See wip-21089-luminous, which 
        should appear at
        
        	https://shaman.ceph.com/builds/ceph/wip-21089-luminous/
        
        in the next hour or so.  If you run fsck with this build you see a line 
        like
        
        2017-08-25 18:01:58.656446 7f9e23825480 -1 bluestore(/home/sage/src/ceph4/build/dev/osd0) fsck warning: fixing leaked block 0x280000000~1000 due to old bug
        2017-08-25 18:01:58.656451 7f9e23825480 -1 bluestore(/home/sage/src/ceph4/build/dev/osd0) fsck  freelist block and key count changed, fixing 0x280001000~7f000
        
        go by in the log on the next fsck, and no such message thereafter.  Can 
        you give it a try on one of those OSDs to confirm it works?  I tested both 
        cases I see locally but want to ensure it works in your case too.
        
        Thanks!
        sage
        
        
    Ok, will report my findings with this code.
    
    Thanks,
    Nitin
        
    
    ?�{.n�+�������+%��lzwm��b�맲��r��yǩ�ׯzX��?��ܨ}���Ơz�&j:+v���?����zZ+��+zf���h���~����i���z�?�w���?����&�)ߢ?f

Re: [Ceph-maintainers] [ceph-users] v12.1.4 Luminous (RC) released

[Henrik Korkuc]

On 17-08-17 04:02, Kamble, Nitin A wrote:
>
> On 8/16/17, 4:18 PM, "ceph-devel-owner@vger.kernel.org on behalf of Dan Mick" <ceph-devel-owner@vger.kernel.org on behalf of dmick@redhat.com> wrote:
>
>      On 08/16/2017 04:01 PM, Kamble, Nitin A wrote:
>      > After upgrading all the ceph nodes to v12.1.4 I see that OSD versions are still shown as older version.
>      > # ceph versions
>      > {
>      >     "mon": {
>      >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
>      >     },
>      >     "mgr": {
>      >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
>      >     },
>      >     "osd": {
>      >         "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50
>      >     },
>      >     "mds": {
>      >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 1
>      >     },
>      >     "overall": {
>      >         "ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev)": 50,
>      >         "ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)": 3
>      >     }
>      > }
>      >
>      >
>      > # ceph-osd --version
>      > ceph version 12.1.4 (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
>      >
>      > Why is it showing old 12.1.0 version for the 50 osds, while all the nodes show ceph osd version as 12.1.4?
>      
>      You didn't say you restarted the osds.  Did you?
>
> Yes, I did restart them, and that too more than once.
can you check what ceph-osd --version reports on one of affected hosts? 
Also "ceph daemon osd.OSD_NUM_ON_HOST version"? And "ps aux |grep 
ceph-osd" to check OSD start time, just in case
>
>      --
>      Dan Mick
>      Red Hat, Inc.
>      Ceph docs: http://ceph.com/docs
>      --
>      To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>      the body of a message to majordomo@vger.kernel.org
>      More majordomo info at  http://vger.kernel.org/majordomo-info.html
>      
>
> N�����r��y���b�X��ǧv�^�)޺{.n�+���z�]z���{ay�\x1dʇڙ�,j\a��f���h���z�\x1e�w���\f���j:+v���w�j�m����\a����zZ+�����ݢj"��!tml=

Re: [ceph-users] v12.1.4 Luminous (RC) released

[Nathan Cutler]

> *I* think that this is getting to a point where we should just have
> nightly development releases.

What if that night's development release is totally broken?

The point of having a release is that someone (a human - preferably one 
of the lead developers) has signed off on it. Releasing is inherently 
unautomatable, because when you automate it you lose that assurance.

Nathan

Re: [ceph-users] v12.1.4 Luminous (RC) released

[Alfredo Deza]

On Fri, Aug 18, 2017 at 1:39 PM, Nathan Cutler <ncutler@suse.cz> wrote:
>> *I* think that this is getting to a point where we should just have
>> nightly development releases.
>
>
> What if that night's development release is totally broken?

Broken like needing a new release? That is my point :) You wait until
the next day.

>
> The point of having a release is that someone (a human - preferably one of
> the lead developers) has signed off on it. Releasing is inherently
> unautomatable, because when you automate it you lose that assurance.

A regular, stable release, yes. Not a nightly/dev release.

>
> Nathan
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html