v0.56.2 released

v0.56.2 released

[Sage Weil]

The next bobtail point release is ready, and it's looking pretty good. 
This is an important update for the 0.56.x backport series that fixes a 
number of bugs and several performance issues. All v0.56.x users are 
encouraged to upgrade. 

Notable changes since v0.56.1:

 * osd: snapshot trimming fixes
 * osd: scrub snapshot metadata
 * osd: fix osdmap trimming
 * osd: misc peering fixes
 * osd: stop heartbeating with peers if internal threads are stuck/hung
 * osd: PG removal is friendlier to other workloads
 * osd: fix recovery start delay (was causing very slow recovery)
 * osd: fix scheduling of explicitly requested scrubs
 * osd: fix scrub interval config options
 * osd: improve recovery vs client io tuning
 * osd: improve 'slow request' warning detail for better diagnosis
 * osd: default CRUSH map now distributes across hosts, not OSDs
 * osd: fix crash on 32-bit hosts triggered by librbd clients
 * librbd: fix error handling when talking to older OSDs
 * mon: fix a few rare crashes
 * ceph command: ability to easily adjust CRUSH tunables
 * radosgw: object copy does not copy source ACLs
 * rados command: fix omap command usage
 * sysvinit script: set ulimit -n properly on remote hosts
 * msgr: fix narrow race with message queuing
 * fixed compilation on some old distros (e.g., RHEL 5.x)

There are a small number of interface changes related to the default CRUSH 
rule and scrub interval configuration options. Please see the full release 
notes.

You can get v0.56.2 in the usual fashion:

 * Git at git://github.com/ceph/ceph.git
 * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
 * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
 * For RPMs, see http://ceph.com/docs/master/install/rpm

Re: v0.56.2 released

[Stefan Priebe - Profihost AG]

Hello,

is it correct that the actual upstream/bobtail branch does not match the
v0.56.2 tag? The upstream/bobtail branch has 6 more commits from 25.01.
and 28.01.

Greets,
Stefan

Am 31.01.2013 00:46, schrieb Sage Weil:
> The next bobtail point release is ready, and it's looking pretty good. 
> This is an important update for the 0.56.x backport series that fixes a 
> number of bugs and several performance issues. All v0.56.x users are 
> encouraged to upgrade. 
> 
> Notable changes since v0.56.1:
> 
>  * osd: snapshot trimming fixes
>  * osd: scrub snapshot metadata
>  * osd: fix osdmap trimming
>  * osd: misc peering fixes
>  * osd: stop heartbeating with peers if internal threads are stuck/hung
>  * osd: PG removal is friendlier to other workloads
>  * osd: fix recovery start delay (was causing very slow recovery)
>  * osd: fix scheduling of explicitly requested scrubs
>  * osd: fix scrub interval config options
>  * osd: improve recovery vs client io tuning
>  * osd: improve 'slow request' warning detail for better diagnosis
>  * osd: default CRUSH map now distributes across hosts, not OSDs
>  * osd: fix crash on 32-bit hosts triggered by librbd clients
>  * librbd: fix error handling when talking to older OSDs
>  * mon: fix a few rare crashes
>  * ceph command: ability to easily adjust CRUSH tunables
>  * radosgw: object copy does not copy source ACLs
>  * rados command: fix omap command usage
>  * sysvinit script: set ulimit -n properly on remote hosts
>  * msgr: fix narrow race with message queuing
>  * fixed compilation on some old distros (e.g., RHEL 5.x)
> 
> There are a small number of interface changes related to the default CRUSH 
> rule and scrub interval configuration options. Please see the full release 
> notes.
> 
> You can get v0.56.2 in the usual fashion:
> 
>  * Git at git://github.com/ceph/ceph.git
>  * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
>  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>  * For RPMs, see http://ceph.com/docs/master/install/rpm
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: v0.56.2 released

[Stefan Priebe - Profihost AG]

Hello,

while compiling the bobtail branch i've seen this warning:
mon/PGMap.cc: In member function ‘void
PGMap::apply_incremental(CephContext*, const PGMap::Incremental&)’:
mon/PGMap.cc:247: warning: comparison between signed and unsigned
integer expressions
  CXX    libmon_a-LogMonitor.o

Greets,
Stefan

Am 31.01.2013 00:46, schrieb Sage Weil:
> The next bobtail point release is ready, and it's looking pretty good. 
> This is an important update for the 0.56.x backport series that fixes a 
> number of bugs and several performance issues. All v0.56.x users are 
> encouraged to upgrade. 
> 
> Notable changes since v0.56.1:
> 
>  * osd: snapshot trimming fixes
>  * osd: scrub snapshot metadata
>  * osd: fix osdmap trimming
>  * osd: misc peering fixes
>  * osd: stop heartbeating with peers if internal threads are stuck/hung
>  * osd: PG removal is friendlier to other workloads
>  * osd: fix recovery start delay (was causing very slow recovery)
>  * osd: fix scheduling of explicitly requested scrubs
>  * osd: fix scrub interval config options
>  * osd: improve recovery vs client io tuning
>  * osd: improve 'slow request' warning detail for better diagnosis
>  * osd: default CRUSH map now distributes across hosts, not OSDs
>  * osd: fix crash on 32-bit hosts triggered by librbd clients
>  * librbd: fix error handling when talking to older OSDs
>  * mon: fix a few rare crashes
>  * ceph command: ability to easily adjust CRUSH tunables
>  * radosgw: object copy does not copy source ACLs
>  * rados command: fix omap command usage
>  * sysvinit script: set ulimit -n properly on remote hosts
>  * msgr: fix narrow race with message queuing
>  * fixed compilation on some old distros (e.g., RHEL 5.x)
> 
> There are a small number of interface changes related to the default CRUSH 
> rule and scrub interval configuration options. Please see the full release 
> notes.
> 
> You can get v0.56.2 in the usual fashion:
> 
>  * Git at git://github.com/ceph/ceph.git
>  * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
>  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>  * For RPMs, see http://ceph.com/docs/master/install/rpm
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: v0.56.2 released

[Danny Al-Gaaf]

Am 31.01.2013 08:43, schrieb Stefan Priebe - Profihost AG:
> Hello,
> 
> while compiling the bobtail branch i've seen this warning:
> mon/PGMap.cc: In member function ‘void
> PGMap::apply_incremental(CephContext*, const PGMap::Incremental&)’:
> mon/PGMap.cc:247: warning: comparison between signed and unsigned
> integer expressions
>   CXX    libmon_a-LogMonitor.o

For fix see:
http://ceph.com/git/?p=ceph.git;a=commit;h=b571f8ee2d22a3894120204bc5f119ff37e1de53

@sage: please cherry pick b571f8ee2d22a3894120204bc5f119ff37e1de53 to
the bobtail branch.

Cheers,

Danny
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: v0.56.2 released

[Stefan Priebe - Profihost AG]

Hi,

great to see that we now have op/s and B/s output in ceph -w / ceph -s.

But is it reading or writing or both? Also if there are not ops the ;
and the rest of the line is missing instead of printing zeros. This
makes parsing harder.

See:
2013-01-31 10:46:42.045874 mon.0 [INF] pgmap v4037097: 8128 pgs: 8128
active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail; 8086B/s
wr, 1op/s

2013-01-31 10:46:43.056919 mon.0 [INF] pgmap v4037098: 8128 pgs: 8128
active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail

Stefan

Am 31.01.2013 08:43, schrieb Stefan Priebe - Profihost AG:
> Hello,
> 
> while compiling the bobtail branch i've seen this warning:
> mon/PGMap.cc: In member function ‘void
> PGMap::apply_incremental(CephContext*, const PGMap::Incremental&)’:
> mon/PGMap.cc:247: warning: comparison between signed and unsigned
> integer expressions
>   CXX    libmon_a-LogMonitor.o
> 
> Greets,
> Stefan
> 
> Am 31.01.2013 00:46, schrieb Sage Weil:
>> The next bobtail point release is ready, and it's looking pretty good. 
>> This is an important update for the 0.56.x backport series that fixes a 
>> number of bugs and several performance issues. All v0.56.x users are 
>> encouraged to upgrade. 
>>
>> Notable changes since v0.56.1:
>>
>>  * osd: snapshot trimming fixes
>>  * osd: scrub snapshot metadata
>>  * osd: fix osdmap trimming
>>  * osd: misc peering fixes
>>  * osd: stop heartbeating with peers if internal threads are stuck/hung
>>  * osd: PG removal is friendlier to other workloads
>>  * osd: fix recovery start delay (was causing very slow recovery)
>>  * osd: fix scheduling of explicitly requested scrubs
>>  * osd: fix scrub interval config options
>>  * osd: improve recovery vs client io tuning
>>  * osd: improve 'slow request' warning detail for better diagnosis
>>  * osd: default CRUSH map now distributes across hosts, not OSDs
>>  * osd: fix crash on 32-bit hosts triggered by librbd clients
>>  * librbd: fix error handling when talking to older OSDs
>>  * mon: fix a few rare crashes
>>  * ceph command: ability to easily adjust CRUSH tunables
>>  * radosgw: object copy does not copy source ACLs
>>  * rados command: fix omap command usage
>>  * sysvinit script: set ulimit -n properly on remote hosts
>>  * msgr: fix narrow race with message queuing
>>  * fixed compilation on some old distros (e.g., RHEL 5.x)
>>
>> There are a small number of interface changes related to the default CRUSH 
>> rule and scrub interval configuration options. Please see the full release 
>> notes.
>>
>> You can get v0.56.2 in the usual fashion:
>>
>>  * Git at git://github.com/ceph/ceph.git
>>  * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
>>  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>>  * For RPMs, see http://ceph.com/docs/master/install/rpm
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: v0.56.2 released

[Sage Weil]

On Thu, 31 Jan 2013, Stefan Priebe - Profihost AG wrote:
> Hi,
> 
> great to see that we now have op/s and B/s output in ceph -w / ceph -s.
> 
> But is it reading or writing or both? Also if there are not ops the ;
> and the rest of the line is missing instead of printing zeros. This
> makes parsing harder.
> 
> See:
> 2013-01-31 10:46:42.045874 mon.0 [INF] pgmap v4037097: 8128 pgs: 8128
> active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail; 8086B/s
> wr, 1op/s
> 
> 2013-01-31 10:46:43.056919 mon.0 [INF] pgmap v4037098: 8128 pgs: 8128
> active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail

This output is meant for a human.  If you need to parse it, we should be 
adding a --format=json option for ceph -s and/or -w so that's not 
necessary...

sage


> 
> Stefan
> 
> Am 31.01.2013 08:43, schrieb Stefan Priebe - Profihost AG:
> > Hello,
> > 
> > while compiling the bobtail branch i've seen this warning:
> > mon/PGMap.cc: In member function ?void
> > PGMap::apply_incremental(CephContext*, const PGMap::Incremental&)?:
> > mon/PGMap.cc:247: warning: comparison between signed and unsigned
> > integer expressions
> >   CXX    libmon_a-LogMonitor.o
> > 
> > Greets,
> > Stefan
> > 
> > Am 31.01.2013 00:46, schrieb Sage Weil:
> >> The next bobtail point release is ready, and it's looking pretty good. 
> >> This is an important update for the 0.56.x backport series that fixes a 
> >> number of bugs and several performance issues. All v0.56.x users are 
> >> encouraged to upgrade. 
> >>
> >> Notable changes since v0.56.1:
> >>
> >>  * osd: snapshot trimming fixes
> >>  * osd: scrub snapshot metadata
> >>  * osd: fix osdmap trimming
> >>  * osd: misc peering fixes
> >>  * osd: stop heartbeating with peers if internal threads are stuck/hung
> >>  * osd: PG removal is friendlier to other workloads
> >>  * osd: fix recovery start delay (was causing very slow recovery)
> >>  * osd: fix scheduling of explicitly requested scrubs
> >>  * osd: fix scrub interval config options
> >>  * osd: improve recovery vs client io tuning
> >>  * osd: improve 'slow request' warning detail for better diagnosis
> >>  * osd: default CRUSH map now distributes across hosts, not OSDs
> >>  * osd: fix crash on 32-bit hosts triggered by librbd clients
> >>  * librbd: fix error handling when talking to older OSDs
> >>  * mon: fix a few rare crashes
> >>  * ceph command: ability to easily adjust CRUSH tunables
> >>  * radosgw: object copy does not copy source ACLs
> >>  * rados command: fix omap command usage
> >>  * sysvinit script: set ulimit -n properly on remote hosts
> >>  * msgr: fix narrow race with message queuing
> >>  * fixed compilation on some old distros (e.g., RHEL 5.x)
> >>
> >> There are a small number of interface changes related to the default CRUSH 
> >> rule and scrub interval configuration options. Please see the full release 
> >> notes.
> >>
> >> You can get v0.56.2 in the usual fashion:
> >>
> >>  * Git at git://github.com/ceph/ceph.git
> >>  * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
> >>  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
> >>  * For RPMs, see http://ceph.com/docs/master/install/rpm
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> 
> 

Re: v0.56.2 released

[Joao Eduardo Luis]

On 01/31/2013 12:16 PM, Sage Weil wrote:
> On Thu, 31 Jan 2013, Stefan Priebe - Profihost AG wrote:
>> Hi,
>>
>> great to see that we now have op/s and B/s output in ceph -w / ceph -s.
>>
>> But is it reading or writing or both? Also if there are not ops the ;
>> and the rest of the line is missing instead of printing zeros. This
>> makes parsing harder.
>>
>> See:
>> 2013-01-31 10:46:42.045874 mon.0 [INF] pgmap v4037097: 8128 pgs: 8128
>> active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail; 8086B/s
>> wr, 1op/s
>>
>> 2013-01-31 10:46:43.056919 mon.0 [INF] pgmap v4037098: 8128 pgs: 8128
>> active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail
>
> This output is meant for a human.  If you need to parse it, we should be
> adding a --format=json option for ceph -s and/or -w so that's not
> necessary...
>
> sage

One of the timecheck-related patches on master introduced the 
possibility of outputting 'health' and 'status' in json.

'ceph status --format=json' however is only outputting the same info as 
its plain counterpart, but with a different entry for each map, sort of 
like this:

{
"monmap": "3 mons at 
{a=10.0.0.1:6789/0,b=10.0.0.2:6789/0,c=10.0.0.3:6789/0}",
"election_epoch": 2,
"quorum", "0 1 2",
...
}

This could be easily converted into something more useful though.

   -Joao


>
>
>>
>> Stefan
>>
>> Am 31.01.2013 08:43, schrieb Stefan Priebe - Profihost AG:
>>> Hello,
>>>
>>> while compiling the bobtail branch i've seen this warning:
>>> mon/PGMap.cc: In member function ?void
>>> PGMap::apply_incremental(CephContext*, const PGMap::Incremental&)?:
>>> mon/PGMap.cc:247: warning: comparison between signed and unsigned
>>> integer expressions
>>>    CXX    libmon_a-LogMonitor.o
>>>
>>> Greets,
>>> Stefan
>>>
>>> Am 31.01.2013 00:46, schrieb Sage Weil:
>>>> The next bobtail point release is ready, and it's looking pretty good.
>>>> This is an important update for the 0.56.x backport series that fixes a
>>>> number of bugs and several performance issues. All v0.56.x users are
>>>> encouraged to upgrade.
>>>>
>>>> Notable changes since v0.56.1:
>>>>
>>>>   * osd: snapshot trimming fixes
>>>>   * osd: scrub snapshot metadata
>>>>   * osd: fix osdmap trimming
>>>>   * osd: misc peering fixes
>>>>   * osd: stop heartbeating with peers if internal threads are stuck/hung
>>>>   * osd: PG removal is friendlier to other workloads
>>>>   * osd: fix recovery start delay (was causing very slow recovery)
>>>>   * osd: fix scheduling of explicitly requested scrubs
>>>>   * osd: fix scrub interval config options
>>>>   * osd: improve recovery vs client io tuning
>>>>   * osd: improve 'slow request' warning detail for better diagnosis
>>>>   * osd: default CRUSH map now distributes across hosts, not OSDs
>>>>   * osd: fix crash on 32-bit hosts triggered by librbd clients
>>>>   * librbd: fix error handling when talking to older OSDs
>>>>   * mon: fix a few rare crashes
>>>>   * ceph command: ability to easily adjust CRUSH tunables
>>>>   * radosgw: object copy does not copy source ACLs
>>>>   * rados command: fix omap command usage
>>>>   * sysvinit script: set ulimit -n properly on remote hosts
>>>>   * msgr: fix narrow race with message queuing
>>>>   * fixed compilation on some old distros (e.g., RHEL 5.x)
>>>>
>>>> There are a small number of interface changes related to the default CRUSH
>>>> rule and scrub interval configuration options. Please see the full release
>>>> notes.
>>>>
>>>> You can get v0.56.2 in the usual fashion:
>>>>
>>>>   * Git at git://github.com/ceph/ceph.git
>>>>   * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
>>>>   * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>>>>   * For RPMs, see http://ceph.com/docs/master/install/rpm
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: v0.56.2 released

[Stefan Priebe - Profihost AG]

this does not work:

#~ ceph --format=json -s

   health HEALTH_OK
   monmap e1: 3 mons at
{a=10.255.0.100:6789/0,b=10.255.0.101:6789/0,c=10.255.0.102:6789/0},
election epoch 2502, quorum 0,1,2 a,b,c
   osdmap e14994: 24 osds: 24 up, 24 in
    pgmap v4046683: 8128 pgs: 8128 active+clean; 172 GB data, 367 GB
used, 4968 GB / 5336 GB avail; 56588B/s wr, 8op/s
   mdsmap e1: 0/0/1 up

Stefan

Am 31.01.2013 13:16, schrieb Sage Weil:
> On Thu, 31 Jan 2013, Stefan Priebe - Profihost AG wrote:
>> Hi,
>>
>> great to see that we now have op/s and B/s output in ceph -w / ceph -s.
>>
>> But is it reading or writing or both? Also if there are not ops the ;
>> and the rest of the line is missing instead of printing zeros. This
>> makes parsing harder.
>>
>> See:
>> 2013-01-31 10:46:42.045874 mon.0 [INF] pgmap v4037097: 8128 pgs: 8128
>> active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail; 8086B/s
>> wr, 1op/s
>>
>> 2013-01-31 10:46:43.056919 mon.0 [INF] pgmap v4037098: 8128 pgs: 8128
>> active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail
> 
> This output is meant for a human.  If you need to parse it, we should be 
> adding a --format=json option for ceph -s and/or -w so that's not 
> necessary...
> 
> sage
> 
> 
>>
>> Stefan
>>
>> Am 31.01.2013 08:43, schrieb Stefan Priebe - Profihost AG:
>>> Hello,
>>>
>>> while compiling the bobtail branch i've seen this warning:
>>> mon/PGMap.cc: In member function ?void
>>> PGMap::apply_incremental(CephContext*, const PGMap::Incremental&)?:
>>> mon/PGMap.cc:247: warning: comparison between signed and unsigned
>>> integer expressions
>>>   CXX    libmon_a-LogMonitor.o
>>>
>>> Greets,
>>> Stefan
>>>
>>> Am 31.01.2013 00:46, schrieb Sage Weil:
>>>> The next bobtail point release is ready, and it's looking pretty good. 
>>>> This is an important update for the 0.56.x backport series that fixes a 
>>>> number of bugs and several performance issues. All v0.56.x users are 
>>>> encouraged to upgrade. 
>>>>
>>>> Notable changes since v0.56.1:
>>>>
>>>>  * osd: snapshot trimming fixes
>>>>  * osd: scrub snapshot metadata
>>>>  * osd: fix osdmap trimming
>>>>  * osd: misc peering fixes
>>>>  * osd: stop heartbeating with peers if internal threads are stuck/hung
>>>>  * osd: PG removal is friendlier to other workloads
>>>>  * osd: fix recovery start delay (was causing very slow recovery)
>>>>  * osd: fix scheduling of explicitly requested scrubs
>>>>  * osd: fix scrub interval config options
>>>>  * osd: improve recovery vs client io tuning
>>>>  * osd: improve 'slow request' warning detail for better diagnosis
>>>>  * osd: default CRUSH map now distributes across hosts, not OSDs
>>>>  * osd: fix crash on 32-bit hosts triggered by librbd clients
>>>>  * librbd: fix error handling when talking to older OSDs
>>>>  * mon: fix a few rare crashes
>>>>  * ceph command: ability to easily adjust CRUSH tunables
>>>>  * radosgw: object copy does not copy source ACLs
>>>>  * rados command: fix omap command usage
>>>>  * sysvinit script: set ulimit -n properly on remote hosts
>>>>  * msgr: fix narrow race with message queuing
>>>>  * fixed compilation on some old distros (e.g., RHEL 5.x)
>>>>
>>>> There are a small number of interface changes related to the default CRUSH 
>>>> rule and scrub interval configuration options. Please see the full release 
>>>> notes.
>>>>
>>>> You can get v0.56.2 in the usual fashion:
>>>>
>>>>  * Git at git://github.com/ceph/ceph.git
>>>>  * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
>>>>  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>>>>  * For RPMs, see http://ceph.com/docs/master/install/rpm
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>
>>

Re: v0.56.2 released

[Joao Eduardo Luis]

On 01/31/2013 12:46 PM, Stefan Priebe - Profihost AG wrote:
> this does not work:
>
> #~ ceph --format=json -s
>
>     health HEALTH_OK
>     monmap e1: 3 mons at
> {a=10.255.0.100:6789/0,b=10.255.0.101:6789/0,c=10.255.0.102:6789/0},
> election epoch 2502, quorum 0,1,2 a,b,c
>     osdmap e14994: 24 osds: 24 up, 24 in
>      pgmap v4046683: 8128 pgs: 8128 active+clean; 172 GB data, 367 GB
> used, 4968 GB / 5336 GB avail; 56588B/s wr, 8op/s
>     mdsmap e1: 0/0/1 up
>
> Stefan


The patches that would allow this are not on v0.56.2; they're in master 
though.

   -Joao

>
> Am 31.01.2013 13:16, schrieb Sage Weil:
>> On Thu, 31 Jan 2013, Stefan Priebe - Profihost AG wrote:
>>> Hi,
>>>
>>> great to see that we now have op/s and B/s output in ceph -w / ceph -s.
>>>
>>> But is it reading or writing or both? Also if there are not ops the ;
>>> and the rest of the line is missing instead of printing zeros. This
>>> makes parsing harder.
>>>
>>> See:
>>> 2013-01-31 10:46:42.045874 mon.0 [INF] pgmap v4037097: 8128 pgs: 8128
>>> active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail; 8086B/s
>>> wr, 1op/s
>>>
>>> 2013-01-31 10:46:43.056919 mon.0 [INF] pgmap v4037098: 8128 pgs: 8128
>>> active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail
>>
>> This output is meant for a human.  If you need to parse it, we should be
>> adding a --format=json option for ceph -s and/or -w so that's not
>> necessary...
>>
>> sage
>>
>>
>>>
>>> Stefan
>>>
>>> Am 31.01.2013 08:43, schrieb Stefan Priebe - Profihost AG:
>>>> Hello,
>>>>
>>>> while compiling the bobtail branch i've seen this warning:
>>>> mon/PGMap.cc: In member function ?void
>>>> PGMap::apply_incremental(CephContext*, const PGMap::Incremental&)?:
>>>> mon/PGMap.cc:247: warning: comparison between signed and unsigned
>>>> integer expressions
>>>>    CXX    libmon_a-LogMonitor.o
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>> Am 31.01.2013 00:46, schrieb Sage Weil:
>>>>> The next bobtail point release is ready, and it's looking pretty good.
>>>>> This is an important update for the 0.56.x backport series that fixes a
>>>>> number of bugs and several performance issues. All v0.56.x users are
>>>>> encouraged to upgrade.
>>>>>
>>>>> Notable changes since v0.56.1:
>>>>>
>>>>>   * osd: snapshot trimming fixes
>>>>>   * osd: scrub snapshot metadata
>>>>>   * osd: fix osdmap trimming
>>>>>   * osd: misc peering fixes
>>>>>   * osd: stop heartbeating with peers if internal threads are stuck/hung
>>>>>   * osd: PG removal is friendlier to other workloads
>>>>>   * osd: fix recovery start delay (was causing very slow recovery)
>>>>>   * osd: fix scheduling of explicitly requested scrubs
>>>>>   * osd: fix scrub interval config options
>>>>>   * osd: improve recovery vs client io tuning
>>>>>   * osd: improve 'slow request' warning detail for better diagnosis
>>>>>   * osd: default CRUSH map now distributes across hosts, not OSDs
>>>>>   * osd: fix crash on 32-bit hosts triggered by librbd clients
>>>>>   * librbd: fix error handling when talking to older OSDs
>>>>>   * mon: fix a few rare crashes
>>>>>   * ceph command: ability to easily adjust CRUSH tunables
>>>>>   * radosgw: object copy does not copy source ACLs
>>>>>   * rados command: fix omap command usage
>>>>>   * sysvinit script: set ulimit -n properly on remote hosts
>>>>>   * msgr: fix narrow race with message queuing
>>>>>   * fixed compilation on some old distros (e.g., RHEL 5.x)
>>>>>
>>>>> There are a small number of interface changes related to the default CRUSH
>>>>> rule and scrub interval configuration options. Please see the full release
>>>>> notes.
>>>>>
>>>>> You can get v0.56.2 in the usual fashion:
>>>>>
>>>>>   * Git at git://github.com/ceph/ceph.git
>>>>>   * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
>>>>>   * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>>>>>   * For RPMs, see http://ceph.com/docs/master/install/rpm
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: v0.56.2 released

[Joao Eduardo Luis]

On 01/31/2013 12:54 PM, Joao Eduardo Luis wrote:
> On 01/31/2013 12:46 PM, Stefan Priebe - Profihost AG wrote:
>> this does not work:
>>
>> #~ ceph --format=json -s
>>
>>     health HEALTH_OK
>>     monmap e1: 3 mons at
>> {a=10.255.0.100:6789/0,b=10.255.0.101:6789/0,c=10.255.0.102:6789/0},
>> election epoch 2502, quorum 0,1,2 a,b,c
>>     osdmap e14994: 24 osds: 24 up, 24 in
>>      pgmap v4046683: 8128 pgs: 8128 active+clean; 172 GB data, 367 GB
>> used, 4968 GB / 5336 GB avail; 56588B/s wr, 8op/s
>>     mdsmap e1: 0/0/1 up
>>
>> Stefan
>
>
> The patches that would allow this are not on v0.56.2; they're in master
> though.

And I just realized that it would not work with 'ceph -s', but would 
with 'ceph status'; this is due to the way the 'ceph' tool handles 
arguments and the fact that '-s' is a special case.

So if you're trying it on master, try 'ceph status --format=json' 
instead of 'ceph -s --format=json' :)

   -Joao

>>
>> Am 31.01.2013 13:16, schrieb Sage Weil:
>>> On Thu, 31 Jan 2013, Stefan Priebe - Profihost AG wrote:
>>>> Hi,
>>>>
>>>> great to see that we now have op/s and B/s output in ceph -w / ceph -s.
>>>>
>>>> But is it reading or writing or both? Also if there are not ops the ;
>>>> and the rest of the line is missing instead of printing zeros. This
>>>> makes parsing harder.
>>>>
>>>> See:
>>>> 2013-01-31 10:46:42.045874 mon.0 [INF] pgmap v4037097: 8128 pgs: 8128
>>>> active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail;
>>>> 8086B/s
>>>> wr, 1op/s
>>>>
>>>> 2013-01-31 10:46:43.056919 mon.0 [INF] pgmap v4037098: 8128 pgs: 8128
>>>> active+clean; 172 GB data, 366 GB used, 4970 GB / 5336 GB avail
>>>
>>> This output is meant for a human.  If you need to parse it, we should be
>>> adding a --format=json option for ceph -s and/or -w so that's not
>>> necessary...
>>>
>>> sage
>>>
>>>
>>>>
>>>> Stefan
>>>>
>>>> Am 31.01.2013 08:43, schrieb Stefan Priebe - Profihost AG:
>>>>> Hello,
>>>>>
>>>>> while compiling the bobtail branch i've seen this warning:
>>>>> mon/PGMap.cc: In member function ?void
>>>>> PGMap::apply_incremental(CephContext*, const PGMap::Incremental&)?:
>>>>> mon/PGMap.cc:247: warning: comparison between signed and unsigned
>>>>> integer expressions
>>>>>    CXX    libmon_a-LogMonitor.o
>>>>>
>>>>> Greets,
>>>>> Stefan
>>>>>
>>>>> Am 31.01.2013 00:46, schrieb Sage Weil:
>>>>>> The next bobtail point release is ready, and it's looking pretty
>>>>>> good.
>>>>>> This is an important update for the 0.56.x backport series that
>>>>>> fixes a
>>>>>> number of bugs and several performance issues. All v0.56.x users are
>>>>>> encouraged to upgrade.
>>>>>>
>>>>>> Notable changes since v0.56.1:
>>>>>>
>>>>>>   * osd: snapshot trimming fixes
>>>>>>   * osd: scrub snapshot metadata
>>>>>>   * osd: fix osdmap trimming
>>>>>>   * osd: misc peering fixes
>>>>>>   * osd: stop heartbeating with peers if internal threads are
>>>>>> stuck/hung
>>>>>>   * osd: PG removal is friendlier to other workloads
>>>>>>   * osd: fix recovery start delay (was causing very slow recovery)
>>>>>>   * osd: fix scheduling of explicitly requested scrubs
>>>>>>   * osd: fix scrub interval config options
>>>>>>   * osd: improve recovery vs client io tuning
>>>>>>   * osd: improve 'slow request' warning detail for better diagnosis
>>>>>>   * osd: default CRUSH map now distributes across hosts, not OSDs
>>>>>>   * osd: fix crash on 32-bit hosts triggered by librbd clients
>>>>>>   * librbd: fix error handling when talking to older OSDs
>>>>>>   * mon: fix a few rare crashes
>>>>>>   * ceph command: ability to easily adjust CRUSH tunables
>>>>>>   * radosgw: object copy does not copy source ACLs
>>>>>>   * rados command: fix omap command usage
>>>>>>   * sysvinit script: set ulimit -n properly on remote hosts
>>>>>>   * msgr: fix narrow race with message queuing
>>>>>>   * fixed compilation on some old distros (e.g., RHEL 5.x)
>>>>>>
>>>>>> There are a small number of interface changes related to the
>>>>>> default CRUSH
>>>>>> rule and scrub interval configuration options. Please see the full
>>>>>> release
>>>>>> notes.
>>>>>>
>>>>>> You can get v0.56.2 in the usual fashion:
>>>>>>
>>>>>>   * Git at git://github.com/ceph/ceph.git
>>>>>>   * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
>>>>>>   * For Debian/Ubuntu packages, see
>>>>>> http://ceph.com/docs/master/install/debian
>>>>>>   * For RPMs, see http://ceph.com/docs/master/install/rpm
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> ceph-devel" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>
>>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: v0.56.2 released

[ArtemGr]

Sage Weil <sage <at> inktank.com> writes:
> The next bobtail point release is ready, and it's looking pretty good. 
> This is an important update for the 0.56.x backport series that fixes a 
> number of bugs and several performance issues. All v0.56.x users are 
> encouraged to upgrade.

Is it just me, or is the Debian release missing?
I have "deb http://ceph.com/debian-testing/ wheezy main" in "sources.list"
and am not seeing the v0.56.2.

Re: v0.56.2 released

[Sage Weil]

On Sun, 3 Feb 2013, ArtemGr wrote:
> Sage Weil <sage <at> inktank.com> writes:
> > The next bobtail point release is ready, and it's looking pretty good. 
> > This is an important update for the 0.56.x backport series that fixes a 
> > number of bugs and several performance issues. All v0.56.x users are 
> > encouraged to upgrade.
> 
> Is it just me, or is the Debian release missing?
> I have "deb http://ceph.com/debian-testing/ wheezy main" in "sources.list"
> and am not seeing the v0.56.2.

Try 'deb http://ceph.com/debian-bobtail/ wheezy main'.

It only got pushed to the bobtail repo, which is slightly awkward only 
because 0.57 isn't out yet.

sage

Re: v0.56.2 released

[Mandell Degerness]

We are using v0.56.2 and it still seems that the default crushmap is
osd centered.  Here is the crushmap as dumped:

[root@node-172-19-0-15 ~]# cat crush.txt
# begin crush map

# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5
device 6 osd.6
device 7 osd.7
device 8 osd.8
device 9 osd.9
device 10 osd.10
device 11 osd.11
device 12 osd.12
device 13 osd.13
device 14 osd.14

# types
type 0 osd
type 1 host
type 2 rack
type 3 row
type 4 room
type 5 datacenter
type 6 root

# buckets
host 172.19.0.14 {
id -2 # do not change unnecessarily
# weight 6.000
alg straw
hash 0 # rjenkins1
item osd.0 weight 1.000
item osd.4 weight 1.000
item osd.5 weight 1.000
item osd.12 weight 1.000
item osd.13 weight 1.000
item osd.14 weight 1.000
}
host 172.19.0.13 {
id -4 # do not change unnecessarily
# weight 3.000
alg straw
hash 0 # rjenkins1
item osd.2 weight 1.000
item osd.7 weight 1.000
item osd.9 weight 1.000
}
host 172.19.0.16 {
id -5 # do not change unnecessarily
# weight 3.000
alg straw
hash 0 # rjenkins1
item osd.3 weight 1.000
item osd.6 weight 1.000
item osd.10 weight 1.000
}
host 172.19.0.15 {
id -6 # do not change unnecessarily
# weight 3.000
alg straw
hash 0 # rjenkins1
item osd.1 weight 1.000
item osd.8 weight 1.000
item osd.11 weight 1.000
}
rack 0 {
id -3 # do not change unnecessarily
# weight 15.000
alg straw
hash 0 # rjenkins1
item 172.19.0.14 weight 6.000
item 172.19.0.13 weight 3.000
item 172.19.0.16 weight 3.000
item 172.19.0.15 weight 3.000
}
root default {
id -1 # do not change unnecessarily
# weight 15.000
alg straw
hash 0 # rjenkins1
item 0 weight 15.000
}

# rules
rule data {
ruleset 0
type replicated
min_size 1
max_size 10
step take default
step choose firstn 0 type osd
step emit
}
rule metadata {
ruleset 1
type replicated
min_size 1
max_size 10
step take default
step choose firstn 0 type osd
step emit
}
rule rbd {
ruleset 2
type replicated
min_size 1
max_size 10
step take default
step choose firstn 0 type osd
step emit
}

# end crush map
[root@node-172-19-0-15 ~]# ceph --version
ceph version 0.56.2 (586538e22afba85c59beda49789ec42024e7a061)

We do not run any explicit crushtool commands as part of our start up
at this time.  Should we be?

Regards,
Mandell Degerness

On Wed, Jan 30, 2013 at 3:46 PM, Sage Weil <sage@inktank.com> wrote:
> The next bobtail point release is ready, and it's looking pretty good.
> This is an important update for the 0.56.x backport series that fixes a
> number of bugs and several performance issues. All v0.56.x users are
> encouraged to upgrade.
>
> Notable changes since v0.56.1:
>
>  * osd: snapshot trimming fixes
>  * osd: scrub snapshot metadata
>  * osd: fix osdmap trimming
>  * osd: misc peering fixes
>  * osd: stop heartbeating with peers if internal threads are stuck/hung
>  * osd: PG removal is friendlier to other workloads
>  * osd: fix recovery start delay (was causing very slow recovery)
>  * osd: fix scheduling of explicitly requested scrubs
>  * osd: fix scrub interval config options
>  * osd: improve recovery vs client io tuning
>  * osd: improve 'slow request' warning detail for better diagnosis
>  * osd: default CRUSH map now distributes across hosts, not OSDs
>  * osd: fix crash on 32-bit hosts triggered by librbd clients
>  * librbd: fix error handling when talking to older OSDs
>  * mon: fix a few rare crashes
>  * ceph command: ability to easily adjust CRUSH tunables
>  * radosgw: object copy does not copy source ACLs
>  * rados command: fix omap command usage
>  * sysvinit script: set ulimit -n properly on remote hosts
>  * msgr: fix narrow race with message queuing
>  * fixed compilation on some old distros (e.g., RHEL 5.x)
>
> There are a small number of interface changes related to the default CRUSH
> rule and scrub interval configuration options. Please see the full release
> notes.
>
> You can get v0.56.2 in the usual fashion:
>
>  * Git at git://github.com/ceph/ceph.git
>  * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
>  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>  * For RPMs, see http://ceph.com/docs/master/install/rpm
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: v0.56.2 released

[Sage Weil]

On Tue, 5 Feb 2013, Mandell Degerness wrote:
> We are using v0.56.2 and it still seems that the default crushmap is
> osd centered.  Here is the crushmap as dumped:
> 
> [root@node-172-19-0-15 ~]# cat crush.txt
> # begin crush map
> 
> # devices
> device 0 osd.0
> device 1 osd.1
> device 2 osd.2
> device 3 osd.3
> device 4 osd.4
> device 5 osd.5
> device 6 osd.6
> device 7 osd.7
> device 8 osd.8
> device 9 osd.9
> device 10 osd.10
> device 11 osd.11
> device 12 osd.12
> device 13 osd.13
> device 14 osd.14
> 
> # types
> type 0 osd
> type 1 host
> type 2 rack
> type 3 row
> type 4 room
> type 5 datacenter
> type 6 root
> 
> # buckets
> host 172.19.0.14 {
> id -2 # do not change unnecessarily
> # weight 6.000
> alg straw
> hash 0 # rjenkins1
> item osd.0 weight 1.000
> item osd.4 weight 1.000
> item osd.5 weight 1.000
> item osd.12 weight 1.000
> item osd.13 weight 1.000
> item osd.14 weight 1.000
> }
> host 172.19.0.13 {
> id -4 # do not change unnecessarily
> # weight 3.000
> alg straw
> hash 0 # rjenkins1
> item osd.2 weight 1.000
> item osd.7 weight 1.000
> item osd.9 weight 1.000
> }
> host 172.19.0.16 {
> id -5 # do not change unnecessarily
> # weight 3.000
> alg straw
> hash 0 # rjenkins1
> item osd.3 weight 1.000
> item osd.6 weight 1.000
> item osd.10 weight 1.000
> }
> host 172.19.0.15 {
> id -6 # do not change unnecessarily
> # weight 3.000
> alg straw
> hash 0 # rjenkins1
> item osd.1 weight 1.000
> item osd.8 weight 1.000
> item osd.11 weight 1.000
> }
> rack 0 {
> id -3 # do not change unnecessarily
> # weight 15.000
> alg straw
> hash 0 # rjenkins1
> item 172.19.0.14 weight 6.000
> item 172.19.0.13 weight 3.000
> item 172.19.0.16 weight 3.000
> item 172.19.0.15 weight 3.000
> }
> root default {
> id -1 # do not change unnecessarily
> # weight 15.000
> alg straw
> hash 0 # rjenkins1
> item 0 weight 15.000
> }
> 
> # rules
> rule data {
> ruleset 0
> type replicated
> min_size 1
> max_size 10
> step take default
> step choose firstn 0 type osd
> step emit
> }
> rule metadata {
> ruleset 1
> type replicated
> min_size 1
> max_size 10
> step take default
> step choose firstn 0 type osd
> step emit
> }
> rule rbd {
> ruleset 2
> type replicated
> min_size 1
> max_size 10
> step take default
> step choose firstn 0 type osd
> step emit
> }
> 
> # end crush map
> [root@node-172-19-0-15 ~]# ceph --version
> ceph version 0.56.2 (586538e22afba85c59beda49789ec42024e7a061)
> 
> We do not run any explicit crushtool commands as part of our start up
> at this time.  Should we be?

Do you run mkcephfs?  If you are passing in a ceph.conf to mkcephfs, it is 
still dynamically choosing a rule based on whether you have enough osds (3 
i think?  i forget).  If you are running ceph-mon --mkfs directly (as 
ceph-deploy, chef, juju do), it will always default to osds.

sage

> 
> Regards,
> Mandell Degerness
> 
> On Wed, Jan 30, 2013 at 3:46 PM, Sage Weil <sage@inktank.com> wrote:
> > The next bobtail point release is ready, and it's looking pretty good.
> > This is an important update for the 0.56.x backport series that fixes a
> > number of bugs and several performance issues. All v0.56.x users are
> > encouraged to upgrade.
> >
> > Notable changes since v0.56.1:
> >
> >  * osd: snapshot trimming fixes
> >  * osd: scrub snapshot metadata
> >  * osd: fix osdmap trimming
> >  * osd: misc peering fixes
> >  * osd: stop heartbeating with peers if internal threads are stuck/hung
> >  * osd: PG removal is friendlier to other workloads
> >  * osd: fix recovery start delay (was causing very slow recovery)
> >  * osd: fix scheduling of explicitly requested scrubs
> >  * osd: fix scrub interval config options
> >  * osd: improve recovery vs client io tuning
> >  * osd: improve 'slow request' warning detail for better diagnosis
> >  * osd: default CRUSH map now distributes across hosts, not OSDs
> >  * osd: fix crash on 32-bit hosts triggered by librbd clients
> >  * librbd: fix error handling when talking to older OSDs
> >  * mon: fix a few rare crashes
> >  * ceph command: ability to easily adjust CRUSH tunables
> >  * radosgw: object copy does not copy source ACLs
> >  * rados command: fix omap command usage
> >  * sysvinit script: set ulimit -n properly on remote hosts
> >  * msgr: fix narrow race with message queuing
> >  * fixed compilation on some old distros (e.g., RHEL 5.x)
> >
> > There are a small number of interface changes related to the default CRUSH
> > rule and scrub interval configuration options. Please see the full release
> > notes.
> >
> > You can get v0.56.2 in the usual fashion:
> >
> >  * Git at git://github.com/ceph/ceph.git
> >  * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
> >  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
> >  * For RPMs, see http://ceph.com/docs/master/install/rpm
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: v0.56.2 released

[Mandell Degerness]

We are doing the latter.  What is the best way to force it to host
first selection?

On Tue, Feb 5, 2013 at 11:34 AM, Sage Weil <sage@inktank.com> wrote:
> On Tue, 5 Feb 2013, Mandell Degerness wrote:
>> We are using v0.56.2 and it still seems that the default crushmap is
>> osd centered.  Here is the crushmap as dumped:
>>
>> [root@node-172-19-0-15 ~]# cat crush.txt
>> # begin crush map
>>
>> # devices
>> device 0 osd.0
>> device 1 osd.1
>> device 2 osd.2
>> device 3 osd.3
>> device 4 osd.4
>> device 5 osd.5
>> device 6 osd.6
>> device 7 osd.7
>> device 8 osd.8
>> device 9 osd.9
>> device 10 osd.10
>> device 11 osd.11
>> device 12 osd.12
>> device 13 osd.13
>> device 14 osd.14
>>
>> # types
>> type 0 osd
>> type 1 host
>> type 2 rack
>> type 3 row
>> type 4 room
>> type 5 datacenter
>> type 6 root
>>
>> # buckets
>> host 172.19.0.14 {
>> id -2 # do not change unnecessarily
>> # weight 6.000
>> alg straw
>> hash 0 # rjenkins1
>> item osd.0 weight 1.000
>> item osd.4 weight 1.000
>> item osd.5 weight 1.000
>> item osd.12 weight 1.000
>> item osd.13 weight 1.000
>> item osd.14 weight 1.000
>> }
>> host 172.19.0.13 {
>> id -4 # do not change unnecessarily
>> # weight 3.000
>> alg straw
>> hash 0 # rjenkins1
>> item osd.2 weight 1.000
>> item osd.7 weight 1.000
>> item osd.9 weight 1.000
>> }
>> host 172.19.0.16 {
>> id -5 # do not change unnecessarily
>> # weight 3.000
>> alg straw
>> hash 0 # rjenkins1
>> item osd.3 weight 1.000
>> item osd.6 weight 1.000
>> item osd.10 weight 1.000
>> }
>> host 172.19.0.15 {
>> id -6 # do not change unnecessarily
>> # weight 3.000
>> alg straw
>> hash 0 # rjenkins1
>> item osd.1 weight 1.000
>> item osd.8 weight 1.000
>> item osd.11 weight 1.000
>> }
>> rack 0 {
>> id -3 # do not change unnecessarily
>> # weight 15.000
>> alg straw
>> hash 0 # rjenkins1
>> item 172.19.0.14 weight 6.000
>> item 172.19.0.13 weight 3.000
>> item 172.19.0.16 weight 3.000
>> item 172.19.0.15 weight 3.000
>> }
>> root default {
>> id -1 # do not change unnecessarily
>> # weight 15.000
>> alg straw
>> hash 0 # rjenkins1
>> item 0 weight 15.000
>> }
>>
>> # rules
>> rule data {
>> ruleset 0
>> type replicated
>> min_size 1
>> max_size 10
>> step take default
>> step choose firstn 0 type osd
>> step emit
>> }
>> rule metadata {
>> ruleset 1
>> type replicated
>> min_size 1
>> max_size 10
>> step take default
>> step choose firstn 0 type osd
>> step emit
>> }
>> rule rbd {
>> ruleset 2
>> type replicated
>> min_size 1
>> max_size 10
>> step take default
>> step choose firstn 0 type osd
>> step emit
>> }
>>
>> # end crush map
>> [root@node-172-19-0-15 ~]# ceph --version
>> ceph version 0.56.2 (586538e22afba85c59beda49789ec42024e7a061)
>>
>> We do not run any explicit crushtool commands as part of our start up
>> at this time.  Should we be?
>
> Do you run mkcephfs?  If you are passing in a ceph.conf to mkcephfs, it is
> still dynamically choosing a rule based on whether you have enough osds (3
> i think?  i forget).  If you are running ceph-mon --mkfs directly (as
> ceph-deploy, chef, juju do), it will always default to osds.
>
> sage
>
>>
>> Regards,
>> Mandell Degerness
>>
>> On Wed, Jan 30, 2013 at 3:46 PM, Sage Weil <sage@inktank.com> wrote:
>> > The next bobtail point release is ready, and it's looking pretty good.
>> > This is an important update for the 0.56.x backport series that fixes a
>> > number of bugs and several performance issues. All v0.56.x users are
>> > encouraged to upgrade.
>> >
>> > Notable changes since v0.56.1:
>> >
>> >  * osd: snapshot trimming fixes
>> >  * osd: scrub snapshot metadata
>> >  * osd: fix osdmap trimming
>> >  * osd: misc peering fixes
>> >  * osd: stop heartbeating with peers if internal threads are stuck/hung
>> >  * osd: PG removal is friendlier to other workloads
>> >  * osd: fix recovery start delay (was causing very slow recovery)
>> >  * osd: fix scheduling of explicitly requested scrubs
>> >  * osd: fix scrub interval config options
>> >  * osd: improve recovery vs client io tuning
>> >  * osd: improve 'slow request' warning detail for better diagnosis
>> >  * osd: default CRUSH map now distributes across hosts, not OSDs
>> >  * osd: fix crash on 32-bit hosts triggered by librbd clients
>> >  * librbd: fix error handling when talking to older OSDs
>> >  * mon: fix a few rare crashes
>> >  * ceph command: ability to easily adjust CRUSH tunables
>> >  * radosgw: object copy does not copy source ACLs
>> >  * rados command: fix omap command usage
>> >  * sysvinit script: set ulimit -n properly on remote hosts
>> >  * msgr: fix narrow race with message queuing
>> >  * fixed compilation on some old distros (e.g., RHEL 5.x)
>> >
>> > There are a small number of interface changes related to the default CRUSH
>> > rule and scrub interval configuration options. Please see the full release
>> > notes.
>> >
>> > You can get v0.56.2 in the usual fashion:
>> >
>> >  * Git at git://github.com/ceph/ceph.git
>> >  * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
>> >  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>> >  * For RPMs, see http://ceph.com/docs/master/install/rpm
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>

Re: v0.56.2 released

[Sage Weil]

On Tue, 5 Feb 2013, Mandell Degerness wrote:
> We are doing the latter.  What is the best way to force it to host
> first selection?

Checking the code, you should be seeing either hosts (unless you've 
changed osd_crush_chooseleaf_type from its default of 1).  Or, if the map 
generation is being done from a prepared ceph.conf, it should do hosts if 
there is more than one host in the initial conf.

Maybe you're doing the ceph-mon --mkfs from an initial .conf that has only 
1 host to begin with, and others are added after?

I can make it observe the config setting in all cases (instead of trying 
to guess what the user wants).  I pushed a patch to wip-osdmap-chooseleaf 
that you can cherry-pick that does this.. can you let me know if that 
resolves the issue, and/or if you think this makes the most sense for 
users?

sage




> 
> On Tue, Feb 5, 2013 at 11:34 AM, Sage Weil <sage@inktank.com> wrote:
> > On Tue, 5 Feb 2013, Mandell Degerness wrote:
> >> We are using v0.56.2 and it still seems that the default crushmap is
> >> osd centered.  Here is the crushmap as dumped:
> >>
> >> [root@node-172-19-0-15 ~]# cat crush.txt
> >> # begin crush map
> >>
> >> # devices
> >> device 0 osd.0
> >> device 1 osd.1
> >> device 2 osd.2
> >> device 3 osd.3
> >> device 4 osd.4
> >> device 5 osd.5
> >> device 6 osd.6
> >> device 7 osd.7
> >> device 8 osd.8
> >> device 9 osd.9
> >> device 10 osd.10
> >> device 11 osd.11
> >> device 12 osd.12
> >> device 13 osd.13
> >> device 14 osd.14
> >>
> >> # types
> >> type 0 osd
> >> type 1 host
> >> type 2 rack
> >> type 3 row
> >> type 4 room
> >> type 5 datacenter
> >> type 6 root
> >>
> >> # buckets
> >> host 172.19.0.14 {
> >> id -2 # do not change unnecessarily
> >> # weight 6.000
> >> alg straw
> >> hash 0 # rjenkins1
> >> item osd.0 weight 1.000
> >> item osd.4 weight 1.000
> >> item osd.5 weight 1.000
> >> item osd.12 weight 1.000
> >> item osd.13 weight 1.000
> >> item osd.14 weight 1.000
> >> }
> >> host 172.19.0.13 {
> >> id -4 # do not change unnecessarily
> >> # weight 3.000
> >> alg straw
> >> hash 0 # rjenkins1
> >> item osd.2 weight 1.000
> >> item osd.7 weight 1.000
> >> item osd.9 weight 1.000
> >> }
> >> host 172.19.0.16 {
> >> id -5 # do not change unnecessarily
> >> # weight 3.000
> >> alg straw
> >> hash 0 # rjenkins1
> >> item osd.3 weight 1.000
> >> item osd.6 weight 1.000
> >> item osd.10 weight 1.000
> >> }
> >> host 172.19.0.15 {
> >> id -6 # do not change unnecessarily
> >> # weight 3.000
> >> alg straw
> >> hash 0 # rjenkins1
> >> item osd.1 weight 1.000
> >> item osd.8 weight 1.000
> >> item osd.11 weight 1.000
> >> }
> >> rack 0 {
> >> id -3 # do not change unnecessarily
> >> # weight 15.000
> >> alg straw
> >> hash 0 # rjenkins1
> >> item 172.19.0.14 weight 6.000
> >> item 172.19.0.13 weight 3.000
> >> item 172.19.0.16 weight 3.000
> >> item 172.19.0.15 weight 3.000
> >> }
> >> root default {
> >> id -1 # do not change unnecessarily
> >> # weight 15.000
> >> alg straw
> >> hash 0 # rjenkins1
> >> item 0 weight 15.000
> >> }
> >>
> >> # rules
> >> rule data {
> >> ruleset 0
> >> type replicated
> >> min_size 1
> >> max_size 10
> >> step take default
> >> step choose firstn 0 type osd
> >> step emit
> >> }
> >> rule metadata {
> >> ruleset 1
> >> type replicated
> >> min_size 1
> >> max_size 10
> >> step take default
> >> step choose firstn 0 type osd
> >> step emit
> >> }
> >> rule rbd {
> >> ruleset 2
> >> type replicated
> >> min_size 1
> >> max_size 10
> >> step take default
> >> step choose firstn 0 type osd
> >> step emit
> >> }
> >>
> >> # end crush map
> >> [root@node-172-19-0-15 ~]# ceph --version
> >> ceph version 0.56.2 (586538e22afba85c59beda49789ec42024e7a061)
> >>
> >> We do not run any explicit crushtool commands as part of our start up
> >> at this time.  Should we be?
> >
> > Do you run mkcephfs?  If you are passing in a ceph.conf to mkcephfs, it is
> > still dynamically choosing a rule based on whether you have enough osds (3
> > i think?  i forget).  If you are running ceph-mon --mkfs directly (as
> > ceph-deploy, chef, juju do), it will always default to osds.
> >
> > sage
> >
> >>
> >> Regards,
> >> Mandell Degerness
> >>
> >> On Wed, Jan 30, 2013 at 3:46 PM, Sage Weil <sage@inktank.com> wrote:
> >> > The next bobtail point release is ready, and it's looking pretty good.
> >> > This is an important update for the 0.56.x backport series that fixes a
> >> > number of bugs and several performance issues. All v0.56.x users are
> >> > encouraged to upgrade.
> >> >
> >> > Notable changes since v0.56.1:
> >> >
> >> >  * osd: snapshot trimming fixes
> >> >  * osd: scrub snapshot metadata
> >> >  * osd: fix osdmap trimming
> >> >  * osd: misc peering fixes
> >> >  * osd: stop heartbeating with peers if internal threads are stuck/hung
> >> >  * osd: PG removal is friendlier to other workloads
> >> >  * osd: fix recovery start delay (was causing very slow recovery)
> >> >  * osd: fix scheduling of explicitly requested scrubs
> >> >  * osd: fix scrub interval config options
> >> >  * osd: improve recovery vs client io tuning
> >> >  * osd: improve 'slow request' warning detail for better diagnosis
> >> >  * osd: default CRUSH map now distributes across hosts, not OSDs
> >> >  * osd: fix crash on 32-bit hosts triggered by librbd clients
> >> >  * librbd: fix error handling when talking to older OSDs
> >> >  * mon: fix a few rare crashes
> >> >  * ceph command: ability to easily adjust CRUSH tunables
> >> >  * radosgw: object copy does not copy source ACLs
> >> >  * rados command: fix omap command usage
> >> >  * sysvinit script: set ulimit -n properly on remote hosts
> >> >  * msgr: fix narrow race with message queuing
> >> >  * fixed compilation on some old distros (e.g., RHEL 5.x)
> >> >
> >> > There are a small number of interface changes related to the default CRUSH
> >> > rule and scrub interval configuration options. Please see the full release
> >> > notes.
> >> >
> >> > You can get v0.56.2 in the usual fashion:
> >> >
> >> >  * Git at git://github.com/ceph/ceph.git
> >> >  * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
> >> >  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
> >> >  * For RPMs, see http://ceph.com/docs/master/install/rpm
> >> >
> >> > --
> >> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> > the body of a message to majordomo@vger.kernel.org
> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >>
> 
> 

Re: v0.56.2 released

[Mandell Degerness]

I'm doing a ceph-mon --mkfs from an initial .conf that has no OSDs
defined in it.  All osds are added dynamically.  We'll try your patch.
 I would think that we would want to have a way to specify in the
.conf file which option to use.  It would be important to be able to
revert back to OSD first in test scenarios.

On Tue, Feb 5, 2013 at 12:37 PM, Sage Weil <sage@inktank.com> wrote:
> On Tue, 5 Feb 2013, Mandell Degerness wrote:
>> We are doing the latter.  What is the best way to force it to host
>> first selection?
>
> Checking the code, you should be seeing either hosts (unless you've
> changed osd_crush_chooseleaf_type from its default of 1).  Or, if the map
> generation is being done from a prepared ceph.conf, it should do hosts if
> there is more than one host in the initial conf.
>
> Maybe you're doing the ceph-mon --mkfs from an initial .conf that has only
> 1 host to begin with, and others are added after?
>
> I can make it observe the config setting in all cases (instead of trying
> to guess what the user wants).  I pushed a patch to wip-osdmap-chooseleaf
> that you can cherry-pick that does this.. can you let me know if that
> resolves the issue, and/or if you think this makes the most sense for
> users?
>
> sage
>
>
>
>
>>
>> On Tue, Feb 5, 2013 at 11:34 AM, Sage Weil <sage@inktank.com> wrote:
>> > On Tue, 5 Feb 2013, Mandell Degerness wrote:
>> >> We are using v0.56.2 and it still seems that the default crushmap is
>> >> osd centered.  Here is the crushmap as dumped:
>> >>
>> >> [root@node-172-19-0-15 ~]# cat crush.txt
>> >> # begin crush map
>> >>
>> >> # devices
>> >> device 0 osd.0
>> >> device 1 osd.1
>> >> device 2 osd.2
>> >> device 3 osd.3
>> >> device 4 osd.4
>> >> device 5 osd.5
>> >> device 6 osd.6
>> >> device 7 osd.7
>> >> device 8 osd.8
>> >> device 9 osd.9
>> >> device 10 osd.10
>> >> device 11 osd.11
>> >> device 12 osd.12
>> >> device 13 osd.13
>> >> device 14 osd.14
>> >>
>> >> # types
>> >> type 0 osd
>> >> type 1 host
>> >> type 2 rack
>> >> type 3 row
>> >> type 4 room
>> >> type 5 datacenter
>> >> type 6 root
>> >>
>> >> # buckets
>> >> host 172.19.0.14 {
>> >> id -2 # do not change unnecessarily
>> >> # weight 6.000
>> >> alg straw
>> >> hash 0 # rjenkins1
>> >> item osd.0 weight 1.000
>> >> item osd.4 weight 1.000
>> >> item osd.5 weight 1.000
>> >> item osd.12 weight 1.000
>> >> item osd.13 weight 1.000
>> >> item osd.14 weight 1.000
>> >> }
>> >> host 172.19.0.13 {
>> >> id -4 # do not change unnecessarily
>> >> # weight 3.000
>> >> alg straw
>> >> hash 0 # rjenkins1
>> >> item osd.2 weight 1.000
>> >> item osd.7 weight 1.000
>> >> item osd.9 weight 1.000
>> >> }
>> >> host 172.19.0.16 {
>> >> id -5 # do not change unnecessarily
>> >> # weight 3.000
>> >> alg straw
>> >> hash 0 # rjenkins1
>> >> item osd.3 weight 1.000
>> >> item osd.6 weight 1.000
>> >> item osd.10 weight 1.000
>> >> }
>> >> host 172.19.0.15 {
>> >> id -6 # do not change unnecessarily
>> >> # weight 3.000
>> >> alg straw
>> >> hash 0 # rjenkins1
>> >> item osd.1 weight 1.000
>> >> item osd.8 weight 1.000
>> >> item osd.11 weight 1.000
>> >> }
>> >> rack 0 {
>> >> id -3 # do not change unnecessarily
>> >> # weight 15.000
>> >> alg straw
>> >> hash 0 # rjenkins1
>> >> item 172.19.0.14 weight 6.000
>> >> item 172.19.0.13 weight 3.000
>> >> item 172.19.0.16 weight 3.000
>> >> item 172.19.0.15 weight 3.000
>> >> }
>> >> root default {
>> >> id -1 # do not change unnecessarily
>> >> # weight 15.000
>> >> alg straw
>> >> hash 0 # rjenkins1
>> >> item 0 weight 15.000
>> >> }
>> >>
>> >> # rules
>> >> rule data {
>> >> ruleset 0
>> >> type replicated
>> >> min_size 1
>> >> max_size 10
>> >> step take default
>> >> step choose firstn 0 type osd
>> >> step emit
>> >> }
>> >> rule metadata {
>> >> ruleset 1
>> >> type replicated
>> >> min_size 1
>> >> max_size 10
>> >> step take default
>> >> step choose firstn 0 type osd
>> >> step emit
>> >> }
>> >> rule rbd {
>> >> ruleset 2
>> >> type replicated
>> >> min_size 1
>> >> max_size 10
>> >> step take default
>> >> step choose firstn 0 type osd
>> >> step emit
>> >> }
>> >>
>> >> # end crush map
>> >> [root@node-172-19-0-15 ~]# ceph --version
>> >> ceph version 0.56.2 (586538e22afba85c59beda49789ec42024e7a061)
>> >>
>> >> We do not run any explicit crushtool commands as part of our start up
>> >> at this time.  Should we be?
>> >
>> > Do you run mkcephfs?  If you are passing in a ceph.conf to mkcephfs, it is
>> > still dynamically choosing a rule based on whether you have enough osds (3
>> > i think?  i forget).  If you are running ceph-mon --mkfs directly (as
>> > ceph-deploy, chef, juju do), it will always default to osds.
>> >
>> > sage
>> >
>> >>
>> >> Regards,
>> >> Mandell Degerness
>> >>
>> >> On Wed, Jan 30, 2013 at 3:46 PM, Sage Weil <sage@inktank.com> wrote:
>> >> > The next bobtail point release is ready, and it's looking pretty good.
>> >> > This is an important update for the 0.56.x backport series that fixes a
>> >> > number of bugs and several performance issues. All v0.56.x users are
>> >> > encouraged to upgrade.
>> >> >
>> >> > Notable changes since v0.56.1:
>> >> >
>> >> >  * osd: snapshot trimming fixes
>> >> >  * osd: scrub snapshot metadata
>> >> >  * osd: fix osdmap trimming
>> >> >  * osd: misc peering fixes
>> >> >  * osd: stop heartbeating with peers if internal threads are stuck/hung
>> >> >  * osd: PG removal is friendlier to other workloads
>> >> >  * osd: fix recovery start delay (was causing very slow recovery)
>> >> >  * osd: fix scheduling of explicitly requested scrubs
>> >> >  * osd: fix scrub interval config options
>> >> >  * osd: improve recovery vs client io tuning
>> >> >  * osd: improve 'slow request' warning detail for better diagnosis
>> >> >  * osd: default CRUSH map now distributes across hosts, not OSDs
>> >> >  * osd: fix crash on 32-bit hosts triggered by librbd clients
>> >> >  * librbd: fix error handling when talking to older OSDs
>> >> >  * mon: fix a few rare crashes
>> >> >  * ceph command: ability to easily adjust CRUSH tunables
>> >> >  * radosgw: object copy does not copy source ACLs
>> >> >  * rados command: fix omap command usage
>> >> >  * sysvinit script: set ulimit -n properly on remote hosts
>> >> >  * msgr: fix narrow race with message queuing
>> >> >  * fixed compilation on some old distros (e.g., RHEL 5.x)
>> >> >
>> >> > There are a small number of interface changes related to the default CRUSH
>> >> > rule and scrub interval configuration options. Please see the full release
>> >> > notes.
>> >> >
>> >> > You can get v0.56.2 in the usual fashion:
>> >> >
>> >> >  * Git at git://github.com/ceph/ceph.git
>> >> >  * Tarball at http://ceph.com/download/ceph-0.56.2.tar.gz
>> >> >  * For Debian/Ubuntu packages, see http://ceph.com/docs/master/install/debian
>> >> >  * For RPMs, see http://ceph.com/docs/master/install/rpm
>> >> >
>> >> > --
>> >> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >> > the body of a message to majordomo@vger.kernel.org
>> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >>
>> >>
>>
>>