v0.83 released

v0.83 released

[Sage Weil]

Another Ceph development release!  This has been a longer cycle, so there 
has been quite a bit of bug fixing and stabilization in this round.  
There is also a bunch of packaging fixes for RPM distros (RHEL/CentOS, 
Fedora, and SUSE) and for systemd.  We've also added a new 
librados-striper library from Sebastien Ponce that provides a generic 
striping API for applications to code to.

Upgrading
---------

* The experimental keyvaluestore-dev OSD backend had an on-disk format
  change that prevents existing OSD data from being upgraded.  This
  affects developers and testers only.

* mon-specific and osd-specific leveldb options have been removed.
  From this point onward users should use 'leveldb_' generic options and add
  the options in the appropriate sections of their configuration files.
  Monitors will still maintain the following monitor-specific defaults:

    leveldb_write_buffer_size = 32*1024*1024  = 33554432  // 32MB
    leveldb_cache_size        = 512*1024*1204 = 536870912 // 512MB
    leveldb_block_size        = 64*1024       = 65536     // 64KB
    leveldb_compression       = false
    leveldb_log               = ""

  OSDs will still maintain the following osd-specific defaults:

    leveldb_log               = ""

Notable Changes
---------------

* ceph-disk: fix dmcrypt support (Stephen Taylor)
* cephtool: fix help (Yilong Zhao)
* cephtool: test cleanup (Joao Eduardo Luis)
* doc: librados example fixes (Kevin Dalley)
* doc: many doc updates (John Wilkins)
* doc: update erasure docs (Loic Dachary, Venky Shankar)
* filestore: disable use of XFS hint (buggy on old kernels) (Samuel Just)
* filestore: fix xattr spillout (Greg Farnum, Haomai Wang)
* keyvaluestore: header cache (Haomai Wang)
* librados_striper: striping library for librados (Sebastien Ponce)
* libs3: update to latest (Danny Al-Gaaf)
* log: fix derr level (Joao Eduardo Luis)
* logrotate: fix osd log rotation on ubuntu (Sage Weil)
* mds: fix xattr bug triggered by ACLs (Yan, Zheng)
* misc memory leaks, cleanups, fixes (Danny Al-Gaaf, Sahid Ferdjaoui)
* misc suse fixes (Danny Al-Gaaf)
* misc word size fixes (Kevin Cox)
* mon: drop mon- and osd- specific leveldb options (Joao Eduardo Luis)
* mon: ec pool profile fixes (Loic Dachary)
* mon: fix health down messages (Sage Weil)
* mon: fix quorum feature check (#8738, Greg Farnum)
* mon: 'osd crush reweight-subtree ...' (Sage Weil)
* mon, osd: relax client EC support requirements (Sage Weil)
* mon: some instrumentation (Sage Weil)
* objecter: flag operations that are redirected by caching (Sage Weil)
* osd: clean up shard_id_t, shard_t (Loic Dachary)
* osd: fix connection reconnect race (Greg Farnum)
* osd: fix dumps (Joao Eduardo Luis)
* osd: fix erasure-code lib initialization (Loic Dachary)
* osd: fix extent normalization (Adam Crume)
* osd: fix loopback msgr issue (Ma Jianpeng)
* osd: fix LSB release parsing (Danny Al-Gaaf)
* osd: improved backfill priorities (Sage Weil)
* osd: many many core fixes (Samuel Just)
* osd, mon: config sanity checks on start (Sage Weil, Joao Eduardo Luis)
* osd: sharded threadpool to improve parallelism (Somnath Roy)
* osd: simple io prioritization for scrub (Sage Weil)
* osd: simple scrub throttling (Sage Weil)
* osd: tests for bench command (Loic Dachary)
* osd: use xfs hint less frequently (Ilya Dryomov)
* pybind/rados: fix small timeouts (John Spray)
* qa: xfstests updates (Ilya Dryomov)
* rgw: cache bucket info (Yehuda Sadeh)
* rgw: cache decoded user info (Yehuda Sadeh)
* rgw: fix multipart object attr regression (#8452, Yehuda Sadeh)
* rgw: fix radosgw-admin 'show log' command (#8553, Yehuda Sadeh)
* rgw: fix URL decoding (#8702, Brian Rak)
* rgw: handle empty extra pool name (Yehuda Sadeh)
* rpm: do not restart daemons on upgrade (Alfredo Deza)
* rpm: misc packaging fixes for rhel7 (Sandon Van Ness)
* rpm: split ceph-common from ceph (Sandon Van Ness)
* systemd: wrap started daemons in new systemd environment (Sage Weil, Dan 
  Mick)
* sysvinit: less sensitive to failures (Sage Weil)
* upstart: increase max open files limit (Sage Weil)

Getting Ceph
------------

* Git at git://github.com/ceph/ceph.git
* Tarball at http://ceph.com/download/ceph-0.83.tar.gz
* For packages, see http://ceph.com/docs/master/install/get-packages
* For ceph-deploy, see http://ceph.com/docs/master/install/install-ceph-deploy

Re: v0.83 released

[debian Only]

[-- Attachment #1.1: Type: text/plain, Size: 4890 bytes --]

Good news.  when this release will public in Debian Wheezy pkglist ?
thanks for ur good job


2014-07-30 8:21 GMT+07:00 Sage Weil <sage-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>:

> Another Ceph development release!  This has been a longer cycle, so there
> has been quite a bit of bug fixing and stabilization in this round.
> There is also a bunch of packaging fixes for RPM distros (RHEL/CentOS,
> Fedora, and SUSE) and for systemd.  We've also added a new
> librados-striper library from Sebastien Ponce that provides a generic
> striping API for applications to code to.
>
> Upgrading
> ---------
>
> * The experimental keyvaluestore-dev OSD backend had an on-disk format
>   change that prevents existing OSD data from being upgraded.  This
>   affects developers and testers only.
>
> * mon-specific and osd-specific leveldb options have been removed.
>   From this point onward users should use 'leveldb_' generic options and
> add
>   the options in the appropriate sections of their configuration files.
>   Monitors will still maintain the following monitor-specific defaults:
>
>     leveldb_write_buffer_size = 32*1024*1024  = 33554432  // 32MB
>     leveldb_cache_size        = 512*1024*1204 = 536870912 // 512MB
>     leveldb_block_size        = 64*1024       = 65536     // 64KB
>     leveldb_compression       = false
>     leveldb_log               = ""
>
>   OSDs will still maintain the following osd-specific defaults:
>
>     leveldb_log               = ""
>
> Notable Changes
> ---------------
>
> * ceph-disk: fix dmcrypt support (Stephen Taylor)
> * cephtool: fix help (Yilong Zhao)
> * cephtool: test cleanup (Joao Eduardo Luis)
> * doc: librados example fixes (Kevin Dalley)
> * doc: many doc updates (John Wilkins)
> * doc: update erasure docs (Loic Dachary, Venky Shankar)
> * filestore: disable use of XFS hint (buggy on old kernels) (Samuel Just)
> * filestore: fix xattr spillout (Greg Farnum, Haomai Wang)
> * keyvaluestore: header cache (Haomai Wang)
> * librados_striper: striping library for librados (Sebastien Ponce)
> * libs3: update to latest (Danny Al-Gaaf)
> * log: fix derr level (Joao Eduardo Luis)
> * logrotate: fix osd log rotation on ubuntu (Sage Weil)
> * mds: fix xattr bug triggered by ACLs (Yan, Zheng)
> * misc memory leaks, cleanups, fixes (Danny Al-Gaaf, Sahid Ferdjaoui)
> * misc suse fixes (Danny Al-Gaaf)
> * misc word size fixes (Kevin Cox)
> * mon: drop mon- and osd- specific leveldb options (Joao Eduardo Luis)
> * mon: ec pool profile fixes (Loic Dachary)
> * mon: fix health down messages (Sage Weil)
> * mon: fix quorum feature check (#8738, Greg Farnum)
> * mon: 'osd crush reweight-subtree ...' (Sage Weil)
> * mon, osd: relax client EC support requirements (Sage Weil)
> * mon: some instrumentation (Sage Weil)
> * objecter: flag operations that are redirected by caching (Sage Weil)
> * osd: clean up shard_id_t, shard_t (Loic Dachary)
> * osd: fix connection reconnect race (Greg Farnum)
> * osd: fix dumps (Joao Eduardo Luis)
> * osd: fix erasure-code lib initialization (Loic Dachary)
> * osd: fix extent normalization (Adam Crume)
> * osd: fix loopback msgr issue (Ma Jianpeng)
> * osd: fix LSB release parsing (Danny Al-Gaaf)
> * osd: improved backfill priorities (Sage Weil)
> * osd: many many core fixes (Samuel Just)
> * osd, mon: config sanity checks on start (Sage Weil, Joao Eduardo Luis)
> * osd: sharded threadpool to improve parallelism (Somnath Roy)
> * osd: simple io prioritization for scrub (Sage Weil)
> * osd: simple scrub throttling (Sage Weil)
> * osd: tests for bench command (Loic Dachary)
> * osd: use xfs hint less frequently (Ilya Dryomov)
> * pybind/rados: fix small timeouts (John Spray)
> * qa: xfstests updates (Ilya Dryomov)
> * rgw: cache bucket info (Yehuda Sadeh)
> * rgw: cache decoded user info (Yehuda Sadeh)
> * rgw: fix multipart object attr regression (#8452, Yehuda Sadeh)
> * rgw: fix radosgw-admin 'show log' command (#8553, Yehuda Sadeh)
> * rgw: fix URL decoding (#8702, Brian Rak)
> * rgw: handle empty extra pool name (Yehuda Sadeh)
> * rpm: do not restart daemons on upgrade (Alfredo Deza)
> * rpm: misc packaging fixes for rhel7 (Sandon Van Ness)
> * rpm: split ceph-common from ceph (Sandon Van Ness)
> * systemd: wrap started daemons in new systemd environment (Sage Weil, Dan
>   Mick)
> * sysvinit: less sensitive to failures (Sage Weil)
> * upstart: increase max open files limit (Sage Weil)
>
> Getting Ceph
> ------------
>
> * Git at git://github.com/ceph/ceph.git
> * Tarball at http://ceph.com/download/ceph-0.83.tar.gz
> * For packages, see http://ceph.com/docs/master/install/get-packages
> * For ceph-deploy, see
> http://ceph.com/docs/master/install/install-ceph-deploy
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>

[-- Attachment #1.2: Type: text/html, Size: 6084 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: v0.83 released

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 6284 bytes --]

On Tue, 5 Aug 2014, debian Only wrote:
> Good news.  when this release will public in Debian Wheezy pkglist ?thanks
> for ur good job

I think that, in general, the strategy should be to keep the stable 
packages (firefly 0.80.x currently) in the stable downstream repos 
(wheezy).  These development releases can either be found at ceph.com, or 
perhaps they can go into debian unstable or rawhide or similar.  I don't 
think it is the release that most users will want when they 'apt-get 
install ceph' on a generic system, though!

sage


> 
> 
> 2014-07-30 8:21 GMT+07:00 Sage Weil <sage-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>:
>       Another Ceph development release!  This has been a longer cycle,
>       so there
>       has been quite a bit of bug fixing and stabilization in this
>       round.
>       There is also a bunch of packaging fixes for RPM distros
>       (RHEL/CentOS,
>       Fedora, and SUSE) and for systemd.  We've also added a new
>       librados-striper library from Sebastien Ponce that provides a
>       generic
>       striping API for applications to code to.
> 
>       Upgrading
>       ---------
> 
>       * The experimental keyvaluestore-dev OSD backend had an on-disk
>       format
>         change that prevents existing OSD data from being upgraded.
>        This
>         affects developers and testers only.
> 
>       * mon-specific and osd-specific leveldb options have been
>       removed.
>         From this point onward users should use 'leveldb_' generic
>       options and add
>         the options in the appropriate sections of their configuration
>       files.
>         Monitors will still maintain the following monitor-specific
>       defaults:
> 
>           leveldb_write_buffer_size = 32*1024*1024  = 33554432  //
>       32MB
>           leveldb_cache_size        = 512*1024*1204 = 536870912 //
>       512MB
>           leveldb_block_size        = 64*1024       = 65536     //
>       64KB
>           leveldb_compression       = false
>           leveldb_log               = ""
> 
>         OSDs will still maintain the following osd-specific defaults:
> 
>           leveldb_log               = ""
> 
>       Notable Changes
>       ---------------
> 
>       * ceph-disk: fix dmcrypt support (Stephen Taylor)
>       * cephtool: fix help (Yilong Zhao)
>       * cephtool: test cleanup (Joao Eduardo Luis)
>       * doc: librados example fixes (Kevin Dalley)
>       * doc: many doc updates (John Wilkins)
>       * doc: update erasure docs (Loic Dachary, Venky Shankar)
>       * filestore: disable use of XFS hint (buggy on old kernels)
>       (Samuel Just)
>       * filestore: fix xattr spillout (Greg Farnum, Haomai Wang)
>       * keyvaluestore: header cache (Haomai Wang)
>       * librados_striper: striping library for librados (Sebastien
>       Ponce)
>       * libs3: update to latest (Danny Al-Gaaf)
>       * log: fix derr level (Joao Eduardo Luis)
>       * logrotate: fix osd log rotation on ubuntu (Sage Weil)
>       * mds: fix xattr bug triggered by ACLs (Yan, Zheng)
>       * misc memory leaks, cleanups, fixes (Danny Al-Gaaf, Sahid
>       Ferdjaoui)
>       * misc suse fixes (Danny Al-Gaaf)
>       * misc word size fixes (Kevin Cox)
>       * mon: drop mon- and osd- specific leveldb options (Joao Eduardo
>       Luis)
>       * mon: ec pool profile fixes (Loic Dachary)
>       * mon: fix health down messages (Sage Weil)
>       * mon: fix quorum feature check (#8738, Greg Farnum)
>       * mon: 'osd crush reweight-subtree ...' (Sage Weil)
>       * mon, osd: relax client EC support requirements (Sage Weil)
>       * mon: some instrumentation (Sage Weil)
>       * objecter: flag operations that are redirected by caching (Sage
>       Weil)
>       * osd: clean up shard_id_t, shard_t (Loic Dachary)
>       * osd: fix connection reconnect race (Greg Farnum)
>       * osd: fix dumps (Joao Eduardo Luis)
>       * osd: fix erasure-code lib initialization (Loic Dachary)
>       * osd: fix extent normalization (Adam Crume)
>       * osd: fix loopback msgr issue (Ma Jianpeng)
>       * osd: fix LSB release parsing (Danny Al-Gaaf)
>       * osd: improved backfill priorities (Sage Weil)
>       * osd: many many core fixes (Samuel Just)
>       * osd, mon: config sanity checks on start (Sage Weil, Joao
>       Eduardo Luis)
>       * osd: sharded threadpool to improve parallelism (Somnath Roy)
>       * osd: simple io prioritization for scrub (Sage Weil)
>       * osd: simple scrub throttling (Sage Weil)
>       * osd: tests for bench command (Loic Dachary)
>       * osd: use xfs hint less frequently (Ilya Dryomov)
>       * pybind/rados: fix small timeouts (John Spray)
>       * qa: xfstests updates (Ilya Dryomov)
>       * rgw: cache bucket info (Yehuda Sadeh)
>       * rgw: cache decoded user info (Yehuda Sadeh)
>       * rgw: fix multipart object attr regression (#8452, Yehuda
>       Sadeh)
>       * rgw: fix radosgw-admin 'show log' command (#8553, Yehuda
>       Sadeh)
>       * rgw: fix URL decoding (#8702, Brian Rak)
>       * rgw: handle empty extra pool name (Yehuda Sadeh)
>       * rpm: do not restart daemons on upgrade (Alfredo Deza)
>       * rpm: misc packaging fixes for rhel7 (Sandon Van Ness)
>       * rpm: split ceph-common from ceph (Sandon Van Ness)
>       * systemd: wrap started daemons in new systemd environment (Sage
>       Weil, Dan
>         Mick)
>       * sysvinit: less sensitive to failures (Sage Weil)
>       * upstart: increase max open files limit (Sage Weil)
> 
>       Getting Ceph
>       ------------
> 
>       * Git at git://github.com/ceph/ceph.git
>       * Tarball at http://ceph.com/download/ceph-0.83.tar.gz
>       * For packages, see
>       http://ceph.com/docs/master/install/get-packages
>       * For ceph-deploy, see
>       http://ceph.com/docs/master/install/install-ceph-deploy
>       _______________________________________________
>       ceph-users mailing list
>       ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
>       http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> 
> 

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: v0.83 released

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 6654 bytes --]

Oops, fixing ceph-maintainers address.

On Tue, 5 Aug 2014, Sage Weil wrote:

> On Tue, 5 Aug 2014, debian Only wrote:
> > Good news.  when this release will public in Debian Wheezy pkglist ?thanks
> > for ur good job
> 
> I think that, in general, the strategy should be to keep the stable 
> packages (firefly 0.80.x currently) in the stable downstream repos 
> (wheezy).  These development releases can either be found at ceph.com, or 
> perhaps they can go into debian unstable or rawhide or similar.  I don't 
> think it is the release that most users will want when they 'apt-get 
> install ceph' on a generic system, though!
> 
> sage
> 
> 
> > 
> > 
> > 2014-07-30 8:21 GMT+07:00 Sage Weil <sage-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>:
> >       Another Ceph development release!  This has been a longer cycle,
> >       so there
> >       has been quite a bit of bug fixing and stabilization in this
> >       round.
> >       There is also a bunch of packaging fixes for RPM distros
> >       (RHEL/CentOS,
> >       Fedora, and SUSE) and for systemd.  We've also added a new
> >       librados-striper library from Sebastien Ponce that provides a
> >       generic
> >       striping API for applications to code to.
> > 
> >       Upgrading
> >       ---------
> > 
> >       * The experimental keyvaluestore-dev OSD backend had an on-disk
> >       format
> >         change that prevents existing OSD data from being upgraded.
> >        This
> >         affects developers and testers only.
> > 
> >       * mon-specific and osd-specific leveldb options have been
> >       removed.
> >         From this point onward users should use 'leveldb_' generic
> >       options and add
> >         the options in the appropriate sections of their configuration
> >       files.
> >         Monitors will still maintain the following monitor-specific
> >       defaults:
> > 
> >           leveldb_write_buffer_size = 32*1024*1024  = 33554432  //
> >       32MB
> >           leveldb_cache_size        = 512*1024*1204 = 536870912 //
> >       512MB
> >           leveldb_block_size        = 64*1024       = 65536     //
> >       64KB
> >           leveldb_compression       = false
> >           leveldb_log               = ""
> > 
> >         OSDs will still maintain the following osd-specific defaults:
> > 
> >           leveldb_log               = ""
> > 
> >       Notable Changes
> >       ---------------
> > 
> >       * ceph-disk: fix dmcrypt support (Stephen Taylor)
> >       * cephtool: fix help (Yilong Zhao)
> >       * cephtool: test cleanup (Joao Eduardo Luis)
> >       * doc: librados example fixes (Kevin Dalley)
> >       * doc: many doc updates (John Wilkins)
> >       * doc: update erasure docs (Loic Dachary, Venky Shankar)
> >       * filestore: disable use of XFS hint (buggy on old kernels)
> >       (Samuel Just)
> >       * filestore: fix xattr spillout (Greg Farnum, Haomai Wang)
> >       * keyvaluestore: header cache (Haomai Wang)
> >       * librados_striper: striping library for librados (Sebastien
> >       Ponce)
> >       * libs3: update to latest (Danny Al-Gaaf)
> >       * log: fix derr level (Joao Eduardo Luis)
> >       * logrotate: fix osd log rotation on ubuntu (Sage Weil)
> >       * mds: fix xattr bug triggered by ACLs (Yan, Zheng)
> >       * misc memory leaks, cleanups, fixes (Danny Al-Gaaf, Sahid
> >       Ferdjaoui)
> >       * misc suse fixes (Danny Al-Gaaf)
> >       * misc word size fixes (Kevin Cox)
> >       * mon: drop mon- and osd- specific leveldb options (Joao Eduardo
> >       Luis)
> >       * mon: ec pool profile fixes (Loic Dachary)
> >       * mon: fix health down messages (Sage Weil)
> >       * mon: fix quorum feature check (#8738, Greg Farnum)
> >       * mon: 'osd crush reweight-subtree ...' (Sage Weil)
> >       * mon, osd: relax client EC support requirements (Sage Weil)
> >       * mon: some instrumentation (Sage Weil)
> >       * objecter: flag operations that are redirected by caching (Sage
> >       Weil)
> >       * osd: clean up shard_id_t, shard_t (Loic Dachary)
> >       * osd: fix connection reconnect race (Greg Farnum)
> >       * osd: fix dumps (Joao Eduardo Luis)
> >       * osd: fix erasure-code lib initialization (Loic Dachary)
> >       * osd: fix extent normalization (Adam Crume)
> >       * osd: fix loopback msgr issue (Ma Jianpeng)
> >       * osd: fix LSB release parsing (Danny Al-Gaaf)
> >       * osd: improved backfill priorities (Sage Weil)
> >       * osd: many many core fixes (Samuel Just)
> >       * osd, mon: config sanity checks on start (Sage Weil, Joao
> >       Eduardo Luis)
> >       * osd: sharded threadpool to improve parallelism (Somnath Roy)
> >       * osd: simple io prioritization for scrub (Sage Weil)
> >       * osd: simple scrub throttling (Sage Weil)
> >       * osd: tests for bench command (Loic Dachary)
> >       * osd: use xfs hint less frequently (Ilya Dryomov)
> >       * pybind/rados: fix small timeouts (John Spray)
> >       * qa: xfstests updates (Ilya Dryomov)
> >       * rgw: cache bucket info (Yehuda Sadeh)
> >       * rgw: cache decoded user info (Yehuda Sadeh)
> >       * rgw: fix multipart object attr regression (#8452, Yehuda
> >       Sadeh)
> >       * rgw: fix radosgw-admin 'show log' command (#8553, Yehuda
> >       Sadeh)
> >       * rgw: fix URL decoding (#8702, Brian Rak)
> >       * rgw: handle empty extra pool name (Yehuda Sadeh)
> >       * rpm: do not restart daemons on upgrade (Alfredo Deza)
> >       * rpm: misc packaging fixes for rhel7 (Sandon Van Ness)
> >       * rpm: split ceph-common from ceph (Sandon Van Ness)
> >       * systemd: wrap started daemons in new systemd environment (Sage
> >       Weil, Dan
> >         Mick)
> >       * sysvinit: less sensitive to failures (Sage Weil)
> >       * upstart: increase max open files limit (Sage Weil)
> > 
> >       Getting Ceph
> >       ------------
> > 
> >       * Git at git://github.com/ceph/ceph.git
> >       * Tarball at http://ceph.com/download/ceph-0.83.tar.gz
> >       * For packages, see
> >       http://ceph.com/docs/master/install/get-packages
> >       * For ceph-deploy, see
> >       http://ceph.com/docs/master/install/install-ceph-deploy
> >       _______________________________________________
> >       ceph-users mailing list
> >       ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> >       http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> > 
> > 
> > 
> > 

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

krbd kernel 3.16.0-1 with v0.83 got stuck during write

[Kasper Dieter]

Hi,

I'm running a 3 node cluster with 126 OSDs in total under CentOS-6.5 with 
ceph version 0.83 (78ff1f0a5dfd3c5850805b4021738564c36c92b8)

On the client side it's 0.83, too 
with kernel 3.16.0-1.el6.elrepo.x86_64

rbd showmapped
id pool   image           snap device    
0  SAS-r2 sas2-r2-1T-4m.0 -    /dev/rbd0 
1  SAS-r2 sas2-r2-1T-4m.1 -    /dev/rbd1 
2  SAS-r2 sas2-r2-1T-4m.2 -    /dev/rbd2 

After a couple of minutes (trying to fill the 1TB volume)
fio --filename=/dev/rbd0 --direct=1 --rw=write --bs=8M --size=8G --numjobs=128 --offset_increment=8G --runtime=3600 --group_reporting --name=file1
got stuck.

/var/log/message:
(...)
Aug  7 19:22:34 rx37-0 kernel: libceph: osd118 192.168.113.54:6902 socket closed (con state OPEN)
Aug  7 19:22:34 rx37-0 kernel: libceph: osd40 192.168.113.52:6920 socket closed (con state OPEN)
Aug  7 19:22:34 rx37-0 kernel: libceph: osd109 192.168.113.54:6875 socket closed (con state OPEN)
Aug  7 19:22:34 rx37-0 kernel: libceph: osd67 192.168.113.53:6875 socket closed (con state OPEN)
Aug  7 19:22:34 rx37-0 kernel: libceph: osd37 192.168.113.52:6911 socket closed (con state OPEN)
Aug  7 19:22:34 rx37-0 kernel: libceph: osd98 192.168.113.54:6842 socket closed (con state OPEN)
Aug  7 19:22:34 rx37-0 kernel: libceph: osd26 192.168.113.52:6878 socket closed (con state OPEN)
Aug  7 19:24:43 rx37-0 kernel: INFO: task kworker/2:0:19 blocked for more than 120 seconds.
Aug  7 19:24:43 rx37-0 kernel:      Not tainted 3.16.0-1.el6.elrepo.x86_64 #1
Aug  7 19:24:43 rx37-0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug  7 19:24:43 rx37-0 kernel: kworker/2:0     D 0000000000000002     0    19      2 0x00000000
Aug  7 19:24:43 rx37-0 kernel: Workqueue: ceph-msgr con_work [libceph]
Aug  7 19:24:43 rx37-0 kernel: ffff8810307bfb68 0000000000000046 ffff8810307bfb18 ffff8810307bc010
Aug  7 19:24:43 rx37-0 kernel: 0000000000014380 0000000000014380 ffff8810307ae390 ffff880079678250
Aug  7 19:24:43 rx37-0 kernel: 0000003500004040 ffff88102a1fd7c8 ffff88102a1fd7cc ffff8810307ae390
Aug  7 19:24:43 rx37-0 kernel: Call Trace:
Aug  7 19:24:43 rx37-0 kernel: [<ffffffff81647629>] schedule+0x29/0x70
Aug  7 19:24:43 rx37-0 kernel: [<ffffffff8164778e>] schedule_preempt_disabled+0xe/0x10
Aug  7 19:24:43 rx37-0 kernel: [<ffffffff816490fb>] __mutex_lock_slowpath+0xdb/0x1d0
Aug  7 19:24:43 rx37-0 kernel: [<ffffffff81649213>] mutex_lock+0x23/0x40
Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa0615e0f>] get_reply+0x3f/0x200 [libceph]
Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa0616058>] alloc_msg+0x88/0x90 [libceph]
Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa060d8f1>] ceph_con_in_msg_alloc+0x71/0x240 [libceph]
Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa060eba8>] read_partial_message+0x1e8/0x3d0 [libceph]
Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa060d278>] ? ceph_tcp_recvmsg+0x48/0x60 [libceph]
Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa06101d6>] try_read+0x2b6/0x430 [libceph]
Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa0610688>] con_work+0x78/0x220 [libceph]
Aug  7 19:24:43 rx37-0 kernel: [<ffffffff8108d60c>] process_one_work+0x17c/0x420
Aug  7 19:24:43 rx37-0 kernel: [<ffffffff8108e7d3>] worker_thread+0x123/0x420
Aug  7 19:24:43 rx37-0 kernel: [<ffffffff8108e6b0>] ? maybe_create_worker+0x180/0x180
Aug  7 19:24:43 rx37-0 kernel: [<ffffffff810943be>] kthread+0xce/0xf0
Aug  7 19:24:43 rx37-0 kernel: [<ffffffff810942f0>] ? kthread_freezable_should_stop+0x70/0x70
Aug  7 19:24:43 rx37-0 kernel: [<ffffffff8164ae3c>] ret_from_fork+0x7c/0xb0
Aug  7 19:24:43 rx37-0 kernel: [<ffffffff810942f0>] ? kthread_freezable_should_stop+0x70/0x70
Aug  7 19:24:43 rx37-0 kernel: INFO: task kworker/3:0:24 blocked for more than 120 seconds.
Aug  7 19:24:43 rx37-0 kernel:      Not tainted 3.16.0-1.el6.elrepo.x86_64 #1
Aug  7 19:24:43 rx37-0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug  7 19:24:43 rx37-0 kernel: kworker/3:0     D 0000000000000003     0    24      2 0x00000000
Aug  7 19:24:43 rx37-0 kernel: Workqueue: ceph-msgr con_work [libceph]
Aug  7 19:24:43 rx37-0 kernel: ffff881030027c98 0000000000000046 ffff881019afe330 ffff881030024010
(...)


Any ideas ?

With Kernel 3.10.32 on the client side everythink worked fine.


Mit freundlichen Grüßen / Best regards
Dieter Kasper
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: krbd kernel 3.16.0-1 with v0.83 got stuck during write

[Ilya Dryomov]

On Thu, Aug 7, 2014 at 9:36 PM, Kasper Dieter
<dieter.kasper@ts.fujitsu.com> wrote:
> Hi,
>
> I'm running a 3 node cluster with 126 OSDs in total under CentOS-6.5 with
> ceph version 0.83 (78ff1f0a5dfd3c5850805b4021738564c36c92b8)
>
> On the client side it's 0.83, too
> with kernel 3.16.0-1.el6.elrepo.x86_64
>
> rbd showmapped
> id pool   image           snap device
> 0  SAS-r2 sas2-r2-1T-4m.0 -    /dev/rbd0
> 1  SAS-r2 sas2-r2-1T-4m.1 -    /dev/rbd1
> 2  SAS-r2 sas2-r2-1T-4m.2 -    /dev/rbd2
>
> After a couple of minutes (trying to fill the 1TB volume)
> fio --filename=/dev/rbd0 --direct=1 --rw=write --bs=8M --size=8G --numjobs=128 --offset_increment=8G --runtime=3600 --group_reporting --name=file1
> got stuck.
>
> /var/log/message:
> (...)
> Aug  7 19:22:34 rx37-0 kernel: libceph: osd118 192.168.113.54:6902 socket closed (con state OPEN)
> Aug  7 19:22:34 rx37-0 kernel: libceph: osd40 192.168.113.52:6920 socket closed (con state OPEN)
> Aug  7 19:22:34 rx37-0 kernel: libceph: osd109 192.168.113.54:6875 socket closed (con state OPEN)
> Aug  7 19:22:34 rx37-0 kernel: libceph: osd67 192.168.113.53:6875 socket closed (con state OPEN)
> Aug  7 19:22:34 rx37-0 kernel: libceph: osd37 192.168.113.52:6911 socket closed (con state OPEN)
> Aug  7 19:22:34 rx37-0 kernel: libceph: osd98 192.168.113.54:6842 socket closed (con state OPEN)
> Aug  7 19:22:34 rx37-0 kernel: libceph: osd26 192.168.113.52:6878 socket closed (con state OPEN)
> Aug  7 19:24:43 rx37-0 kernel: INFO: task kworker/2:0:19 blocked for more than 120 seconds.
> Aug  7 19:24:43 rx37-0 kernel:      Not tainted 3.16.0-1.el6.elrepo.x86_64 #1
> Aug  7 19:24:43 rx37-0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Aug  7 19:24:43 rx37-0 kernel: kworker/2:0     D 0000000000000002     0    19      2 0x00000000
> Aug  7 19:24:43 rx37-0 kernel: Workqueue: ceph-msgr con_work [libceph]
> Aug  7 19:24:43 rx37-0 kernel: ffff8810307bfb68 0000000000000046 ffff8810307bfb18 ffff8810307bc010
> Aug  7 19:24:43 rx37-0 kernel: 0000000000014380 0000000000014380 ffff8810307ae390 ffff880079678250
> Aug  7 19:24:43 rx37-0 kernel: 0000003500004040 ffff88102a1fd7c8 ffff88102a1fd7cc ffff8810307ae390
> Aug  7 19:24:43 rx37-0 kernel: Call Trace:
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffff81647629>] schedule+0x29/0x70
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffff8164778e>] schedule_preempt_disabled+0xe/0x10
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffff816490fb>] __mutex_lock_slowpath+0xdb/0x1d0
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffff81649213>] mutex_lock+0x23/0x40
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa0615e0f>] get_reply+0x3f/0x200 [libceph]
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa0616058>] alloc_msg+0x88/0x90 [libceph]
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa060d8f1>] ceph_con_in_msg_alloc+0x71/0x240 [libceph]
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa060eba8>] read_partial_message+0x1e8/0x3d0 [libceph]
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa060d278>] ? ceph_tcp_recvmsg+0x48/0x60 [libceph]
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa06101d6>] try_read+0x2b6/0x430 [libceph]
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffffa0610688>] con_work+0x78/0x220 [libceph]
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffff8108d60c>] process_one_work+0x17c/0x420
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffff8108e7d3>] worker_thread+0x123/0x420
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffff8108e6b0>] ? maybe_create_worker+0x180/0x180
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffff810943be>] kthread+0xce/0xf0
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffff810942f0>] ? kthread_freezable_should_stop+0x70/0x70
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffff8164ae3c>] ret_from_fork+0x7c/0xb0
> Aug  7 19:24:43 rx37-0 kernel: [<ffffffff810942f0>] ? kthread_freezable_should_stop+0x70/0x70
> Aug  7 19:24:43 rx37-0 kernel: INFO: task kworker/3:0:24 blocked for more than 120 seconds.
> Aug  7 19:24:43 rx37-0 kernel:      Not tainted 3.16.0-1.el6.elrepo.x86_64 #1
> Aug  7 19:24:43 rx37-0 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Aug  7 19:24:43 rx37-0 kernel: kworker/3:0     D 0000000000000003     0    24      2 0x00000000
> Aug  7 19:24:43 rx37-0 kernel: Workqueue: ceph-msgr con_work [libceph]
> Aug  7 19:24:43 rx37-0 kernel: ffff881030027c98 0000000000000046 ffff881019afe330 ffff881030024010
> (...)
>
>
> Any ideas ?
>
> With Kernel 3.10.32 on the client side everythink worked fine.

http://tracker.ceph.com/issues/8818, the fix is on its way to mainline
and will be backported to 3.15 and 3.16.

Thanks,

                Ilya