v12.2.7 Luminous released

[Abhishek Lekshmanan <abhishek-IBi9RG/b67k@public.gmane.org>]

This is the seventh bugfix release of Luminous v12.2.x long term
stable release series. This release contains several fixes for
regressions in the v12.2.6 and v12.2.5 releases.  We recommend that
all users upgrade. 

*NOTE* The v12.2.6 release has serious known regressions, while 12.2.6
wasn't formally announced in the mailing lists or blog, the packages
were built and available on download.ceph.com since last week. If you
installed this release, please see the upgrade procedure below.

*NOTE* The v12.2.5 release has a potential data corruption issue with
erasure coded pools. If you ran v12.2.5 with erasure coding, please see
below.

The full blog post alongwith the complete changelog is published at the
official ceph blog at https://ceph.com/releases/12-2-7-luminous-released/

Upgrading from v12.2.6
----------------------

v12.2.6 included an incomplete backport of an optimization for
BlueStore OSDs that avoids maintaining both the per-object checksum
and the internal BlueStore checksum.  Due to the accidental omission
of a critical follow-on patch, v12.2.6 corrupts (fails to update) the
stored per-object checksum value for some objects.  This can result in
an EIO error when trying to read those objects.

#. If your cluster uses FileStore only, no special action is required.
   This problem only affects clusters with BlueStore.

#. If your cluster has only BlueStore OSDs (no FileStore), then you
   should enable the following OSD option::

     osd skip data digest = true

   This will avoid setting and start ignoring the full-object digests
   whenever the primary for a PG is BlueStore.

#. If you have a mix of BlueStore and FileStore OSDs, then you should
   enable the following OSD option::

     osd distrust data digest = true

   This will avoid setting and start ignoring the full-object digests
   in all cases.  This weakens the data integrity checks for
   FileStore (although those checks were always only opportunistic).

If your cluster includes BlueStore OSDs and was affected, deep scrubs
will generate errors about mismatched CRCs for affected objects.
Currently the repair operation does not know how to correct them
(since all replicas do not match the expected checksum it does not
know how to proceed).  These warnings are harmless in the sense that
IO is not affected and the replicas are all still in sync.  The number
of affected objects is likely to drop (possibly to zero) on their own
over time as those objects are modified.  We expect to include a scrub
improvement in v12.2.8 to clean up any remaining objects.

Additionally, see the notes below, which apply to both v12.2.5 and v12.2.6.

Upgrading from v12.2.5 or v12.2.6
---------------------------------

If you used v12.2.5 or v12.2.6 in combination with erasure coded
pools, there is a small risk of corruption under certain workloads.
Specifically, when:

* An erasure coded pool is in use
* The pool is busy with successful writes
* The pool is also busy with updates that result in an error result to
  the librados user.  RGW garbage collection is the most common
  example of this (it sends delete operations on objects that don't
  always exist.)
* Some OSDs are reasonably busy.  One known example of such load is
  FileStore splitting, although in principle any load on the cluster
  could also trigger the behavior.
* One or more OSDs restarts.

This combination can trigger an OSD crash and possibly leave PGs in a state
where they fail to peer.

Notably, upgrading a cluster involves OSD restarts and as such may
increase the risk of encountering this bug.  For this reason, for
clusters with erasure coded pools, we recommend the following upgrade
procedure to minimize risk:

1. Install the v12.2.7 packages.
2. Temporarily quiesce IO to cluster::

     ceph osd pause

3. Restart all OSDs and wait for all PGs to become active.
4. Resume IO::

     ceph osd unpause

This will cause an availability outage for the duration of the OSD
restarts.  If this in unacceptable, an *more risky* alternative is to
disable RGW garbage collection (the primary known cause of these rados
operations) for the duration of the upgrade::

1. Set ``rgw_enable_gc_threads = false`` in ceph.conf
2. Restart all radosgw daemons
3. Upgrade and restart all OSDs
4. Remove ``rgw_enable_gc_threads = false`` from ceph.conf
5. Restart all radosgw daemons

Upgrading from other versions
-----------------------------

If your cluster did not run v12.2.5 or v12.2.6 then none of the above
issues apply to you and you should upgrade normally.

v12.2.7 Changelog
-----------------

* mon/AuthMonitor: improve error message (issue#21765, pr#22963, Douglas Fuller)
* osd/PG: do not blindly roll forward to log.head (issue#24597, pr#22976, Sage Weil)
* osd/PrimaryLogPG: rebuild attrs from clients (issue#24768 , pr#22962, Sage Weil)
* osd: work around data digest problems in 12.2.6 (version 2) (issue#24922, pr#23055, Sage Weil)
* rgw: objects in cache never refresh after rgw_cache_expiry_interval (issue#24346, pr#22369, Casey Bodley, Matt Benjamin)

Notable changes in v12.2.6 Luminous
===================================

:note: This is a broken release with serious known regressions.  Do not
install it. The release notes below are to help track the changes that
went in 12.2.6 and hence a part of 12.2.7


- *Auth*:

  * In 12.2.4 and earlier releases, keyring caps were not checked for validity,
    so the caps string could be anything. As of 12.2.6, caps strings are
    validated and providing a keyring with an invalid caps string to, e.g.,
    "ceph auth add" will result in an error.
  * CVE 2018-1128: auth: cephx authorizer subject to replay attack (issue#24836, Sage Weil)
  * CVE 2018-1129: auth: cephx signature check is weak (issue#24837, Sage Weil)
  * CVE 2018-10861: mon: auth checks not correct for pool ops (issue#24838, Jason Dillaman)


- The config-key interface can store arbitrary binary blobs but JSON
  can only express printable strings.  If binary blobs are present,
  the 'ceph config-key dump' command will show them as something like
  ``<<< binary blob of length N >>>``.

The full changelog for 12.2.6 is published in the release blog.

Getting ceph:
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-12.2.7.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: 3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5

-- 
Abhishek Lekshmanan
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
_______________________________________________
Ceph-announce mailing list
Ceph-announce@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-announce-ceph.com