Re: issue #8752 (inconsistent PGs on RBD caching pool)

[Dmitry Smirnov <onlyjob@debian.org>]

[-- Attachment #1: Type: text/plain, Size: 1702 bytes --]

On Thu, 2 Oct 2014 08:28:16 Sage Weil wrote:
> My guess is a btrfs issue.  The weird thing about your report is the byte
> totals are off by an uneven number of bytes (3 bytes, 9 bytes, etc.).
> We haven't ever seen this.  We do test RBD over cache tiers on btrfs,
> but not with EC on the base.  I'll add that combo to the matrix.  My first
> guess is a btrfs issue, honestly.

I think I found where it is happening: for a while I was using Btrfs-based 
OSDs with journals on ext4 partition on SSD. As an experiment I've decided to 
try moving all journal files back to their OSDs and it eliminated 
inconsistencies. I've updated the ticket with this information.
This behaviour is reproducible on 0.80.6.

It looks like Btrfs snapshotting do not affect this issue.


> Does it continue to come up after the kernels are upgraded (and after a
> full cycle of scrub and repairs have been done to clear out
> inconsistencies introduced while running the older kernel)?

Yes, I tried many times after every kernel update or any change in cluster 
whatsoever. Repair is usually ineffective and doesn't change anything: it 
would log "repair 1 errors, 1 fixed" but "ceph pg scrub" will find an error 
right away. Moreover repair is not even necessary -- inconsistencies stay on 
some PGs for a while then "move" to different PGs. For example "ceph pg scrub 
19.NN" sometimes would be clearing affected pg from "inconsistent" state or 
discover a new inconsistency seemingly at random.

Thank you.

-- 
Cheers,
 Dmitry Smirnov.

---

Odious ideas are not entitled to hide from criticism behind the human
shield of their believers' feelings.
        -- Richard Stallman

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]