Failed to repair pg

CEPH filesystem development
 help / color / mirror / Atom feed

* Failed to repair pg
@ 2019-03-07 16:37 Herbert Alexander Faleiros
       [not found] ` <20190307163755.GV6574-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>
  0 siblings, 1 reply; 6+ messages in thread
From: Herbert Alexander Faleiros @ 2019-03-07 16:37 UTC (permalink / raw)
  To: ceph-users-idqoXFIVOFJgJs9I8MT0rw; +Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA

Hi,

# ceph health detail
HEALTH_ERR 3 scrub errors; Possible data damage: 1 pg inconsistent
OSD_SCRUB_ERRORS 3 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
    pg 2.2bb is active+clean+inconsistent, acting [36,12,80]

# ceph pg repair 2.2bb
instructing pg 2.2bb on osd.36 to repair

But:

2019-03-07 13:23:38.636881 [ERR]  Health check update: Possible data damage: 1 pg inconsistent, 1 pg repair (PG_DAMAGED) 
2019-03-07 13:20:38.373431 [ERR]  2.2bb deep-scrub 3 errors 
2019-03-07 13:20:38.373426 [ERR]  2.2bb deep-scrub 0 missing, 1 inconsistent objects 
2019-03-07 13:20:43.486860 [ERR]  Health check update: 3 scrub errors (OSD_SCRUB_ERRORS) 
2019-03-07 13:19:17.741350 [ERR]  deep-scrub 2.2bb 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : is an unexpected clone 
2019-03-07 13:19:17.523042 [ERR]  2.2bb shard 36 soid 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : data_digest 0xffffffff != data_digest 0xfc6b9538 from shard 12, size 0 != size 4194304 from auth oi 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986(482757'14986708 client.112595650.0:344888465 dirty|omap_digest s 4194304 uv 14974021 od ffffffff alloc_hint [0 0 0]), size 0 != size 4194304 from shard 12 
2019-03-07 13:19:17.523038 [ERR]  2.2bb shard 36 soid 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : candidate size 0 info size 4194304 mismatch 
2019-03-07 13:16:48.542673 [ERR]  2.2bb repair 2 errors, 1 fixed 
2019-03-07 13:16:48.542656 [ERR]  2.2bb repair 1 missing, 0 inconsistent objects 
2019-03-07 13:16:53.774956 [ERR]  Health check update: Possible data damage: 1 pg inconsistent (PG_DAMAGED) 
2019-03-07 13:16:53.774916 [ERR]  Health check update: 2 scrub errors (OSD_SCRUB_ERRORS) 
2019-03-07 13:15:16.986872 [ERR]  repair 2.2bb 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : is an unexpected clone 
2019-03-07 13:15:16.986817 [ERR]  2.2bb shard 36 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : missing 
2019-03-07 13:12:18.517442 [ERR]  Health check update: Possible data damage: 1 pg inconsistent, 1 pg repair (PG_DAMAGED) 

Also tried deep-scrub and scrub, same results.

Also set noscrub,nodeep-scrub, kicked currently active scrubs one at
a time using 'ceph osd down <id>'. After the last scrub was kicked,
forced scrub ran immediately then 'ceph pg repair', no luck.

Finally tryed the manual aproach:

 - stop osd.36
 - flush-journal
 - rm rbd\udata.dfd5e2235befd0.000000000001c299__4f986_CBDE52BB__2
 - start osd.36
 - ceph pg repair 2.2bb

Also no luck...

rbd\udata.dfd5e2235befd0.000000000001c299__4f986_CBDE52BB__2 at osd.36
is empty (0 size). At osd.80 4.0M, osd.2 is bluestore (can't find it).

Ceph is 12.2.10, I'm currently migrating all my OSDs to bluestore.

Is there anything else I can do?

# rados list-inconsistent-obj 2.2bb | jq
{
  "epoch": 484655,
  "inconsistents": [
    {
      "object": {
        "name": "rbd_data.dfd5e2235befd0.000000000001c299",
        "nspace": "",
        "locator": "",
        "snap": 326022,
        "version": 14974021
      },
      "errors": [
        "data_digest_mismatch",
        "size_mismatch"
      ],
      "union_shard_errors": [
        "size_mismatch_info",
        "obj_size_info_mismatch"
      ],
      "selected_object_info": {
        "oid": {
          "oid": "rbd_data.dfd5e2235befd0.000000000001c299",
          "key": "",
          "snapid": 326022,
          "hash": 3420345019,
          "max": 0,
          "pool": 2,
          "namespace": ""
        },
        "version": "482757'14986708",
        "prior_version": "482697'14980304",
        "last_reqid": "client.112595650.0:344888465",
        "user_version": 14974021,
        "size": 4194304,
        "mtime": "2019-03-02 22:30:23.812849",
        "local_mtime": "2019-03-02 22:30:23.813281",
        "lost": 0,
        "flags": [
          "dirty",
          "omap_digest"
        ],
        "legacy_snaps": [],
        "truncate_seq": 0,
        "truncate_size": 0,
        "data_digest": "0xffffffff",
        "omap_digest": "0xffffffff",
        "expected_object_size": 0,
        "expected_write_size": 0,
        "alloc_hint_flags": 0,
        "manifest": {
          "type": 0,
          "redirect_target": {
            "oid": "",
            "key": "",
            "snapid": 0,
            "hash": 0,
            "max": 0,
            "pool": -9223372036854776000,
            "namespace": ""
          }
        },
        "watchers": {}
      },
      "shards": [
        {
          "osd": 12,
          "primary": false,
          "errors": [],
          "size": 4194304,
          "omap_digest": "0xffffffff",
          "data_digest": "0xfc6b9538"
        },
        {
          "osd": 36,
          "primary": true,
          "errors": [
            "size_mismatch_info",
            "obj_size_info_mismatch"
          ],
          "size": 0,
          "omap_digest": "0xffffffff",
          "data_digest": "0xffffffff",
          "object_info": {
            "oid": {
              "oid": "rbd_data.dfd5e2235befd0.000000000001c299",
              "key": "",
              "snapid": 326022,
              "hash": 3420345019,
              "max": 0,
              "pool": 2,
              "namespace": ""
            },
            "version": "482757'14986708",
            "prior_version": "482697'14980304",
            "last_reqid": "client.112595650.0:344888465",
            "user_version": 14974021,
            "size": 4194304,
            "mtime": "2019-03-02 22:30:23.812849",
            "local_mtime": "2019-03-02 22:30:23.813281",
            "lost": 0,
            "flags": [
              "dirty",
              "omap_digest"
            ],
            "legacy_snaps": [],
            "truncate_seq": 0,
            "truncate_size": 0,
            "data_digest": "0xffffffff",
            "omap_digest": "0xffffffff",
            "expected_object_size": 0,
            "expected_write_size": 0,
            "alloc_hint_flags": 0,
            "manifest": {
              "type": 0,
              "redirect_target": {
                "oid": "",
                "key": "",
                "snapid": 0,
                "hash": 0,
                "max": 0,
                "pool": -9223372036854776000,
                "namespace": ""
              }
            },
            "watchers": {}
          }
        },
        {
          "osd": 80,
          "primary": false,
          "errors": [],
          "size": 4194304,
          "omap_digest": "0xffffffff",
          "data_digest": "0xfc6b9538"
        }
      ]
    }
  ]
}

-- 
Herbert

^ permalink raw reply	[flat|nested] 6+ messages in thread

[parent not found: <20190307163755.GV6574-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>]

* Re: Failed to repair pg
       [not found] ` <20190307163755.GV6574-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>
@ 2019-03-07 17:32   ` Herbert Alexander Faleiros
       [not found]     ` <20190307173242.GB6574-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>
  0 siblings, 1 reply; 6+ messages in thread
From: Herbert Alexander Faleiros @ 2019-03-07 17:32 UTC (permalink / raw)
  To: ceph-users-idqoXFIVOFJgJs9I8MT0rw; +Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA

On Thu, Mar 07, 2019 at 01:37:55PM -0300, Herbert Alexander Faleiros wrote:
> Hi,
> 
> # ceph health detail
> HEALTH_ERR 3 scrub errors; Possible data damage: 1 pg inconsistent
> OSD_SCRUB_ERRORS 3 scrub errors
> PG_DAMAGED Possible data damage: 1 pg inconsistent
>     pg 2.2bb is active+clean+inconsistent, acting [36,12,80]
> 
> # ceph pg repair 2.2bb
> instructing pg 2.2bb on osd.36 to repair
> 
> But:
> 
> 2019-03-07 13:23:38.636881 [ERR]  Health check update: Possible data damage: 1 pg inconsistent, 1 pg repair (PG_DAMAGED) 
> 2019-03-07 13:20:38.373431 [ERR]  2.2bb deep-scrub 3 errors 
> 2019-03-07 13:20:38.373426 [ERR]  2.2bb deep-scrub 0 missing, 1 inconsistent objects 
> 2019-03-07 13:20:43.486860 [ERR]  Health check update: 3 scrub errors (OSD_SCRUB_ERRORS) 
> 2019-03-07 13:19:17.741350 [ERR]  deep-scrub 2.2bb 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : is an unexpected clone 
> 2019-03-07 13:19:17.523042 [ERR]  2.2bb shard 36 soid 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : data_digest 0xffffffff != data_digest 0xfc6b9538 from shard 12, size 0 != size 4194304 from auth oi 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986(482757'14986708 client.112595650.0:344888465 dirty|omap_digest s 4194304 uv 14974021 od ffffffff alloc_hint [0 0 0]), size 0 != size 4194304 from shard 12 
> 2019-03-07 13:19:17.523038 [ERR]  2.2bb shard 36 soid 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : candidate size 0 info size 4194304 mismatch 
> 2019-03-07 13:16:48.542673 [ERR]  2.2bb repair 2 errors, 1 fixed 
> 2019-03-07 13:16:48.542656 [ERR]  2.2bb repair 1 missing, 0 inconsistent objects 
> 2019-03-07 13:16:53.774956 [ERR]  Health check update: Possible data damage: 1 pg inconsistent (PG_DAMAGED) 
> 2019-03-07 13:16:53.774916 [ERR]  Health check update: 2 scrub errors (OSD_SCRUB_ERRORS) 
> 2019-03-07 13:15:16.986872 [ERR]  repair 2.2bb 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : is an unexpected clone 
> 2019-03-07 13:15:16.986817 [ERR]  2.2bb shard 36 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : missing 
> 2019-03-07 13:12:18.517442 [ERR]  Health check update: Possible data damage: 1 pg inconsistent, 1 pg repair (PG_DAMAGED) 
> 
> Also tried deep-scrub and scrub, same results.
> 
> Also set noscrub,nodeep-scrub, kicked currently active scrubs one at
> a time using 'ceph osd down <id>'. After the last scrub was kicked,
> forced scrub ran immediately then 'ceph pg repair', no luck.
> 
> Finally tryed the manual aproach:
> 
>  - stop osd.36
>  - flush-journal
>  - rm rbd\udata.dfd5e2235befd0.000000000001c299__4f986_CBDE52BB__2
>  - start osd.36
>  - ceph pg repair 2.2bb
> 
> Also no luck...
> 
> rbd\udata.dfd5e2235befd0.000000000001c299__4f986_CBDE52BB__2 at osd.36
> is empty (0 size). At osd.80 4.0M, osd.2 is bluestore (can't find it).
> 
> Ceph is 12.2.10, I'm currently migrating all my OSDs to bluestore.
> 
> Is there anything else I can do?

Should I do something like this? (below, after stop osd.36)

# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-36/ --journal-path /dev/sdc1 rbd_data.dfd5e2235befd0.000000000001c299 remove-clone-metadata 326022

I'm no sure about rbd_data.$RBD and $CLONEID (took from rados
list-inconsistent-obj, also below).

> # rados list-inconsistent-obj 2.2bb | jq
> {
>   "epoch": 484655,
>   "inconsistents": [
>     {
>       "object": {
>         "name": "rbd_data.dfd5e2235befd0.000000000001c299",
>         "nspace": "",
>         "locator": "",
>         "snap": 326022,
>         "version": 14974021
>       },
>       "errors": [
>         "data_digest_mismatch",
>         "size_mismatch"
>       ],
>       "union_shard_errors": [
>         "size_mismatch_info",
>         "obj_size_info_mismatch"
>       ],
>       "selected_object_info": {
>         "oid": {
>           "oid": "rbd_data.dfd5e2235befd0.000000000001c299",
>           "key": "",
>           "snapid": 326022,
>           "hash": 3420345019,
>           "max": 0,
>           "pool": 2,
>           "namespace": ""
>         },
>         "version": "482757'14986708",
>         "prior_version": "482697'14980304",
>         "last_reqid": "client.112595650.0:344888465",
>         "user_version": 14974021,
>         "size": 4194304,
>         "mtime": "2019-03-02 22:30:23.812849",
>         "local_mtime": "2019-03-02 22:30:23.813281",
>         "lost": 0,
>         "flags": [
>           "dirty",
>           "omap_digest"
>         ],
>         "legacy_snaps": [],
>         "truncate_seq": 0,
>         "truncate_size": 0,
>         "data_digest": "0xffffffff",
>         "omap_digest": "0xffffffff",
>         "expected_object_size": 0,
>         "expected_write_size": 0,
>         "alloc_hint_flags": 0,
>         "manifest": {
>           "type": 0,
>           "redirect_target": {
>             "oid": "",
>             "key": "",
>             "snapid": 0,
>             "hash": 0,
>             "max": 0,
>             "pool": -9223372036854776000,
>             "namespace": ""
>           }
>         },
>         "watchers": {}
>       },
>       "shards": [
>         {
>           "osd": 12,
>           "primary": false,
>           "errors": [],
>           "size": 4194304,
>           "omap_digest": "0xffffffff",
>           "data_digest": "0xfc6b9538"
>         },
>         {
>           "osd": 36,
>           "primary": true,
>           "errors": [
>             "size_mismatch_info",
>             "obj_size_info_mismatch"
>           ],
>           "size": 0,
>           "omap_digest": "0xffffffff",
>           "data_digest": "0xffffffff",
>           "object_info": {
>             "oid": {
>               "oid": "rbd_data.dfd5e2235befd0.000000000001c299",
>               "key": "",
>               "snapid": 326022,
>               "hash": 3420345019,
>               "max": 0,
>               "pool": 2,
>               "namespace": ""
>             },
>             "version": "482757'14986708",
>             "prior_version": "482697'14980304",
>             "last_reqid": "client.112595650.0:344888465",
>             "user_version": 14974021,
>             "size": 4194304,
>             "mtime": "2019-03-02 22:30:23.812849",
>             "local_mtime": "2019-03-02 22:30:23.813281",
>             "lost": 0,
>             "flags": [
>               "dirty",
>               "omap_digest"
>             ],
>             "legacy_snaps": [],
>             "truncate_seq": 0,
>             "truncate_size": 0,
>             "data_digest": "0xffffffff",
>             "omap_digest": "0xffffffff",
>             "expected_object_size": 0,
>             "expected_write_size": 0,
>             "alloc_hint_flags": 0,
>             "manifest": {
>               "type": 0,
>               "redirect_target": {
>                 "oid": "",
>                 "key": "",
>                 "snapid": 0,
>                 "hash": 0,
>                 "max": 0,
>                 "pool": -9223372036854776000,
>                 "namespace": ""
>               }
>             },
>             "watchers": {}
>           }
>         },
>         {
>           "osd": 80,
>           "primary": false,
>           "errors": [],
>           "size": 4194304,
>           "omap_digest": "0xffffffff",
>           "data_digest": "0xfc6b9538"
>         }
>       ]
>     }
>   ]
> }

^ permalink raw reply	[flat|nested] 6+ messages in thread

[parent not found: <20190307173242.GB6574-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>]

* Re: Failed to repair pg
       [not found]     ` <20190307173242.GB6574-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>
@ 2019-03-07 23:18       ` Brad Hubbard
  2019-03-08  3:48       ` David Zafman
  1 sibling, 0 replies; 6+ messages in thread
From: Brad Hubbard @ 2019-03-07 23:18 UTC (permalink / raw)
  To: Herbert Alexander Faleiros; +Cc: Ceph Users, ceph-devel

you could try reading the data from this object and write it again
using rados get then rados put.

On Fri, Mar 8, 2019 at 3:32 AM Herbert Alexander Faleiros
<herbert-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org> wrote:
>
> On Thu, Mar 07, 2019 at 01:37:55PM -0300, Herbert Alexander Faleiros wrote:
> > Hi,
> >
> > # ceph health detail
> > HEALTH_ERR 3 scrub errors; Possible data damage: 1 pg inconsistent
> > OSD_SCRUB_ERRORS 3 scrub errors
> > PG_DAMAGED Possible data damage: 1 pg inconsistent
> >     pg 2.2bb is active+clean+inconsistent, acting [36,12,80]
> >
> > # ceph pg repair 2.2bb
> > instructing pg 2.2bb on osd.36 to repair
> >
> > But:
> >
> > 2019-03-07 13:23:38.636881 [ERR]  Health check update: Possible data damage: 1 pg inconsistent, 1 pg repair (PG_DAMAGED)
> > 2019-03-07 13:20:38.373431 [ERR]  2.2bb deep-scrub 3 errors
> > 2019-03-07 13:20:38.373426 [ERR]  2.2bb deep-scrub 0 missing, 1 inconsistent objects
> > 2019-03-07 13:20:43.486860 [ERR]  Health check update: 3 scrub errors (OSD_SCRUB_ERRORS)
> > 2019-03-07 13:19:17.741350 [ERR]  deep-scrub 2.2bb 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : is an unexpected clone
> > 2019-03-07 13:19:17.523042 [ERR]  2.2bb shard 36 soid 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : data_digest 0xffffffff != data_digest 0xfc6b9538 from shard 12, size 0 != size 4194304 from auth oi 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986(482757'14986708 client.112595650.0:344888465 dirty|omap_digest s 4194304 uv 14974021 od ffffffff alloc_hint [0 0 0]), size 0 != size 4194304 from shard 12
> > 2019-03-07 13:19:17.523038 [ERR]  2.2bb shard 36 soid 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : candidate size 0 info size 4194304 mismatch
> > 2019-03-07 13:16:48.542673 [ERR]  2.2bb repair 2 errors, 1 fixed
> > 2019-03-07 13:16:48.542656 [ERR]  2.2bb repair 1 missing, 0 inconsistent objects
> > 2019-03-07 13:16:53.774956 [ERR]  Health check update: Possible data damage: 1 pg inconsistent (PG_DAMAGED)
> > 2019-03-07 13:16:53.774916 [ERR]  Health check update: 2 scrub errors (OSD_SCRUB_ERRORS)
> > 2019-03-07 13:15:16.986872 [ERR]  repair 2.2bb 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : is an unexpected clone
> > 2019-03-07 13:15:16.986817 [ERR]  2.2bb shard 36 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : missing
> > 2019-03-07 13:12:18.517442 [ERR]  Health check update: Possible data damage: 1 pg inconsistent, 1 pg repair (PG_DAMAGED)
> >
> > Also tried deep-scrub and scrub, same results.
> >
> > Also set noscrub,nodeep-scrub, kicked currently active scrubs one at
> > a time using 'ceph osd down <id>'. After the last scrub was kicked,
> > forced scrub ran immediately then 'ceph pg repair', no luck.
> >
> > Finally tryed the manual aproach:
> >
> >  - stop osd.36
> >  - flush-journal
> >  - rm rbd\udata.dfd5e2235befd0.000000000001c299__4f986_CBDE52BB__2
> >  - start osd.36
> >  - ceph pg repair 2.2bb
> >
> > Also no luck...
> >
> > rbd\udata.dfd5e2235befd0.000000000001c299__4f986_CBDE52BB__2 at osd.36
> > is empty (0 size). At osd.80 4.0M, osd.2 is bluestore (can't find it).
> >
> > Ceph is 12.2.10, I'm currently migrating all my OSDs to bluestore.
> >
> > Is there anything else I can do?
>
> Should I do something like this? (below, after stop osd.36)
>
> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-36/ --journal-path /dev/sdc1 rbd_data.dfd5e2235befd0.000000000001c299 remove-clone-metadata 326022
>
> I'm no sure about rbd_data.$RBD and $CLONEID (took from rados
> list-inconsistent-obj, also below).
>
> > # rados list-inconsistent-obj 2.2bb | jq
> > {
> >   "epoch": 484655,
> >   "inconsistents": [
> >     {
> >       "object": {
> >         "name": "rbd_data.dfd5e2235befd0.000000000001c299",
> >         "nspace": "",
> >         "locator": "",
> >         "snap": 326022,
> >         "version": 14974021
> >       },
> >       "errors": [
> >         "data_digest_mismatch",
> >         "size_mismatch"
> >       ],
> >       "union_shard_errors": [
> >         "size_mismatch_info",
> >         "obj_size_info_mismatch"
> >       ],
> >       "selected_object_info": {
> >         "oid": {
> >           "oid": "rbd_data.dfd5e2235befd0.000000000001c299",
> >           "key": "",
> >           "snapid": 326022,
> >           "hash": 3420345019,
> >           "max": 0,
> >           "pool": 2,
> >           "namespace": ""
> >         },
> >         "version": "482757'14986708",
> >         "prior_version": "482697'14980304",
> >         "last_reqid": "client.112595650.0:344888465",
> >         "user_version": 14974021,
> >         "size": 4194304,
> >         "mtime": "2019-03-02 22:30:23.812849",
> >         "local_mtime": "2019-03-02 22:30:23.813281",
> >         "lost": 0,
> >         "flags": [
> >           "dirty",
> >           "omap_digest"
> >         ],
> >         "legacy_snaps": [],
> >         "truncate_seq": 0,
> >         "truncate_size": 0,
> >         "data_digest": "0xffffffff",
> >         "omap_digest": "0xffffffff",
> >         "expected_object_size": 0,
> >         "expected_write_size": 0,
> >         "alloc_hint_flags": 0,
> >         "manifest": {
> >           "type": 0,
> >           "redirect_target": {
> >             "oid": "",
> >             "key": "",
> >             "snapid": 0,
> >             "hash": 0,
> >             "max": 0,
> >             "pool": -9223372036854776000,
> >             "namespace": ""
> >           }
> >         },
> >         "watchers": {}
> >       },
> >       "shards": [
> >         {
> >           "osd": 12,
> >           "primary": false,
> >           "errors": [],
> >           "size": 4194304,
> >           "omap_digest": "0xffffffff",
> >           "data_digest": "0xfc6b9538"
> >         },
> >         {
> >           "osd": 36,
> >           "primary": true,
> >           "errors": [
> >             "size_mismatch_info",
> >             "obj_size_info_mismatch"
> >           ],
> >           "size": 0,
> >           "omap_digest": "0xffffffff",
> >           "data_digest": "0xffffffff",
> >           "object_info": {
> >             "oid": {
> >               "oid": "rbd_data.dfd5e2235befd0.000000000001c299",
> >               "key": "",
> >               "snapid": 326022,
> >               "hash": 3420345019,
> >               "max": 0,
> >               "pool": 2,
> >               "namespace": ""
> >             },
> >             "version": "482757'14986708",
> >             "prior_version": "482697'14980304",
> >             "last_reqid": "client.112595650.0:344888465",
> >             "user_version": 14974021,
> >             "size": 4194304,
> >             "mtime": "2019-03-02 22:30:23.812849",
> >             "local_mtime": "2019-03-02 22:30:23.813281",
> >             "lost": 0,
> >             "flags": [
> >               "dirty",
> >               "omap_digest"
> >             ],
> >             "legacy_snaps": [],
> >             "truncate_seq": 0,
> >             "truncate_size": 0,
> >             "data_digest": "0xffffffff",
> >             "omap_digest": "0xffffffff",
> >             "expected_object_size": 0,
> >             "expected_write_size": 0,
> >             "alloc_hint_flags": 0,
> >             "manifest": {
> >               "type": 0,
> >               "redirect_target": {
> >                 "oid": "",
> >                 "key": "",
> >                 "snapid": 0,
> >                 "hash": 0,
> >                 "max": 0,
> >                 "pool": -9223372036854776000,
> >                 "namespace": ""
> >               }
> >             },
> >             "watchers": {}
> >           }
> >         },
> >         {
> >           "osd": 80,
> >           "primary": false,
> >           "errors": [],
> >           "size": 4194304,
> >           "omap_digest": "0xffffffff",
> >           "data_digest": "0xfc6b9538"
> >         }
> >       ]
> >     }
> >   ]
> > }
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Cheers,
Brad

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: Failed to repair pg
       [not found]     ` <20190307173242.GB6574-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>
  2019-03-07 23:18       ` Brad Hubbard
@ 2019-03-08  3:48       ` David Zafman
       [not found]         ` <ab4615ef-6064-52b6-4442-faa45cbc84f5-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
  1 sibling, 1 reply; 6+ messages in thread
From: David Zafman @ 2019-03-08  3:48 UTC (permalink / raw)
  To: Herbert Alexander Faleiros, ceph-users-idqoXFIVOFJgJs9I8MT0rw
  Cc: ceph-devel-u79uwXL29TY76Z2rM5mHXA


On 3/7/19 9:32 AM, Herbert Alexander Faleiros wrote:
> On Thu, Mar 07, 2019 at 01:37:55PM -0300, Herbert Alexander Faleiros wrote:
> Should I do something like this? (below, after stop osd.36)
>
> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-36/ --journal-path /dev/sdc1 rbd_data.dfd5e2235befd0.000000000001c299 remove-clone-metadata 326022
>
> I'm no sure about rbd_data.$RBD and $CLONEID (took from rados
> list-inconsistent-obj, also below).


See what results you get from this command.

# rados list-inconsistent-snapset 2.2bb --format=json-pretty

You might see this, so nothing interesting.  If you don't get json, then 
re-run a scrub again.

{
     "epoch": ######,
     "inconsistents": []
}

I don't think you need to do the remove-clone-metadata because you got 
"unexpected clone" so I think you'd get "Clone 326022 not present"

I think you need to remove the clone object from osd.12 and osd.80.  For 
example:

# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12/ 
--journal-path /dev/sdXX --op list rbd_data.dfd5e2235befd0.000000000001c299

["2.2bb",{"oid":"rbd_data.dfd5e2235befd0.000000000001c299","key":"","snapid":-2,"hash":########,"max":0,"pool":2,"namespace":"","max":0}]
["2.2bb",{"oid":"rbd_data.dfd5e2235befd0.000000000001c299","key":"","snapid":326022,"hash":#########,"max":0,"pool":2,"namespace":"","max":0}]

Use the json for snapid 326022 to remove it.

# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12/ 
--journal-path /dev/sdXX 
'["2.2bb",{"oid":"rbd_data.dfd5e2235befd0.000000000001c299","key":"","snapid":326022,"hash":#########,"max":0,"pool":2,"namespace":"","max":0}]' 
remove


David

_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

^ permalink raw reply	[flat|nested] 6+ messages in thread

[parent not found: <ab4615ef-6064-52b6-4442-faa45cbc84f5-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>]

* Re: Failed to repair pg
       [not found]         ` <ab4615ef-6064-52b6-4442-faa45cbc84f5-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
@ 2019-03-08 12:52           ` Herbert Alexander Faleiros
       [not found]             ` <20190308125224.GA92844-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>
  0 siblings, 1 reply; 6+ messages in thread
From: Herbert Alexander Faleiros @ 2019-03-08 12:52 UTC (permalink / raw)
  To: David Zafman
  Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw,
	ceph-devel-u79uwXL29TY76Z2rM5mHXA

Hi,

thanks for the answer.

On Thu, Mar 07, 2019 at 07:48:59PM -0800, David Zafman wrote:
> See what results you get from this command.
> 
> # rados list-inconsistent-snapset 2.2bb --format=json-pretty
> 
> You might see this, so nothing interesting.  If you don't get json, then 
> re-run a scrub again.
> 
> {
>      "epoch": ######,
>      "inconsistents": []
> }

# rados list-inconsistent-snapset 2.2bb --format=json-pretty
{
    "epoch": 485065,
    "inconsistents": [
        {
            "name": "rbd_data.dfd5e2235befd0.000000000001c299",
            "nspace": "",
            "locator": "",
            "snap": 326022,
            "errors": [
                "headless"
            ]
        },
        {
            "name": "rbd_data.dfd5e2235befd0.000000000001c299",
            "nspace": "",
            "locator": "",
            "snap": "head",
            "snapset": {
                "snap_context": {
                    "seq": 327360,
                    "snaps": []
                },
                "head_exists": 1,
                "clones": []
            },
            "errors": [
                "extra_clones"
            ],
            "extra clones": [
                326022
            ]
        }
    ]
}

> I don't think you need to do the remove-clone-metadata because you got 
> "unexpected clone" so I think you'd get "Clone 326022 not present"
> 
> I think you need to remove the clone object from osd.12 and osd.80.  For 
> example:
> 
> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12/ 
> --journal-path /dev/sdXX --op list rbd_data.dfd5e2235befd0.000000000001c299
> 
> ["2.2bb",{"oid":"rbd_data.dfd5e2235befd0.000000000001c299","key":"","snapid":-2,"hash":########,"max":0,"pool":2,"namespace":"","max":0}]
> ["2.2bb",{"oid":"rbd_data.dfd5e2235befd0.000000000001c299","key":"","snapid":326022,"hash":#########,"max":0,"pool":2,"namespace":"","max":0}]
> 
> Use the json for snapid 326022 to remove it.
> 
> # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12/ 
> --journal-path /dev/sdXX 
> '["2.2bb",{"oid":"rbd_data.dfd5e2235befd0.000000000001c299","key":"","snapid":326022,"hash":#########,"max":0,"pool":2,"namespace":"","max":0}]' 
> remove

# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-80/ --journal-path /dev/sda1 --op list rbd_data.dfd5e2235befd0.000000000001c299 --pgid 2.2bb
["2.2bb",{"oid":"rbd_data.dfd5e2235befd0.000000000001c299","key":"","snapid":326022,"hash":3420345019,"max":0,"pool":2,"namespace":"","max":0}]
["2.2bb",{"oid":"rbd_data.dfd5e2235befd

I added --pgid 2.2bb because it is taking to long to finish.

# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-80/ --journal-path /dev/sda1 '["2.2bb",{"oid":"rbd_data.dfd5e2235befd0.000000000001c299","key":"","snapid":326022,"hash":3420345019,"max":0,"pool":2,"namespace":"","max":0}]' remove
remove #2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986#

osd.12 was a slight diferent because it is bluestore:

# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12/ --op list rbd_data.dfd5e2235befd0.000000000001c299 --pgid 2.2bb
["2.2bb",{"oid":"rbd_data.dfd5e2235befd0.000000000001c299","key":"","snapid":326022,"hash":3420345019,"max":0,"pool":2,"namespace":"","max":0}]
["2.2bb",{"oid":"rbd_data.dfd5e2235befd0.000000000001c299","key":"","snapid":-2,"hash":3420345019,"max":0,"pool":2,"namespace":"","max":0}]

# ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12/ '["2.2bb",{"oid":"rbd_data.dfd5e2235befd0.000000000001c299","key":"","snapid":326022,"hash":3420345019,"max":0,"pool":2,"namespace":"","max":0}]' remove
remove #2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986#

But nothing changed, so I tried to repair the pg again and from osd.36
I got now:

2019-03-08 09:09:11.786038 7f920c40d700 -1 log_channel(cluster) log [ERR] : 2.2bb shard 36 soid 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : candidate size 0 info size 4194304 mismatch
2019-03-08 09:09:11.786041 7f920c40d700 -1 log_channel(cluster) log [ERR] : 2.2bb soid 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : failed to pick suitable object info
2019-03-08 09:09:11.786182 7f920c40d700 -1 log_channel(cluster) log [ERR] : repair 2.2bb 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : on disk size (0) does not match object info size (4194304) adjusted for ondisk to (4194304)
2019-03-08 09:09:11.786191 7f920c40d700 -1 log_channel(cluster) log [ERR] : repair 2.2bb 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 : is an unexpected clone
2019-03-08 09:09:11.786213 7f920c40d700 -1 osd.36 pg_epoch: 485254 pg[2.2bb( v 485253'15080921 (485236'15079373,485253'15080921] local-lis/les=485251/485252 n=3836 ec=38/38 lis/c 485251/485251 les/c/f 485252/485252/0 485251/485251/484996) [36,12,80] r=0 lpr=485251 crt=485253'15080921 lcod 485252'15080920 mlcod 485252'15080920 active+clean+scrubbing+deep+inconsistent+repair snaptrimq=[5022c~1,50230~1]] _scan_snaps no clone_snaps for 2:dd4a7bd3:::rbd_data.dfd5e2235befd0.000000000001c299:4f986 in 4fec0=[]:{}

And:

# rados list-inconsistent-snapset 2.2bb --format=json-pretty
{
    "epoch": 485251,
    "inconsistents": []
}

Now I have:

HEALTH_ERR 5 scrub errors; Possible data damage: 1 pg inconsistent
OSD_SCRUB_ERRORS 5 scrub errors
PG_DAMAGED Possible data damage: 1 pg inconsistent
    pg 2.2bb is active+clean+inconsistent, acting [36,12,80]

Jumped from 3 to 5 scrub errors now.

Any clues?

Thanks again,

--
Herbert

^ permalink raw reply	[flat|nested] 6+ messages in thread

[parent not found: <20190308125224.GA92844-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>]

* Re: Failed to repair pg
       [not found]             ` <20190308125224.GA92844-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>
@ 2019-03-08 16:19               ` Herbert Alexander Faleiros
  0 siblings, 0 replies; 6+ messages in thread
From: Herbert Alexander Faleiros @ 2019-03-08 16:19 UTC (permalink / raw)
  To: David Zafman
  Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw,
	ceph-devel-u79uwXL29TY76Z2rM5mHXA

Hi,

[...]
> Now I have:
>
> HEALTH_ERR 5 scrub errors; Possible data damage: 1 pg inconsistent
> OSD_SCRUB_ERRORS 5 scrub errors
> PG_DAMAGED Possible data damage: 1 pg inconsistent
>     pg 2.2bb is active+clean+inconsistent, acting [36,12,80]
>
> Jumped from 3 to 5 scrub errors now.

did the same on osd.36 and a repair worked.

Health OK again.

Thank you,

--
Herbert

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2019-03-08 16:19 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-03-07 16:37 Failed to repair pg Herbert Alexander Faleiros
     [not found] ` <20190307163755.GV6574-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>
2019-03-07 17:32   ` Herbert Alexander Faleiros
     [not found]     ` <20190307173242.GB6574-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>
2019-03-07 23:18       ` Brad Hubbard
2019-03-08  3:48       ` David Zafman
     [not found]         ` <ab4615ef-6064-52b6-4442-faa45cbc84f5-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2019-03-08 12:52           ` Herbert Alexander Faleiros
     [not found]             ` <20190308125224.GA92844-b2jWxOBV4z0IdKJ7tpkyPg@public.gmane.org>
2019-03-08 16:19               ` Herbert Alexander Faleiros

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox