osd crash when deep-scrubbing

osd crash when deep-scrubbing

[Jiaying Ren]

Hi, cephers:

I've encountered a problem that a pg stuck in inconsistent status:

$ ceph -s
    cluster 27d39faa-48ae-4356-a8e3-19d5b81e179e
     health HEALTH_ERR 1 pgs inconsistent; 34 near full osd(s); 1
scrub errors; noout flag(s) set
     monmap e4: 3 mons at
{server-61.0.yyyy.xxxxxxxxx.in=10.8.0.61:6789/0,server-62.0.yyyy.xxxxxxxxx.in=10.8.0.62:6789/0,server-63.0.yyyy.xxxxxxxxx.in=10.8.0.63:6789/0},
election epoch 6706, quorum 0,1,2
server-61.0.yyyy.xxxxxxxxx.in,server-62.0.yyyy.xxxxxxxxx.in,server-63.0.yyyy.xxxxxxxxx.in
     osdmap e87808: 180 osds: 180 up, 180 in
            flags noout
      pgmap v29322850: 35026 pgs, 15 pools, 27768 GB data, 1905 kobjects
            83575 GB used, 114 TB / 196 TB avail
               35025 active+clean
                   1 active+clean+inconsistent
  client io 120 kB/s rd, 216 MB/s wr, 6398 op/s

`pg repair` cmd doesn't work, so I manually repaired a inconsistent object(pool
size is 3,I removed the object different from other two copys).after that pg
still in inconsistent status:

$ ceph pg dump | grep active+clean+inconsistent
dumped all in format plain
3.d70   290     0       0       0       4600869888      3050    3050
  stale+active+clean+inconsistent 2015-10-18 13:05:43.320451
  87798'7631234   87798:10758311        [131,119,132]   131
  [131,119,132]   131     85161'7599152   2015-10-16 14:34:21.283303
  85161'7599152   2015-10-16 14:34:21.283303

And after restarted osd.131, the primary osd osd.131 would crash,the straceback:

 1: /usr/bin/ceph-osd() [0x9c6de1]
 2: (()+0xf790) [0x7f384b6b8790]
 3: (gsignal()+0x35) [0x7f384a58a625]
 4: (abort()+0x175) [0x7f384a58be05]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f384ae44a5d]
 6: (()+0xbcbe6) [0x7f384ae42be6]
 7: (()+0xbcc13) [0x7f384ae42c13]
 8: (()+0xbcd0e) [0x7f384ae42d0e]
 9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x13e) [0x9cd0de]
 10: (object_info_t::decode(ceph::buffer::list::iterator&)+0x81) [0x7dfaf1]
 11: (PG::_scan_snaps(ScrubMap&)+0x394) [0x84b8c4]
 12: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool,
ThreadPool::TPHandle&)+0x27b) [0x84cdab]
 13: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x5c4) [0x85c1b4]
 14: (PG::scrub(ThreadPool::TPHandle&)+0x181) [0x85d691]
 15: (OSD::ScrubWQ::_process(PG*, ThreadPool::TPHandle&)+0x1c) [0x6737cc]
 16: (ThreadPool::worker(ThreadPool::WorkThread*)+0x53d) [0x9e05dd]
 17: (ThreadPool::WorkThread::entry()+0x10) [0x9e1760]
 18: (()+0x7a51) [0x7f384b6b0a51]
 19: (clone()+0x6d) [0x7f384a6409ad]

ceph version is v0.80.9, manually executes `ceph pg deep-scrub 3.d70` would also
cause osd crash.

Any ideas? or did I missed some logs necessary for further investigation?

Thx.

--
Best Regards!
Jiaying Ren(mikulely)

Re: osd crash when deep-scrubbing

[changtao381]

Jiaying Ren <mikulely <at> gmail.com> writes:

> 
> Hi, cephers:
> 
> I've encountered a problem that a pg stuck in inconsistent status:
> 
> $ ceph -s
>     cluster 27d39faa-48ae-4356-a8e3-19d5b81e179e
>      health HEALTH_ERR 1 pgs inconsistent; 34 near full osd(s); 1
> scrub errors; noout flag(s) set
>      monmap e4: 3 mons at
>
{server-61.0.yyyy.xxxxxxxxx.in=10.8.0.61:6789/0,server-62.0.yyyy.xxxxxxxxx.i
n=10.8.0.62:6789/0,server-63.0.yyyy.xxxxxxxxx.in=10.8.0.63:6789/0},
> election epoch 6706, quorum 0,1,2
>
server-61.0.yyyy.xxxxxxxxx.in,server-62.0.yyyy.xxxxxxxxx.in,server-63.0.yyyy
.xxxxxxxxx.in
>      osdmap e87808: 180 osds: 180 up, 180 in
>             flags noout
>       pgmap v29322850: 35026 pgs, 15 pools, 27768 GB data, 1905 kobjects
>             83575 GB used, 114 TB / 196 TB avail
>                35025 active+clean
>                    1 active+clean+inconsistent
>   client io 120 kB/s rd, 216 MB/s wr, 6398 op/s
> 
> `pg repair` cmd doesn't work, so I manually repaired a inconsistent
object(pool
> size is 3,I removed the object different from other two copys).after that
pg
> still in inconsistent status:
> 
> $ ceph pg dump | grep active+clean+inconsistent
> dumped all in format plain
> 3.d70   290     0       0       0       4600869888      3050    3050
>   stale+active+clean+inconsistent 2015-10-18 13:05:43.320451
>   87798'7631234   87798:10758311        [131,119,132]   131
>   [131,119,132]   131     85161'7599152   2015-10-16 14:34:21.283303
>   85161'7599152   2015-10-16 14:34:21.283303
> 
> And after restarted osd.131, the primary osd osd.131 would crash,the
straceback:
> 
>  1: /usr/bin/ceph-osd() [0x9c6de1]
>  2: (()+0xf790) [0x7f384b6b8790]
>  3: (gsignal()+0x35) [0x7f384a58a625]
>  4: (abort()+0x175) [0x7f384a58be05]
>  5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f384ae44a5d]
>  6: (()+0xbcbe6) [0x7f384ae42be6]
>  7: (()+0xbcc13) [0x7f384ae42c13]
>  8: (()+0xbcd0e) [0x7f384ae42d0e]
>  9: (ceph::buffer::list::iterator::copy(unsigned int, char*)+0x13e)
[0x9cd0de]
>  10: (object_info_t::decode(ceph::buffer::list::iterator&)+0x81)
[0x7dfaf1]
>  11: (PG::_scan_snaps(ScrubMap&)+0x394) [0x84b8c4]
>  12: (PG::build_scrub_map_chunk(ScrubMap&, hobject_t, hobject_t, bool,
> ThreadPool::TPHandle&)+0x27b) [0x84cdab]
>  13: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x5c4) [0x85c1b4]
>  14: (PG::scrub(ThreadPool::TPHandle&)+0x181) [0x85d691]
>  15: (OSD::ScrubWQ::_process(PG*, ThreadPool::TPHandle&)+0x1c) [0x6737cc]
>  16: (ThreadPool::worker(ThreadPool::WorkThread*)+0x53d) [0x9e05dd]
>  17: (ThreadPool::WorkThread::entry()+0x10) [0x9e1760]
>  18: (()+0x7a51) [0x7f384b6b0a51]
>  19: (clone()+0x6d) [0x7f384a6409ad]
> 
> ceph version is v0.80.9, manually executes `ceph pg deep-scrub 3.d70`
would also
> cause osd crash.
> 
> Any ideas? or did I missed some logs necessary for further investigation?
> 
> Thx.
> 
> --
> Best Regards!
> Jiaying Ren(mikulely)
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

I have met a problem when run 'ceph pg deep-scrub' command. It also causes
osd crash. And finally i find some sector of the disk have corrupted .so
please check dmesg info to check weather there is some disk errors