* Re: [ceph-users] Inconsistent PGs that ceph pg repair does not fix [not found] ` <D1E53B6A.8DC6%apataki@simonsfoundation.org> @ 2015-08-03 19:38 ` Samuel Just 2015-08-03 20:09 ` Andras Pataki 0 siblings, 1 reply; 6+ messages in thread From: Samuel Just @ 2015-08-03 19:38 UTC (permalink / raw) To: Andras Pataki; +Cc: ceph-users@lists.ceph.com, ceph-devel@vger.kernel.org Hrm, that's certainly supposed to work. Can you make a bug? Be sure to note what version you are running (output of ceph-osd -v). -Sam On Mon, Aug 3, 2015 at 12:34 PM, Andras Pataki <apataki@simonsfoundation.org> wrote: > Summary: I am having problems with inconsistent PG's that the 'ceph pg > repair' command does not fix. Below are the details. Any help would be > appreciated. > > # Find the inconsistent PG's > ~# ceph pg dump | grep inconsistent > dumped all in format plain > 2.439 42080 00 017279507143 31033103 active+clean+inconsistent2015-08-03 > 14:49:17.29288477323'2250145 77480:890566 [78,54]78 [78,54]78 > 77323'22501452015-08-03 14:49:17.29253877323'2250145 2015-08-03 > 14:49:17.292538 > 2.8b9 40830 00 016669590823 30513051 active+clean+inconsistent2015-08-03 > 14:46:05.14006377323'2249886 77473:897325 [7,72]7 [7,72]7 > 77323'22498862015-08-03 14:22:47.83406377323'2249886 2015-08-03 > 14:22:47.834063 > > # Look at the first one: > ~# ceph pg deep-scrub 2.439 > instructing pg 2.439 on osd.78 to deep-scrub > > # The logs of osd.78 show: > 2015-08-03 15:16:34.409738 7f09ec04a700 0 log_channel(cluster) log [INF] : > 2.439 deep-scrub starts > 2015-08-03 15:16:51.364229 7f09ec04a700 -1 log_channel(cluster) log [ERR] : > deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest > 0xb3d78a6e != 0xa3944ad0 > 2015-08-03 15:16:52.763977 7f09ec04a700 -1 log_channel(cluster) log [ERR] : > 2.439 deep-scrub 1 errors > > # Finding the object in question: > ~# find ~ceph/osd/ceph-78/current/2.439_head -name 10000022d93.00000f0c* -ls > 21510412310 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 > /var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/10000022d93.00000f0c__head_B029E439__2 > ~# md5sum > /var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/10000022d93.00000f0c__head_B029E439__2 > 4e4523244deec051cfe53dd48489a5db > /var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/10000022d93.00000f0c__head_B029E439__2 > > # The object on the backup osd: > ~# find ~ceph/osd/ceph-54/current/2.439_head -name 10000022d93.00000f0c* -ls > 6442614367 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 > /var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/10000022d93.00000f0c__head_B029E439__2 > ~# md5sum > /var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/10000022d93.00000f0c__head_B029E439__2 > 4e4523244deec051cfe53dd48489a5db > /var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/10000022d93.00000f0c__head_B029E439__2 > > # They don't seem to be different. > # When I try repair: > ~# ceph pg repair 2.439 > instructing pg 2.439 on osd.78 to repair > > # The osd.78 logs show: > 2015-08-03 15:19:21.775933 7f09ec04a700 0 log_channel(cluster) log [INF] : > 2.439 repair starts > 2015-08-03 15:19:38.088673 7f09ec04a700 -1 log_channel(cluster) log [ERR] : > repair 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest > 0xb3d78a6e != 0xa3944ad0 > 2015-08-03 15:19:39.958019 7f09ec04a700 -1 log_channel(cluster) log [ERR] : > 2.439 repair 1 errors, 0 fixed > 2015-08-03 15:19:39.962406 7f09ec04a700 0 log_channel(cluster) log [INF] : > 2.439 deep-scrub starts > 2015-08-03 15:19:56.510874 7f09ec04a700 -1 log_channel(cluster) log [ERR] : > deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest > 0xb3d78a6e != 0xa3944ad0 > 2015-08-03 15:19:58.348083 7f09ec04a700 -1 log_channel(cluster) log [ERR] : > 2.439 deep-scrub 1 errors > > The inconsistency is not fixed. Any hints of what should be done next? > I have tried a few things: > * Stop the primary osd, remove the object from the filesystem, restart the > OSD and issue a repair. It didn't work - it sais that one object is > missing, but did not copy it from the backup. > * I tried the same on the backup (remove the file) - it also didn't get > copied back from the primary in a repair. > > Any help would be appreciated. > > Thanks, > > Andras > apataki@simonsfoundation.org > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Inconsistent PGs that ceph pg repair does not fix 2015-08-03 19:38 ` [ceph-users] Inconsistent PGs that ceph pg repair does not fix Samuel Just @ 2015-08-03 20:09 ` Andras Pataki [not found] ` <D1E543BF.8DD2%apataki-0QEYAsm1mgjsfHDXvbKv3WD2FQJk+8+b@public.gmane.org> 0 siblings, 1 reply; 6+ messages in thread From: Andras Pataki @ 2015-08-03 20:09 UTC (permalink / raw) To: Samuel Just Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Done: http://tracker.ceph.com/issues/12577 BTW, I¹m using the latest release 0.94.2 on all machines. Andras On 8/3/15, 3:38 PM, "Samuel Just" <sjust-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote: >Hrm, that's certainly supposed to work. Can you make a bug? Be sure >to note what version you are running (output of ceph-osd -v). >-Sam > >On Mon, Aug 3, 2015 at 12:34 PM, Andras Pataki ><apataki-0QEYAsm1mgjsfHDXvbKv3WD2FQJk+8+b@public.gmane.org> wrote: >> Summary: I am having problems with inconsistent PG's that the 'ceph pg >> repair' command does not fix. Below are the details. Any help would be >> appreciated. >> >> # Find the inconsistent PG's >> ~# ceph pg dump | grep inconsistent >> dumped all in format plain >> 2.439 42080 00 017279507143 31033103 active+clean+inconsistent2015-08-03 >> 14:49:17.29288477323'2250145 77480:890566 [78,54]78 [78,54]78 >> 77323'22501452015-08-03 14:49:17.29253877323'2250145 2015-08-03 >> 14:49:17.292538 >> 2.8b9 40830 00 016669590823 30513051 active+clean+inconsistent2015-08-03 >> 14:46:05.14006377323'2249886 77473:897325 [7,72]7 [7,72]7 >> 77323'22498862015-08-03 14:22:47.83406377323'2249886 2015-08-03 >> 14:22:47.834063 >> >> # Look at the first one: >> ~# ceph pg deep-scrub 2.439 >> instructing pg 2.439 on osd.78 to deep-scrub >> >> # The logs of osd.78 show: >> 2015-08-03 15:16:34.409738 7f09ec04a700 0 log_channel(cluster) log >>[INF] : >> 2.439 deep-scrub starts >> 2015-08-03 15:16:51.364229 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data >>digest >> 0xb3d78a6e != 0xa3944ad0 >> 2015-08-03 15:16:52.763977 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> 2.439 deep-scrub 1 errors >> >> # Finding the object in question: >> ~# find ~ceph/osd/ceph-78/current/2.439_head -name >>10000022d93.00000f0c* -ls >> 21510412310 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 >> >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> ~# md5sum >> >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> 4e4523244deec051cfe53dd48489a5db >> >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> >> # The object on the backup osd: >> ~# find ~ceph/osd/ceph-54/current/2.439_head -name >>10000022d93.00000f0c* -ls >> 6442614367 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 >> >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> ~# md5sum >> >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> 4e4523244deec051cfe53dd48489a5db >> >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> >> # They don't seem to be different. >> # When I try repair: >> ~# ceph pg repair 2.439 >> instructing pg 2.439 on osd.78 to repair >> >> # The osd.78 logs show: >> 2015-08-03 15:19:21.775933 7f09ec04a700 0 log_channel(cluster) log >>[INF] : >> 2.439 repair starts >> 2015-08-03 15:19:38.088673 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> repair 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest >> 0xb3d78a6e != 0xa3944ad0 >> 2015-08-03 15:19:39.958019 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> 2.439 repair 1 errors, 0 fixed >> 2015-08-03 15:19:39.962406 7f09ec04a700 0 log_channel(cluster) log >>[INF] : >> 2.439 deep-scrub starts >> 2015-08-03 15:19:56.510874 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data >>digest >> 0xb3d78a6e != 0xa3944ad0 >> 2015-08-03 15:19:58.348083 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> 2.439 deep-scrub 1 errors >> >> The inconsistency is not fixed. Any hints of what should be done next? >> I have tried a few things: >> * Stop the primary osd, remove the object from the filesystem, restart >>the >> OSD and issue a repair. It didn't work - it sais that one object is >> missing, but did not copy it from the backup. >> * I tried the same on the backup (remove the file) - it also didn't get >> copied back from the primary in a repair. >> >> Any help would be appreciated. >> >> Thanks, >> >> Andras >> apataki-0QEYAsm1mgjsfHDXvbKv3WD2FQJk+8+b@public.gmane.org >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ^ permalink raw reply [flat|nested] 6+ messages in thread
[parent not found: <D1E543BF.8DD2%apataki-0QEYAsm1mgjsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>]
* Re: Inconsistent PGs that ceph pg repair does not fix [not found] ` <D1E543BF.8DD2%apataki-0QEYAsm1mgjsfHDXvbKv3WD2FQJk+8+b@public.gmane.org> @ 2015-09-08 17:50 ` Andras Pataki 2015-09-08 18:07 ` [ceph-users] " Sage Weil 0 siblings, 1 reply; 6+ messages in thread From: Andras Pataki @ 2015-09-08 17:50 UTC (permalink / raw) To: Samuel Just Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org, ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Hi Sam, I saw that ceph 0.94.3 is out and it contains a resolution to the issue below (http://tracker.ceph.com/issues/12577). I installed it on our cluster, but unfortunately it didn't resolve the issue. Same as before, I have a couple of inconsistent pg's, and run ceph pg repair on them - the OSD says: 2015-09-08 11:21:53.930324 7f49c17ea700 0 log_channel(cluster) log [INF] : 2.439 repair starts 2015-09-08 11:27:57.708394 7f49c17ea700 -1 log_channel(cluster) log [ERR] : repair 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest 0xb3d78a6e != 0xa3944ad0 2015-09-08 11:28:32.359938 7f49c17ea700 -1 log_channel(cluster) log [ERR] : 2.439 repair 1 errors, 0 fixed 2015-09-08 11:28:32.364506 7f49c17ea700 0 log_channel(cluster) log [INF] : 2.439 deep-scrub starts 2015-09-08 11:29:18.650876 7f49c17ea700 -1 log_channel(cluster) log [ERR] : deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest 0xb3d78a6e != 0xa3944ad0 2015-09-08 11:29:23.136109 7f49c17ea700 -1 log_channel(cluster) log [ERR] : 2.439 deep-scrub 1 errors $ ceph tell osd.* version | grep version | sort | uniq -c 94 "version": "ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)" Could you have another look? Thanks, Andras ________________________________________ From: Andras Pataki Sent: Monday, August 3, 2015 4:09 PM To: Samuel Just Cc: ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org; ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: Re: [ceph-users] Inconsistent PGs that ceph pg repair does not fix Done: http://tracker.ceph.com/issues/12577 BTW, I¹m using the latest release 0.94.2 on all machines. Andras On 8/3/15, 3:38 PM, "Samuel Just" <sjust-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote: >Hrm, that's certainly supposed to work. Can you make a bug? Be sure >to note what version you are running (output of ceph-osd -v). >-Sam > >On Mon, Aug 3, 2015 at 12:34 PM, Andras Pataki ><apataki-0QEYAsm1mgjsfHDXvbKv3WD2FQJk+8+b@public.gmane.org> wrote: >> Summary: I am having problems with inconsistent PG's that the 'ceph pg >> repair' command does not fix. Below are the details. Any help would be >> appreciated. >> >> # Find the inconsistent PG's >> ~# ceph pg dump | grep inconsistent >> dumped all in format plain >> 2.439 42080 00 017279507143 31033103 active+clean+inconsistent2015-08-03 >> 14:49:17.29288477323'2250145 77480:890566 [78,54]78 [78,54]78 >> 77323'22501452015-08-03 14:49:17.29253877323'2250145 2015-08-03 >> 14:49:17.292538 >> 2.8b9 40830 00 016669590823 30513051 active+clean+inconsistent2015-08-03 >> 14:46:05.14006377323'2249886 77473:897325 [7,72]7 [7,72]7 >> 77323'22498862015-08-03 14:22:47.83406377323'2249886 2015-08-03 >> 14:22:47.834063 >> >> # Look at the first one: >> ~# ceph pg deep-scrub 2.439 >> instructing pg 2.439 on osd.78 to deep-scrub >> >> # The logs of osd.78 show: >> 2015-08-03 15:16:34.409738 7f09ec04a700 0 log_channel(cluster) log >>[INF] : >> 2.439 deep-scrub starts >> 2015-08-03 15:16:51.364229 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data >>digest >> 0xb3d78a6e != 0xa3944ad0 >> 2015-08-03 15:16:52.763977 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> 2.439 deep-scrub 1 errors >> >> # Finding the object in question: >> ~# find ~ceph/osd/ceph-78/current/2.439_head -name >>10000022d93.00000f0c* -ls >> 21510412310 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 >> >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> ~# md5sum >> >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> 4e4523244deec051cfe53dd48489a5db >> >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> >> # The object on the backup osd: >> ~# find ~ceph/osd/ceph-54/current/2.439_head -name >>10000022d93.00000f0c* -ls >> 6442614367 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 >> >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> ~# md5sum >> >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> 4e4523244deec051cfe53dd48489a5db >> >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 >>0022d93.00000f0c__head_B029E439__2 >> >> # They don't seem to be different. >> # When I try repair: >> ~# ceph pg repair 2.439 >> instructing pg 2.439 on osd.78 to repair >> >> # The osd.78 logs show: >> 2015-08-03 15:19:21.775933 7f09ec04a700 0 log_channel(cluster) log >>[INF] : >> 2.439 repair starts >> 2015-08-03 15:19:38.088673 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> repair 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest >> 0xb3d78a6e != 0xa3944ad0 >> 2015-08-03 15:19:39.958019 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> 2.439 repair 1 errors, 0 fixed >> 2015-08-03 15:19:39.962406 7f09ec04a700 0 log_channel(cluster) log >>[INF] : >> 2.439 deep-scrub starts >> 2015-08-03 15:19:56.510874 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data >>digest >> 0xb3d78a6e != 0xa3944ad0 >> 2015-08-03 15:19:58.348083 7f09ec04a700 -1 log_channel(cluster) log >>[ERR] : >> 2.439 deep-scrub 1 errors >> >> The inconsistency is not fixed. Any hints of what should be done next? >> I have tried a few things: >> * Stop the primary osd, remove the object from the filesystem, restart >>the >> OSD and issue a repair. It didn't work - it sais that one object is >> missing, but did not copy it from the backup. >> * I tried the same on the backup (remove the file) - it also didn't get >> copied back from the primary in a repair. >> >> Any help would be appreciated. >> >> Thanks, >> >> Andras >> apataki-0QEYAsm1mgjsfHDXvbKv3WD2FQJk+8+b@public.gmane.org >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [ceph-users] Inconsistent PGs that ceph pg repair does not fix 2015-09-08 17:50 ` Andras Pataki @ 2015-09-08 18:07 ` Sage Weil 2015-09-08 18:17 ` Andras Pataki 2015-09-08 21:42 ` Shinobu Kinjo 0 siblings, 2 replies; 6+ messages in thread From: Sage Weil @ 2015-09-08 18:07 UTC (permalink / raw) To: Andras Pataki Cc: Samuel Just, ceph-users@lists.ceph.com, ceph-devel@vger.kernel.org [-- Attachment #1: Type: TEXT/PLAIN, Size: 7014 bytes --] On Tue, 8 Sep 2015, Andras Pataki wrote: > Hi Sam, > > I saw that ceph 0.94.3 is out and it contains a resolution to the issue below (http://tracker.ceph.com/issues/12577). I installed it on our cluster, but unfortunately it didn't resolve the issue. Same as before, I have a couple of inconsistent pg's, and run ceph pg repair on them - the OSD says: > > 2015-09-08 11:21:53.930324 7f49c17ea700 0 log_channel(cluster) log [INF] : 2.439 repair starts > 2015-09-08 11:27:57.708394 7f49c17ea700 -1 log_channel(cluster) log [ERR] : repair 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest 0xb3d78a6e != 0xa3944ad0 > 2015-09-08 11:28:32.359938 7f49c17ea700 -1 log_channel(cluster) log [ERR] : 2.439 repair 1 errors, 0 fixed > 2015-09-08 11:28:32.364506 7f49c17ea700 0 log_channel(cluster) log [INF] : 2.439 deep-scrub starts > 2015-09-08 11:29:18.650876 7f49c17ea700 -1 log_channel(cluster) log [ERR] : deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest 0xb3d78a6e != 0xa3944ad0 > 2015-09-08 11:29:23.136109 7f49c17ea700 -1 log_channel(cluster) log [ERR] : 2.439 deep-scrub 1 errors > > $ ceph tell osd.* version | grep version | sort | uniq -c > 94 "version": "ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)" > > Could you have another look? The fix was merged into master in 6a949e10198a1787f2008b6c537b7060d191d236, after v0.94.3 was released. It will be in v0.94.4. Note that we had a bunch of similar errors on our internal lab cluster and this resolved them. We installed the test build from gitbuilder, available at http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/hammer/ (or similar, adjust URL for your distro). sage > > Thanks, > > Andras > > > ________________________________________ > From: Andras Pataki > Sent: Monday, August 3, 2015 4:09 PM > To: Samuel Just > Cc: ceph-users@lists.ceph.com; ceph-devel@vger.kernel.org > Subject: Re: [ceph-users] Inconsistent PGs that ceph pg repair does not fix > > Done: http://tracker.ceph.com/issues/12577 > BTW, I¹m using the latest release 0.94.2 on all machines. > > Andras > > > On 8/3/15, 3:38 PM, "Samuel Just" <sjust@redhat.com> wrote: > > >Hrm, that's certainly supposed to work. Can you make a bug? Be sure > >to note what version you are running (output of ceph-osd -v). > >-Sam > > > >On Mon, Aug 3, 2015 at 12:34 PM, Andras Pataki > ><apataki@simonsfoundation.org> wrote: > >> Summary: I am having problems with inconsistent PG's that the 'ceph pg > >> repair' command does not fix. Below are the details. Any help would be > >> appreciated. > >> > >> # Find the inconsistent PG's > >> ~# ceph pg dump | grep inconsistent > >> dumped all in format plain > >> 2.439 42080 00 017279507143 31033103 active+clean+inconsistent2015-08-03 > >> 14:49:17.29288477323'2250145 77480:890566 [78,54]78 [78,54]78 > >> 77323'22501452015-08-03 14:49:17.29253877323'2250145 2015-08-03 > >> 14:49:17.292538 > >> 2.8b9 40830 00 016669590823 30513051 active+clean+inconsistent2015-08-03 > >> 14:46:05.14006377323'2249886 77473:897325 [7,72]7 [7,72]7 > >> 77323'22498862015-08-03 14:22:47.83406377323'2249886 2015-08-03 > >> 14:22:47.834063 > >> > >> # Look at the first one: > >> ~# ceph pg deep-scrub 2.439 > >> instructing pg 2.439 on osd.78 to deep-scrub > >> > >> # The logs of osd.78 show: > >> 2015-08-03 15:16:34.409738 7f09ec04a700 0 log_channel(cluster) log > >>[INF] : > >> 2.439 deep-scrub starts > >> 2015-08-03 15:16:51.364229 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data > >>digest > >> 0xb3d78a6e != 0xa3944ad0 > >> 2015-08-03 15:16:52.763977 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> 2.439 deep-scrub 1 errors > >> > >> # Finding the object in question: > >> ~# find ~ceph/osd/ceph-78/current/2.439_head -name > >>10000022d93.00000f0c* -ls > >> 21510412310 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 > >> > >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> ~# md5sum > >> > >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> 4e4523244deec051cfe53dd48489a5db > >> > >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> > >> # The object on the backup osd: > >> ~# find ~ceph/osd/ceph-54/current/2.439_head -name > >>10000022d93.00000f0c* -ls > >> 6442614367 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 > >> > >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> ~# md5sum > >> > >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> 4e4523244deec051cfe53dd48489a5db > >> > >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> > >> # They don't seem to be different. > >> # When I try repair: > >> ~# ceph pg repair 2.439 > >> instructing pg 2.439 on osd.78 to repair > >> > >> # The osd.78 logs show: > >> 2015-08-03 15:19:21.775933 7f09ec04a700 0 log_channel(cluster) log > >>[INF] : > >> 2.439 repair starts > >> 2015-08-03 15:19:38.088673 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> repair 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest > >> 0xb3d78a6e != 0xa3944ad0 > >> 2015-08-03 15:19:39.958019 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> 2.439 repair 1 errors, 0 fixed > >> 2015-08-03 15:19:39.962406 7f09ec04a700 0 log_channel(cluster) log > >>[INF] : > >> 2.439 deep-scrub starts > >> 2015-08-03 15:19:56.510874 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data > >>digest > >> 0xb3d78a6e != 0xa3944ad0 > >> 2015-08-03 15:19:58.348083 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> 2.439 deep-scrub 1 errors > >> > >> The inconsistency is not fixed. Any hints of what should be done next? > >> I have tried a few things: > >> * Stop the primary osd, remove the object from the filesystem, restart > >>the > >> OSD and issue a repair. It didn't work - it sais that one object is > >> missing, but did not copy it from the backup. > >> * I tried the same on the backup (remove the file) - it also didn't get > >> copied back from the primary in a repair. > >> > >> Any help would be appreciated. > >> > >> Thanks, > >> > >> Andras > >> apataki@simonsfoundation.org > >> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [ceph-users] Inconsistent PGs that ceph pg repair does not fix 2015-09-08 18:07 ` [ceph-users] " Sage Weil @ 2015-09-08 18:17 ` Andras Pataki 2015-09-08 21:42 ` Shinobu Kinjo 1 sibling, 0 replies; 6+ messages in thread From: Andras Pataki @ 2015-09-08 18:17 UTC (permalink / raw) To: Sage Weil Cc: Samuel Just, ceph-users@lists.ceph.com, ceph-devel@vger.kernel.org Cool, thanks! Andras ________________________________________ From: Sage Weil <sweil@redhat.com> Sent: Tuesday, September 8, 2015 2:07 PM To: Andras Pataki Cc: Samuel Just; ceph-users@lists.ceph.com; ceph-devel@vger.kernel.org Subject: Re: [ceph-users] Inconsistent PGs that ceph pg repair does not fix On Tue, 8 Sep 2015, Andras Pataki wrote: > Hi Sam, > > I saw that ceph 0.94.3 is out and it contains a resolution to the issue below (http://tracker.ceph.com/issues/12577). I installed it on our cluster, but unfortunately it didn't resolve the issue. Same as before, I have a couple of inconsistent pg's, and run ceph pg repair on them - the OSD says: > > 2015-09-08 11:21:53.930324 7f49c17ea700 0 log_channel(cluster) log [INF] : 2.439 repair starts > 2015-09-08 11:27:57.708394 7f49c17ea700 -1 log_channel(cluster) log [ERR] : repair 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest 0xb3d78a6e != 0xa3944ad0 > 2015-09-08 11:28:32.359938 7f49c17ea700 -1 log_channel(cluster) log [ERR] : 2.439 repair 1 errors, 0 fixed > 2015-09-08 11:28:32.364506 7f49c17ea700 0 log_channel(cluster) log [INF] : 2.439 deep-scrub starts > 2015-09-08 11:29:18.650876 7f49c17ea700 -1 log_channel(cluster) log [ERR] : deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest 0xb3d78a6e != 0xa3944ad0 > 2015-09-08 11:29:23.136109 7f49c17ea700 -1 log_channel(cluster) log [ERR] : 2.439 deep-scrub 1 errors > > $ ceph tell osd.* version | grep version | sort | uniq -c > 94 "version": "ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)" > > Could you have another look? The fix was merged into master in 6a949e10198a1787f2008b6c537b7060d191d236, after v0.94.3 was released. It will be in v0.94.4. Note that we had a bunch of similar errors on our internal lab cluster and this resolved them. We installed the test build from gitbuilder, available at http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/hammer/ (or similar, adjust URL for your distro). sage > > Thanks, > > Andras > > > ________________________________________ > From: Andras Pataki > Sent: Monday, August 3, 2015 4:09 PM > To: Samuel Just > Cc: ceph-users@lists.ceph.com; ceph-devel@vger.kernel.org > Subject: Re: [ceph-users] Inconsistent PGs that ceph pg repair does not fix > > Done: http://tracker.ceph.com/issues/12577 > BTW, I¹m using the latest release 0.94.2 on all machines. > > Andras > > > On 8/3/15, 3:38 PM, "Samuel Just" <sjust@redhat.com> wrote: > > >Hrm, that's certainly supposed to work. Can you make a bug? Be sure > >to note what version you are running (output of ceph-osd -v). > >-Sam > > > >On Mon, Aug 3, 2015 at 12:34 PM, Andras Pataki > ><apataki@simonsfoundation.org> wrote: > >> Summary: I am having problems with inconsistent PG's that the 'ceph pg > >> repair' command does not fix. Below are the details. Any help would be > >> appreciated. > >> > >> # Find the inconsistent PG's > >> ~# ceph pg dump | grep inconsistent > >> dumped all in format plain > >> 2.439 42080 00 017279507143 31033103 active+clean+inconsistent2015-08-03 > >> 14:49:17.29288477323'2250145 77480:890566 [78,54]78 [78,54]78 > >> 77323'22501452015-08-03 14:49:17.29253877323'2250145 2015-08-03 > >> 14:49:17.292538 > >> 2.8b9 40830 00 016669590823 30513051 active+clean+inconsistent2015-08-03 > >> 14:46:05.14006377323'2249886 77473:897325 [7,72]7 [7,72]7 > >> 77323'22498862015-08-03 14:22:47.83406377323'2249886 2015-08-03 > >> 14:22:47.834063 > >> > >> # Look at the first one: > >> ~# ceph pg deep-scrub 2.439 > >> instructing pg 2.439 on osd.78 to deep-scrub > >> > >> # The logs of osd.78 show: > >> 2015-08-03 15:16:34.409738 7f09ec04a700 0 log_channel(cluster) log > >>[INF] : > >> 2.439 deep-scrub starts > >> 2015-08-03 15:16:51.364229 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data > >>digest > >> 0xb3d78a6e != 0xa3944ad0 > >> 2015-08-03 15:16:52.763977 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> 2.439 deep-scrub 1 errors > >> > >> # Finding the object in question: > >> ~# find ~ceph/osd/ceph-78/current/2.439_head -name > >>10000022d93.00000f0c* -ls > >> 21510412310 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 > >> > >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> ~# md5sum > >> > >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> 4e4523244deec051cfe53dd48489a5db > >> > >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> > >> # The object on the backup osd: > >> ~# find ~ceph/osd/ceph-54/current/2.439_head -name > >>10000022d93.00000f0c* -ls > >> 6442614367 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 > >> > >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> ~# md5sum > >> > >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> 4e4523244deec051cfe53dd48489a5db > >> > >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> > >> # They don't seem to be different. > >> # When I try repair: > >> ~# ceph pg repair 2.439 > >> instructing pg 2.439 on osd.78 to repair > >> > >> # The osd.78 logs show: > >> 2015-08-03 15:19:21.775933 7f09ec04a700 0 log_channel(cluster) log > >>[INF] : > >> 2.439 repair starts > >> 2015-08-03 15:19:38.088673 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> repair 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest > >> 0xb3d78a6e != 0xa3944ad0 > >> 2015-08-03 15:19:39.958019 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> 2.439 repair 1 errors, 0 fixed > >> 2015-08-03 15:19:39.962406 7f09ec04a700 0 log_channel(cluster) log > >>[INF] : > >> 2.439 deep-scrub starts > >> 2015-08-03 15:19:56.510874 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data > >>digest > >> 0xb3d78a6e != 0xa3944ad0 > >> 2015-08-03 15:19:58.348083 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> 2.439 deep-scrub 1 errors > >> > >> The inconsistency is not fixed. Any hints of what should be done next? > >> I have tried a few things: > >> * Stop the primary osd, remove the object from the filesystem, restart > >>the > >> OSD and issue a repair. It didn't work - it sais that one object is > >> missing, but did not copy it from the backup. > >> * I tried the same on the backup (remove the file) - it also didn't get > >> copied back from the primary in a repair. > >> > >> Any help would be appreciated. > >> > >> Thanks, > >> > >> Andras > >> apataki@simonsfoundation.org > >> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [ceph-users] Inconsistent PGs that ceph pg repair does not fix 2015-09-08 18:07 ` [ceph-users] " Sage Weil 2015-09-08 18:17 ` Andras Pataki @ 2015-09-08 21:42 ` Shinobu Kinjo 1 sibling, 0 replies; 6+ messages in thread From: Shinobu Kinjo @ 2015-09-08 21:42 UTC (permalink / raw) To: Sage Weil; +Cc: Andras Pataki, ceph-users, ceph-devel That's a good news. Shinobu ----- Original Message ----- From: "Sage Weil" <sweil@redhat.com> To: "Andras Pataki" <apataki@simonsfoundation.org> Cc: ceph-users@lists.ceph.com, ceph-devel@vger.kernel.org Sent: Wednesday, September 9, 2015 3:07:29 AM Subject: Re: [ceph-users] Inconsistent PGs that ceph pg repair does not fix On Tue, 8 Sep 2015, Andras Pataki wrote: > Hi Sam, > > I saw that ceph 0.94.3 is out and it contains a resolution to the issue below (http://tracker.ceph.com/issues/12577). I installed it on our cluster, but unfortunately it didn't resolve the issue. Same as before, I have a couple of inconsistent pg's, and run ceph pg repair on them - the OSD says: > > 2015-09-08 11:21:53.930324 7f49c17ea700 0 log_channel(cluster) log [INF] : 2.439 repair starts > 2015-09-08 11:27:57.708394 7f49c17ea700 -1 log_channel(cluster) log [ERR] : repair 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest 0xb3d78a6e != 0xa3944ad0 > 2015-09-08 11:28:32.359938 7f49c17ea700 -1 log_channel(cluster) log [ERR] : 2.439 repair 1 errors, 0 fixed > 2015-09-08 11:28:32.364506 7f49c17ea700 0 log_channel(cluster) log [INF] : 2.439 deep-scrub starts > 2015-09-08 11:29:18.650876 7f49c17ea700 -1 log_channel(cluster) log [ERR] : deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest 0xb3d78a6e != 0xa3944ad0 > 2015-09-08 11:29:23.136109 7f49c17ea700 -1 log_channel(cluster) log [ERR] : 2.439 deep-scrub 1 errors > > $ ceph tell osd.* version | grep version | sort | uniq -c > 94 "version": "ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)" > > Could you have another look? The fix was merged into master in 6a949e10198a1787f2008b6c537b7060d191d236, after v0.94.3 was released. It will be in v0.94.4. Note that we had a bunch of similar errors on our internal lab cluster and this resolved them. We installed the test build from gitbuilder, available at http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/hammer/ (or similar, adjust URL for your distro). sage > > Thanks, > > Andras > > > ________________________________________ > From: Andras Pataki > Sent: Monday, August 3, 2015 4:09 PM > To: Samuel Just > Cc: ceph-users@lists.ceph.com; ceph-devel@vger.kernel.org > Subject: Re: [ceph-users] Inconsistent PGs that ceph pg repair does not fix > > Done: http://tracker.ceph.com/issues/12577 > BTW, I¹m using the latest release 0.94.2 on all machines. > > Andras > > > On 8/3/15, 3:38 PM, "Samuel Just" <sjust@redhat.com> wrote: > > >Hrm, that's certainly supposed to work. Can you make a bug? Be sure > >to note what version you are running (output of ceph-osd -v). > >-Sam > > > >On Mon, Aug 3, 2015 at 12:34 PM, Andras Pataki > ><apataki@simonsfoundation.org> wrote: > >> Summary: I am having problems with inconsistent PG's that the 'ceph pg > >> repair' command does not fix. Below are the details. Any help would be > >> appreciated. > >> > >> # Find the inconsistent PG's > >> ~# ceph pg dump | grep inconsistent > >> dumped all in format plain > >> 2.439 42080 00 017279507143 31033103 active+clean+inconsistent2015-08-03 > >> 14:49:17.29288477323'2250145 77480:890566 [78,54]78 [78,54]78 > >> 77323'22501452015-08-03 14:49:17.29253877323'2250145 2015-08-03 > >> 14:49:17.292538 > >> 2.8b9 40830 00 016669590823 30513051 active+clean+inconsistent2015-08-03 > >> 14:46:05.14006377323'2249886 77473:897325 [7,72]7 [7,72]7 > >> 77323'22498862015-08-03 14:22:47.83406377323'2249886 2015-08-03 > >> 14:22:47.834063 > >> > >> # Look at the first one: > >> ~# ceph pg deep-scrub 2.439 > >> instructing pg 2.439 on osd.78 to deep-scrub > >> > >> # The logs of osd.78 show: > >> 2015-08-03 15:16:34.409738 7f09ec04a700 0 log_channel(cluster) log > >>[INF] : > >> 2.439 deep-scrub starts > >> 2015-08-03 15:16:51.364229 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data > >>digest > >> 0xb3d78a6e != 0xa3944ad0 > >> 2015-08-03 15:16:52.763977 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> 2.439 deep-scrub 1 errors > >> > >> # Finding the object in question: > >> ~# find ~ceph/osd/ceph-78/current/2.439_head -name > >>10000022d93.00000f0c* -ls > >> 21510412310 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 > >> > >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> ~# md5sum > >> > >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> 4e4523244deec051cfe53dd48489a5db > >> > >>/var/lib/ceph/osd/ceph-78/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> > >> # The object on the backup osd: > >> ~# find ~ceph/osd/ceph-54/current/2.439_head -name > >>10000022d93.00000f0c* -ls > >> 6442614367 4100 -rw-r--r-- 1 root root 4194304 Jun 30 17:09 > >> > >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> ~# md5sum > >> > >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> 4e4523244deec051cfe53dd48489a5db > >> > >>/var/lib/ceph/osd/ceph-54/current/2.439_head/DIR_9/DIR_3/DIR_4/DIR_E/1000 > >>0022d93.00000f0c__head_B029E439__2 > >> > >> # They don't seem to be different. > >> # When I try repair: > >> ~# ceph pg repair 2.439 > >> instructing pg 2.439 on osd.78 to repair > >> > >> # The osd.78 logs show: > >> 2015-08-03 15:19:21.775933 7f09ec04a700 0 log_channel(cluster) log > >>[INF] : > >> 2.439 repair starts > >> 2015-08-03 15:19:38.088673 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> repair 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data digest > >> 0xb3d78a6e != 0xa3944ad0 > >> 2015-08-03 15:19:39.958019 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> 2.439 repair 1 errors, 0 fixed > >> 2015-08-03 15:19:39.962406 7f09ec04a700 0 log_channel(cluster) log > >>[INF] : > >> 2.439 deep-scrub starts > >> 2015-08-03 15:19:56.510874 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> deep-scrub 2.439 b029e439/10000022d93.00000f0c/head//2 on disk data > >>digest > >> 0xb3d78a6e != 0xa3944ad0 > >> 2015-08-03 15:19:58.348083 7f09ec04a700 -1 log_channel(cluster) log > >>[ERR] : > >> 2.439 deep-scrub 1 errors > >> > >> The inconsistency is not fixed. Any hints of what should be done next? > >> I have tried a few things: > >> * Stop the primary osd, remove the object from the filesystem, restart > >>the > >> OSD and issue a repair. It didn't work - it sais that one object is > >> missing, but did not copy it from the backup. > >> * I tried the same on the backup (remove the file) - it also didn't get > >> copied back from the primary in a repair. > >> > >> Any help would be appreciated. > >> > >> Thanks, > >> > >> Andras > >> apataki@simonsfoundation.org > >> > >> > >> _______________________________________________ > >> ceph-users mailing list > >> ceph-users@lists.ceph.com > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > >> > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-09-08 21:42 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <D1E53A36.8DB6%apataki@simonsfoundation.org>
[not found] ` <D1E53B6A.8DC6%apataki@simonsfoundation.org>
2015-08-03 19:38 ` [ceph-users] Inconsistent PGs that ceph pg repair does not fix Samuel Just
2015-08-03 20:09 ` Andras Pataki
[not found] ` <D1E543BF.8DD2%apataki-0QEYAsm1mgjsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>
2015-09-08 17:50 ` Andras Pataki
2015-09-08 18:07 ` [ceph-users] " Sage Weil
2015-09-08 18:17 ` Andras Pataki
2015-09-08 21:42 ` Shinobu Kinjo
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.