inconsistent chunk

inconsistent chunk

[Fyodor Ustinov]

Hi!

I have inconsistent chunk:

2011-06-17 08:11:33.844030    pg v26069: 2376 pgs: 2375 active+clean, 1 
active+clean+inconsistent; 285 GB data, 572 GB used, 24180 GB / 25494 GB 
avail

Questions:
1. What does this mean?
2. How to learn what kind of file is damaged?
3. Delete damaged file should fix it?
4. Can I recover the data without backup?


WBR,
     Fyodor.

Re: inconsistent chunk

[Gregory Farnum]

On Jun 16, 2011, at 10:20 PM, Fyodor Ustinov wrote:

> Hi!
> 
> I have inconsistent chunk:
> 
> 2011-06-17 08:11:33.844030    pg v26069: 2376 pgs: 2375 active+clean, 1 active+clean+inconsistent; 285 GB data, 572 GB used, 24180 GB / 25494 GB avail
> 
> Questions:
> 1. What does this mean?
This isn't actually an inconsistent chunk. It's an inconsistent "placement group", which is a logical grouping of RADOS objects which are moved between OSDs and share the same RADOS metadata. It being labeled inconsistent means that the background scrub has identified a difference in the PG's contents. Right now, I think that means a difference in the file contents, or in one of their sizes or modification times.

> 2. How to learn what kind of file is damaged?
That's not readily-available data, I'm afraid. There may be a way to find out which objects caused it to get flagged inconsistent (which would tell you the inode of the damaged files), Sam would be able to tell you more about this...

> 3. Delete damaged file should fix it?
You're unlikely to be able to delete things that are marked inconsistent, since the system wants to resolve the inconsistency before it does anything else with them.

> 4. Can I recover the data without backup?
Possibly. If the inconsistency is just that one of the OSDs is storing the object and another OSD says the object shouldn't exist, that can be recovered from by working out which one is correct.

Have you manually adjusted the contents of any OSDs? Can you think of anything you've done that might have triggered this?

Re: inconsistent chunk

[Fyodor Ustinov]

On 06/17/2011 07:27 PM, Gregory Farnum wrote:
> 4. Can I recover the data without backup?
> Possibly. If the inconsistency is just that one of the OSDs is storing the object and another OSD says the object shouldn't exist, that can be recovered from by working out which one is correct.
>
> Have you manually adjusted the contents of any OSDs? Can you think of anything you've done that might have triggered this?
  I step by step down osd servers until only one remains (hmm, it is not 
clear how this is consistent with "pg_size 2"). After one day I step by 
step up osd servers.

WBR,
     Fyodor.

Re: inconsistent chunk

[Gregory Farnum]

On Jun 18, 2011, at 8:27 AM, Fyodor Ustinov wrote:

> On 06/17/2011 07:27 PM, Gregory Farnum wrote:
>> 4. Can I recover the data without backup?
>> Possibly. If the inconsistency is just that one of the OSDs is storing the object and another OSD says the object shouldn't exist, that can be recovered from by working out which one is correct.
>> 
>> Have you manually adjusted the contents of any OSDs? Can you think of anything you've done that might have triggered this?
> I step by step down osd servers until only one remains (hmm, it is not clear how this is consistent with "pg_size 2"). After one day I step by step up osd servers.

Well if you've got enough space and step down slowly enough all the data will simply exist on the one up OSD in a degraded form. :)

Is it still marked as inconsistent? If that's really all you did it shouldn't be, but maybe restarting the cluster would let it fix itself. :/
-Greg

Re: inconsistent chunk

[Fyodor Ustinov]

On 06/20/2011 08:16 PM, Gregory Farnum wrote:
>>> 4. Can I recover the data without backup?
>>> Possibly. If the inconsistency is just that one of the OSDs is storing the object and another OSD says the object shouldn't exist, that can be recovered from by working out which one is correct.
>>>
>>> Have you manually adjusted the contents of any OSDs? Can you think of anything you've done that might have triggered this?
>> I step by step down osd servers until only one remains (hmm, it is not clear how this is consistent with "pg_size 2"). After one day I step by step up osd servers.
> Well if you've got enough space and step down slowly enough all the data will simply exist on the one up OSD in a degraded form. :)
>
> Is it still marked as inconsistent? If that's really all you did it shouldn't be, but maybe restarting the cluster would let it fix itself. :/
> -Greg

Yes, I restarted osd server with inconsistent pg and cluster now in 
consistent state.

WBR,
     Fyodor.