Pg stuck stale...why?

All of lore.kernel.org
 help / color / mirror / Atom feed

* Pg stuck stale...why?
@ 2012-07-11  1:11 Mark Kirkwood
  2012-07-11  1:22 ` Josh Durgin
  0 siblings, 1 reply; 7+ messages in thread
From: Mark Kirkwood @ 2012-07-11  1:11 UTC (permalink / raw)
  To: ceph-devel

I am seeing this:

# ceph -s
    health HEALTH_WARN 256 pgs stale; 256 pgs stuck stale
    monmap e1: 3 mons at 
{ved1=192.168.122.11:6789/0,ved2=192.168.122.12:6789/0,ved3=192.168.122.13:6789/0}, 
election epoch 18, quorum 0,1,2 ved1,ved2,ved3
    osdmap e62: 4 osds: 4 up, 4 in
     pgmap v47148: 768 pgs: 512 active+clean, 256 stale+active+clean; 
2224 MB data, 15442 MB used, 86907 MB / 102350 MB avail
    mdsmap e1: 0/0/1

In particular 256 pgs stuck stale - I've tried a) waiting a while 
(overnight),  b) a rolling restart of all 4 osd's,  c) restarting all 
ceph services on all 4 nodes. All without changing this.

As far as I understand what stuck state means, I can't see why they need 
to stay that way, given all osd's and mon's are up. (I have no mds 
configured)....any ideas? Or is this just expected?

Regards

Mark

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Pg stuck stale...why?
  2012-07-11  1:11 Pg stuck stale...why? Mark Kirkwood
@ 2012-07-11  1:22 ` Josh Durgin
  2012-07-11  1:32   ` Mark Kirkwood
  0 siblings, 1 reply; 7+ messages in thread
From: Josh Durgin @ 2012-07-11  1:22 UTC (permalink / raw)
  To: Mark Kirkwood; +Cc: ceph-devel

On 07/10/2012 06:11 PM, Mark Kirkwood wrote:
> I am seeing this:
>
> # ceph -s
> health HEALTH_WARN 256 pgs stale; 256 pgs stuck stale
> monmap e1: 3 mons at
> {ved1=192.168.122.11:6789/0,ved2=192.168.122.12:6789/0,ved3=192.168.122.13:6789/0},
> election epoch 18, quorum 0,1,2 ved1,ved2,ved3
> osdmap e62: 4 osds: 4 up, 4 in
> pgmap v47148: 768 pgs: 512 active+clean, 256 stale+active+clean; 2224 MB
> data, 15442 MB used, 86907 MB / 102350 MB avail
> mdsmap e1: 0/0/1
>
> In particular 256 pgs stuck stale - I've tried a) waiting a while
> (overnight), b) a rolling restart of all 4 osd's, c) restarting all ceph
> services on all 4 nodes. All without changing this.
>
> As far as I understand what stuck state means, I can't see why they need
> to stay that way, given all osd's and mon's are up. (I have no mds
> configured)....any ideas? Or is this just expected?
>
> Regards
>
> Mark

What does 'ceph pg dump_stuck stale' show? Stale means that the
monitors haven't gotten updates about those pgs from the osds within
the a certain period of time (default is 300 seconds), so something may
be wrong with your crushmap or those pgs themselves.

Josh

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Pg stuck stale...why?
  2012-07-11  1:22 ` Josh Durgin
@ 2012-07-11  1:32   ` Mark Kirkwood
  2012-07-11  1:35     ` Mark Kirkwood
  2012-07-11  1:55     ` Josh Durgin
  0 siblings, 2 replies; 7+ messages in thread
From: Mark Kirkwood @ 2012-07-11  1:32 UTC (permalink / raw)
  To: Josh Durgin; +Cc: ceph-devel

[-- Attachment #1: Type: text/plain, Size: 1424 bytes --]

On 11/07/12 13:22, Josh Durgin wrote:
> On 07/10/2012 06:11 PM, Mark Kirkwood wrote:
>> I am seeing this:
>>
>> # ceph -s
>> health HEALTH_WARN 256 pgs stale; 256 pgs stuck stale
>> monmap e1: 3 mons at
>> {ved1=192.168.122.11:6789/0,ved2=192.168.122.12:6789/0,ved3=192.168.122.13:6789/0}, 
>>
>> election epoch 18, quorum 0,1,2 ved1,ved2,ved3
>> osdmap e62: 4 osds: 4 up, 4 in
>> pgmap v47148: 768 pgs: 512 active+clean, 256 stale+active+clean; 2224 MB
>> data, 15442 MB used, 86907 MB / 102350 MB avail
>> mdsmap e1: 0/0/1
>>
>> In particular 256 pgs stuck stale - I've tried a) waiting a while
>> (overnight), b) a rolling restart of all 4 osd's, c) restarting all ceph
>> services on all 4 nodes. All without changing this.
>>
>> As far as I understand what stuck state means, I can't see why they need
>> to stay that way, given all osd's and mon's are up. (I have no mds
>> configured)....any ideas? Or is this just expected?
>>
>> Regards
>>
>> Mark
>
> What does 'ceph pg dump_stuck stale' show? Stale means that the
> monitors haven't gotten updates about those pgs from the osds within
> the a certain period of time (default is 300 seconds), so something may
> be wrong with your crushmap or those pgs themselves.
>
> Josh

I have attached the dump of stuck stale pgs, and the crushmap in use.

I did wonder if it is to do with not using any mds - i.e could this mean 
the meta data pgs never get touched?

Mark


[-- Attachment #2: pg-stale.dump --]
[-- Type: text/plain, Size: 27489 bytes --]

ok
pg_stat	objects	mip	degr	unf	bytes	log	disklog	state	state_stamp	v	reported	up	acting	last_scrub	scrub_stamp
0.c	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.398997	0'0	5'13	[1,0,2,3]	[1,0,2,3]	0'0	0.000000
0.b	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.547700	0'0	4'13	[3,2,0,1]	[3,2,0,1]	0'0	0.000000
0.a	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.548638	0'0	4'13	[3,2,1,0]	[3,2,1,0]	0'0	0.000000
0.9	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.399058	0'0	5'13	[1,3,2,0]	[1,3,2,0]	0'0	0.000000
0.8	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.549203	0'0	4'13	[3,0,1,2]	[3,0,1,2]	0'0	0.000000
0.7	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.225656	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.6	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.549348	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.5	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.399093	0'0	5'13	[1,2,3,0]	[1,2,3,0]	0'0	0.000000
0.4	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.550016	0'0	4'13	[3,2,0,1]	[3,2,0,1]	0'0	0.000000
0.3	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.550764	0'0	4'13	[3,2,0,1]	[3,2,0,1]	0'0	0.000000
0.2	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.399135	0'0	5'13	[1,3,2,0]	[1,3,2,0]	0'0	0.000000
0.1	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.551391	0'0	4'13	[3,2,0,1]	[3,2,0,1]	0'0	0.000000
0.0	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.377207	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.ff	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.377591	0'0	3'13	[2,0,3,1]	[2,0,3,1]	0'0	0.000000
0.fe	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.548875	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.fd	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.549702	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000
0.fc	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.399168	0'0	5'13	[1,2,0,3]	[1,2,0,3]	0'0	0.000000
0.fb	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.377897	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.fa	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.378378	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.f9	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.399204	0'0	5'13	[1,3,2,0]	[1,3,2,0]	0'0	0.000000
0.f8	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.551621	0'0	4'13	[3,1,0,2]	[3,1,0,2]	0'0	0.000000
0.f7	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.399237	0'0	5'13	[1,3,2,0]	[1,3,2,0]	0'0	0.000000
0.f6	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.551991	0'0	4'13	[3,0,2,1]	[3,0,2,1]	0'0	0.000000
0.f5	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.399273	0'0	5'13	[1,2,0,3]	[1,2,0,3]	0'0	0.000000
0.f4	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.399305	0'0	5'13	[1,2,3,0]	[1,2,3,0]	0'0	0.000000
0.f3	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.550251	0'0	2'16	[0,3,2,1]	[0,3,2,1]	0'0	0.000000
0.f2	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.550929	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.f1	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.399338	0'0	5'13	[1,2,3,0]	[1,2,3,0]	0'0	0.000000
0.f0	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.551940	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.ef	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.378957	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.ee	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.379533	0'0	3'13	[2,0,3,1]	[2,0,3,1]	0'0	0.000000
0.ed	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.552712	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.ec	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.554542	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.eb	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.555242	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.ea	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.552494	0'0	4'13	[3,0,1,2]	[3,0,1,2]	0'0	0.000000
0.e9	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.399372	0'0	5'13	[1,3,0,2]	[1,3,0,2]	0'0	0.000000
0.e8	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.553346	0'0	4'13	[3,0,2,1]	[3,0,2,1]	0'0	0.000000
0.e7	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.555787	0'0	2'16	[0,1,2,3]	[0,1,2,3]	0'0	0.000000
0.e6	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.553848	0'0	4'13	[3,2,1,0]	[3,2,1,0]	0'0	0.000000
0.e5	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.379900	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.e4	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.556264	0'0	2'16	[0,3,2,1]	[0,3,2,1]	0'0	0.000000
0.e3	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.486976	0'0	5'13	[1,0,3,2]	[1,0,3,2]	0'0	0.000000
0.e2	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.556925	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.e1	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.554403	0'0	4'13	[3,0,1,2]	[3,0,1,2]	0'0	0.000000
0.e0	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.557482	0'0	2'16	[0,1,3,2]	[0,1,3,2]	0'0	0.000000
0.df	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.380084	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.de	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.555001	0'0	4'13	[3,1,0,2]	[3,1,0,2]	0'0	0.000000
0.dd	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.558353	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000
0.dc	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.559043	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.db	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.559800	0'0	2'16	[0,3,2,1]	[0,3,2,1]	0'0	0.000000
0.da	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.380490	0'0	3'13	[2,3,0,1]	[2,3,0,1]	0'0	0.000000
0.d9	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.380751	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.d8	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.487108	0'0	5'13	[1,2,0,3]	[1,2,0,3]	0'0	0.000000
0.d7	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.487218	0'0	5'13	[1,3,2,0]	[1,3,2,0]	0'0	0.000000
0.d6	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.487324	0'0	5'13	[1,2,0,3]	[1,2,0,3]	0'0	0.000000
0.d5	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.380805	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.d4	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.555555	0'0	4'13	[3,2,1,0]	[3,2,1,0]	0'0	0.000000
0.d3	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.560628	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.d2	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.556332	0'0	4'13	[3,0,1,2]	[3,0,1,2]	0'0	0.000000
0.d1	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.487434	0'0	5'13	[1,0,3,2]	[1,0,3,2]	0'0	0.000000
0.d0	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.487545	0'0	5'13	[1,0,2,3]	[1,0,2,3]	0'0	0.000000
0.cf	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.562376	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000
0.ce	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.488773	0'0	5'13	[1,2,3,0]	[1,2,3,0]	0'0	0.000000
0.cd	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.380842	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.cc	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.488970	0'0	5'13	[1,3,0,2]	[1,3,0,2]	0'0	0.000000
0.cb	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.380955	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.ca	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.563030	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000
0.c9	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.381041	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.c8	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.489080	0'0	5'13	[1,0,3,2]	[1,0,3,2]	0'0	0.000000
0.c7	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.381189	0'0	3'13	[2,0,3,1]	[2,0,3,1]	0'0	0.000000
0.c6	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.563908	0'0	2'16	[0,1,2,3]	[0,1,2,3]	0'0	0.000000
0.c5	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.564494	0'0	2'16	[0,1,2,3]	[0,1,2,3]	0'0	0.000000
0.c4	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.556848	0'0	4'13	[3,0,2,1]	[3,0,2,1]	0'0	0.000000
0.c3	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.565018	0'0	2'16	[0,1,3,2]	[0,1,3,2]	0'0	0.000000
0.c2	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.381294	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.c1	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.565542	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000
0.c0	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.566142	0'0	2'16	[0,3,2,1]	[0,3,2,1]	0'0	0.000000
0.bf	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.489177	0'0	5'13	[1,3,0,2]	[1,3,0,2]	0'0	0.000000
0.be	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.382223	0'0	3'13	[2,3,0,1]	[2,3,0,1]	0'0	0.000000
0.bd	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.489302	0'0	5'13	[1,2,3,0]	[1,2,3,0]	0'0	0.000000
0.bc	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.382601	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.bb	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.557404	0'0	4'13	[3,2,0,1]	[3,2,0,1]	0'0	0.000000
0.ba	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.489410	0'0	5'13	[1,0,2,3]	[1,0,2,3]	0'0	0.000000
0.b9	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.557867	0'0	4'13	[3,1,0,2]	[3,1,0,2]	0'0	0.000000
0.b8	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.566749	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.b7	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.558295	0'0	4'13	[3,1,0,2]	[3,1,0,2]	0'0	0.000000
0.b6	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.489512	0'0	5'13	[1,0,2,3]	[1,0,2,3]	0'0	0.000000
0.b5	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.382939	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.b4	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.567778	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000
0.b3	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.568995	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000
0.b2	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.489720	0'0	5'13	[1,3,0,2]	[1,3,0,2]	0'0	0.000000
0.b1	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.569605	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.b0	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.383154	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.af	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.383421	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.ae	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.558791	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.ad	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.571039	0'0	2'16	[0,3,2,1]	[0,3,2,1]	0'0	0.000000
0.ac	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.383737	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.ab	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.383994	0'0	3'13	[2,0,3,1]	[2,0,3,1]	0'0	0.000000
0.aa	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.505084	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.a9	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.489833	0'0	5'13	[1,3,2,0]	[1,3,2,0]	0'0	0.000000
0.a8	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.505207	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.a7	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.489951	0'0	5'13	[1,2,0,3]	[1,2,0,3]	0'0	0.000000
0.a6	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.505312	0'0	3'13	[2,0,3,1]	[2,0,3,1]	0'0	0.000000
0.a5	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.559997	0'0	4'13	[3,1,0,2]	[3,1,0,2]	0'0	0.000000
0.a4	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.505421	0'0	3'13	[2,0,3,1]	[2,0,3,1]	0'0	0.000000
0.a3	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.505524	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.a2	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.560282	0'0	4'13	[3,1,0,2]	[3,1,0,2]	0'0	0.000000
0.a1	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.560511	0'0	4'13	[3,0,1,2]	[3,0,1,2]	0'0	0.000000
0.a0	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.505635	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.9f	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.490060	0'0	5'13	[1,3,0,2]	[1,3,0,2]	0'0	0.000000
0.9e	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.560679	0'0	4'13	[3,2,1,0]	[3,2,1,0]	0'0	0.000000
0.9d	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.571888	0'0	2'16	[0,1,2,3]	[0,1,2,3]	0'0	0.000000
0.9c	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.490164	0'0	5'13	[1,2,0,3]	[1,2,0,3]	0'0	0.000000
0.9b	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.505739	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.9a	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.572393	0'0	2'16	[0,1,3,2]	[0,1,3,2]	0'0	0.000000
0.99	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.490272	0'0	5'13	[1,2,0,3]	[1,2,0,3]	0'0	0.000000
0.98	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.490395	0'0	5'13	[1,2,0,3]	[1,2,0,3]	0'0	0.000000
0.97	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.572933	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.96	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.560887	0'0	4'13	[3,1,0,2]	[3,1,0,2]	0'0	0.000000
0.95	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.573397	0'0	2'16	[0,3,2,1]	[0,3,2,1]	0'0	0.000000
0.94	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.561045	0'0	4'13	[3,2,1,0]	[3,2,1,0]	0'0	0.000000
0.93	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.505843	0'0	3'13	[2,3,0,1]	[2,3,0,1]	0'0	0.000000
0.92	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.505945	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.91	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.561219	0'0	4'13	[3,0,1,2]	[3,0,1,2]	0'0	0.000000
0.90	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.561379	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.8f	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.506049	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.8e	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.574844	0'0	2'16	[0,1,2,3]	[0,1,2,3]	0'0	0.000000
0.8d	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.491064	0'0	5'13	[1,3,0,2]	[1,3,0,2]	0'0	0.000000
0.8c	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.561558	0'0	4'13	[3,0,1,2]	[3,0,1,2]	0'0	0.000000
0.8b	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.561707	0'0	4'13	[3,0,2,1]	[3,0,2,1]	0'0	0.000000
0.8a	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.509347	0'0	3'13	[2,3,0,1]	[2,3,0,1]	0'0	0.000000
0.89	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.509492	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.88	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.509599	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.87	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.575476	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.86	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.561932	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.85	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.575693	0'0	2'16	[0,1,2,3]	[0,1,2,3]	0'0	0.000000
0.84	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.562961	0'0	4'13	[3,2,0,1]	[3,2,0,1]	0'0	0.000000
0.83	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.509707	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.82	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.576348	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.81	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.563243	0'0	4'13	[3,0,2,1]	[3,0,2,1]	0'0	0.000000
0.80	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.576671	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.7f	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.510869	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.7e	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.491434	0'0	5'13	[1,3,2,0]	[1,3,2,0]	0'0	0.000000
0.7d	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.511088	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.7c	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.566688	0'0	4'13	[3,0,1,2]	[3,0,1,2]	0'0	0.000000
0.7b	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.567048	0'0	4'13	[3,0,1,2]	[3,0,1,2]	0'0	0.000000
0.7a	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.491675	0'0	5'13	[1,0,3,2]	[1,0,3,2]	0'0	0.000000
0.79	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.491972	0'0	5'13	[1,2,3,0]	[1,2,3,0]	0'0	0.000000
0.78	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.511216	0'0	3'13	[2,3,0,1]	[2,3,0,1]	0'0	0.000000
0.77	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.511305	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.76	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.567322	0'0	4'13	[3,2,1,0]	[3,2,1,0]	0'0	0.000000
0.75	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.567583	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.74	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.567815	0'0	4'13	[3,2,0,1]	[3,2,0,1]	0'0	0.000000
0.73	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.568076	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.72	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.568317	0'0	4'13	[3,2,1,0]	[3,2,1,0]	0'0	0.000000
0.71	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.492295	0'0	5'13	[1,3,2,0]	[1,3,2,0]	0'0	0.000000
0.70	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.511377	0'0	3'13	[2,3,0,1]	[2,3,0,1]	0'0	0.000000
0.6f	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.568566	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.6e	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.511449	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.6d	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.511525	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.6c	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.569747	0'0	4'13	[3,1,0,2]	[3,1,0,2]	0'0	0.000000
0.6b	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.576967	0'0	2'16	[0,1,3,2]	[0,1,3,2]	0'0	0.000000
0.6a	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.492539	0'0	5'13	[1,0,3,2]	[1,0,3,2]	0'0	0.000000
0.69	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.511596	0'0	3'13	[2,0,3,1]	[2,0,3,1]	0'0	0.000000
0.68	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.511668	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.67	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.577272	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000
0.66	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.577657	0'0	2'16	[0,3,2,1]	[0,3,2,1]	0'0	0.000000
0.65	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.577960	0'0	2'16	[0,1,2,3]	[0,1,2,3]	0'0	0.000000
0.64	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.511740	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.63	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.578287	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.62	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.570034	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.61	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.511824	0'0	3'13	[2,3,0,1]	[2,3,0,1]	0'0	0.000000
0.60	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.578769	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000
0.5f	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.492914	0'0	5'13	[1,0,3,2]	[1,0,3,2]	0'0	0.000000
0.5e	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.511981	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.5d	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.512060	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.5c	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.524809	0'0	3'13	[2,3,0,1]	[2,3,0,1]	0'0	0.000000
0.5b	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.493371	0'0	5'13	[1,3,2,0]	[1,3,2,0]	0'0	0.000000
0.5a	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.493630	0'0	5'13	[1,2,3,0]	[1,2,3,0]	0'0	0.000000
0.59	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.579193	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.58	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.525403	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.57	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.525848	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.56	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.570314	0'0	4'13	[3,2,1,0]	[3,2,1,0]	0'0	0.000000
0.55	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.579632	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.54	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.570660	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.53	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.526338	0'0	3'13	[2,0,3,1]	[2,0,3,1]	0'0	0.000000
0.52	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.570919	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.51	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.526842	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.50	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.527131	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.4f	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.580034	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.4e	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.527444	0'0	3'13	[2,0,3,1]	[2,0,3,1]	0'0	0.000000
0.4d	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.493979	0'0	5'13	[1,3,0,2]	[1,3,0,2]	0'0	0.000000
0.4c	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.494295	0'0	5'13	[1,2,0,3]	[1,2,0,3]	0'0	0.000000
0.4b	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.527677	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.4a	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.571185	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.49	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.580358	0'0	2'16	[0,1,3,2]	[0,1,3,2]	0'0	0.000000
0.48	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.528093	0'0	3'13	[2,3,0,1]	[2,3,0,1]	0'0	0.000000
0.47	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.529074	0'0	3'13	[2,3,0,1]	[2,3,0,1]	0'0	0.000000
0.46	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.529248	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.45	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.580649	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000
0.44	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.529387	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.43	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.581019	0'0	2'16	[0,1,2,3]	[0,1,2,3]	0'0	0.000000
0.42	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.529519	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.41	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.763560	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000
0.40	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.763684	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.3f	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.571389	0'0	4'13	[3,2,1,0]	[3,2,1,0]	0'0	0.000000
0.3e	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.529654	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.3d	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.763788	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.3c	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.529787	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.3b	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.763891	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.3a	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.494559	0'0	5'13	[1,0,3,2]	[1,0,3,2]	0'0	0.000000
0.39	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.529929	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.38	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.571652	0'0	4'13	[3,2,1,0]	[3,2,1,0]	0'0	0.000000
0.37	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.494790	0'0	5'13	[1,2,3,0]	[1,2,3,0]	0'0	0.000000
0.36	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.764003	0'0	2'16	[0,1,2,3]	[0,1,2,3]	0'0	0.000000
0.35	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.764107	0'0	2'16	[0,1,2,3]	[0,1,2,3]	0'0	0.000000
0.34	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.530061	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.33	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.767830	0'0	4'13	[3,1,0,2]	[3,1,0,2]	0'0	0.000000
0.32	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.495022	0'0	5'13	[1,0,2,3]	[1,0,2,3]	0'0	0.000000
0.31	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.530198	0'0	3'13	[2,0,3,1]	[2,0,3,1]	0'0	0.000000
0.30	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.531072	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.2f	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.531217	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.2e	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.495284	0'0	5'13	[1,2,0,3]	[1,2,0,3]	0'0	0.000000
0.2d	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.769244	0'0	4'13	[3,2,1,0]	[3,2,1,0]	0'0	0.000000
0.2c	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.769389	0'0	4'13	[3,2,0,1]	[3,2,0,1]	0'0	0.000000
0.2b	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.495582	0'0	5'13	[1,3,2,0]	[1,3,2,0]	0'0	0.000000
0.2a	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.764216	0'0	2'16	[0,1,3,2]	[0,1,3,2]	0'0	0.000000
0.29	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.495800	0'0	5'13	[1,2,3,0]	[1,2,3,0]	0'0	0.000000
0.28	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.531335	0'0	3'13	[2,3,0,1]	[2,3,0,1]	0'0	0.000000
0.27	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.531584	0'0	3'13	[2,1,3,0]	[2,1,3,0]	0'0	0.000000
0.26	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.764318	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.25	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.531655	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.24	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.764420	0'0	2'16	[0,3,2,1]	[0,3,2,1]	0'0	0.000000
0.23	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.531708	0'0	3'13	[2,0,1,3]	[2,0,1,3]	0'0	0.000000
0.22	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.769765	0'0	4'13	[3,2,0,1]	[3,2,0,1]	0'0	0.000000
0.21	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.531760	0'0	3'13	[2,1,0,3]	[2,1,0,3]	0'0	0.000000
0.20	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.531809	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.1f	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.531858	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.1e	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.764520	0'0	2'16	[0,2,1,3]	[0,2,1,3]	0'0	0.000000
0.1d	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.764626	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.1c	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.764731	0'0	2'16	[0,3,2,1]	[0,3,2,1]	0'0	0.000000
0.1b	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.764818	0'0	2'16	[0,1,3,2]	[0,1,3,2]	0'0	0.000000
0.1a	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.769937	0'0	4'13	[3,1,0,2]	[3,1,0,2]	0'0	0.000000
0.19	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.495930	0'0	5'13	[1,3,2,0]	[1,3,2,0]	0'0	0.000000
0.18	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.496497	0'0	5'13	[1,3,0,2]	[1,3,0,2]	0'0	0.000000
0.17	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.770146	0'0	4'13	[3,1,2,0]	[3,1,2,0]	0'0	0.000000
0.16	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.496825	0'0	5'13	[1,0,3,2]	[1,0,3,2]	0'0	0.000000
0.15	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.764909	0'0	2'16	[0,2,3,1]	[0,2,3,1]	0'0	0.000000
0.14	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.531905	0'0	3'13	[2,3,1,0]	[2,3,1,0]	0'0	0.000000
0.13	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.497045	0'0	5'13	[1,0,2,3]	[1,0,2,3]	0'0	0.000000
0.12	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.770357	0'0	4'13	[3,1,0,2]	[3,1,0,2]	0'0	0.000000
0.11	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.770512	0'0	4'13	[3,2,0,1]	[3,2,0,1]	0'0	0.000000
0.10	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:55:00.198661	0'0	5'13	[1,3,0,2]	[1,3,0,2]	0'0	0.000000
0.f	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.251880	0'0	2'16	[0,1,2,3]	[0,1,2,3]	0'0	0.000000
0.e	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.771042	0'0	4'13	[3,2,0,1]	[3,2,0,1]	0'0	0.000000
0.d	0	0	0	0	0	0	0	stale+active+clean	2012-07-05 18:54:53.547897	0'0	2'16	[0,3,1,2]	[0,3,1,2]	0'0	0.000000


^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Pg stuck stale...why?
  2012-07-11  1:32   ` Mark Kirkwood
@ 2012-07-11  1:35     ` Mark Kirkwood
  2012-07-11  1:55     ` Josh Durgin
  1 sibling, 0 replies; 7+ messages in thread
From: Mark Kirkwood @ 2012-07-11  1:35 UTC (permalink / raw)
  Cc: ceph-devel

[-- Attachment #1: Type: text/plain, Size: 213 bytes --]

On 11/07/12 13:32, Mark Kirkwood wrote:
>
> I have attached the dump of stuck stale pgs, and the crushmap in use.


...of course I left off the crushmap, so here it is, plus my ceph.conf 
for good measure.


Mark

[-- Attachment #2: crush.txt --]
[-- Type: text/plain, Size: 1575 bytes --]

# begin crush map

# devices
device 0 osd0
device 1 osd1
device 2 osd2
device 3 osd3

# types
type 0 device
type 1 host
type 2 datacenter
type 3 root

# buckets
host ved1 {
	id -1		# do not change unnecessarily
	# weight 1.000
	alg straw
	hash 0	# rjenkins1
	item osd0 weight 1.000
}
host ved2 {
	id -2		# do not change unnecessarily
	# weight 1.000
	alg straw
	hash 0	# rjenkins1
	item osd1 weight 1.000
}
host ved3 {
	id -3		# do not change unnecessarily
	# weight 1.000
	alg straw
	hash 0	# rjenkins1
	item osd2 weight 1.000
}
host ved4 {
	id -4		# do not change unnecessarily
	# weight 1.000
	alg straw
	hash 0	# rjenkins1
	item osd3 weight 1.000
}
datacenter datacenter0 {
	id -5		# do not change unnecessarily
	# weight 2.000
	alg straw
	hash 0	# rjenkins1
	item ved1 weight 1.000
	item ved2 weight 1.000
}
datacenter datacenter1 {
	id -6		# do not change unnecessarily
	# weight 2.000
	alg straw
	hash 0	# rjenkins1
	item ved3 weight 1.000
	item ved4 weight 1.000
}
root root {
	id -7		# do not change unnecessarily
	# weight 4.000
	alg straw
	hash 0	# rjenkins1
	item datacenter0 weight 2.000
	item datacenter1 weight 2.000
}

# rules
rule data {
	ruleset 1
	type replicated
	min_size 2
	max_size 10
	step take datacenter0
	step chooseleaf firstn -2 type host
	step emit
	step take datacenter1
	step chooseleaf firstn 2 type host
	step emit
}
rule rbd {
	ruleset 2
	type replicated
	min_size 2
	max_size 10
	step take datacenter0
	step chooseleaf firstn -2 type host
	step emit
	step take datacenter1
	step chooseleaf firstn 2 type host
	step emit
}

# end crush map

[-- Attachment #3: ceph.conf --]
[-- Type: text/plain, Size: 4197 bytes --]

;
; Sample ceph ceph.conf file.
;
; This file defines cluster membership, the various locations
; that Ceph stores data, and any other runtime options.

;
; Experimental setup for 4 osd, 3 mon and 0 mds. 
; Will experiment with crush rules later...
;

; If a 'host' is defined for a daemon, the start/stop script will
; verify that it matches the hostname (or else ignore it).  If it is
; not defined, it is assumed that the daemon is intended to start on
; the current host (e.g., in a setup with a startup.conf on each
; node).

; The variables $type, $id and $name are available to use in paths
; $type = The type of daemon, possible values: mon, mds and osd
; $id = The ID of the daemon, for mon.alpha, $id will be alpha
; $name = $type.$id

; For example:
; osd.0
;  $type = osd
;  $id = 0
;  $name = osd.0

; mon.beta
;  $type = mon
;  $id = beta
;  $name = mon.beta

; global
[global]
	; enable secure authentication
	;auth supported = cephx
	;keyring = /etc/ceph/ceph.keyring

        ; allow ourselves to open a lot of files
        max open files = 131072

        ; set log file
        log file = /var/log/ceph/$name.log
        ; log_to_syslog = true        ; uncomment this line to log to syslog

        ; set up pid files
        pid file = /var/run/ceph/$name.pid

        ; If you want to run a IPv6 cluster, set this to true. Dual-stack isn't possible
        ;ms bind ipv6 = true

; monitors
;  You need at least one.  You need at least three if you want to
;  tolerate any node failures.  Always create an odd number.
[mon]
        mon data = /var/data/$name
	;keyring = /var/data/keyring.$name

        ; If you are using for example the RADOS Gateway and want to have your newly created
        ; pools a higher replication level, you can set a default
        ;osd pool default size = 3

        ; You can also specify a CRUSH rule for new pools
        ; Wiki: http://ceph.newdream.net/wiki/Custom_data_placement_with_CRUSH
        ;osd pool default crush rule = 0

        ; Timing is critical for monitors, but if you want to allow the clocks to drift a
        ; bit more, you can specify the max drift.
        ;mon clock drift allowed = 1

        ; Tell the monitor to backoff from this warning for 30 seconds
        ;mon clock drift warn backoff = 30

	; logging, for debugging monitor crashes, in order of
	; their likelihood of being helpful :)
	;debug ms = 1
	;debug mon = 20
	;debug paxos = 20
	;debug auth = 20

[mon.ved1]
	host = ved1
	mon addr = 192.168.122.11:6789

[mon.ved2]
	host = ved2
	mon addr = 192.168.122.12:6789

[mon.ved3]
	host = ved3
	mon addr = 192.168.122.13:6789

; osd
;  You need at least one.  Two if you want data to be replicated.
;  Define as many as you like.
[osd]
	; This is where the btrfs volume will be mounted.
	osd data = /var/data/$name
	;keyring = /var/data/keyring.$name

	; Ideally, make this a separate disk or partition.  A few
 	; hundred MB should be enough; more if you have fast or many
 	; disks.  You can use a file under the osd data dir if need be
 	; (e.g. /data/$name/journal), but it will be slower than a
 	; separate disk or partition.

        ; This is an example of a file-based journal.
	osd journal = /var/data/$name/journal
	osd journal size = 1000 ; journal size, in megabytes

        ; If you want to run the journal on a tmpfs, disable DirectIO
        ;journal dio = false

        ; You can change the number of recovery operations to speed up recovery
        ; or slow it down if your machines can't handle it
        ; osd recovery max active = 3

	; osd logging to debug osd issues, in order of likelihood of being
	; helpful
	;debug ms = 1
	;debug osd = 20
	;debug filestore = 20
	;debug journal = 20

[osd.0]
	host = ved1

	; if 'btrfs devs' is not specified, you're responsible for
	; setting up the 'osd data' dir.  if it is not btrfs, things
	; will behave up until you try to recover from a crash (which
	; usually fine for basic testing).
	;btrfs devs = /dev/sdx

        ; If you want to specify some other mount options, you can do so.
        ; The default values are rw,noatime
        ;btrfs options = rw,noatime

[osd.1]
	host = ved2

[osd.2]
	host = ved3

[osd.3]
	host = ved4

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Pg stuck stale...why?
  2012-07-11  1:32   ` Mark Kirkwood
  2012-07-11  1:35     ` Mark Kirkwood
@ 2012-07-11  1:55     ` Josh Durgin
  2012-07-11  2:09       ` Mark Kirkwood
  1 sibling, 1 reply; 7+ messages in thread
From: Josh Durgin @ 2012-07-11  1:55 UTC (permalink / raw)
  To: Mark Kirkwood; +Cc: ceph-devel

On 07/10/2012 06:32 PM, Mark Kirkwood wrote:
> On 11/07/12 13:22, Josh Durgin wrote:
>> On 07/10/2012 06:11 PM, Mark Kirkwood wrote:
>>> I am seeing this:
>>>
>>> # ceph -s
>>> health HEALTH_WARN 256 pgs stale; 256 pgs stuck stale
>>> monmap e1: 3 mons at
>>> {ved1=192.168.122.11:6789/0,ved2=192.168.122.12:6789/0,ved3=192.168.122.13:6789/0},
>>>
>>> election epoch 18, quorum 0,1,2 ved1,ved2,ved3
>>> osdmap e62: 4 osds: 4 up, 4 in
>>> pgmap v47148: 768 pgs: 512 active+clean, 256 stale+active+clean; 2224 MB
>>> data, 15442 MB used, 86907 MB / 102350 MB avail
>>> mdsmap e1: 0/0/1
>>>
>>> In particular 256 pgs stuck stale - I've tried a) waiting a while
>>> (overnight), b) a rolling restart of all 4 osd's, c) restarting all ceph
>>> services on all 4 nodes. All without changing this.
>>>
>>> As far as I understand what stuck state means, I can't see why they need
>>> to stay that way, given all osd's and mon's are up. (I have no mds
>>> configured)....any ideas? Or is this just expected?
>>>
>>> Regards
>>>
>>> Mark
>>
>> What does 'ceph pg dump_stuck stale' show? Stale means that the
>> monitors haven't gotten updates about those pgs from the osds within
>> the a certain period of time (default is 300 seconds), so something may
>> be wrong with your crushmap or those pgs themselves.
>>
>> Josh
>
> I have attached the dump of stuck stale pgs, and the crushmap in use.
>
> I did wonder if it is to do with not using any mds - i.e could this mean
> the meta data pgs never get touched?
>
> Mark
>

It doesn't look like a problem with your crushmap - the pgs are all 
mapped to osds, and there's no common osd holding things up.

Not using the mds doesn't affect the pgs. They should still be active.
All the stuck ones are in pool 0 though. Is there anything special about 
that pool? Were there any changes before the pgs became stuck?

I don't think it should work in this case, but you might try 'ceph pg 
force_create_pg 0.c'.

Josh

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Pg stuck stale...why?
  2012-07-11  1:55     ` Josh Durgin
@ 2012-07-11  2:09       ` Mark Kirkwood
  2012-07-11  3:23         ` Mark Kirkwood
  0 siblings, 1 reply; 7+ messages in thread
From: Mark Kirkwood @ 2012-07-11  2:09 UTC (permalink / raw)
  To: Josh Durgin; +Cc: ceph-devel

On 11/07/12 13:55, Josh Durgin wrote:
> On 07/10/2012 06:32 PM, Mark Kirkwood wrote:
>> On 11/07/12 13:22, Josh Durgin wrote:
>>> On 07/10/2012 06:11 PM, Mark Kirkwood wrote:
>>>> I am seeing this:
>>>>
>>>> # ceph -s
>>>> health HEALTH_WARN 256 pgs stale; 256 pgs stuck stale
>>>> monmap e1: 3 mons at
>>>> {ved1=192.168.122.11:6789/0,ved2=192.168.122.12:6789/0,ved3=192.168.122.13:6789/0}, 
>>>>
>>>>
>>>> election epoch 18, quorum 0,1,2 ved1,ved2,ved3
>>>> osdmap e62: 4 osds: 4 up, 4 in
>>>> pgmap v47148: 768 pgs: 512 active+clean, 256 stale+active+clean; 
>>>> 2224 MB
>>>> data, 15442 MB used, 86907 MB / 102350 MB avail
>>>> mdsmap e1: 0/0/1
>>>>
>>>> In particular 256 pgs stuck stale - I've tried a) waiting a while
>>>> (overnight), b) a rolling restart of all 4 osd's, c) restarting all 
>>>> ceph
>>>> services on all 4 nodes. All without changing this.
>>>>
>>>> As far as I understand what stuck state means, I can't see why they 
>>>> need
>>>> to stay that way, given all osd's and mon's are up. (I have no mds
>>>> configured)....any ideas? Or is this just expected?
>>>>
>>>> Regards
>>>>
>>>> Mark
>>>
>>> What does 'ceph pg dump_stuck stale' show? Stale means that the
>>> monitors haven't gotten updates about those pgs from the osds within
>>> the a certain period of time (default is 300 seconds), so something may
>>> be wrong with your crushmap or those pgs themselves.
>>>
>>> Josh
>>
>> I have attached the dump of stuck stale pgs, and the crushmap in use.
>>
>> I did wonder if it is to do with not using any mds - i.e could this mean
>> the meta data pgs never get touched?
>>
>> Mark
>>
>
> It doesn't look like a problem with your crushmap - the pgs are all 
> mapped to osds, and there's no common osd holding things up.
>
> Not using the mds doesn't affect the pgs. They should still be active.
> All the stuck ones are in pool 0 though. Is there anything special 
> about that pool? Were there any changes before the pgs became stuck?
>
> I don't think it should work in this case, but you might try 'ceph pg 
> force_create_pg 0.c'.
>
>

Hmm - good observation - pool 0 is data, and I am only using rbd at the 
moment, and:

$ ceph osd dump -o -|grep size
pool 0 'data' rep size 4 crush_ruleset 0 object_hash rjenkins pg_num 256 
pgp_num 256 last_change 8 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins pg_num 
256 pgp_num 256 last_change 1 owner 0
pool 2 'rbd' rep size 4 crush_ruleset 2 object_hash rjenkins pg_num 256 
pgp_num 256 last_change 11 owner 0

... pool 0 is mapped to crush ruleset 0 - and I don't have a ruleset 0. 
Could that be the problem?

Mark




^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Pg stuck stale...why?
  2012-07-11  2:09       ` Mark Kirkwood
@ 2012-07-11  3:23         ` Mark Kirkwood
  0 siblings, 0 replies; 7+ messages in thread
From: Mark Kirkwood @ 2012-07-11  3:23 UTC (permalink / raw)
  To: Josh Durgin; +Cc: ceph-devel

On 11/07/12 14:09, Mark Kirkwood wrote:
> On 11/07/12 13:55, Josh Durgin wrote:
>> On 07/10/2012 06:32 PM, Mark Kirkwood wrote:
>>> On 11/07/12 13:22, Josh Durgin wrote:
>>>> On 07/10/2012 06:11 PM, Mark Kirkwood wrote:
>>>>> I am seeing this:
>>>>>
>>>>> # ceph -s
>>>>> health HEALTH_WARN 256 pgs stale; 256 pgs stuck stale
>>>>> monmap e1: 3 mons at
>>>>> {ved1=192.168.122.11:6789/0,ved2=192.168.122.12:6789/0,ved3=192.168.122.13:6789/0}, 
>>>>>
>>>>>
>>>>> election epoch 18, quorum 0,1,2 ved1,ved2,ved3
>>>>> osdmap e62: 4 osds: 4 up, 4 in
>>>>> pgmap v47148: 768 pgs: 512 active+clean, 256 stale+active+clean; 
>>>>> 2224 MB
>>>>> data, 15442 MB used, 86907 MB / 102350 MB avail
>>>>> mdsmap e1: 0/0/1
>>>>>
>>>>> In particular 256 pgs stuck stale - I've tried a) waiting a while
>>>>> (overnight), b) a rolling restart of all 4 osd's, c) restarting 
>>>>> all ceph
>>>>> services on all 4 nodes. All without changing this.
>>>>>
>>>>> As far as I understand what stuck state means, I can't see why 
>>>>> they need
>>>>> to stay that way, given all osd's and mon's are up. (I have no mds
>>>>> configured)....any ideas? Or is this just expected?
>>>>>
>>>>> Regards
>>>>>
>>>>> Mark
>>>>
>>>> What does 'ceph pg dump_stuck stale' show? Stale means that the
>>>> monitors haven't gotten updates about those pgs from the osds within
>>>> the a certain period of time (default is 300 seconds), so something 
>>>> may
>>>> be wrong with your crushmap or those pgs themselves.
>>>>
>>>> Josh
>>>
>>> I have attached the dump of stuck stale pgs, and the crushmap in use.
>>>
>>> I did wonder if it is to do with not using any mds - i.e could this 
>>> mean
>>> the meta data pgs never get touched?
>>>
>>> Mark
>>>
>>
>> It doesn't look like a problem with your crushmap - the pgs are all 
>> mapped to osds, and there's no common osd holding things up.
>>
>> Not using the mds doesn't affect the pgs. They should still be active.
>> All the stuck ones are in pool 0 though. Is there anything special 
>> about that pool? Were there any changes before the pgs became stuck?
>>
>> I don't think it should work in this case, but you might try 'ceph pg 
>> force_create_pg 0.c'.
>>
>>
>
> Hmm - good observation - pool 0 is data, and I am only using rbd at 
> the moment, and:
>
> $ ceph osd dump -o -|grep size
> pool 0 'data' rep size 4 crush_ruleset 0 object_hash rjenkins pg_num 
> 256 pgp_num 256 last_change 8 owner 0 crash_replay_interval 45
> pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins 
> pg_num 256 pgp_num 256 last_change 1 owner 0
> pool 2 'rbd' rep size 4 crush_ruleset 2 object_hash rjenkins pg_num 
> 256 pgp_num 256 last_change 11 owner 0
>
> ... pool 0 is mapped to crush ruleset 0 - and I don't have a ruleset 
> 0. Could that be the problem?
>
> Mark

...and that was it:

$ ceph osd pool set data crush_ruleset 1

$ ceph health
HEALTH_OK

Thanks for making me think about this more carefully!

Mark

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2012-07-11  3:23 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-07-11  1:11 Pg stuck stale...why? Mark Kirkwood
2012-07-11  1:22 ` Josh Durgin
2012-07-11  1:32   ` Mark Kirkwood
2012-07-11  1:35     ` Mark Kirkwood
2012-07-11  1:55     ` Josh Durgin
2012-07-11  2:09       ` Mark Kirkwood
2012-07-11  3:23         ` Mark Kirkwood

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.