From: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
To: Josh Durgin <josh.durgin@inktank.com>
Cc: ceph-devel@vger.kernel.org
Subject: Re: Pg stuck stale...why?
Date: Wed, 11 Jul 2012 13:32:39 +1200 [thread overview]
Message-ID: <4FFCD7B7.5020707@catalyst.net.nz> (raw)
In-Reply-To: <4FFCD53F.108@inktank.com>
[-- Attachment #1: Type: text/plain, Size: 1424 bytes --]
On 11/07/12 13:22, Josh Durgin wrote:
> On 07/10/2012 06:11 PM, Mark Kirkwood wrote:
>> I am seeing this:
>>
>> # ceph -s
>> health HEALTH_WARN 256 pgs stale; 256 pgs stuck stale
>> monmap e1: 3 mons at
>> {ved1=192.168.122.11:6789/0,ved2=192.168.122.12:6789/0,ved3=192.168.122.13:6789/0},
>>
>> election epoch 18, quorum 0,1,2 ved1,ved2,ved3
>> osdmap e62: 4 osds: 4 up, 4 in
>> pgmap v47148: 768 pgs: 512 active+clean, 256 stale+active+clean; 2224 MB
>> data, 15442 MB used, 86907 MB / 102350 MB avail
>> mdsmap e1: 0/0/1
>>
>> In particular 256 pgs stuck stale - I've tried a) waiting a while
>> (overnight), b) a rolling restart of all 4 osd's, c) restarting all ceph
>> services on all 4 nodes. All without changing this.
>>
>> As far as I understand what stuck state means, I can't see why they need
>> to stay that way, given all osd's and mon's are up. (I have no mds
>> configured)....any ideas? Or is this just expected?
>>
>> Regards
>>
>> Mark
>
> What does 'ceph pg dump_stuck stale' show? Stale means that the
> monitors haven't gotten updates about those pgs from the osds within
> the a certain period of time (default is 300 seconds), so something may
> be wrong with your crushmap or those pgs themselves.
>
> Josh
I have attached the dump of stuck stale pgs, and the crushmap in use.
I did wonder if it is to do with not using any mds - i.e could this mean
the meta data pgs never get touched?
Mark
[-- Attachment #2: pg-stale.dump --]
[-- Type: text/plain, Size: 27489 bytes --]
ok
pg_stat objects mip degr unf bytes log disklog state state_stamp v reported up acting last_scrub scrub_stamp
0.c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.398997 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.547700 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.548638 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399058 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.549203 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.225656 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.549348 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399093 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.550016 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.550764 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399135 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.551391 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.377207 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.ff 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.377591 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.fe 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.548875 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.fd 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.549702 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.fc 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399168 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.fb 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.377897 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.fa 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.378378 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.f9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399204 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.f8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.551621 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.f7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399237 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.f6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.551991 0'0 4'13 [3,0,2,1] [3,0,2,1] 0'0 0.000000
0.f5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399273 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.f4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399305 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.f3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.550251 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.f2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.550929 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.f1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399338 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.f0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.551940 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.ef 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.378957 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.ee 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.379533 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.ed 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.552712 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.ec 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.554542 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.eb 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.555242 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.ea 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.552494 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.e9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399372 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.e8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.553346 0'0 4'13 [3,0,2,1] [3,0,2,1] 0'0 0.000000
0.e7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.555787 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.e6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.553848 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.e5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.379900 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.e4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.556264 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.e3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.486976 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.e2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.556925 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.e1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.554403 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.e0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.557482 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.df 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380084 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.de 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.555001 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.dd 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.558353 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.dc 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.559043 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.db 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.559800 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.da 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380490 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.d9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380751 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.d8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.487108 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.d7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.487218 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.d6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.487324 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.d5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380805 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.d4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.555555 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.d3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.560628 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.d2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.556332 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.d1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.487434 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.d0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.487545 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.cf 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.562376 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.ce 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.488773 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.cd 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380842 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.cc 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.488970 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.cb 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380955 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.ca 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.563030 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.c9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.381041 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.c8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489080 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.c7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.381189 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.c6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.563908 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.c5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.564494 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.c4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.556848 0'0 4'13 [3,0,2,1] [3,0,2,1] 0'0 0.000000
0.c3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.565018 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.c2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.381294 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.c1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.565542 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.c0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.566142 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.bf 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489177 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.be 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.382223 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.bd 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489302 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.bc 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.382601 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.bb 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.557404 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.ba 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489410 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.b9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.557867 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.b8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.566749 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.b7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.558295 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.b6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489512 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.b5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.382939 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.b4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.567778 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.b3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.568995 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.b2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489720 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.b1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.569605 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.b0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.383154 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.af 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.383421 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.ae 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.558791 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.ad 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.571039 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.ac 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.383737 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.ab 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.383994 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.aa 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505084 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.a9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489833 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.a8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505207 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.a7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489951 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.a6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505312 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.a5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.559997 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.a4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505421 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.a3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505524 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.a2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.560282 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.a1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.560511 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.a0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505635 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.9f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.490060 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.9e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.560679 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.9d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.571888 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.9c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.490164 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.9b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505739 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.9a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.572393 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.99 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.490272 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.98 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.490395 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.97 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.572933 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.96 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.560887 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.95 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.573397 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.94 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561045 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.93 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505843 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.92 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505945 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.91 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561219 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.90 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561379 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.8f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.506049 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.8e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.574844 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.8d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.491064 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.8c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561558 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.8b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561707 0'0 4'13 [3,0,2,1] [3,0,2,1] 0'0 0.000000
0.8a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.509347 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.89 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.509492 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.88 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.509599 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.87 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.575476 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.86 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561932 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.85 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.575693 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.84 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.562961 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.83 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.509707 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.82 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.576348 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.81 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.563243 0'0 4'13 [3,0,2,1] [3,0,2,1] 0'0 0.000000
0.80 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.576671 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.7f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.510869 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.7e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.491434 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.7d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511088 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.7c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.566688 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.7b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.567048 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.7a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.491675 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.79 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.491972 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.78 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511216 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.77 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511305 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.76 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.567322 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.75 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.567583 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.74 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.567815 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.73 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.568076 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.72 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.568317 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.71 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.492295 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.70 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511377 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.6f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.568566 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.6e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511449 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.6d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511525 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.6c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.569747 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.6b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.576967 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.6a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.492539 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.69 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511596 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.68 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511668 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.67 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.577272 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.66 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.577657 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.65 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.577960 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.64 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511740 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.63 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.578287 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.62 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.570034 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.61 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511824 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.60 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.578769 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.5f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.492914 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.5e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511981 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.5d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.512060 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.5c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.524809 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.5b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.493371 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.5a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.493630 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.59 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.579193 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.58 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.525403 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.57 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.525848 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.56 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.570314 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.55 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.579632 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.54 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.570660 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.53 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.526338 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.52 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.570919 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.51 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.526842 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.50 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.527131 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.4f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.580034 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.4e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.527444 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.4d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.493979 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.4c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.494295 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.4b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.527677 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.4a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.571185 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.49 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.580358 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.48 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.528093 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.47 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529074 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.46 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529248 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.45 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.580649 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.44 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529387 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.43 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.581019 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.42 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529519 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.41 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.763560 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.40 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.763684 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.3f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.571389 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.3e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529654 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.3d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.763788 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.3c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529787 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.3b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.763891 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.3a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.494559 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.39 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529929 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.38 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.571652 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.37 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.494790 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.36 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764003 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.35 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764107 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.34 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.530061 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.33 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.767830 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.32 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.495022 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.31 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.530198 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.30 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531072 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.2f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531217 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.2e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.495284 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.2d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.769244 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.2c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.769389 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.2b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.495582 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.2a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764216 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.29 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.495800 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.28 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531335 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.27 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531584 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.26 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764318 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.25 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531655 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.24 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764420 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.23 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531708 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.22 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.769765 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.21 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531760 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.20 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531809 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.1f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531858 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.1e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764520 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.1d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764626 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.1c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764731 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.1b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764818 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.1a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.769937 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.19 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.495930 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.18 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.496497 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.17 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.770146 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.16 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.496825 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.15 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764909 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.14 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531905 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.13 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.497045 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.12 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.770357 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.11 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.770512 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.10 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.198661 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.251880 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.771042 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.547897 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
next prev parent reply other threads:[~2012-07-11 1:32 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-07-11 1:11 Pg stuck stale...why? Mark Kirkwood
2012-07-11 1:22 ` Josh Durgin
2012-07-11 1:32 ` Mark Kirkwood [this message]
2012-07-11 1:35 ` Mark Kirkwood
2012-07-11 1:55 ` Josh Durgin
2012-07-11 2:09 ` Mark Kirkwood
2012-07-11 3:23 ` Mark Kirkwood
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4FFCD7B7.5020707@catalyst.net.nz \
--to=mark.kirkwood@catalyst.net.nz \
--cc=ceph-devel@vger.kernel.org \
--cc=josh.durgin@inktank.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox