From: Mark Kirkwood <mark.kirkwood@catalyst.net.nz>
To: Josh Durgin <josh.durgin@inktank.com>
Cc: ceph-devel@vger.kernel.org
Subject: Re: Pg stuck stale...why?
Date: Wed, 11 Jul 2012 13:32:39 +1200 [thread overview]
Message-ID: <4FFCD7B7.5020707@catalyst.net.nz> (raw)
In-Reply-To: <4FFCD53F.108@inktank.com>
[-- Attachment #1: Type: text/plain, Size: 1424 bytes --]
On 11/07/12 13:22, Josh Durgin wrote:
> On 07/10/2012 06:11 PM, Mark Kirkwood wrote:
>> I am seeing this:
>>
>> # ceph -s
>> health HEALTH_WARN 256 pgs stale; 256 pgs stuck stale
>> monmap e1: 3 mons at
>> {ved1=192.168.122.11:6789/0,ved2=192.168.122.12:6789/0,ved3=192.168.122.13:6789/0},
>>
>> election epoch 18, quorum 0,1,2 ved1,ved2,ved3
>> osdmap e62: 4 osds: 4 up, 4 in
>> pgmap v47148: 768 pgs: 512 active+clean, 256 stale+active+clean; 2224 MB
>> data, 15442 MB used, 86907 MB / 102350 MB avail
>> mdsmap e1: 0/0/1
>>
>> In particular 256 pgs stuck stale - I've tried a) waiting a while
>> (overnight), b) a rolling restart of all 4 osd's, c) restarting all ceph
>> services on all 4 nodes. All without changing this.
>>
>> As far as I understand what stuck state means, I can't see why they need
>> to stay that way, given all osd's and mon's are up. (I have no mds
>> configured)....any ideas? Or is this just expected?
>>
>> Regards
>>
>> Mark
>
> What does 'ceph pg dump_stuck stale' show? Stale means that the
> monitors haven't gotten updates about those pgs from the osds within
> the a certain period of time (default is 300 seconds), so something may
> be wrong with your crushmap or those pgs themselves.
>
> Josh
I have attached the dump of stuck stale pgs, and the crushmap in use.
I did wonder if it is to do with not using any mds - i.e could this mean
the meta data pgs never get touched?
Mark
[-- Attachment #2: pg-stale.dump --]
[-- Type: text/plain, Size: 27489 bytes --]
ok
pg_stat objects mip degr unf bytes log disklog state state_stamp v reported up acting last_scrub scrub_stamp
0.c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.398997 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.547700 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.548638 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399058 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.549203 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.225656 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.549348 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399093 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.550016 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.550764 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399135 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.551391 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.377207 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.ff 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.377591 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.fe 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.548875 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.fd 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.549702 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.fc 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399168 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.fb 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.377897 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.fa 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.378378 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.f9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399204 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.f8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.551621 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.f7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399237 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.f6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.551991 0'0 4'13 [3,0,2,1] [3,0,2,1] 0'0 0.000000
0.f5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399273 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.f4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399305 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.f3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.550251 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.f2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.550929 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.f1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399338 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.f0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.551940 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.ef 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.378957 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.ee 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.379533 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.ed 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.552712 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.ec 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.554542 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.eb 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.555242 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.ea 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.552494 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.e9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.399372 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.e8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.553346 0'0 4'13 [3,0,2,1] [3,0,2,1] 0'0 0.000000
0.e7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.555787 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.e6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.553848 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.e5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.379900 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.e4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.556264 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.e3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.486976 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.e2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.556925 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.e1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.554403 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.e0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.557482 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.df 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380084 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.de 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.555001 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.dd 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.558353 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.dc 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.559043 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.db 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.559800 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.da 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380490 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.d9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380751 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.d8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.487108 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.d7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.487218 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.d6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.487324 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.d5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380805 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.d4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.555555 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.d3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.560628 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.d2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.556332 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.d1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.487434 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.d0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.487545 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.cf 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.562376 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.ce 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.488773 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.cd 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380842 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.cc 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.488970 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.cb 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.380955 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.ca 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.563030 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.c9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.381041 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.c8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489080 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.c7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.381189 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.c6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.563908 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.c5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.564494 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.c4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.556848 0'0 4'13 [3,0,2,1] [3,0,2,1] 0'0 0.000000
0.c3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.565018 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.c2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.381294 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.c1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.565542 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.c0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.566142 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.bf 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489177 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.be 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.382223 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.bd 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489302 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.bc 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.382601 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.bb 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.557404 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.ba 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489410 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.b9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.557867 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.b8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.566749 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.b7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.558295 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.b6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489512 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.b5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.382939 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.b4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.567778 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.b3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.568995 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.b2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489720 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.b1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.569605 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.b0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.383154 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.af 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.383421 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.ae 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.558791 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.ad 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.571039 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.ac 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.383737 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.ab 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.383994 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.aa 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505084 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.a9 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489833 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.a8 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505207 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.a7 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.489951 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.a6 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505312 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.a5 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.559997 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.a4 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505421 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.a3 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505524 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.a2 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.560282 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.a1 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.560511 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.a0 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505635 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.9f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.490060 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.9e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.560679 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.9d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.571888 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.9c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.490164 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.9b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505739 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.9a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.572393 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.99 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.490272 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.98 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.490395 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.97 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.572933 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.96 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.560887 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.95 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.573397 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.94 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561045 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.93 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505843 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.92 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.505945 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.91 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561219 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.90 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561379 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.8f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.506049 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.8e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.574844 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.8d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.491064 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.8c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561558 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.8b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561707 0'0 4'13 [3,0,2,1] [3,0,2,1] 0'0 0.000000
0.8a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.509347 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.89 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.509492 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.88 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.509599 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.87 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.575476 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.86 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.561932 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.85 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.575693 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.84 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.562961 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.83 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.509707 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.82 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.576348 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.81 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.563243 0'0 4'13 [3,0,2,1] [3,0,2,1] 0'0 0.000000
0.80 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.576671 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.7f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.510869 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.7e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.491434 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.7d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511088 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.7c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.566688 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.7b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.567048 0'0 4'13 [3,0,1,2] [3,0,1,2] 0'0 0.000000
0.7a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.491675 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.79 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.491972 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.78 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511216 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.77 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511305 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.76 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.567322 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.75 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.567583 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.74 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.567815 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.73 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.568076 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.72 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.568317 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.71 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.492295 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.70 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511377 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.6f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.568566 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.6e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511449 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.6d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511525 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.6c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.569747 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.6b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.576967 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.6a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.492539 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.69 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511596 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.68 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511668 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.67 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.577272 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.66 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.577657 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.65 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.577960 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.64 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511740 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.63 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.578287 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.62 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.570034 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.61 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511824 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.60 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.578769 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.5f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.492914 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.5e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.511981 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.5d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.512060 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.5c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.524809 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.5b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.493371 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.5a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.493630 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.59 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.579193 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.58 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.525403 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.57 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.525848 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.56 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.570314 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.55 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.579632 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.54 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.570660 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.53 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.526338 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.52 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.570919 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.51 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.526842 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.50 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.527131 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.4f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.580034 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.4e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.527444 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.4d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.493979 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.4c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.494295 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.4b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.527677 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.4a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.571185 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.49 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.580358 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.48 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.528093 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.47 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529074 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.46 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529248 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.45 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.580649 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.44 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529387 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.43 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.581019 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.42 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529519 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.41 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.763560 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
0.40 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.763684 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.3f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.571389 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.3e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529654 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.3d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.763788 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.3c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529787 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.3b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.763891 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.3a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.494559 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.39 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.529929 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.38 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.571652 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.37 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.494790 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.36 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764003 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.35 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764107 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.34 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.530061 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.33 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.767830 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.32 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.495022 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.31 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.530198 0'0 3'13 [2,0,3,1] [2,0,3,1] 0'0 0.000000
0.30 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531072 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.2f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531217 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.2e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.495284 0'0 5'13 [1,2,0,3] [1,2,0,3] 0'0 0.000000
0.2d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.769244 0'0 4'13 [3,2,1,0] [3,2,1,0] 0'0 0.000000
0.2c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.769389 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.2b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.495582 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.2a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764216 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.29 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.495800 0'0 5'13 [1,2,3,0] [1,2,3,0] 0'0 0.000000
0.28 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531335 0'0 3'13 [2,3,0,1] [2,3,0,1] 0'0 0.000000
0.27 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531584 0'0 3'13 [2,1,3,0] [2,1,3,0] 0'0 0.000000
0.26 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764318 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.25 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531655 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.24 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764420 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.23 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531708 0'0 3'13 [2,0,1,3] [2,0,1,3] 0'0 0.000000
0.22 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.769765 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.21 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531760 0'0 3'13 [2,1,0,3] [2,1,0,3] 0'0 0.000000
0.20 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531809 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.1f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531858 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.1e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764520 0'0 2'16 [0,2,1,3] [0,2,1,3] 0'0 0.000000
0.1d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764626 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.1c 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764731 0'0 2'16 [0,3,2,1] [0,3,2,1] 0'0 0.000000
0.1b 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764818 0'0 2'16 [0,1,3,2] [0,1,3,2] 0'0 0.000000
0.1a 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.769937 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.19 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.495930 0'0 5'13 [1,3,2,0] [1,3,2,0] 0'0 0.000000
0.18 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.496497 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.17 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.770146 0'0 4'13 [3,1,2,0] [3,1,2,0] 0'0 0.000000
0.16 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.496825 0'0 5'13 [1,0,3,2] [1,0,3,2] 0'0 0.000000
0.15 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.764909 0'0 2'16 [0,2,3,1] [0,2,3,1] 0'0 0.000000
0.14 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.531905 0'0 3'13 [2,3,1,0] [2,3,1,0] 0'0 0.000000
0.13 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.497045 0'0 5'13 [1,0,2,3] [1,0,2,3] 0'0 0.000000
0.12 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.770357 0'0 4'13 [3,1,0,2] [3,1,0,2] 0'0 0.000000
0.11 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.770512 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.10 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:55:00.198661 0'0 5'13 [1,3,0,2] [1,3,0,2] 0'0 0.000000
0.f 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.251880 0'0 2'16 [0,1,2,3] [0,1,2,3] 0'0 0.000000
0.e 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.771042 0'0 4'13 [3,2,0,1] [3,2,0,1] 0'0 0.000000
0.d 0 0 0 0 0 0 0 stale+active+clean 2012-07-05 18:54:53.547897 0'0 2'16 [0,3,1,2] [0,3,1,2] 0'0 0.000000
next prev parent reply other threads:[~2012-07-11 1:32 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-07-11 1:11 Pg stuck stale...why? Mark Kirkwood
2012-07-11 1:22 ` Josh Durgin
2012-07-11 1:32 ` Mark Kirkwood [this message]
2012-07-11 1:35 ` Mark Kirkwood
2012-07-11 1:55 ` Josh Durgin
2012-07-11 2:09 ` Mark Kirkwood
2012-07-11 3:23 ` Mark Kirkwood
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4FFCD7B7.5020707@catalyst.net.nz \
--to=mark.kirkwood@catalyst.net.nz \
--cc=ceph-devel@vger.kernel.org \
--cc=josh.durgin@inktank.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.