* Ceph remap/recovery stuck
@ 2012-08-23 13:52 Sławomir Skowron
2012-08-24 11:54 ` Sławomir Skowron
0 siblings, 1 reply; 4+ messages in thread
From: Sławomir Skowron @ 2012-08-23 13:52 UTC (permalink / raw)
To: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 1599 bytes --]
3 osd after crash rebuilds ok, but rebuild of two more osd (12 and
30), i can't make cluster to be active+clean
I do rebuild like in doc:
stop osd,
remove from crush,
rm from map,
recreate a osd, after cluster get stable
But now, all osd are in, and up, and data won't remap, and some of PG,
have only two osd in chain with replication level 3 for this pool.
2012-08-23 15:26:46.073685 mon.0 [INF] pgmap v117192: 6472 pgs: 63
active, 4457 active+clean, 1942 active+remapped, 10 active+degraded;
596 GB data, 1650 GB used, 20059 GB / 21710 GB avail; 57815/4705888
degraded (1.229%)
In attachment output from:
ceph osd dump -o -
I can't find any info in doc for this situation.
HEALTH_WARN 10 pgs degraded; 2015 pgs stuck unclean; recovery
57871/4706179 degraded (1.230%)
root@s3-10-177-64-6:~# ceph -s
health HEALTH_WARN 10 pgs degraded; 2015 pgs stuck unclean;
recovery 57871/4706179 degraded (1.230%)
monmap e4: 3 mons at
{0=10.177.64.4:6789/0,1=10.177.64.6:6789/0,2=10.177.64.8:6789/0},
election epoch 16, quorum 0,1,2 0,1,2
osdmap e1300: 78 osds: 78 up, 78 in
pgmap v117464: 6472 pgs: 63 active, 4457 active+clean, 1942
active+remapped, 10 active+degraded; 596 GB data, 1651 GB used, 20059
GB / 21710 GB avail; 57871/4706179 degraded (1.230%)
mdsmap e1: 0/0/1 up
Please help, i will try to give you any output you need.
And one more thing, little bug in 0.48.1:
ceph health blabla command, does same thing, as ceph health details.
Whatever is after health, means details.
--
-----
Regards
Sławek "sZiBis" Skowron
[-- Attachment #2: osd_dump.txt --]
[-- Type: text/plain, Size: 68774 bytes --]
ceph osd dump -o -
dumped osdmap epoch 1300
epoch 1300
fsid 16773c25-5054-4451-bf9f-efc1f7f21b89
created 2012-08-17 13:27:38.230943
modifed 2012-08-23 15:06:08.992622
flags
pool 0 'data' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 10 owner 0 crash_replay_interval 45
pool 1 'metadata' rep size 3 crush_ruleset 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 9 owner 0
pool 2 'rbd' rep size 3 crush_ruleset 2 object_hash rjenkins pg_num 64 pgp_num 64 last_change 8 owner 0
pool 3 '.rgw.buckets' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num 4800 pgp_num 4800 last_change 908 owner 0
pool 4 '.log' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num 1440 pgp_num 1440 last_change 19 owner 0
pool 5 '.rgw' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 21 owner 0
pool 6 '.users.uid' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 17 owner 0
pool 7 '.users' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 18 owner 0
pool 8 '.usage' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 158 owner 18446744073709551615
pool 9 '.intent-log' rep size 3 crush_ruleset 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 384 owner 18446744073709551615
max_osd 78
osd.0 up in weight 1 up_from 969 up_thru 1297 down_at 968 last_clean_interval [935,967) 10.177.64.4:6851/14555 10.177.64.4:6852/14555 10.177.64.4:6853/14555 exists,up 63cf7d70-99cb-0ab1-4004-000000000000
osd.1 up in weight 1 up_from 971 up_thru 1295 down_at 970 last_clean_interval [934,967) 10.177.64.4:6854/14899 10.177.64.4:6855/14899 10.177.64.4:6856/14899 exists,up 63cf7d70-99cb-0ab1-4004-000000000001
osd.2 up in weight 1 up_from 1006 up_thru 1294 down_at 990 last_clean_interval [962,982) 10.177.64.4:6812/28326 10.177.64.4:6813/28326 10.177.64.4:6814/28326 exists,up 63cf7d70-99cb-0ab1-4004-000000000002
osd.3 up in weight 1 up_from 1010 up_thru 1295 down_at 997 last_clean_interval [32,987) 10.177.64.4:6833/29830 10.177.64.4:6834/29830 10.177.64.4:6835/29830 exists,up 63cf7d70-99cb-0ab1-4004-000000000003
osd.4 up in weight 1 up_from 1010 up_thru 1295 down_at 997 last_clean_interval [37,987) 10.177.64.4:6836/30060 10.177.64.4:6837/30060 10.177.64.4:6838/30060 exists,up 63cf7d70-99cb-0ab1-4004-000000000004
osd.5 up in weight 1 up_from 1010 up_thru 1295 down_at 997 last_clean_interval [857,989) 10.177.64.4:6839/30282 10.177.64.4:6840/30282 10.177.64.4:6841/30282 exists,up 63cf7d70-99cb-0ab1-4004-000000000005
osd.6 up in weight 1 up_from 1008 up_thru 1296 down_at 997 last_clean_interval [929,990) 10.177.64.4:6842/31900 10.177.64.4:6843/31900 10.177.64.4:6844/31900 exists,up 63cf7d70-99cb-0ab1-4004-000000000006
osd.7 up in weight 1 up_from 1010 up_thru 1295 down_at 1001 last_clean_interval [55,992) 10.177.64.4:6845/389 10.177.64.4:6846/389 10.177.64.4:6847/389 exists,up 63cf7d70-99cb-0ab1-4004-000000000007
osd.8 up in weight 1 up_from 1010 up_thru 1295 down_at 1002 last_clean_interval [852,993) 10.177.64.4:6848/590 10.177.64.4:6849/590 10.177.64.4:6850/590 exists,up 63cf7d70-99cb-0ab1-4004-000000000008
osd.9 up in weight 1 up_from 1011 up_thru 1294 down_at 1001 last_clean_interval [66,993) 10.177.64.4:6863/686 10.177.64.4:6864/686 10.177.64.4:6865/686 exists,up 63cf7d70-99cb-0ab1-4004-000000000009
osd.10 up in weight 1 up_from 1129 up_thru 1294 down_at 1128 last_clean_interval [972,1127) 10.177.64.4:6866/852 10.177.64.4:6867/852 10.177.64.4:6868/852 exists,up 63cf7d70-99cb-0ab1-4004-00000000000a
osd.11 up in weight 1 up_from 974 up_thru 1295 down_at 973 last_clean_interval [937,970) 10.177.64.4:6860/16855 10.177.64.4:6861/16855 10.177.64.4:6862/16855 exists,up 63cf7d70-99cb-0ab1-4004-00000000000b
osd.12 up in weight 1 up_from 1233 up_thru 1295 down_at 1180 last_clean_interval [1150,1154) 10.177.64.4:6819/20558 10.177.64.4:6820/20558 10.177.64.4:6858/20558 exists,up 63cf7d70-99cb-0ab1-4004-00000000000c
osd.13 up in weight 1 up_from 977 up_thru 1295 down_at 976 last_clean_interval [940,973) 10.177.64.4:6882/18472 10.177.64.4:6883/18472 10.177.64.4:6884/18472 exists,up 63cf7d70-99cb-0ab1-4004-00000000000d
osd.14 up in weight 1 up_from 1132 up_thru 1295 down_at 1131 last_clean_interval [979,1128) 10.177.64.4:6873/2123 10.177.64.4:6874/2123 10.177.64.4:6875/2123 exists,up 63cf7d70-99cb-0ab1-4004-00000000000e
osd.15 up in weight 1 up_from 980 up_thru 1281 down_at 979 last_clean_interval [962,976) 10.177.64.4:6885/19586 10.177.64.4:6886/19586 10.177.64.4:6887/19586 exists,up 63cf7d70-99cb-0ab1-4004-00000000000f
osd.16 up in weight 1 up_from 1006 up_thru 1295 down_at 985 last_clean_interval [962,977) 10.177.64.4:6803/27926 10.177.64.4:6804/27926 10.177.64.4:6805/27926 exists,up 63cf7d70-99cb-0ab1-4004-000000000010
osd.17 up in weight 1 up_from 1005 up_thru 1294 down_at 986 last_clean_interval [962,979) 10.177.64.4:6806/28002 10.177.64.4:6807/28002 10.177.64.4:6808/28002 exists,up 63cf7d70-99cb-0ab1-4004-000000000011
osd.18 up in weight 1 up_from 1005 up_thru 1294 down_at 988 last_clean_interval [962,980) 10.177.64.4:6800/28095 10.177.64.4:6801/28095 10.177.64.4:6802/28095 exists,up 63cf7d70-99cb-0ab1-4004-000000000012
osd.19 up in weight 1 up_from 1006 up_thru 1295 down_at 992 last_clean_interval [962,981) 10.177.64.4:6809/28186 10.177.64.4:6810/28186 10.177.64.4:6811/28186 exists,up 63cf7d70-99cb-0ab1-4004-000000000013
osd.20 up in weight 1 up_from 1126 up_thru 1295 down_at 1125 last_clean_interval [1006,1124) 10.177.64.4:6827/23650 10.177.64.4:6828/23650 10.177.64.4:6829/23650 exists,up 63cf7d70-99cb-0ab1-4004-000000000014
osd.21 up in weight 1 up_from 1142 up_thru 1295 down_at 1141 last_clean_interval [1006,1140) 10.177.64.4:6815/5974 10.177.64.4:6816/5974 10.177.64.4:6817/5974 exists,up 63cf7d70-99cb-0ab1-4004-000000000015
osd.22 up in weight 1 up_from 1006 up_thru 1295 down_at 994 last_clean_interval [961,984) 10.177.64.4:6821/28684 10.177.64.4:6822/28684 10.177.64.4:6823/28684 exists,up 63cf7d70-99cb-0ab1-4004-000000000016
osd.23 up in weight 1 up_from 1007 up_thru 1295 down_at 994 last_clean_interval [963,984) 10.177.64.4:6824/29254 10.177.64.4:6825/29254 10.177.64.4:6826/29254 exists,up 63cf7d70-99cb-0ab1-4004-000000000017
osd.24 up in weight 1 up_from 991 up_thru 1295 down_at 990 last_clean_interval [149,984) 10.177.64.4:6872/22770 10.177.64.4:6893/22770 10.177.64.4:6900/22770 exists,up 63cf7d70-99cb-0ab1-4004-000000000018
osd.25 up in weight 1 up_from 1009 up_thru 1295 down_at 996 last_clean_interval [154,985) 10.177.64.4:6830/29688 10.177.64.4:6831/29688 10.177.64.4:6832/29688 exists,up 63cf7d70-99cb-0ab1-4004-000000000019
osd.26 up in weight 1 up_from 1069 up_thru 1293 down_at 1068 last_clean_interval [848,1067) 10.177.64.6:6833/28064 10.177.64.6:6837/28064 10.177.64.6:6838/28064 exists,up 63cf7d70-99cb-0ab1-4006-00000000001a
osd.27 up in weight 1 up_from 1071 up_thru 1295 down_at 1070 last_clean_interval [846,1067) 10.177.64.6:6851/28470 10.177.64.6:6852/28470 10.177.64.6:6853/28470 exists,up 63cf7d70-99cb-0ab1-4006-00000000001b
osd.28 up in weight 1 up_from 1073 up_thru 1294 down_at 1072 last_clean_interval [845,1067) 10.177.64.6:6860/28740 10.177.64.6:6882/28740 10.177.64.6:6883/28740 exists,up 63cf7d70-99cb-0ab1-4006-00000000001c
osd.29 up in weight 1 up_from 1072 up_thru 1293 down_at 1071 last_clean_interval [842,1067) 10.177.64.6:6884/29009 10.177.64.6:6885/29009 10.177.64.6:6886/29009 exists,up 63cf7d70-99cb-0ab1-4006-00000000001d
osd.30 up in weight 1 up_from 1221 up_thru 1294 down_at 1191 last_clean_interval [838,1068) 10.177.64.6:6863/30614 10.177.64.6:6864/30614 10.177.64.6:6865/30614 exists,up 63cf7d70-99cb-0ab1-4006-00000000001e
osd.31 up in weight 1 up_from 1076 up_thru 1295 down_at 1075 last_clean_interval [835,1068) 10.177.64.6:6893/30004 10.177.64.6:6894/30004 10.177.64.6:6895/30004 exists,up 63cf7d70-99cb-0ab1-4006-00000000001f
osd.32 up in weight 1 up_from 1078 up_thru 1294 down_at 1077 last_clean_interval [837,1071) 10.177.64.6:6896/31293 10.177.64.6:6897/31293 10.177.64.6:6898/31293 exists,up 63cf7d70-99cb-0ab1-4006-000000000020
osd.33 up in weight 1 up_from 1112 up_thru 1295 down_at 1081 last_clean_interval [834,1072) 10.177.64.6:6800/9543 10.177.64.6:6801/9543 10.177.64.6:6802/9543 exists,up 63cf7d70-99cb-0ab1-4006-000000000021
osd.34 up in weight 1 up_from 1113 up_thru 1295 down_at 1086 last_clean_interval [906,1073) 10.177.64.6:6803/9614 10.177.64.6:6804/9614 10.177.64.6:6805/9614 exists,up 63cf7d70-99cb-0ab1-4006-000000000022
osd.35 up in weight 1 up_from 1113 up_thru 1295 down_at 1086 last_clean_interval [830,1074) 10.177.64.6:6806/9686 10.177.64.6:6807/9686 10.177.64.6:6808/9686 exists,up 63cf7d70-99cb-0ab1-4006-000000000023
osd.36 up in weight 1 up_from 1113 up_thru 1295 down_at 1084 last_clean_interval [826,1076) 10.177.64.6:6809/9756 10.177.64.6:6810/9756 10.177.64.6:6811/9756 exists,up 63cf7d70-99cb-0ab1-4006-000000000024
osd.37 up in weight 1 up_from 1114 up_thru 1295 down_at 1086 last_clean_interval [828,1077) 10.177.64.6:6812/9826 10.177.64.6:6813/9826 10.177.64.6:6814/9826 exists,up 63cf7d70-99cb-0ab1-4006-000000000025
osd.38 up in weight 1 up_from 1113 up_thru 1295 down_at 1093 last_clean_interval [877,1078) 10.177.64.6:6815/9898 10.177.64.6:6816/9898 10.177.64.6:6817/9898 exists,up 63cf7d70-99cb-0ab1-4006-000000000026
osd.39 up in weight 1 up_from 1113 up_thru 1293 down_at 1091 last_clean_interval [822,1080) 10.177.64.6:6818/9969 10.177.64.6:6819/9969 10.177.64.6:6820/9969 exists,up 63cf7d70-99cb-0ab1-4006-000000000027
osd.40 up in weight 1 up_from 1113 up_thru 1295 down_at 1093 last_clean_interval [919,1080) 10.177.64.6:6821/10073 10.177.64.6:6822/10073 10.177.64.6:6823/10073 exists,up 63cf7d70-99cb-0ab1-4006-000000000028
osd.41 up in weight 1 up_from 1114 up_thru 1294 down_at 1093 last_clean_interval [923,1080) 10.177.64.6:6824/10155 10.177.64.6:6825/10155 10.177.64.6:6826/10155 exists,up 63cf7d70-99cb-0ab1-4006-000000000029
osd.42 up in weight 1 up_from 1113 up_thru 1295 down_at 1095 last_clean_interval [813,1082) 10.177.64.6:6827/10225 10.177.64.6:6828/10225 10.177.64.6:6829/10225 exists,up 63cf7d70-99cb-0ab1-4006-00000000002a
osd.43 up in weight 1 up_from 1089 up_thru 1295 down_at 1088 last_clean_interval [812,1083) 10.177.64.6:6832/4875 10.177.64.6:6899/4875 10.177.64.6:6900/4875 exists,up 63cf7d70-99cb-0ab1-4006-00000000002b
osd.44 up in weight 1 up_from 1113 up_thru 1293 down_at 1097 last_clean_interval [809,1084) 10.177.64.6:6830/10394 10.177.64.6:6831/10394 10.177.64.6:6834/10394 exists,up 63cf7d70-99cb-0ab1-4006-00000000002c
osd.45 up in weight 1 up_from 1114 up_thru 1293 down_at 1101 last_clean_interval [809,1085) 10.177.64.6:6835/10487 10.177.64.6:6836/10487 10.177.64.6:6839/10487 exists,up 63cf7d70-99cb-0ab1-4006-00000000002d
osd.46 up in weight 1 up_from 1113 up_thru 1295 down_at 1101 last_clean_interval [805,1087) 10.177.64.6:6840/10568 10.177.64.6:6841/10568 10.177.64.6:6842/10568 exists,up 63cf7d70-99cb-0ab1-4006-00000000002e
osd.47 up in weight 1 up_from 1114 up_thru 1293 down_at 1102 last_clean_interval [865,1088) 10.177.64.6:6843/10674 10.177.64.6:6844/10674 10.177.64.6:6845/10674 exists,up 63cf7d70-99cb-0ab1-4006-00000000002f
osd.48 up in weight 1 up_from 1114 up_thru 1293 down_at 1101 last_clean_interval [803,1089) 10.177.64.6:6846/10756 10.177.64.6:6847/10756 10.177.64.6:6848/10756 exists,up 63cf7d70-99cb-0ab1-4006-000000000030
osd.49 up in weight 1 up_from 1116 up_thru 1295 down_at 1101 last_clean_interval [801,1090) 10.177.64.6:6849/10892 10.177.64.6:6850/10892 10.177.64.6:6854/10892 exists,up 63cf7d70-99cb-0ab1-4006-000000000031
osd.50 up in weight 1 up_from 1115 up_thru 1294 down_at 1108 last_clean_interval [799,1091) 10.177.64.6:6855/11020 10.177.64.6:6856/11020 10.177.64.6:6857/11020 exists,up 63cf7d70-99cb-0ab1-4006-000000000032
osd.51 up in weight 1 up_from 1114 up_thru 1295 down_at 1107 last_clean_interval [796,1093) 10.177.64.6:6858/11103 10.177.64.6:6859/11103 10.177.64.6:6861/11103 exists,up 63cf7d70-99cb-0ab1-4006-000000000033
osd.52 up in weight 1 up_from 1016 up_thru 1295 down_at 1015 last_clean_interval [266,1014) 10.177.64.8:6818/13877 10.177.64.8:6837/13877 10.177.64.8:6838/13877 exists,up 63cf7d70-99cb-0ab1-4008-000000000034
osd.53 up in weight 1 up_from 1018 up_thru 1295 down_at 1017 last_clean_interval [270,1014) 10.177.64.8:6839/14277 10.177.64.8:6846/14277 10.177.64.8:6871/14277 exists,up 63cf7d70-99cb-0ab1-4008-000000000035
osd.54 up in weight 1 up_from 1019 up_thru 1293 down_at 1018 last_clean_interval [274,1015) 10.177.64.8:6872/14974 10.177.64.8:6874/14974 10.177.64.8:6886/14974 exists,up 63cf7d70-99cb-0ab1-4008-000000000036
osd.55 up in weight 1 up_from 1021 up_thru 1295 down_at 1020 last_clean_interval [278,1016) 10.177.64.8:6887/16478 10.177.64.8:6888/16478 10.177.64.8:6889/16478 exists,up 63cf7d70-99cb-0ab1-4008-000000000037
osd.56 up in weight 1 up_from 1022 up_thru 1295 down_at 1021 last_clean_interval [282,1017) 10.177.64.8:6890/17334 10.177.64.8:6891/17334 10.177.64.8:6892/17334 exists,up 63cf7d70-99cb-0ab1-4008-000000000038
osd.57 up in weight 1 up_from 1023 up_thru 1293 down_at 1022 last_clean_interval [902,1019) 10.177.64.8:6893/17978 10.177.64.8:6894/17978 10.177.64.8:6895/17978 exists,up 63cf7d70-99cb-0ab1-4008-000000000039
osd.58 up in weight 1 up_from 1162 up_thru 1295 down_at 1161 last_clean_interval [1024,1160) 10.177.64.8:6815/27677 10.177.64.8:6816/27677 10.177.64.8:6817/27677 exists,up 63cf7d70-99cb-0ab1-4008-00000000003a
osd.59 up in weight 1 up_from 1059 up_thru 1295 down_at 1026 last_clean_interval [294,1021) 10.177.64.8:6800/4433 10.177.64.8:6801/4433 10.177.64.8:6802/4433 exists,up 63cf7d70-99cb-0ab1-4008-00000000003b
osd.60 up in weight 1 up_from 1057 up_thru 1295 down_at 1030 last_clean_interval [298,1023) 10.177.64.8:6803/4515 10.177.64.8:6804/4515 10.177.64.8:6805/4515 exists,up 63cf7d70-99cb-0ab1-4008-00000000003c
osd.61 up in weight 1 up_from 1057 up_thru 1294 down_at 1031 last_clean_interval [302,1024) 10.177.64.8:6806/4586 10.177.64.8:6807/4586 10.177.64.8:6808/4586 exists,up 63cf7d70-99cb-0ab1-4008-00000000003d
osd.62 up in weight 1 up_from 1136 up_thru 1295 down_at 1135 last_clean_interval [1123,1134) 10.177.64.8:6809/2881 10.177.64.8:6810/2881 10.177.64.8:6811/2881 exists,up 63cf7d70-99cb-0ab1-4008-00000000003e
osd.63 up in weight 1 up_from 1058 up_thru 1294 down_at 1036 last_clean_interval [310,1025) 10.177.64.8:6812/4753 10.177.64.8:6813/4753 10.177.64.8:6814/4753 exists,up 63cf7d70-99cb-0ab1-4008-00000000003f
osd.64 up in weight 1 up_from 1159 up_thru 1295 down_at 1158 last_clean_interval [1058,1157) 10.177.64.8:6822/31981 10.177.64.8:6823/31981 10.177.64.8:6824/31981 exists,up 63cf7d70-99cb-0ab1-4008-000000000040
osd.65 up in weight 1 up_from 1058 up_thru 1295 down_at 1038 last_clean_interval [318,1025) 10.177.64.8:6819/4928 10.177.64.8:6820/4928 10.177.64.8:6821/4928 exists,up 63cf7d70-99cb-0ab1-4008-000000000041
osd.66 up in weight 1 up_from 1139 up_thru 1295 down_at 1138 last_clean_interval [1058,1137) 10.177.64.8:6866/4694 10.177.64.8:6867/4694 10.177.64.8:6869/4694 exists,up 63cf7d70-99cb-0ab1-4008-000000000042
osd.67 up in weight 1 up_from 1059 up_thru 1293 down_at 1039 last_clean_interval [328,1025) 10.177.64.8:6825/5148 10.177.64.8:6826/5148 10.177.64.8:6827/5148 exists,up 63cf7d70-99cb-0ab1-4008-000000000043
osd.68 up in weight 1 up_from 1059 up_thru 1299 down_at 1039 last_clean_interval [332,1025) 10.177.64.8:6828/5296 10.177.64.8:6829/5296 10.177.64.8:6830/5296 exists,up 63cf7d70-99cb-0ab1-4008-000000000044
osd.69 up in weight 1 up_from 1059 up_thru 1293 down_at 1040 last_clean_interval [337,1027) 10.177.64.8:6831/5377 10.177.64.8:6832/5377 10.177.64.8:6833/5377 exists,up 63cf7d70-99cb-0ab1-4008-000000000045
osd.70 up in weight 1 up_from 1058 up_thru 1295 down_at 1043 last_clean_interval [342,1028) 10.177.64.8:6834/5471 10.177.64.8:6835/5471 10.177.64.8:6836/5471 exists,up 63cf7d70-99cb-0ab1-4008-000000000046
osd.71 up in weight 1 up_from 1059 up_thru 1294 down_at 1051 last_clean_interval [346,1029) 10.177.64.8:6840/5619 10.177.64.8:6841/5619 10.177.64.8:6842/5619 exists,up 63cf7d70-99cb-0ab1-4008-000000000047
osd.72 up in weight 1 up_from 1059 up_thru 1293 down_at 1044 last_clean_interval [351,1030) 10.177.64.8:6843/5725 10.177.64.8:6844/5725 10.177.64.8:6845/5725 exists,up 63cf7d70-99cb-0ab1-4008-000000000048
osd.73 up in weight 1 up_from 1059 up_thru 1293 down_at 1044 last_clean_interval [356,1031) 10.177.64.8:6847/6265 10.177.64.8:6848/6265 10.177.64.8:6849/6265 exists,up 63cf7d70-99cb-0ab1-4008-000000000049
osd.74 up in weight 1 up_from 1145 up_thru 1295 down_at 1144 last_clean_interval [1059,1143) 10.177.64.8:6870/7632 10.177.64.8:6875/7632 10.177.64.8:6876/7632 exists,up 63cf7d70-99cb-0ab1-4008-00000000004a
osd.75 up in weight 1 up_from 1148 up_thru 1295 down_at 1147 last_clean_interval [1059,1143) 10.177.64.8:6877/8764 10.177.64.8:6878/8764 10.177.64.8:6879/8764 exists,up 63cf7d70-99cb-0ab1-4008-00000000004b
osd.76 up in weight 1 up_from 1061 up_thru 1295 down_at 1048 last_clean_interval [369,1034) 10.177.64.8:6856/6799 10.177.64.8:6857/6799 10.177.64.8:6858/6799 exists,up 63cf7d70-99cb-0ab1-4008-00000000004c
osd.77 up in weight 1 up_from 1059 up_thru 1295 down_at 1051 last_clean_interval [374,1035) 10.177.64.8:6859/6898 10.177.64.8:6860/6898 10.177.64.8:6861/6898 exists,up 63cf7d70-99cb-0ab1-4008-00000000004d
pg_temp 0.0 [45,52,28]
pg_temp 0.5 [21,47,25]
pg_temp 0.8 [62,12,6]
pg_temp 0.9 [46,68,35]
pg_temp 0.d [61,22,76]
pg_temp 0.13 [9,76,10]
pg_temp 0.15 [67,5,57]
pg_temp 0.16 [30,66,23]
pg_temp 0.1a [59,9,63]
pg_temp 0.1b [76,40,31]
pg_temp 0.29 [14,28,67]
pg_temp 0.32 [57,34,35]
pg_temp 0.3a [68,1,63]
pg_temp 0.3c [33,54,3]
pg_temp 0.3d [54,39,6]
pg_temp 0.3e [66,0,8]
pg_temp 3.2 [21,47,25]
pg_temp 3.5 [62,12,6]
pg_temp 3.6 [46,68,35]
pg_temp 3.a [61,22,76]
pg_temp 3.10 [9,76,10]
pg_temp 3.12 [67,5,57]
pg_temp 3.13 [30,66,23]
pg_temp 3.17 [59,9,63]
pg_temp 3.18 [76,40,31]
pg_temp 3.26 [14,28,67]
pg_temp 3.2f [57,34,35]
pg_temp 3.37 [68,1,63]
pg_temp 3.39 [33,54,3]
pg_temp 3.3a [54,39,6]
pg_temp 3.3b [66,0,8]
pg_temp 3.3d [58,30,35]
pg_temp 3.40 [29,5,11]
pg_temp 3.47 [69,7,74]
pg_temp 3.49 [55,35,62]
pg_temp 3.4b [31,21,1]
pg_temp 3.51 [53,49,8]
pg_temp 3.52 [50,55,7]
pg_temp 3.57 [14,27,47]
pg_temp 3.59 [31,13,9]
pg_temp 3.5e [35,4,26]
pg_temp 3.62 [56,4,76]
pg_temp 3.63 [7,50,34]
pg_temp 3.66 [34,6,14]
pg_temp 3.68 [74,41,28]
pg_temp 3.69 [7,43,2]
pg_temp 3.6d [47,16,0]
pg_temp 3.71 [68,28,1]
pg_temp 3.72 [5,50,22]
pg_temp 3.75 [44,53,14]
pg_temp 3.78 [4,39,13]
pg_temp 3.7b [26,3,28]
pg_temp 3.81 [69,50,1]
pg_temp 3.82 [34,20,31]
pg_temp 3.84 [35,75,16]
pg_temp 3.85 [43,75,22]
pg_temp 3.8b [70,17,73]
pg_temp 3.8c [51,10,77]
pg_temp 3.8f [38,12,1]
pg_temp 3.90 [2,28,1]
pg_temp 3.93 [24,43,12]
pg_temp 3.94 [8,31,62]
pg_temp 3.95 [9,39,18]
pg_temp 3.99 [76,30,2]
pg_temp 3.9a [54,28,64]
pg_temp 3.9c [19,45,48]
pg_temp 3.a0 [32,69,31]
pg_temp 3.a4 [13,32,40]
pg_temp 3.a5 [68,40,47]
pg_temp 3.a7 [28,64,32]
pg_temp 3.a9 [33,3,14]
pg_temp 3.ab [4,61,25]
pg_temp 3.ac [23,46,1]
pg_temp 3.ad [50,1,31]
pg_temp 3.ae [8,29,3]
pg_temp 3.b0 [1,58,60]
pg_temp 3.b1 [3,67,7]
pg_temp 3.b4 [16,76,26]
pg_temp 3.b5 [20,74,15]
pg_temp 3.bd [17,48,31]
pg_temp 3.be [0,66,9]
pg_temp 3.c0 [13,60,30]
pg_temp 3.c1 [10,49,6]
pg_temp 3.c2 [18,49,26]
pg_temp 3.c7 [4,59,24]
pg_temp 3.cc [73,31,58]
pg_temp 3.cd [63,23,16]
pg_temp 3.ce [3,66,69]
pg_temp 3.d0 [26,9,11]
pg_temp 3.d8 [10,68,0]
pg_temp 3.e0 [13,47,3]
pg_temp 3.e1 [37,69,28]
pg_temp 3.e4 [30,56,15]
pg_temp 3.e6 [47,76,17]
pg_temp 3.ed [63,3,60]
pg_temp 3.f1 [77,45,0]
pg_temp 3.f6 [2,56,1]
pg_temp 3.f8 [74,17,52]
pg_temp 3.fc [48,60,7]
pg_temp 3.103 [71,21,52]
pg_temp 3.105 [72,27,42]
pg_temp 3.106 [66,46,6]
pg_temp 3.107 [16,48,39]
pg_temp 3.109 [61,24,9]
pg_temp 3.10a [43,3,25]
pg_temp 3.10b [5,34,0]
pg_temp 3.111 [7,38,44]
pg_temp 3.113 [22,47,31]
pg_temp 3.115 [31,0,3]
pg_temp 3.11a [48,2,20]
pg_temp 3.11b [27,19,6]
pg_temp 3.11c [63,27,32]
pg_temp 3.11f [56,18,23]
pg_temp 3.121 [20,44,16]
pg_temp 3.123 [30,75,27]
pg_temp 3.125 [1,68,56]
pg_temp 3.126 [6,53,23]
pg_temp 3.12a [51,11,23]
pg_temp 3.12c [19,69,67]
pg_temp 3.12d [22,38,25]
pg_temp 3.12e [53,49,34]
pg_temp 3.12f [0,64,8]
pg_temp 3.132 [33,57,39]
pg_temp 3.133 [4,43,23]
pg_temp 3.135 [33,15,16]
pg_temp 3.136 [31,59,7]
pg_temp 3.137 [45,52,63]
pg_temp 3.13b [17,29,33]
pg_temp 3.13c [24,62,9]
pg_temp 3.13e [1,41,6]
pg_temp 3.141 [21,42,3]
pg_temp 3.14b [10,29,31]
pg_temp 3.14c [60,13,73]
pg_temp 3.14d [37,11,23]
pg_temp 3.150 [5,62,75]
pg_temp 3.151 [66,37,13]
pg_temp 3.155 [27,53,38]
pg_temp 3.156 [33,67,32]
pg_temp 3.159 [10,47,35]
pg_temp 3.160 [66,47,8]
pg_temp 3.162 [14,76,7]
pg_temp 3.163 [58,46,18]
pg_temp 3.165 [65,12,10]
pg_temp 3.167 [28,67,53]
pg_temp 3.16e [37,0,11]
pg_temp 3.179 [39,56,3]
pg_temp 3.17b [26,1,32]
pg_temp 3.17d [15,49,11]
pg_temp 3.17e [11,27,61]
pg_temp 3.17f [53,24,54]
pg_temp 3.185 [72,13,2]
pg_temp 3.186 [21,50,10]
pg_temp 3.18e [0,45,2]
pg_temp 3.18f [21,44,18]
pg_temp 3.190 [55,3,71]
pg_temp 3.193 [68,40,31]
pg_temp 3.197 [1,33,8]
pg_temp 3.19f [1,29,46]
pg_temp 3.1a2 [16,47,22]
pg_temp 3.1a6 [77,50,32]
pg_temp 3.1ac [36,9,25]
pg_temp 3.1b0 [50,71,68]
pg_temp 3.1b2 [26,55,20]
pg_temp 3.1b4 [40,22,4]
pg_temp 3.1b8 [65,24,14]
pg_temp 3.1ba [19,46,15]
pg_temp 3.1bc [16,56,61]
pg_temp 3.1bf [30,19,16]
pg_temp 3.1c0 [66,14,56]
pg_temp 3.1c1 [0,33,1]
pg_temp 3.1c4 [36,55,22]
pg_temp 3.1d0 [61,36,32]
pg_temp 3.1d7 [9,59,16]
pg_temp 3.1e0 [15,41,6]
pg_temp 3.1e1 [18,52,7]
pg_temp 3.1e3 [40,54,25]
pg_temp 3.1e4 [46,53,31]
pg_temp 3.1e7 [22,44,14]
pg_temp 3.1ec [63,37,24]
pg_temp 3.1f3 [1,61,24]
pg_temp 3.1f5 [58,32,28]
pg_temp 3.1f8 [12,77,6]
pg_temp 3.1fc [77,22,23]
pg_temp 3.1fd [13,55,7]
pg_temp 3.200 [72,2,0]
pg_temp 3.205 [63,22,8]
pg_temp 3.208 [25,34,5]
pg_temp 3.209 [4,33,38]
pg_temp 3.20c [0,27,22]
pg_temp 3.20e [48,10,13]
pg_temp 3.210 [27,64,28]
pg_temp 3.214 [48,6,22]
pg_temp 3.218 [36,69,10]
pg_temp 3.219 [76,73,74]
pg_temp 3.21d [7,26,20]
pg_temp 3.21e [43,15,8]
pg_temp 3.221 [53,27,45]
pg_temp 3.225 [10,31,58]
pg_temp 3.229 [52,31,29]
pg_temp 3.22b [70,5,63]
pg_temp 3.22c [17,77,24]
pg_temp 3.230 [12,54,1]
pg_temp 3.234 [23,46,4]
pg_temp 3.235 [17,68,22]
pg_temp 3.236 [65,18,57]
pg_temp 3.238 [76,4,61]
pg_temp 3.23b [70,21,13]
pg_temp 3.23f [20,53,3]
pg_temp 3.244 [6,47,38]
pg_temp 3.245 [72,40,49]
pg_temp 3.246 [5,64,10]
pg_temp 3.248 [29,5,17]
pg_temp 3.249 [20,63,13]
pg_temp 3.251 [25,59,11]
pg_temp 3.255 [24,36,7]
pg_temp 3.256 [18,60,15]
pg_temp 3.257 [3,64,22]
pg_temp 3.259 [33,53,0]
pg_temp 3.260 [66,12,0]
pg_temp 3.261 [48,20,5]
pg_temp 3.267 [3,39,7]
pg_temp 3.272 [77,23,67]
pg_temp 3.274 [48,13,22]
pg_temp 3.276 [2,72,21]
pg_temp 3.279 [64,29,51]
pg_temp 3.27d [49,59,33]
pg_temp 3.27e [53,4,71]
pg_temp 3.27f [47,23,25]
pg_temp 3.281 [53,27,5]
pg_temp 3.288 [30,12,4]
pg_temp 3.28c [34,25,9]
pg_temp 3.28d [26,7,0]
pg_temp 3.28e [21,75,18]
pg_temp 3.28f [75,6,56]
pg_temp 3.290 [11,61,2]
pg_temp 3.294 [75,35,51]
pg_temp 3.29d [44,2,11]
pg_temp 3.29e [68,10,74]
pg_temp 3.29f [36,66,17]
pg_temp 3.2a4 [6,50,26]
pg_temp 3.2a7 [27,61,8]
pg_temp 3.2a9 [46,64,25]
pg_temp 3.2aa [25,27,9]
pg_temp 3.2af [43,7,0]
pg_temp 3.2b0 [34,13,6]
pg_temp 3.2b1 [9,29,18]
pg_temp 3.2b6 [54,38,35]
pg_temp 3.2be [69,22,71]
pg_temp 3.2c0 [9,60,8]
pg_temp 3.2c3 [39,77,75]
pg_temp 3.2c6 [75,49,2]
pg_temp 3.2c7 [62,6,68]
pg_temp 3.2c9 [60,40,24]
pg_temp 3.2d2 [53,47,3]
pg_temp 3.2d8 [29,71,23]
pg_temp 3.2da [35,7,17]
pg_temp 3.2e3 [38,11,25]
pg_temp 3.2e9 [48,25,31]
pg_temp 3.2ec [25,51,2]
pg_temp 3.2ed [69,38,47]
pg_temp 3.2ee [2,64,23]
pg_temp 3.2f2 [63,34,25]
pg_temp 3.2f8 [13,74,16]
pg_temp 3.2f9 [50,14,1]
pg_temp 3.2fb [50,21,15]
pg_temp 3.2fc [64,39,25]
pg_temp 3.2fd [47,74,42]
pg_temp 3.300 [46,52,10]
pg_temp 3.307 [56,37,31]
pg_temp 3.30e [23,67,9]
pg_temp 3.310 [36,70,27]
pg_temp 3.311 [4,31,28]
pg_temp 3.315 [45,24,0]
pg_temp 3.318 [71,14,8]
pg_temp 3.319 [51,60,23]
pg_temp 3.31b [0,71,10]
pg_temp 3.322 [63,2,22]
pg_temp 3.325 [76,12,15]
pg_temp 3.326 [6,47,26]
pg_temp 3.327 [23,35,3]
pg_temp 3.32c [35,66,6]
pg_temp 3.330 [29,75,1]
pg_temp 3.331 [28,5,56]
pg_temp 3.332 [30,76,26]
pg_temp 3.333 [12,62,2]
pg_temp 3.335 [38,17,31]
pg_temp 3.336 [70,0,14]
pg_temp 3.346 [19,69,13]
pg_temp 3.347 [43,64,22]
pg_temp 3.34a [74,30,36]
pg_temp 3.34c [62,2,20]
pg_temp 3.359 [68,22,7]
pg_temp 3.35e [33,68,5]
pg_temp 3.361 [12,43,0]
pg_temp 3.364 [37,64,25]
pg_temp 3.367 [75,17,53]
pg_temp 3.368 [27,6,28]
pg_temp 3.369 [38,64,47]
pg_temp 3.36a [41,67,14]
pg_temp 3.36d [36,64,23]
pg_temp 3.370 [66,17,65]
pg_temp 3.372 [46,62,5]
pg_temp 3.374 [5,39,12]
pg_temp 3.375 [16,64,68]
pg_temp 3.377 [7,72,63]
pg_temp 3.379 [64,2,9]
pg_temp 3.37a [20,68,7]
pg_temp 3.37d [50,22,16]
pg_temp 3.381 [30,55,31]
pg_temp 3.38e [7,30,0]
pg_temp 3.397 [3,39,4]
pg_temp 3.39b [44,66,15]
pg_temp 3.39c [32,23,28]
pg_temp 3.39d [68,10,65]
pg_temp 3.39e [56,12,15]
pg_temp 3.39f [14,55,32]
pg_temp 3.3a5 [48,68,22]
pg_temp 3.3aa [45,74,14]
pg_temp 3.3ac [32,67,4]
pg_temp 3.3ad [4,34,7]
pg_temp 3.3b2 [45,66,11]
pg_temp 3.3b3 [34,62,1]
pg_temp 3.3bb [23,51,7]
pg_temp 3.3bc [42,57,72]
pg_temp 3.3c0 [29,71,0]
pg_temp 3.3c1 [39,15,21]
pg_temp 3.3c5 [47,54,31]
pg_temp 3.3c8 [26,19,8]
pg_temp 3.3c9 [77,38,15]
pg_temp 3.3cb [49,73,17]
pg_temp 3.3d0 [22,48,1]
pg_temp 3.3d5 [76,17,25]
pg_temp 3.3d8 [57,39,2]
pg_temp 3.3de [48,76,4]
pg_temp 3.3df [74,30,37]
pg_temp 3.3e5 [25,31,60]
pg_temp 3.3e6 [33,70,31]
pg_temp 3.3ec [71,50,43]
pg_temp 3.3f1 [71,33,45]
pg_temp 3.3f4 [52,1,30]
pg_temp 3.3fa [22,36,35]
pg_temp 3.3fb [26,18,4]
pg_temp 3.403 [22,26,49]
pg_temp 3.407 [6,44,35]
pg_temp 3.409 [42,15,4]
pg_temp 3.40c [77,43,68]
pg_temp 3.40e [21,59,66]
pg_temp 3.40f [10,61,66]
pg_temp 3.413 [55,24,73]
pg_temp 3.414 [64,6,59]
pg_temp 3.416 [68,43,5]
pg_temp 3.418 [30,66,31]
pg_temp 3.41b [55,44,4]
pg_temp 3.41e [15,67,55]
pg_temp 3.41f [62,1,23]
pg_temp 3.420 [69,32,35]
pg_temp 3.421 [71,7,63]
pg_temp 3.422 [25,68,53]
pg_temp 3.423 [38,15,1]
pg_temp 3.427 [58,31,53]
pg_temp 3.428 [10,41,8]
pg_temp 3.429 [7,52,40]
pg_temp 3.42c [24,58,7]
pg_temp 3.430 [16,65,15]
pg_temp 3.434 [56,11,7]
pg_temp 3.437 [55,0,18]
pg_temp 3.439 [45,55,13]
pg_temp 3.43f [49,70,6]
pg_temp 3.441 [75,36,16]
pg_temp 3.444 [12,76,5]
pg_temp 3.445 [75,8,46]
pg_temp 3.44a [69,29,0]
pg_temp 3.44c [73,39,23]
pg_temp 3.44d [1,71,24]
pg_temp 3.44e [42,71,22]
pg_temp 3.451 [10,27,1]
pg_temp 3.452 [57,44,32]
pg_temp 3.453 [71,3,69]
pg_temp 3.454 [50,53,24]
pg_temp 3.455 [74,32,31]
pg_temp 3.459 [13,41,12]
pg_temp 3.45f [48,5,16]
pg_temp 3.463 [68,30,34]
pg_temp 3.468 [35,55,77]
pg_temp 3.469 [29,16,2]
pg_temp 3.46b [28,7,11]
pg_temp 3.471 [12,59,4]
pg_temp 3.472 [14,49,32]
pg_temp 3.473 [18,30,24]
pg_temp 3.477 [61,23,56]
pg_temp 3.478 [53,41,50]
pg_temp 3.479 [35,3,20]
pg_temp 3.47d [36,74,22]
pg_temp 3.483 [19,42,7]
pg_temp 3.487 [64,48,22]
pg_temp 3.488 [54,45,9]
pg_temp 3.48f [1,54,5]
pg_temp 3.495 [2,60,11]
pg_temp 3.49b [35,22,20]
pg_temp 3.49d [11,53,67]
pg_temp 3.4a0 [57,18,23]
pg_temp 3.4a5 [45,75,34]
pg_temp 3.4a6 [12,41,16]
pg_temp 3.4ab [28,24,11]
pg_temp 3.4ad [54,22,0]
pg_temp 3.4b3 [55,19,66]
pg_temp 3.4b7 [1,32,43]
pg_temp 3.4b8 [63,26,49]
pg_temp 3.4ba [10,42,36]
pg_temp 3.4bb [10,62,0]
pg_temp 3.4be [21,27,11]
pg_temp 3.4c0 [54,40,22]
pg_temp 3.4c1 [50,72,10]
pg_temp 3.4ca [47,64,13]
pg_temp 3.4cb [72,23,67]
pg_temp 3.4cc [62,11,15]
pg_temp 3.4cd [0,77,5]
pg_temp 3.4cf [12,76,7]
pg_temp 3.4d1 [6,67,0]
pg_temp 3.4d3 [23,69,16]
pg_temp 3.4d5 [14,47,13]
pg_temp 3.4d8 [22,59,53]
pg_temp 3.4da [50,66,21]
pg_temp 3.4e4 [18,43,6]
pg_temp 3.4e5 [45,17,2]
pg_temp 3.4e6 [69,13,3]
pg_temp 3.4e8 [50,14,22]
pg_temp 3.4ea [1,31,14]
pg_temp 3.4eb [13,77,21]
pg_temp 3.4ee [64,6,60]
pg_temp 3.4f1 [21,43,17]
pg_temp 3.4f5 [24,39,13]
pg_temp 3.4f8 [61,51,12]
pg_temp 3.4f9 [11,29,38]
pg_temp 3.4fa [14,43,0]
pg_temp 3.4fb [63,10,52]
pg_temp 3.4fc [46,62,11]
pg_temp 3.500 [75,12,16]
pg_temp 3.501 [71,25,24]
pg_temp 3.504 [74,30,16]
pg_temp 3.506 [64,8,31]
pg_temp 3.512 [12,60,3]
pg_temp 3.51c [58,11,8]
pg_temp 3.523 [53,69,75]
pg_temp 3.52b [15,70,77]
pg_temp 3.532 [4,37,31]
pg_temp 3.533 [34,6,31]
pg_temp 3.539 [48,71,31]
pg_temp 3.53a [18,30,42]
pg_temp 3.546 [76,35,66]
pg_temp 3.54d [71,45,72]
pg_temp 3.54f [10,76,54]
pg_temp 3.551 [17,77,9]
pg_temp 3.555 [14,75,56]
pg_temp 3.557 [1,33,0]
pg_temp 3.558 [57,15,1]
pg_temp 3.559 [6,44,27]
pg_temp 3.55b [47,9,19]
pg_temp 3.560 [24,47,18]
pg_temp 3.562 [19,71,74]
pg_temp 3.566 [54,46,43]
pg_temp 3.568 [38,1,16]
pg_temp 3.570 [38,71,31]
pg_temp 3.572 [65,33,49]
pg_temp 3.575 [53,21,2]
pg_temp 3.578 [38,13,31]
pg_temp 3.582 [74,22,69]
pg_temp 3.58c [21,28,7]
pg_temp 3.58d [60,23,1]
pg_temp 3.592 [25,57,9]
pg_temp 3.595 [8,29,39]
pg_temp 3.596 [75,23,64]
pg_temp 3.599 [59,21,22]
pg_temp 3.59c [47,65,42]
pg_temp 3.59d [24,41,20]
pg_temp 3.59e [41,9,68]
pg_temp 3.5a5 [24,57,59]
pg_temp 3.5a6 [55,2,67]
pg_temp 3.5a7 [71,33,35]
pg_temp 3.5a8 [43,3,77]
pg_temp 3.5a9 [24,40,35]
pg_temp 3.5b0 [3,73,7]
pg_temp 3.5b1 [65,13,20]
pg_temp 3.5b5 [8,77,20]
pg_temp 3.5b8 [45,70,0]
pg_temp 3.5c1 [29,25,13]
pg_temp 3.5c2 [12,39,10]
pg_temp 3.5c4 [0,31,21]
pg_temp 3.5cd [61,43,60]
pg_temp 3.5ce [6,64,10]
pg_temp 3.5cf [54,11,67]
pg_temp 3.5d0 [62,48,28]
pg_temp 3.5d1 [6,41,9]
pg_temp 3.5d3 [33,72,2]
pg_temp 3.5d9 [12,71,3]
pg_temp 3.5db [45,76,0]
pg_temp 3.5de [53,27,4]
pg_temp 3.5df [65,27,1]
pg_temp 3.5e0 [27,52,49]
pg_temp 3.5ec [36,20,21]
pg_temp 3.5ef [61,16,56]
pg_temp 3.5f7 [40,0,2]
pg_temp 3.5f9 [45,23,28]
pg_temp 3.5fd [33,15,24]
pg_temp 3.5ff [71,7,11]
pg_temp 3.601 [8,76,16]
pg_temp 3.603 [35,68,6]
pg_temp 3.604 [24,37,6]
pg_temp 3.605 [66,36,25]
pg_temp 3.607 [28,67,74]
pg_temp 3.60b [54,5,53]
pg_temp 3.60c [45,1,5]
pg_temp 3.60e [72,45,10]
pg_temp 3.613 [68,46,0]
pg_temp 3.615 [29,55,33]
pg_temp 3.616 [33,59,50]
pg_temp 3.619 [1,41,20]
pg_temp 3.61a [0,33,10]
pg_temp 3.61b [61,5,15]
pg_temp 3.61c [2,44,34]
pg_temp 3.61e [1,72,9]
pg_temp 3.621 [10,30,42]
pg_temp 3.625 [12,34,11]
pg_temp 3.628 [40,18,31]
pg_temp 3.635 [22,37,25]
pg_temp 3.63a [4,47,35]
pg_temp 3.63b [42,54,13]
pg_temp 3.63e [11,69,6]
pg_temp 3.643 [47,69,16]
pg_temp 3.649 [47,4,10]
pg_temp 3.64e [52,2,4]
pg_temp 3.655 [28,22,16]
pg_temp 3.65a [57,45,56]
pg_temp 3.65f [18,43,4]
pg_temp 3.666 [70,19,57]
pg_temp 3.668 [52,33,7]
pg_temp 3.66b [75,16,58]
pg_temp 3.66e [55,26,19]
pg_temp 3.671 [59,8,16]
pg_temp 3.674 [64,47,42]
pg_temp 3.677 [63,41,0]
pg_temp 3.678 [36,54,7]
pg_temp 3.67c [47,19,31]
pg_temp 3.67d [12,47,23]
pg_temp 3.680 [23,30,43]
pg_temp 3.683 [10,62,64]
pg_temp 3.685 [3,76,7]
pg_temp 3.688 [9,51,8]
pg_temp 3.689 [22,29,16]
pg_temp 3.690 [43,11,3]
pg_temp 3.691 [56,22,73]
pg_temp 3.692 [42,57,21]
pg_temp 3.698 [1,62,2]
pg_temp 3.69a [32,13,1]
pg_temp 3.69b [14,30,9]
pg_temp 3.69e [28,60,34]
pg_temp 3.69f [9,39,0]
pg_temp 3.6a0 [24,40,34]
pg_temp 3.6a1 [55,1,53]
pg_temp 3.6a3 [4,26,6]
pg_temp 3.6a4 [22,69,0]
pg_temp 3.6a5 [67,17,53]
pg_temp 3.6a9 [67,36,26]
pg_temp 3.6ac [23,70,9]
pg_temp 3.6b3 [59,43,26]
pg_temp 3.6b4 [58,25,73]
pg_temp 3.6b5 [7,35,20]
pg_temp 3.6b8 [62,9,14]
pg_temp 3.6b9 [33,14,12]
pg_temp 3.6bc [32,77,19]
pg_temp 3.6bf [4,49,39]
pg_temp 3.6c2 [20,53,0]
pg_temp 3.6c4 [31,8,12]
pg_temp 3.6ce [42,11,31]
pg_temp 3.6cf [0,27,24]
pg_temp 3.6d3 [65,6,13]
pg_temp 3.6d6 [14,52,58]
pg_temp 3.6d8 [6,26,76]
pg_temp 3.6da [50,71,2]
pg_temp 3.6de [28,8,22]
pg_temp 3.6e1 [28,20,1]
pg_temp 3.6e4 [71,16,58]
pg_temp 3.6e6 [43,66,47]
pg_temp 3.6e7 [23,46,48]
pg_temp 3.6e9 [71,50,63]
pg_temp 3.6f2 [4,28,11]
pg_temp 3.6f7 [10,61,53]
pg_temp 3.6ff [38,3,57]
pg_temp 3.700 [34,14,31]
pg_temp 3.701 [34,75,1]
pg_temp 3.704 [63,3,76]
pg_temp 3.70b [65,0,18]
pg_temp 3.70c [2,70,14]
pg_temp 3.70e [27,66,35]
pg_temp 3.710 [53,17,21]
pg_temp 3.713 [32,53,19]
pg_temp 3.71a [51,77,3]
pg_temp 3.71b [32,71,11]
pg_temp 3.71c [30,68,48]
pg_temp 3.71d [55,42,74]
pg_temp 3.721 [30,20,22]
pg_temp 3.722 [59,35,0]
pg_temp 3.725 [20,45,27]
pg_temp 3.726 [37,10,3]
pg_temp 3.729 [7,38,69]
pg_temp 3.72a [2,73,3]
pg_temp 3.72b [57,40,4]
pg_temp 3.72c [77,34,39]
pg_temp 3.732 [24,71,59]
pg_temp 3.735 [62,34,3]
pg_temp 3.739 [23,70,71]
pg_temp 3.743 [28,23,3]
pg_temp 3.74c [3,68,2]
pg_temp 3.757 [1,76,25]
pg_temp 3.75a [75,7,69]
pg_temp 3.75f [64,18,10]
pg_temp 3.760 [31,14,51]
pg_temp 3.764 [49,16,21]
pg_temp 3.767 [9,31,4]
pg_temp 3.768 [23,56,18]
pg_temp 3.769 [2,36,21]
pg_temp 3.76c [73,12,4]
pg_temp 3.76d [46,19,0]
pg_temp 3.76f [6,70,65]
pg_temp 3.770 [14,46,5]
pg_temp 3.773 [8,70,2]
pg_temp 3.774 [46,21,8]
pg_temp 3.775 [19,59,48]
pg_temp 3.776 [43,5,54]
pg_temp 3.777 [34,20,26]
pg_temp 3.77b [65,51,34]
pg_temp 3.77f [75,51,12]
pg_temp 3.782 [7,38,9]
pg_temp 3.784 [61,30,39]
pg_temp 3.78c [23,47,57]
pg_temp 3.78d [21,44,1]
pg_temp 3.790 [57,6,10]
pg_temp 3.79b [24,71,1]
pg_temp 3.79c [5,59,57]
pg_temp 3.79f [27,63,17]
pg_temp 3.7a0 [29,15,5]
pg_temp 3.7a1 [8,49,33]
pg_temp 3.7a2 [49,19,2]
pg_temp 3.7a6 [58,41,24]
pg_temp 3.7a8 [32,14,25]
pg_temp 3.7ab [69,16,6]
pg_temp 3.7af [1,54,8]
pg_temp 3.7b5 [27,20,2]
pg_temp 3.7bc [72,26,1]
pg_temp 3.7c3 [59,41,2]
pg_temp 3.7c4 [44,60,12]
pg_temp 3.7c5 [12,50,4]
pg_temp 3.7c8 [51,53,28]
pg_temp 3.7cc [52,9,10]
pg_temp 3.7ce [50,70,22]
pg_temp 3.7d1 [62,39,58]
pg_temp 3.7d4 [34,63,14]
pg_temp 3.7d6 [7,29,15]
pg_temp 3.7d8 [66,15,22]
pg_temp 3.7df [43,60,4]
pg_temp 3.7e0 [21,47,18]
pg_temp 3.7e5 [64,23,1]
pg_temp 3.7e6 [43,16,22]
pg_temp 3.7e7 [64,21,61]
pg_temp 3.7e9 [44,21,7]
pg_temp 3.7f0 [27,65,3]
pg_temp 3.7f1 [2,67,0]
pg_temp 3.7f6 [42,19,3]
pg_temp 3.7f7 [11,36,18]
pg_temp 3.7fc [66,45,57]
pg_temp 3.7fd [65,4,52]
pg_temp 3.804 [53,0,16]
pg_temp 3.80a [47,56,19]
pg_temp 3.80d [32,14,13]
pg_temp 3.80e [21,65,11]
pg_temp 3.810 [0,57,18]
pg_temp 3.814 [39,2,11]
pg_temp 3.816 [41,56,35]
pg_temp 3.821 [55,46,14]
pg_temp 3.823 [53,41,31]
pg_temp 3.824 [40,63,50]
pg_temp 3.825 [45,22,25]
pg_temp 3.829 [62,41,35]
pg_temp 3.82c [67,17,68]
pg_temp 3.838 [46,74,38]
pg_temp 3.83a [54,36,22]
pg_temp 3.83c [42,12,0]
pg_temp 3.83d [56,37,24]
pg_temp 3.83f [66,12,33]
pg_temp 3.844 [21,62,10]
pg_temp 3.849 [30,67,21]
pg_temp 3.84a [42,5,8]
pg_temp 3.850 [7,41,21]
pg_temp 3.852 [25,73,12]
pg_temp 3.856 [11,33,13]
pg_temp 3.859 [34,7,24]
pg_temp 3.85c [47,73,16]
pg_temp 3.85d [29,54,32]
pg_temp 3.85e [24,3,10]
pg_temp 3.85f [7,52,73]
pg_temp 3.864 [12,26,19]
pg_temp 3.866 [60,36,24]
pg_temp 3.867 [74,34,26]
pg_temp 3.869 [17,26,1]
pg_temp 3.86a [60,42,56]
pg_temp 3.86e [77,24,14]
pg_temp 3.86f [67,9,61]
pg_temp 3.873 [16,72,63]
pg_temp 3.879 [6,27,34]
pg_temp 3.87a [30,16,1]
pg_temp 3.87b [34,16,2]
pg_temp 3.87d [71,16,13]
pg_temp 3.882 [47,77,25]
pg_temp 3.883 [24,39,4]
pg_temp 3.88a [43,75,34]
pg_temp 3.88d [10,65,14]
pg_temp 3.890 [38,64,1]
pg_temp 3.891 [55,49,6]
pg_temp 3.896 [44,58,18]
pg_temp 3.898 [42,70,8]
pg_temp 3.899 [47,54,31]
pg_temp 3.8a0 [28,56,8]
pg_temp 3.8a3 [47,70,24]
pg_temp 3.8a6 [7,36,9]
pg_temp 3.8ad [55,13,70]
pg_temp 3.8b0 [59,6,5]
pg_temp 3.8b2 [37,9,12]
pg_temp 3.8b5 [60,17,52]
pg_temp 3.8b6 [23,47,2]
pg_temp 3.8ba [53,41,9]
pg_temp 3.8bc [8,34,67]
pg_temp 3.8bd [66,9,73]
pg_temp 3.8c3 [5,56,77]
pg_temp 3.8c5 [25,67,20]
pg_temp 3.8ca [48,55,8]
pg_temp 3.8cf [9,52,4]
pg_temp 3.8d0 [42,16,26]
pg_temp 3.8d1 [29,62,26]
pg_temp 3.8d2 [39,14,6]
pg_temp 3.8d6 [6,39,14]
pg_temp 3.8d7 [42,7,23]
pg_temp 3.8d8 [8,60,2]
pg_temp 3.8db [31,76,7]
pg_temp 3.8de [16,72,17]
pg_temp 3.8df [22,58,10]
pg_temp 3.8e0 [32,55,4]
pg_temp 3.8e2 [32,6,18]
pg_temp 3.8e8 [67,28,33]
pg_temp 3.8ee [63,46,50]
pg_temp 3.8ef [53,6,25]
pg_temp 3.8f2 [34,18,6]
pg_temp 3.8fd [25,28,8]
pg_temp 3.900 [28,74,25]
pg_temp 3.901 [38,18,31]
pg_temp 3.902 [42,0,25]
pg_temp 3.904 [47,53,16]
pg_temp 3.908 [1,33,50]
pg_temp 3.909 [67,34,2]
pg_temp 3.90a [28,6,15]
pg_temp 3.90c [12,76,1]
pg_temp 3.90e [33,25,7]
pg_temp 3.90f [32,69,10]
pg_temp 3.912 [35,18,31]
pg_temp 3.913 [45,3,8]
pg_temp 3.918 [72,49,71]
pg_temp 3.91e [13,48,22]
pg_temp 3.921 [32,9,6]
pg_temp 3.923 [32,7,15]
pg_temp 3.92a [20,61,75]
pg_temp 3.92c [51,25,31]
pg_temp 3.92d [7,36,38]
pg_temp 3.92e [36,3,31]
pg_temp 3.92f [5,55,1]
pg_temp 3.930 [58,8,69]
pg_temp 3.931 [58,27,9]
pg_temp 3.932 [43,20,31]
pg_temp 3.936 [42,10,11]
pg_temp 3.938 [67,5,1]
pg_temp 3.93b [13,63,59]
pg_temp 3.940 [32,20,3]
pg_temp 3.943 [5,30,29]
pg_temp 3.949 [57,24,55]
pg_temp 3.94d [1,67,68]
pg_temp 3.94e [4,77,3]
pg_temp 3.950 [68,10,20]
pg_temp 3.952 [63,50,39]
pg_temp 3.958 [27,73,37]
pg_temp 3.959 [28,7,12]
pg_temp 3.95d [2,58,53]
pg_temp 3.95f [50,53,35]
pg_temp 3.966 [9,75,57]
pg_temp 3.967 [5,34,50]
pg_temp 3.96b [58,49,50]
pg_temp 3.96c [53,7,73]
pg_temp 3.96d [38,59,4]
pg_temp 3.975 [69,29,48]
pg_temp 3.97a [26,4,22]
pg_temp 3.97f [31,63,24]
pg_temp 3.981 [67,31,13]
pg_temp 3.983 [38,16,14]
pg_temp 3.987 [65,33,31]
pg_temp 3.98c [25,34,21]
pg_temp 3.98e [2,31,8]
pg_temp 3.990 [41,64,27]
pg_temp 3.992 [10,70,0]
pg_temp 3.994 [72,37,40]
pg_temp 3.9a1 [0,63,11]
pg_temp 3.9a5 [21,63,5]
pg_temp 3.9a6 [46,0,25]
pg_temp 3.9a9 [0,69,11]
pg_temp 3.9aa [29,19,12]
pg_temp 3.9ac [32,22,2]
pg_temp 3.9b0 [71,47,43]
pg_temp 3.9b1 [38,10,15]
pg_temp 3.9b5 [13,48,7]
pg_temp 3.9b8 [61,49,13]
pg_temp 3.9ba [22,55,15]
pg_temp 3.9bd [66,45,36]
pg_temp 3.9bf [18,48,4]
pg_temp 3.9c0 [73,23,54]
pg_temp 3.9c1 [36,73,10]
pg_temp 3.9c2 [35,21,20]
pg_temp 3.9cc [72,38,35]
pg_temp 3.9ce [17,72,69]
pg_temp 3.9d1 [22,50,15]
pg_temp 3.9d2 [17,34,10]
pg_temp 3.9d3 [36,0,19]
pg_temp 3.9d4 [62,39,57]
pg_temp 3.9d7 [21,77,8]
pg_temp 3.9d8 [15,54,9]
pg_temp 3.9da [39,17,16]
pg_temp 3.9df [13,63,3]
pg_temp 3.9e5 [52,22,19]
pg_temp 3.9e8 [57,6,7]
pg_temp 3.9ea [26,72,1]
pg_temp 3.9ee [41,13,6]
pg_temp 3.9f0 [8,45,5]
pg_temp 3.9f3 [11,47,27]
pg_temp 3.9f4 [56,38,39]
pg_temp 3.9f6 [32,20,10]
pg_temp 3.9f9 [22,47,33]
pg_temp 3.9fa [61,26,42]
pg_temp 3.9fc [66,25,65]
pg_temp 3.9ff [55,44,35]
pg_temp 3.a05 [53,9,77]
pg_temp 3.a06 [21,30,9]
pg_temp 3.a09 [56,64,72]
pg_temp 3.a0a [67,42,14]
pg_temp 3.a0b [43,55,73]
pg_temp 3.a0e [60,35,4]
pg_temp 3.a12 [9,34,11]
pg_temp 3.a13 [43,73,10]
pg_temp 3.a19 [13,69,6]
pg_temp 3.a1a [23,52,66]
pg_temp 3.a1b [57,6,52]
pg_temp 3.a24 [18,53,9]
pg_temp 3.a26 [42,6,31]
pg_temp 3.a29 [9,68,8]
pg_temp 3.a2b [8,52,18]
pg_temp 3.a37 [33,68,1]
pg_temp 3.a39 [27,60,20]
pg_temp 3.a3a [36,7,11]
pg_temp 3.a3e [73,2,7]
pg_temp 3.a40 [52,44,9]
pg_temp 3.a42 [40,67,43]
pg_temp 3.a43 [3,45,42]
pg_temp 3.a45 [66,3,68]
pg_temp 3.a47 [47,23,1]
pg_temp 3.a48 [72,7,64]
pg_temp 3.a51 [39,15,28]
pg_temp 3.a53 [4,45,70]
pg_temp 3.a55 [7,34,4]
pg_temp 3.a57 [74,26,3]
pg_temp 3.a59 [3,49,23]
pg_temp 3.a5c [1,59,9]
pg_temp 3.a67 [77,41,31]
pg_temp 3.a68 [26,18,31]
pg_temp 3.a6b [5,48,28]
pg_temp 3.a71 [40,54,50]
pg_temp 3.a77 [30,5,31]
pg_temp 3.a79 [14,69,24]
pg_temp 3.a7e [29,57,16]
pg_temp 3.a7f [10,56,48]
pg_temp 3.a80 [68,38,1]
pg_temp 3.a82 [25,72,61]
pg_temp 3.a85 [40,19,31]
pg_temp 3.a8b [44,68,1]
pg_temp 3.a93 [58,22,3]
pg_temp 3.a94 [76,16,13]
pg_temp 3.a96 [61,42,23]
pg_temp 3.a98 [34,65,14]
pg_temp 3.a99 [10,47,17]
pg_temp 3.aa4 [47,72,21]
pg_temp 3.aa7 [43,10,0]
pg_temp 3.aab [58,44,26]
pg_temp 3.ab1 [0,2,10]
pg_temp 3.ab2 [69,6,77]
pg_temp 3.ab3 [22,41,53]
pg_temp 3.ab5 [15,32,20]
pg_temp 3.ab7 [46,13,56]
pg_temp 3.ab9 [74,51,28]
pg_temp 3.aba [56,19,58]
pg_temp 3.abd [34,52,8]
pg_temp 3.abf [51,61,9]
pg_temp 3.ac0 [22,35,15]
pg_temp 3.ac3 [72,39,17]
pg_temp 3.ac5 [42,24,10]
pg_temp 3.ac9 [73,1,52]
pg_temp 3.aca [69,13,11]
pg_temp 3.acc [17,36,16]
pg_temp 3.ace [16,49,40]
pg_temp 3.ad0 [67,47,2]
pg_temp 3.ad4 [13,51,9]
pg_temp 3.ad5 [24,29,1]
pg_temp 3.ad9 [19,53,15]
pg_temp 3.add [25,54,11]
pg_temp 3.ae0 [26,58,12]
pg_temp 3.ae2 [50,15,7]
pg_temp 3.ae5 [76,11,15]
pg_temp 3.ae8 [70,19,6]
pg_temp 3.aee [66,17,9]
pg_temp 3.aef [64,8,7]
pg_temp 3.af1 [62,35,8]
pg_temp 3.af3 [47,65,12]
pg_temp 3.af4 [19,47,33]
pg_temp 3.af5 [3,26,4]
pg_temp 3.afd [66,41,6]
pg_temp 3.afe [73,21,2]
pg_temp 3.b02 [54,32,24]
pg_temp 3.b06 [37,9,25]
pg_temp 3.b09 [71,6,11]
pg_temp 3.b0c [52,34,38]
pg_temp 3.b0f [66,37,4]
pg_temp 3.b11 [18,44,4]
pg_temp 3.b16 [63,20,4]
pg_temp 3.b1a [30,69,36]
pg_temp 3.b1b [18,46,1]
pg_temp 3.b1c [22,60,19]
pg_temp 3.b1d [26,2,20]
pg_temp 3.b26 [3,44,71]
pg_temp 3.b27 [12,69,13]
pg_temp 3.b2a [4,77,63]
pg_temp 3.b2e [63,10,52]
pg_temp 3.b32 [58,51,50]
pg_temp 3.b38 [6,56,22]
pg_temp 3.b3f [69,9,24]
pg_temp 3.b41 [60,51,4]
pg_temp 3.b42 [31,57,73]
pg_temp 3.b44 [56,4,19]
pg_temp 3.b45 [16,38,11]
pg_temp 3.b4b [0,77,17]
pg_temp 3.b4e [38,70,28]
pg_temp 3.b56 [26,76,5]
pg_temp 3.b5c [41,63,31]
pg_temp 3.b60 [69,37,39]
pg_temp 3.b62 [54,6,3]
pg_temp 3.b63 [66,0,20]
pg_temp 3.b64 [43,61,65]
pg_temp 3.b66 [24,51,9]
pg_temp 3.b68 [73,30,26]
pg_temp 3.b69 [64,4,67]
pg_temp 3.b6a [26,1,7]
pg_temp 3.b6d [49,0,4]
pg_temp 3.b72 [23,33,1]
pg_temp 3.b79 [45,10,31]
pg_temp 3.b7a [57,30,37]
pg_temp 3.b7d [37,53,25]
pg_temp 3.b7f [74,34,9]
pg_temp 3.b81 [28,20,31]
pg_temp 3.b85 [77,47,42]
pg_temp 3.b8f [72,18,58]
pg_temp 3.b92 [26,54,17]
pg_temp 3.b93 [49,63,21]
pg_temp 3.b95 [35,23,6]
pg_temp 3.b97 [8,54,21]
pg_temp 3.b9c [66,22,52]
pg_temp 3.ba0 [22,62,1]
pg_temp 3.ba2 [61,1,58]
pg_temp 3.ba7 [40,2,1]
pg_temp 3.ba9 [73,47,42]
pg_temp 3.baa [42,21,31]
pg_temp 3.bb6 [76,17,77]
pg_temp 3.bb7 [27,74,35]
pg_temp 3.bba [73,24,62]
pg_temp 3.bbc [30,72,20]
pg_temp 3.bbe [77,47,3]
pg_temp 3.bc0 [8,59,14]
pg_temp 3.bc2 [20,44,2]
pg_temp 3.bc3 [76,23,59]
pg_temp 3.bc5 [52,24,5]
pg_temp 3.bcb [6,60,52]
pg_temp 3.bce [72,5,55]
pg_temp 3.bd2 [28,53,16]
pg_temp 3.bd4 [67,22,61]
pg_temp 3.bdc [31,67,61]
pg_temp 3.bde [11,61,2]
pg_temp 3.bdf [51,59,27]
pg_temp 3.be1 [65,28,16]
pg_temp 3.be6 [34,1,16]
pg_temp 3.be7 [42,5,22]
pg_temp 3.be9 [35,55,10]
pg_temp 3.bf2 [13,58,24]
pg_temp 3.bf5 [7,37,5]
pg_temp 3.bf8 [28,56,14]
pg_temp 3.bf9 [51,73,28]
pg_temp 3.bfb [10,36,0]
pg_temp 3.bfd [9,71,12]
pg_temp 3.bfe [64,3,16]
pg_temp 3.c00 [50,63,18]
pg_temp 3.c04 [23,49,7]
pg_temp 3.c0d [74,27,18]
pg_temp 3.c11 [15,32,48]
pg_temp 3.c13 [29,13,16]
pg_temp 3.c17 [66,29,39]
pg_temp 3.c18 [1,76,5]
pg_temp 3.c1d [11,70,12]
pg_temp 3.c1f [49,6,3]
pg_temp 3.c25 [15,77,17]
pg_temp 3.c29 [39,60,3]
pg_temp 3.c2a [35,59,14]
pg_temp 3.c2d [64,2,42]
pg_temp 3.c2f [20,68,18]
pg_temp 3.c36 [75,9,61]
pg_temp 3.c3b [36,55,20]
pg_temp 3.c3d [2,29,8]
pg_temp 3.c40 [50,59,24]
pg_temp 3.c44 [57,40,34]
pg_temp 3.c45 [45,71,23]
pg_temp 3.c49 [59,29,31]
pg_temp 3.c4a [50,75,23]
pg_temp 3.c5b [65,24,12]
pg_temp 3.c60 [5,40,3]
pg_temp 3.c63 [5,72,73]
pg_temp 3.c65 [65,39,24]
pg_temp 3.c6a [48,60,28]
pg_temp 3.c71 [59,9,74]
pg_temp 3.c73 [14,57,21]
pg_temp 3.c74 [45,69,38]
pg_temp 3.c7a [14,54,52]
pg_temp 3.c7b [54,7,24]
pg_temp 3.c7c [68,34,22]
pg_temp 3.c7f [70,40,9]
pg_temp 3.c81 [6,77,63]
pg_temp 3.c86 [14,36,29]
pg_temp 3.c88 [65,36,7]
pg_temp 3.c89 [29,59,31]
pg_temp 3.c8a [4,76,9]
pg_temp 3.c8b [52,8,12]
pg_temp 3.c8e [12,70,13]
pg_temp 3.c90 [75,35,39]
pg_temp 3.c93 [14,72,5]
pg_temp 3.c98 [58,35,6]
pg_temp 3.c9e [58,30,27]
pg_temp 3.c9f [38,66,18]
pg_temp 3.ca1 [52,31,53]
pg_temp 3.ca2 [39,76,41]
pg_temp 3.ca5 [39,19,10]
pg_temp 3.ca6 [65,43,8]
pg_temp 3.cb2 [74,34,31]
pg_temp 3.cb3 [25,56,71]
pg_temp 3.cb6 [20,41,67]
pg_temp 3.cc3 [66,12,3]
pg_temp 3.cc4 [64,15,52]
pg_temp 3.cc9 [55,1,5]
pg_temp 3.ccd [16,62,65]
pg_temp 3.cd3 [12,42,1]
pg_temp 3.cd4 [20,27,18]
pg_temp 3.cd5 [10,61,11]
pg_temp 3.cd8 [10,55,6]
pg_temp 3.cde [52,49,54]
pg_temp 3.ce0 [4,60,62]
pg_temp 3.ce3 [59,26,0]
pg_temp 3.ce4 [22,51,35]
pg_temp 3.ce5 [4,43,1]
pg_temp 3.ce6 [14,74,21]
pg_temp 3.ce9 [50,76,41]
pg_temp 3.ceb [23,63,10]
pg_temp 3.cf8 [61,40,42]
pg_temp 3.cfc [49,55,6]
pg_temp 3.cfe [19,31,22]
pg_temp 3.d05 [16,77,15]
pg_temp 3.d09 [33,53,39]
pg_temp 3.d19 [28,8,3]
pg_temp 3.d1d [1,42,47]
pg_temp 3.d1e [37,60,10]
pg_temp 3.d20 [4,38,3]
pg_temp 3.d21 [76,7,70]
pg_temp 3.d22 [67,5,66]
pg_temp 3.d23 [15,65,62]
pg_temp 3.d27 [54,11,76]
pg_temp 3.d29 [44,4,1]
pg_temp 3.d2e [43,18,31]
pg_temp 3.d32 [56,25,70]
pg_temp 3.d35 [72,34,15]
pg_temp 3.d3a [37,62,25]
pg_temp 3.d3b [46,70,2]
pg_temp 3.d3c [52,46,17]
pg_temp 3.d44 [55,1,20]
pg_temp 3.d48 [8,36,24]
pg_temp 3.d55 [56,27,4]
pg_temp 3.d56 [28,62,43]
pg_temp 3.d57 [77,34,37]
pg_temp 3.d5b [29,73,13]
pg_temp 3.d60 [5,52,38]
pg_temp 3.d63 [62,32,18]
pg_temp 3.d67 [23,61,7]
pg_temp 3.d6a [65,45,23]
pg_temp 3.d6f [22,27,41]
pg_temp 3.d70 [5,29,22]
pg_temp 3.d71 [13,68,3]
pg_temp 3.d72 [7,54,56]
pg_temp 3.d73 [18,77,25]
pg_temp 3.d7c [56,11,6]
pg_temp 3.d86 [50,64,12]
pg_temp 3.d88 [50,59,8]
pg_temp 3.d8a [21,35,40]
pg_temp 3.d8f [68,27,50]
pg_temp 3.d92 [4,29,1]
pg_temp 3.d94 [30,68,15]
pg_temp 3.d96 [22,65,67]
pg_temp 3.d99 [75,40,16]
pg_temp 3.d9e [77,6,23]
pg_temp 3.d9f [10,35,8]
pg_temp 3.da2 [66,11,12]
pg_temp 3.da4 [12,37,6]
pg_temp 3.da5 [26,67,35]
pg_temp 3.da6 [25,32,40]
pg_temp 3.da7 [4,63,9]
pg_temp 3.daa [28,13,25]
pg_temp 3.dad [0,30,18]
pg_temp 3.dae [30,4,3]
pg_temp 3.db3 [69,28,67]
pg_temp 3.db8 [63,23,7]
pg_temp 3.db9 [48,59,10]
pg_temp 3.dba [29,68,15]
pg_temp 3.dbb [51,56,31]
pg_temp 3.dc2 [20,34,39]
pg_temp 3.dc3 [9,69,62]
pg_temp 3.dc4 [51,59,6]
pg_temp 3.dc8 [53,29,1]
pg_temp 3.dca [27,12,2]
pg_temp 3.dcd [43,57,18]
pg_temp 3.dd0 [39,17,7]
pg_temp 3.dd2 [61,4,66]
pg_temp 3.dd5 [69,19,13]
pg_temp 3.dd7 [7,77,24]
pg_temp 3.ddf [73,24,61]
pg_temp 3.de0 [18,69,19]
pg_temp 3.de3 [20,48,28]
pg_temp 3.de7 [23,49,67]
pg_temp 3.deb [42,0,1]
pg_temp 3.ded [43,3,7]
pg_temp 3.df6 [8,53,3]
pg_temp 3.df7 [71,31,17]
pg_temp 3.dfc [28,57,37]
pg_temp 3.e01 [22,71,8]
pg_temp 3.e03 [41,66,6]
pg_temp 3.e04 [3,41,0]
pg_temp 3.e06 [22,73,53]
pg_temp 3.e08 [70,33,50]
pg_temp 3.e09 [48,73,6]
pg_temp 3.e0c [9,77,17]
pg_temp 3.e0e [6,56,59]
pg_temp 3.e11 [3,70,71]
pg_temp 3.e12 [56,34,50]
pg_temp 3.e14 [50,72,8]
pg_temp 3.e20 [0,62,24]
pg_temp 3.e25 [39,56,1]
pg_temp 3.e29 [11,63,62]
pg_temp 3.e38 [65,12,3]
pg_temp 3.e39 [72,43,49]
pg_temp 3.e41 [56,24,68]
pg_temp 3.e43 [39,14,31]
pg_temp 3.e4a [60,18,22]
pg_temp 3.e4c [26,61,42]
pg_temp 3.e4e [76,24,55]
pg_temp 3.e4f [7,64,48]
pg_temp 3.e52 [8,55,63]
pg_temp 3.e53 [48,55,17]
pg_temp 3.e55 [43,17,12]
pg_temp 3.e56 [16,55,18]
pg_temp 3.e57 [22,55,0]
pg_temp 3.e5a [11,39,25]
pg_temp 3.e5e [65,35,5]
pg_temp 3.e60 [71,22,6]
pg_temp 3.e68 [20,63,24]
pg_temp 3.e6c [35,6,8]
pg_temp 3.e6d [46,18,5]
pg_temp 3.e70 [10,74,4]
pg_temp 3.e7b [19,68,8]
pg_temp 3.e7c [3,29,34]
pg_temp 3.e7f [41,77,24]
pg_temp 3.e80 [30,71,15]
pg_temp 3.e88 [57,40,28]
pg_temp 3.e92 [74,41,8]
pg_temp 3.e97 [5,71,6]
pg_temp 3.e98 [36,66,0]
pg_temp 3.e9b [62,1,67]
pg_temp 3.ea1 [58,26,50]
pg_temp 3.ea3 [44,17,1]
pg_temp 3.eaa [16,71,74]
pg_temp 3.eab [53,32,0]
pg_temp 3.eaf [34,18,21]
pg_temp 3.eb4 [40,72,2]
pg_temp 3.eb5 [26,3,10]
pg_temp 3.eb8 [13,26,17]
pg_temp 3.ebc [69,37,21]
pg_temp 3.ec0 [69,11,13]
pg_temp 3.ec6 [12,30,14]
pg_temp 3.ece [40,8,0]
pg_temp 3.ecf [61,18,22]
pg_temp 3.ed3 [55,10,23]
pg_temp 3.ed6 [13,36,20]
pg_temp 3.ed7 [9,31,51]
pg_temp 3.edd [3,73,18]
pg_temp 3.ede [53,27,35]
pg_temp 3.ee2 [30,59,3]
pg_temp 3.ee3 [21,58,22]
pg_temp 3.ee5 [54,42,27]
pg_temp 3.ee6 [69,26,9]
pg_temp 3.eed [72,35,10]
pg_temp 3.eee [36,12,16]
pg_temp 3.ef0 [57,23,74]
pg_temp 3.ef6 [20,71,5]
pg_temp 3.ef8 [3,32,13]
pg_temp 3.efb [20,35,22]
pg_temp 3.efe [28,1,6]
pg_temp 3.f00 [38,0,1]
pg_temp 3.f01 [69,31,25]
pg_temp 3.f08 [63,48,17]
pg_temp 3.f0a [2,37,43]
pg_temp 3.f0d [37,25,0]
pg_temp 3.f0e [64,37,35]
pg_temp 3.f11 [59,17,60]
pg_temp 3.f12 [67,30,34]
pg_temp 3.f16 [23,47,16]
pg_temp 3.f1d [55,41,31]
pg_temp 3.f20 [32,55,19]
pg_temp 3.f24 [23,73,1]
pg_temp 3.f25 [15,39,41]
pg_temp 3.f28 [36,52,9]
pg_temp 3.f29 [53,34,10]
pg_temp 3.f2c [25,40,4]
pg_temp 3.f2d [10,54,75]
pg_temp 3.f33 [68,21,74]
pg_temp 3.f34 [47,3,25]
pg_temp 3.f38 [74,29,15]
pg_temp 3.f3a [6,40,49]
pg_temp 3.f44 [42,19,31]
pg_temp 3.f47 [61,47,26]
pg_temp 3.f49 [72,10,13]
pg_temp 3.f4d [35,9,8]
pg_temp 3.f4f [10,32,19]
pg_temp 3.f50 [39,74,7]
pg_temp 3.f52 [14,1,24]
pg_temp 3.f54 [16,76,58]
pg_temp 3.f55 [38,25,14]
pg_temp 3.f58 [3,32,25]
pg_temp 3.f59 [39,13,15]
pg_temp 3.f5d [68,32,42]
pg_temp 3.f62 [68,26,45]
pg_temp 3.f63 [24,49,44]
pg_temp 3.f64 [51,25,3]
pg_temp 3.f66 [43,6,24]
pg_temp 3.f69 [41,22,28]
pg_temp 3.f6c [20,76,15]
pg_temp 3.f6f [64,51,8]
pg_temp 3.f72 [12,47,8]
pg_temp 3.f73 [20,64,74]
pg_temp 3.f74 [41,53,4]
pg_temp 3.f75 [11,27,46]
pg_temp 3.f7a [14,29,46]
pg_temp 3.f7c [35,0,8]
pg_temp 3.f83 [77,30,26]
pg_temp 3.f85 [6,41,11]
pg_temp 3.f8a [17,56,24]
pg_temp 3.f8f [18,44,7]
pg_temp 3.f95 [19,67,69]
pg_temp 3.f9b [70,44,31]
pg_temp 3.f9f [50,74,27]
pg_temp 3.fa0 [68,41,5]
pg_temp 3.fa1 [37,75,45]
pg_temp 3.fa6 [23,26,27]
pg_temp 3.fab [54,27,3]
pg_temp 3.faf [3,50,43]
pg_temp 3.fb0 [31,71,38]
pg_temp 3.fb6 [33,56,17]
pg_temp 3.fb8 [16,76,58]
pg_temp 3.fbb [53,50,7]
pg_temp 3.fbc [40,21,31]
pg_temp 3.fbe [11,72,3]
pg_temp 3.fc0 [18,71,21]
pg_temp 3.fc1 [47,56,42]
pg_temp 3.fc5 [33,59,21]
pg_temp 3.fc6 [36,20,2]
pg_temp 3.fcd [23,70,17]
pg_temp 3.fd2 [53,24,76]
pg_temp 3.fd3 [37,24,10]
pg_temp 3.fd9 [17,53,16]
pg_temp 3.fdc [65,17,19]
pg_temp 3.fdd [12,55,6]
pg_temp 3.fde [73,26,42]
pg_temp 3.fe2 [9,40,26]
pg_temp 3.fe3 [14,48,4]
pg_temp 3.fe5 [54,1,23]
pg_temp 3.fea [0,42,2]
pg_temp 3.feb [64,46,42]
pg_temp 3.fec [23,43,49]
pg_temp 3.fef [19,55,11]
pg_temp 3.ff2 [33,12,11]
pg_temp 3.ff5 [29,7,12]
pg_temp 3.ffa [34,54,32]
pg_temp 3.ffb [45,13,11]
pg_temp 3.ffd [52,20,4]
pg_temp 3.ffe [74,25,55]
pg_temp 3.1000 [38,9,18]
pg_temp 3.1004 [8,59,61]
pg_temp 3.1007 [56,42,26]
pg_temp 3.100a [40,60,6]
pg_temp 3.1013 [52,12,9]
pg_temp 3.1014 [46,61,24]
pg_temp 3.101c [69,21,64]
pg_temp 3.101f [12,48,28]
pg_temp 3.1020 [20,41,0]
pg_temp 3.1021 [53,23,62]
pg_temp 3.1022 [70,2,16]
pg_temp 3.1023 [76,45,14]
pg_temp 3.1027 [61,10,13]
pg_temp 3.1028 [74,30,35]
pg_temp 3.102e [1,58,72]
pg_temp 3.102f [46,3,15]
pg_temp 3.1034 [75,29,21]
pg_temp 3.1038 [56,47,16]
pg_temp 3.103b [64,24,4]
pg_temp 3.103c [58,28,4]
pg_temp 3.1043 [37,71,27]
pg_temp 3.1045 [35,2,13]
pg_temp 3.1048 [21,75,49]
pg_temp 3.104a [53,37,38]
pg_temp 3.104b [61,3,72]
pg_temp 3.1050 [8,58,22]
pg_temp 3.1051 [77,14,20]
pg_temp 3.1054 [72,33,2]
pg_temp 3.1055 [47,65,25]
pg_temp 3.1056 [20,46,12]
pg_temp 3.105b [12,74,8]
pg_temp 3.105c [6,37,34]
pg_temp 3.105e [57,23,63]
pg_temp 3.1062 [73,36,31]
pg_temp 3.1066 [45,55,17]
pg_temp 3.106a [48,56,50]
pg_temp 3.106d [0,51,19]
pg_temp 3.1070 [51,13,7]
pg_temp 3.1071 [5,35,16]
pg_temp 3.1072 [44,24,15]
pg_temp 3.1073 [49,24,5]
pg_temp 3.1074 [35,9,17]
pg_temp 3.1076 [64,19,71]
pg_temp 3.1078 [54,41,46]
pg_temp 3.107a [37,63,18]
pg_temp 3.1086 [35,9,20]
pg_temp 3.1087 [56,48,12]
pg_temp 3.1088 [75,36,38]
pg_temp 3.108a [25,75,7]
pg_temp 3.108b [32,77,5]
pg_temp 3.108d [54,12,17]
pg_temp 3.1090 [66,24,9]
pg_temp 3.1095 [6,65,9]
pg_temp 3.1098 [56,15,72]
pg_temp 3.1099 [7,31,24]
pg_temp 3.109b [32,54,28]
pg_temp 3.109d [73,6,67]
pg_temp 3.10a0 [18,54,17]
pg_temp 3.10a3 [33,57,8]
pg_temp 3.10a4 [77,1,57]
pg_temp 3.10a7 [29,25,6]
pg_temp 3.10a8 [14,30,1]
pg_temp 3.10ae [70,16,2]
pg_temp 3.10af [10,27,35]
pg_temp 3.10b2 [7,28,65]
pg_temp 3.10b5 [20,68,61]
pg_temp 3.10be [59,40,42]
pg_temp 3.10c1 [18,35,4]
pg_temp 3.10c4 [18,53,25]
pg_temp 3.10c8 [38,0,17]
pg_temp 3.10cf [23,62,69]
pg_temp 3.10d1 [63,33,49]
pg_temp 3.10d9 [53,46,25]
pg_temp 3.10db [73,35,76]
pg_temp 3.10dc [4,62,3]
pg_temp 3.10e6 [54,26,39]
pg_temp 3.10eb [63,16,3]
pg_temp 3.10ed [27,64,20]
pg_temp 3.10ef [31,57,58]
pg_temp 3.10f1 [68,18,11]
pg_temp 3.10f5 [46,69,26]
pg_temp 3.10f6 [50,19,12]
pg_temp 3.10fd [9,71,20]
pg_temp 3.10fe [51,57,19]
pg_temp 3.1100 [13,42,0]
pg_temp 3.1104 [39,71,41]
pg_temp 3.1105 [62,48,50]
pg_temp 3.1106 [41,57,4]
pg_temp 3.1107 [12,73,10]
pg_temp 3.1108 [16,26,67]
pg_temp 3.110f [71,32,49]
pg_temp 3.1113 [21,29,43]
pg_temp 3.1114 [34,17,28]
pg_temp 3.1116 [26,72,4]
pg_temp 3.111a [3,77,1]
pg_temp 3.111c [47,4,59]
pg_temp 3.111e [73,27,20]
pg_temp 3.111f [16,77,13]
pg_temp 3.1121 [17,47,10]
pg_temp 3.1123 [18,39,46]
pg_temp 3.1125 [10,46,18]
pg_temp 3.1128 [29,17,27]
pg_temp 3.1129 [43,11,3]
pg_temp 3.112d [61,16,64]
pg_temp 3.1139 [10,33,46]
pg_temp 3.1143 [76,19,54]
pg_temp 3.1146 [67,9,0]
pg_temp 3.114a [67,48,49]
pg_temp 3.114c [76,42,4]
pg_temp 3.1157 [75,27,31]
pg_temp 3.115e [12,46,3]
pg_temp 3.1161 [20,33,3]
pg_temp 3.116c [55,28,18]
pg_temp 3.116f [8,56,9]
pg_temp 3.1170 [51,55,17]
pg_temp 3.1174 [13,60,15]
pg_temp 3.1178 [37,62,25]
pg_temp 3.117e [52,16,17]
pg_temp 3.117f [61,7,10]
pg_temp 3.1181 [14,54,19]
pg_temp 3.1188 [45,64,4]
pg_temp 3.118a [62,3,58]
pg_temp 3.118f [28,14,38]
pg_temp 3.1191 [52,47,31]
pg_temp 3.1193 [54,50,30]
pg_temp 3.1195 [61,40,3]
pg_temp 3.119e [48,73,27]
pg_temp 3.11a1 [53,41,21]
pg_temp 3.11a4 [50,9,25]
pg_temp 3.11a7 [37,12,2]
pg_temp 3.11b0 [69,24,6]
pg_temp 3.11b2 [14,28,4]
pg_temp 3.11b4 [67,11,12]
pg_temp 3.11b5 [62,14,29]
pg_temp 3.11b8 [48,71,1]
pg_temp 3.11bb [66,36,32]
pg_temp 3.11c1 [27,63,43]
pg_temp 3.11c8 [36,31,67]
pg_temp 3.11cc [46,53,7]
pg_temp 3.11cd [65,10,60]
pg_temp 3.11d0 [33,22,6]
pg_temp 3.11d1 [37,14,31]
pg_temp 3.11d2 [31,1,43]
pg_temp 3.11d9 [10,53,24]
pg_temp 3.11df [6,57,72]
pg_temp 3.11e6 [64,30,4]
pg_temp 3.11e9 [69,36,23]
pg_temp 3.11ea [75,1,24]
pg_temp 3.11eb [60,6,75]
pg_temp 3.11ec [73,4,2]
pg_temp 3.11ef [41,66,24]
pg_temp 3.11f2 [23,41,9]
pg_temp 3.11f5 [66,34,13]
pg_temp 3.11f6 [48,65,42]
pg_temp 3.11f9 [30,75,22]
pg_temp 3.11fb [45,3,13]
pg_temp 3.11fc [11,60,61]
pg_temp 3.11fd [15,50,27]
pg_temp 3.11ff [19,36,21]
pg_temp 3.1202 [65,6,15]
pg_temp 3.1204 [33,4,9]
pg_temp 3.1208 [62,36,28]
pg_temp 3.1210 [46,17,29]
pg_temp 3.1212 [17,76,16]
pg_temp 3.1214 [36,21,20]
pg_temp 3.1215 [72,52,68]
pg_temp 3.1216 [8,30,15]
pg_temp 3.121c [64,44,28]
pg_temp 3.1220 [17,62,74]
pg_temp 3.1222 [32,18,10]
pg_temp 3.1223 [38,77,2]
pg_temp 3.1226 [66,41,22]
pg_temp 3.122b [77,18,65]
pg_temp 3.122e [76,8,68]
pg_temp 3.122f [21,57,67]
pg_temp 3.1233 [64,17,52]
pg_temp 3.123a [12,46,5]
pg_temp 3.123b [17,60,4]
pg_temp 3.123d [10,37,7]
pg_temp 3.1245 [75,37,24]
pg_temp 3.1247 [15,68,16]
pg_temp 3.1249 [74,7,65]
pg_temp 3.124a [5,77,8]
pg_temp 3.1251 [70,14,6]
pg_temp 3.1253 [42,67,26]
pg_temp 3.125b [35,69,5]
pg_temp 3.125d [9,35,14]
pg_temp 3.1260 [30,18,3]
pg_temp 3.1261 [37,77,9]
pg_temp 3.1263 [13,47,16]
pg_temp 3.1265 [34,23,28]
pg_temp 3.1266 [29,52,37]
pg_temp 3.126e [24,39,2]
pg_temp 3.1271 [50,59,20]
pg_temp 3.1274 [24,59,62]
pg_temp 3.1277 [19,46,45]
pg_temp 3.1279 [58,0,5]
pg_temp 3.127b [54,44,17]
pg_temp 3.127c [6,59,57]
pg_temp 3.127d [55,15,10]
pg_temp 3.127f [31,20,27]
pg_temp 3.1280 [6,60,2]
pg_temp 3.1281 [73,25,21]
pg_temp 3.1282 [32,13,31]
pg_temp 3.128a [32,8,4]
pg_temp 3.128b [27,12,23]
pg_temp 3.128c [69,26,23]
pg_temp 3.1292 [2,72,0]
pg_temp 3.1298 [58,39,14]
pg_temp 3.129c [23,27,39]
pg_temp 3.129d [52,61,72]
pg_temp 3.12a7 [57,45,9]
pg_temp 3.12ab [14,51,18]
pg_temp 3.12ae [62,34,8]
pg_temp 3.12b0 [65,15,12]
pg_temp 3.12b6 [35,71,32]
pg_temp 3.12b7 [56,43,75]
pg_temp 3.12bf [31,65,19]
pg_temp 4.1 [21,47,25]
pg_temp 4.4 [62,12,6]
pg_temp 4.5 [46,68,35]
pg_temp 4.9 [61,22,76]
pg_temp 4.f [9,76,10]
pg_temp 4.11 [67,5,57]
pg_temp 4.12 [30,66,23]
pg_temp 4.16 [59,9,63]
pg_temp 4.17 [76,40,31]
pg_temp 4.25 [14,28,67]
pg_temp 4.2e [57,34,35]
pg_temp 4.36 [68,1,63]
pg_temp 4.38 [33,54,3]
pg_temp 4.39 [54,39,6]
pg_temp 4.3a [66,0,8]
pg_temp 4.3c [58,30,35]
pg_temp 4.3f [29,5,11]
pg_temp 4.46 [69,7,74]
pg_temp 4.48 [55,35,62]
pg_temp 4.4a [31,21,1]
pg_temp 4.50 [53,49,8]
pg_temp 4.51 [50,55,7]
pg_temp 4.56 [14,27,47]
pg_temp 4.58 [31,13,9]
pg_temp 4.5d [35,4,26]
pg_temp 4.61 [56,4,76]
pg_temp 4.62 [7,50,34]
pg_temp 4.65 [34,6,14]
pg_temp 4.67 [74,41,28]
pg_temp 4.68 [7,43,2]
pg_temp 4.6c [47,16,0]
pg_temp 4.70 [68,28,1]
pg_temp 4.71 [5,50,22]
pg_temp 4.74 [44,53,14]
pg_temp 4.77 [4,39,13]
pg_temp 4.7a [26,3,28]
pg_temp 4.80 [69,50,1]
pg_temp 4.81 [34,20,31]
pg_temp 4.83 [35,75,16]
pg_temp 4.84 [43,75,22]
pg_temp 4.8a [70,17,73]
pg_temp 4.8b [51,10,77]
pg_temp 4.8e [38,12,1]
pg_temp 4.8f [2,28,1]
pg_temp 4.92 [24,43,12]
pg_temp 4.93 [8,31,62]
pg_temp 4.94 [9,39,18]
pg_temp 4.98 [76,30,2]
pg_temp 4.99 [54,28,64]
pg_temp 4.9b [19,45,48]
pg_temp 4.9f [32,69,31]
pg_temp 4.a3 [13,32,40]
pg_temp 4.a4 [68,40,47]
pg_temp 4.a6 [28,64,32]
pg_temp 4.a8 [33,3,14]
pg_temp 4.aa [4,61,25]
pg_temp 4.ab [23,46,1]
pg_temp 4.ac [50,1,31]
pg_temp 4.ad [8,29,3]
pg_temp 4.af [1,58,60]
pg_temp 4.b0 [3,67,7]
pg_temp 4.b3 [16,76,26]
pg_temp 4.b4 [20,74,15]
pg_temp 4.bc [17,48,31]
pg_temp 4.bd [0,66,9]
pg_temp 4.bf [13,60,30]
pg_temp 4.c0 [10,49,6]
pg_temp 4.c1 [18,49,26]
pg_temp 4.c6 [4,59,24]
pg_temp 4.cb [73,31,58]
pg_temp 4.cc [63,23,16]
pg_temp 4.cd [3,66,69]
pg_temp 4.cf [26,9,11]
pg_temp 4.d7 [10,68,0]
pg_temp 4.df [13,47,3]
pg_temp 4.e0 [37,69,28]
pg_temp 4.e3 [30,56,15]
pg_temp 4.e5 [47,76,17]
pg_temp 4.ec [63,3,60]
pg_temp 4.f0 [77,45,0]
pg_temp 4.f5 [2,56,1]
pg_temp 4.f7 [74,17,52]
pg_temp 4.fb [48,60,7]
pg_temp 4.102 [71,21,52]
pg_temp 4.104 [72,27,42]
pg_temp 4.105 [66,46,6]
pg_temp 4.106 [16,48,39]
pg_temp 4.108 [61,24,9]
pg_temp 4.109 [43,3,25]
pg_temp 4.10a [5,34,0]
pg_temp 4.110 [7,38,44]
pg_temp 4.112 [22,47,31]
pg_temp 4.114 [31,0,3]
pg_temp 4.119 [48,2,20]
pg_temp 4.11a [27,19,6]
pg_temp 4.11b [63,27,32]
pg_temp 4.11e [56,18,23]
pg_temp 4.120 [20,44,16]
pg_temp 4.122 [30,75,27]
pg_temp 4.124 [1,68,56]
pg_temp 4.125 [6,53,23]
pg_temp 4.129 [51,11,23]
pg_temp 4.12b [19,69,67]
pg_temp 4.12c [22,38,25]
pg_temp 4.12d [53,49,34]
pg_temp 4.12e [0,64,8]
pg_temp 4.131 [33,57,39]
pg_temp 4.132 [4,43,23]
pg_temp 4.134 [33,15,16]
pg_temp 4.135 [31,59,7]
pg_temp 4.136 [45,52,63]
pg_temp 4.13a [17,29,33]
pg_temp 4.13b [24,62,9]
pg_temp 4.13d [1,41,6]
pg_temp 4.140 [21,42,3]
pg_temp 4.14a [10,29,31]
pg_temp 4.14b [60,13,73]
pg_temp 4.14c [37,11,23]
pg_temp 4.14f [5,62,75]
pg_temp 4.150 [66,37,13]
pg_temp 4.154 [27,53,38]
pg_temp 4.155 [33,67,35]
pg_temp 4.158 [10,47,35]
pg_temp 4.15f [66,47,8]
pg_temp 4.161 [14,76,7]
pg_temp 4.162 [58,46,18]
pg_temp 4.164 [65,12,10]
pg_temp 4.166 [28,67,53]
pg_temp 4.16d [37,0,11]
pg_temp 4.178 [39,56,3]
pg_temp 4.17a [26,1,32]
pg_temp 4.17c [15,49,11]
pg_temp 4.17d [11,27,61]
pg_temp 4.17e [53,24,54]
pg_temp 4.184 [72,13,2]
pg_temp 4.185 [21,50,10]
pg_temp 4.18d [0,45,2]
pg_temp 4.18e [21,44,18]
pg_temp 4.18f [55,3,71]
pg_temp 4.192 [68,40,31]
pg_temp 4.196 [1,33,8]
pg_temp 4.19e [1,29,46]
pg_temp 4.1a1 [16,47,22]
pg_temp 4.1a5 [77,50,32]
pg_temp 4.1ab [36,9,25]
pg_temp 4.1af [50,71,68]
pg_temp 4.1b1 [26,55,20]
pg_temp 4.1b3 [40,22,4]
pg_temp 4.1b7 [65,24,14]
pg_temp 4.1b9 [19,46,15]
pg_temp 4.1bb [16,56,61]
pg_temp 4.1be [30,19,16]
pg_temp 4.1bf [66,14,56]
pg_temp 4.1c0 [0,33,1]
pg_temp 4.1c3 [36,55,22]
pg_temp 4.1cf [61,36,32]
pg_temp 4.1d6 [9,59,16]
pg_temp 4.1df [15,41,6]
pg_temp 4.1e0 [18,52,7]
pg_temp 4.1e2 [40,54,25]
pg_temp 4.1e3 [46,53,31]
pg_temp 4.1e6 [22,44,14]
pg_temp 4.1eb [63,37,24]
pg_temp 4.1f2 [1,61,24]
pg_temp 4.1f4 [58,32,52]
pg_temp 4.1f7 [12,77,6]
pg_temp 4.1fb [77,22,23]
pg_temp 4.1fc [13,55,7]
pg_temp 4.1ff [72,2,0]
pg_temp 4.204 [63,22,8]
pg_temp 4.207 [25,34,5]
pg_temp 4.208 [4,33,38]
pg_temp 4.20b [0,27,22]
pg_temp 4.20d [48,10,13]
pg_temp 4.20f [27,64,28]
pg_temp 4.213 [48,6,22]
pg_temp 4.217 [36,69,10]
pg_temp 4.218 [76,73,74]
pg_temp 4.21c [7,26,20]
pg_temp 4.21d [43,15,8]
pg_temp 4.220 [53,27,45]
pg_temp 4.224 [10,31,58]
pg_temp 4.228 [52,31,29]
pg_temp 4.22a [70,5,63]
pg_temp 4.22b [17,77,24]
pg_temp 4.22f [12,54,1]
pg_temp 4.233 [23,46,4]
pg_temp 4.234 [17,68,22]
pg_temp 4.235 [65,18,57]
pg_temp 4.237 [76,4,61]
pg_temp 4.23a [70,21,13]
pg_temp 4.23e [20,53,3]
pg_temp 4.243 [6,47,38]
pg_temp 4.244 [72,40,49]
pg_temp 4.245 [5,64,10]
pg_temp 4.247 [29,5,17]
pg_temp 4.248 [20,63,13]
pg_temp 4.250 [25,59,11]
pg_temp 4.254 [24,36,7]
pg_temp 4.255 [18,60,15]
pg_temp 4.256 [3,64,22]
pg_temp 4.258 [33,53,0]
pg_temp 4.25f [66,12,0]
pg_temp 4.260 [48,20,5]
pg_temp 4.266 [3,39,7]
pg_temp 4.271 [77,23,67]
pg_temp 4.273 [48,13,22]
pg_temp 4.275 [2,72,21]
pg_temp 4.278 [64,29,51]
pg_temp 4.27c [49,59,33]
pg_temp 4.27d [53,4,71]
pg_temp 4.27e [47,23,25]
pg_temp 4.280 [53,27,5]
pg_temp 4.287 [30,12,4]
pg_temp 4.28b [34,25,9]
pg_temp 4.28c [26,7,0]
pg_temp 4.28d [21,75,18]
pg_temp 4.28e [75,6,56]
pg_temp 4.28f [11,61,2]
pg_temp 4.293 [75,35,51]
pg_temp 4.29c [44,2,11]
pg_temp 4.29d [68,10,74]
pg_temp 4.29e [36,66,17]
pg_temp 4.2a3 [6,50,26]
pg_temp 4.2a6 [27,61,8]
pg_temp 4.2a8 [46,64,25]
pg_temp 4.2a9 [25,27,9]
pg_temp 4.2ae [43,7,0]
pg_temp 4.2af [34,13,6]
pg_temp 4.2b0 [9,29,18]
pg_temp 4.2b5 [54,38,35]
pg_temp 4.2bd [69,22,71]
pg_temp 4.2bf [9,60,8]
pg_temp 4.2c2 [39,77,75]
pg_temp 4.2c5 [75,49,2]
pg_temp 4.2c6 [62,6,68]
pg_temp 4.2c8 [60,40,24]
pg_temp 4.2d1 [53,47,3]
pg_temp 4.2d7 [29,71,23]
pg_temp 4.2d9 [35,7,17]
pg_temp 4.2e2 [38,11,25]
pg_temp 4.2e8 [48,25,31]
pg_temp 4.2eb [25,51,2]
pg_temp 4.2ec [69,38,47]
pg_temp 4.2ed [2,64,23]
pg_temp 4.2f1 [63,34,25]
pg_temp 4.2f7 [13,74,16]
pg_temp 4.2f8 [50,14,1]
pg_temp 4.2fa [50,21,15]
pg_temp 4.2fb [64,39,25]
pg_temp 4.2fc [47,74,42]
pg_temp 4.2ff [46,52,10]
pg_temp 4.306 [56,37,31]
pg_temp 4.30d [23,67,9]
pg_temp 4.30f [36,70,27]
pg_temp 4.310 [4,31,28]
pg_temp 4.314 [45,24,0]
pg_temp 4.317 [71,14,8]
pg_temp 4.318 [51,60,23]
pg_temp 4.31a [0,71,10]
pg_temp 4.321 [63,2,22]
pg_temp 4.324 [76,12,15]
pg_temp 4.325 [6,47,26]
pg_temp 4.326 [23,35,3]
pg_temp 4.32b [35,66,6]
pg_temp 4.32f [29,75,1]
pg_temp 4.330 [28,5,56]
pg_temp 4.331 [30,76,26]
pg_temp 4.332 [12,62,2]
pg_temp 4.334 [38,17,31]
pg_temp 4.335 [70,0,14]
pg_temp 4.345 [19,69,13]
pg_temp 4.346 [43,64,22]
pg_temp 4.349 [74,30,36]
pg_temp 4.34b [62,2,20]
pg_temp 4.358 [68,22,7]
pg_temp 4.35d [33,68,25]
pg_temp 4.360 [12,43,5]
pg_temp 4.363 [37,64,25]
pg_temp 4.366 [75,17,53]
pg_temp 4.367 [27,6,28]
pg_temp 4.368 [38,64,47]
pg_temp 4.369 [41,67,14]
pg_temp 4.36c [36,64,23]
pg_temp 4.36f [66,17,65]
pg_temp 4.371 [46,62,5]
pg_temp 4.373 [5,39,12]
pg_temp 4.374 [16,64,68]
pg_temp 4.376 [7,72,63]
pg_temp 4.378 [64,2,9]
pg_temp 4.379 [20,68,7]
pg_temp 4.37c [50,22,16]
pg_temp 4.380 [30,55,31]
pg_temp 4.38d [7,30,0]
pg_temp 4.396 [3,39,4]
pg_temp 4.39a [44,66,15]
pg_temp 4.39b [32,23,28]
pg_temp 4.39c [68,10,65]
pg_temp 4.39d [56,12,15]
pg_temp 4.39e [14,55,32]
pg_temp 4.3a4 [48,68,22]
pg_temp 4.3a9 [45,74,14]
pg_temp 4.3ab [32,67,4]
pg_temp 4.3ac [4,34,7]
pg_temp 4.3b1 [45,66,11]
pg_temp 4.3b2 [34,62,1]
pg_temp 4.3ba [23,51,7]
pg_temp 4.3bb [42,57,72]
pg_temp 4.3bf [29,71,0]
pg_temp 4.3c0 [39,15,21]
pg_temp 4.3c4 [47,54,31]
pg_temp 4.3c7 [26,19,8]
pg_temp 4.3c8 [77,38,15]
pg_temp 4.3ca [49,73,17]
pg_temp 4.3cf [22,48,1]
pg_temp 4.3d4 [76,17,25]
pg_temp 4.3d7 [57,39,2]
pg_temp 4.3dd [48,76,4]
pg_temp 4.3de [74,30,37]
pg_temp 4.3e4 [25,31,60]
pg_temp 4.3e5 [33,70,31]
pg_temp 4.3eb [71,50,43]
pg_temp 4.3f0 [71,33,45]
pg_temp 4.3f3 [52,1,30]
pg_temp 4.3f9 [22,36,35]
pg_temp 4.3fa [26,18,4]
pg_temp 4.402 [22,26,49]
pg_temp 4.406 [6,44,35]
pg_temp 4.408 [42,15,4]
pg_temp 4.40b [77,43,68]
pg_temp 4.40d [21,59,66]
pg_temp 4.40e [10,61,66]
pg_temp 4.412 [55,24,73]
pg_temp 4.413 [64,6,59]
pg_temp 4.415 [68,43,5]
pg_temp 4.417 [30,66,31]
pg_temp 4.41a [55,44,4]
pg_temp 4.41d [15,67,55]
pg_temp 4.41e [62,1,23]
pg_temp 4.41f [69,32,35]
pg_temp 4.420 [71,7,63]
pg_temp 4.421 [25,68,53]
pg_temp 4.422 [38,15,1]
pg_temp 4.426 [58,31,53]
pg_temp 4.427 [10,41,8]
pg_temp 4.428 [7,52,40]
pg_temp 4.42b [24,58,7]
pg_temp 4.42f [16,65,15]
pg_temp 4.433 [56,11,7]
pg_temp 4.436 [55,0,18]
pg_temp 4.438 [45,55,13]
pg_temp 4.43e [49,70,6]
pg_temp 4.440 [75,36,16]
pg_temp 4.443 [12,76,5]
pg_temp 4.444 [75,8,46]
pg_temp 4.449 [69,29,0]
pg_temp 4.44b [73,39,23]
pg_temp 4.44c [1,71,24]
pg_temp 4.44d [42,71,22]
pg_temp 4.450 [10,27,1]
pg_temp 4.451 [57,44,32]
pg_temp 4.452 [71,3,69]
pg_temp 4.453 [50,53,24]
pg_temp 4.454 [74,32,31]
pg_temp 4.458 [13,41,12]
pg_temp 4.45e [48,5,16]
pg_temp 4.462 [68,30,34]
pg_temp 4.467 [35,55,77]
pg_temp 4.468 [29,16,2]
pg_temp 4.46a [28,7,11]
pg_temp 4.470 [12,59,4]
pg_temp 4.471 [14,49,32]
pg_temp 4.472 [18,30,24]
pg_temp 4.476 [61,23,56]
pg_temp 4.477 [53,41,50]
pg_temp 4.478 [35,3,20]
pg_temp 4.47c [36,74,22]
pg_temp 4.482 [19,42,7]
pg_temp 4.486 [64,48,22]
pg_temp 4.487 [54,45,9]
pg_temp 4.48e [1,54,5]
pg_temp 4.494 [2,60,11]
pg_temp 4.49a [35,22,20]
pg_temp 4.49c [11,53,67]
pg_temp 4.49f [57,18,23]
pg_temp 4.4a4 [45,75,34]
pg_temp 4.4a5 [12,41,16]
pg_temp 4.4aa [28,24,11]
pg_temp 4.4ac [54,22,0]
pg_temp 4.4b2 [55,19,66]
pg_temp 4.4b6 [1,32,43]
pg_temp 4.4b7 [63,26,49]
pg_temp 4.4b9 [10,42,36]
pg_temp 4.4ba [10,62,0]
pg_temp 4.4bd [21,27,11]
pg_temp 4.4bf [54,40,22]
pg_temp 4.4c0 [50,72,10]
pg_temp 4.4c9 [47,64,13]
pg_temp 4.4ca [72,23,67]
pg_temp 4.4cb [62,11,15]
pg_temp 4.4cc [0,77,5]
pg_temp 4.4ce [12,76,7]
pg_temp 4.4d0 [6,67,0]
pg_temp 4.4d2 [23,69,16]
pg_temp 4.4d4 [14,47,13]
pg_temp 4.4d7 [22,59,53]
pg_temp 4.4d9 [50,66,21]
pg_temp 4.4e3 [18,43,6]
pg_temp 4.4e4 [45,17,2]
pg_temp 4.4e5 [69,13,3]
pg_temp 4.4e7 [50,14,22]
pg_temp 4.4e9 [1,31,14]
pg_temp 4.4ea [13,77,21]
pg_temp 4.4ed [64,6,60]
pg_temp 4.4f0 [21,43,17]
pg_temp 4.4f4 [24,39,13]
pg_temp 4.4f7 [61,51,12]
pg_temp 4.4f8 [11,29,38]
pg_temp 4.4f9 [14,43,0]
pg_temp 4.4fa [63,10,52]
pg_temp 4.4fb [46,62,11]
pg_temp 4.4ff [75,12,16]
pg_temp 4.500 [71,25,24]
pg_temp 4.503 [74,30,16]
pg_temp 4.505 [64,8,31]
pg_temp 4.511 [12,60,3]
pg_temp 4.51b [58,11,8]
pg_temp 4.522 [53,69,75]
pg_temp 4.52a [15,70,77]
pg_temp 4.531 [4,37,31]
pg_temp 4.532 [34,6,31]
pg_temp 4.538 [48,71,31]
pg_temp 4.539 [18,30,42]
pg_temp 4.545 [76,35,66]
pg_temp 4.54c [71,45,72]
pg_temp 4.54e [10,76,54]
pg_temp 4.550 [17,77,9]
pg_temp 4.554 [14,75,56]
pg_temp 4.556 [1,33,0]
pg_temp 4.557 [57,15,1]
pg_temp 4.558 [6,44,27]
pg_temp 4.55a [47,9,19]
pg_temp 4.55f [24,47,18]
pg_temp 4.561 [19,71,74]
pg_temp 4.565 [54,46,43]
pg_temp 4.567 [38,1,16]
pg_temp 4.56f [38,71,31]
pg_temp 4.571 [65,33,49]
pg_temp 4.574 [53,21,2]
pg_temp 4.577 [38,13,31]
pg_temp 4.581 [74,22,69]
pg_temp 4.58b [21,28,7]
pg_temp 4.58c [60,23,1]
pg_temp 4.591 [25,57,9]
pg_temp 4.594 [8,29,39]
pg_temp 4.595 [75,23,64]
pg_temp 4.598 [59,21,22]
pg_temp 4.59b [47,65,42]
pg_temp 4.59c [24,41,20]
pg_temp 4.59d [41,9,68]
pg_temp 5.0 [21,47,25]
pg_temp 5.3 [62,12,6]
pg_temp 5.4 [46,68,35]
pg_temp 6.2 [62,12,6]
pg_temp 6.3 [46,68,35]
pg_temp 6.7 [61,22,76]
pg_temp 7.1 [62,12,6]
pg_temp 7.2 [46,68,35]
pg_temp 7.6 [61,22,76]
pg_temp 8.0 [62,77]
pg_temp 9.0 [46,68,35]
pg_temp 9.4 [61,22,76]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Ceph remap/recovery stuck
2012-08-23 13:52 Ceph remap/recovery stuck Sławomir Skowron
@ 2012-08-24 11:54 ` Sławomir Skowron
2012-08-24 16:35 ` Sage Weil
0 siblings, 1 reply; 4+ messages in thread
From: Sławomir Skowron @ 2012-08-24 11:54 UTC (permalink / raw)
To: ceph-devel
I have found workaround.
Change CRUSH to replication to osd in rule for this pool, and after
recovery, remapped data, i just change same rule into rack awarenes,
and whole cluster, recover again, and back to normal.
Is there any way, to start refill, recovery in this situation for this
specyfic OSD ??
On Thu, Aug 23, 2012 at 3:52 PM, Sławomir Skowron <szibis@gmail.com> wrote:
> 3 osd after crash rebuilds ok, but rebuild of two more osd (12 and
> 30), i can't make cluster to be active+clean
>
> I do rebuild like in doc:
>
> stop osd,
> remove from crush,
> rm from map,
> recreate a osd, after cluster get stable
>
> But now, all osd are in, and up, and data won't remap, and some of PG,
> have only two osd in chain with replication level 3 for this pool.
>
> 2012-08-23 15:26:46.073685 mon.0 [INF] pgmap v117192: 6472 pgs: 63
> active, 4457 active+clean, 1942 active+remapped, 10 active+degraded;
> 596 GB data, 1650 GB used, 20059 GB / 21710 GB avail; 57815/4705888
> degraded (1.229%)
>
> In attachment output from:
>
> ceph osd dump -o -
>
> I can't find any info in doc for this situation.
>
> HEALTH_WARN 10 pgs degraded; 2015 pgs stuck unclean; recovery
> 57871/4706179 degraded (1.230%)
> root@s3-10-177-64-6:~# ceph -s
> health HEALTH_WARN 10 pgs degraded; 2015 pgs stuck unclean;
> recovery 57871/4706179 degraded (1.230%)
> monmap e4: 3 mons at
> {0=10.177.64.4:6789/0,1=10.177.64.6:6789/0,2=10.177.64.8:6789/0},
> election epoch 16, quorum 0,1,2 0,1,2
> osdmap e1300: 78 osds: 78 up, 78 in
> pgmap v117464: 6472 pgs: 63 active, 4457 active+clean, 1942
> active+remapped, 10 active+degraded; 596 GB data, 1651 GB used, 20059
> GB / 21710 GB avail; 57871/4706179 degraded (1.230%)
> mdsmap e1: 0/0/1 up
>
> Please help, i will try to give you any output you need.
>
>
> And one more thing, little bug in 0.48.1:
>
> ceph health blabla command, does same thing, as ceph health details.
> Whatever is after health, means details.
>
> --
> -----
> Regards
>
> Sławek "sZiBis" Skowron
--
-----
Pozdrawiam
Sławek "sZiBis" Skowron
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Ceph remap/recovery stuck
2012-08-24 11:54 ` Sławomir Skowron
@ 2012-08-24 16:35 ` Sage Weil
2012-08-24 16:39 ` Sławomir Skowron
0 siblings, 1 reply; 4+ messages in thread
From: Sage Weil @ 2012-08-24 16:35 UTC (permalink / raw)
To: Sławomir Skowron; +Cc: ceph-devel
On Fri, 24 Aug 2012, S?awomir Skowron wrote:
> I have found workaround.
>
> Change CRUSH to replication to osd in rule for this pool, and after
> recovery, remapped data, i just change same rule into rack awarenes,
> and whole cluster, recover again, and back to normal.
>
> Is there any way, to start refill, recovery in this situation for this
> specyfic OSD ??
This sounds like it might be a problem with the crush retry behavior.
In some cases it would fail to generate teh right number of replicas for a
given input. We fixed this by adding tunables that disable the old/bad
behavior, but haven't enabled it by default because support is only now
showing up in new kernels. If you aren't using older kernel clients, you
can enable the new values on your cluster by following the instructions
at:
http://ceph.com/docs/master/ops/manage/crush/#tunables
FWIW you can test whether this helps by extracting your crushmap from
the cluster, making whatever changes you are planning to the map, and then
running
crushtool -i newmap --test
and verify that you get the right number of results for numrep=3 and
below. There are a bunch of options you can pass to adjust the range of
inputs that are tested (e.g., --min-x 1 --max-x 100000, --num-rep 3,
etc.). crushtool is also used to adjust the tunables to 0, so you can
then verify that it fixes the problem... all before injecting the new map
into the cluster and actually triggering any data migration.
sage
>
> On Thu, Aug 23, 2012 at 3:52 PM, S?awomir Skowron <szibis@gmail.com> wrote:
> > 3 osd after crash rebuilds ok, but rebuild of two more osd (12 and
> > 30), i can't make cluster to be active+clean
> >
> > I do rebuild like in doc:
> >
> > stop osd,
> > remove from crush,
> > rm from map,
> > recreate a osd, after cluster get stable
> >
> > But now, all osd are in, and up, and data won't remap, and some of PG,
> > have only two osd in chain with replication level 3 for this pool.
> >
> > 2012-08-23 15:26:46.073685 mon.0 [INF] pgmap v117192: 6472 pgs: 63
> > active, 4457 active+clean, 1942 active+remapped, 10 active+degraded;
> > 596 GB data, 1650 GB used, 20059 GB / 21710 GB avail; 57815/4705888
> > degraded (1.229%)
> >
> > In attachment output from:
> >
> > ceph osd dump -o -
> >
> > I can't find any info in doc for this situation.
> >
> > HEALTH_WARN 10 pgs degraded; 2015 pgs stuck unclean; recovery
> > 57871/4706179 degraded (1.230%)
> > root@s3-10-177-64-6:~# ceph -s
> > health HEALTH_WARN 10 pgs degraded; 2015 pgs stuck unclean;
> > recovery 57871/4706179 degraded (1.230%)
> > monmap e4: 3 mons at
> > {0=10.177.64.4:6789/0,1=10.177.64.6:6789/0,2=10.177.64.8:6789/0},
> > election epoch 16, quorum 0,1,2 0,1,2
> > osdmap e1300: 78 osds: 78 up, 78 in
> > pgmap v117464: 6472 pgs: 63 active, 4457 active+clean, 1942
> > active+remapped, 10 active+degraded; 596 GB data, 1651 GB used, 20059
> > GB / 21710 GB avail; 57871/4706179 degraded (1.230%)
> > mdsmap e1: 0/0/1 up
> >
> > Please help, i will try to give you any output you need.
> >
> >
> > And one more thing, little bug in 0.48.1:
> >
> > ceph health blabla command, does same thing, as ceph health details.
> > Whatever is after health, means details.
> >
> > --
> > -----
> > Regards
> >
> > S?awek "sZiBis" Skowron
>
>
>
> --
> -----
> Pozdrawiam
>
> S?awek "sZiBis" Skowron
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Ceph remap/recovery stuck
2012-08-24 16:35 ` Sage Weil
@ 2012-08-24 16:39 ` Sławomir Skowron
0 siblings, 0 replies; 4+ messages in thread
From: Sławomir Skowron @ 2012-08-24 16:39 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel@vger.kernel.org
Nice thanks.
Dnia 24 sie 2012 o godz. 18:35 Sage Weil <sage@inktank.com> napisał(a):
> On Fri, 24 Aug 2012, S?awomir Skowron wrote:
>> I have found workaround.
>>
>> Change CRUSH to replication to osd in rule for this pool, and after
>> recovery, remapped data, i just change same rule into rack awarenes,
>> and whole cluster, recover again, and back to normal.
>>
>> Is there any way, to start refill, recovery in this situation for this
>> specyfic OSD ??
>
> This sounds like it might be a problem with the crush retry behavior.
> In some cases it would fail to generate teh right number of replicas for a
> given input. We fixed this by adding tunables that disable the old/bad
> behavior, but haven't enabled it by default because support is only now
> showing up in new kernels. If you aren't using older kernel clients, you
> can enable the new values on your cluster by following the instructions
> at:
>
> http://ceph.com/docs/master/ops/manage/crush/#tunables
>
> FWIW you can test whether this helps by extracting your crushmap from
> the cluster, making whatever changes you are planning to the map, and then
> running
>
> crushtool -i newmap --test
>
> and verify that you get the right number of results for numrep=3 and
> below. There are a bunch of options you can pass to adjust the range of
> inputs that are tested (e.g., --min-x 1 --max-x 100000, --num-rep 3,
> etc.). crushtool is also used to adjust the tunables to 0, so you can
> then verify that it fixes the problem... all before injecting the new map
> into the cluster and actually triggering any data migration.
>
> sage
>
>
>>
>> On Thu, Aug 23, 2012 at 3:52 PM, S?awomir Skowron <szibis@gmail.com> wrote:
>>> 3 osd after crash rebuilds ok, but rebuild of two more osd (12 and
>>> 30), i can't make cluster to be active+clean
>>>
>>> I do rebuild like in doc:
>>>
>>> stop osd,
>>> remove from crush,
>>> rm from map,
>>> recreate a osd, after cluster get stable
>>>
>>> But now, all osd are in, and up, and data won't remap, and some of PG,
>>> have only two osd in chain with replication level 3 for this pool.
>>>
>>> 2012-08-23 15:26:46.073685 mon.0 [INF] pgmap v117192: 6472 pgs: 63
>>> active, 4457 active+clean, 1942 active+remapped, 10 active+degraded;
>>> 596 GB data, 1650 GB used, 20059 GB / 21710 GB avail; 57815/4705888
>>> degraded (1.229%)
>>>
>>> In attachment output from:
>>>
>>> ceph osd dump -o -
>>>
>>> I can't find any info in doc for this situation.
>>>
>>> HEALTH_WARN 10 pgs degraded; 2015 pgs stuck unclean; recovery
>>> 57871/4706179 degraded (1.230%)
>>> root@s3-10-177-64-6:~# ceph -s
>>> health HEALTH_WARN 10 pgs degraded; 2015 pgs stuck unclean;
>>> recovery 57871/4706179 degraded (1.230%)
>>> monmap e4: 3 mons at
>>> {0=10.177.64.4:6789/0,1=10.177.64.6:6789/0,2=10.177.64.8:6789/0},
>>> election epoch 16, quorum 0,1,2 0,1,2
>>> osdmap e1300: 78 osds: 78 up, 78 in
>>> pgmap v117464: 6472 pgs: 63 active, 4457 active+clean, 1942
>>> active+remapped, 10 active+degraded; 596 GB data, 1651 GB used, 20059
>>> GB / 21710 GB avail; 57871/4706179 degraded (1.230%)
>>> mdsmap e1: 0/0/1 up
>>>
>>> Please help, i will try to give you any output you need.
>>>
>>>
>>> And one more thing, little bug in 0.48.1:
>>>
>>> ceph health blabla command, does same thing, as ceph health details.
>>> Whatever is after health, means details.
>>>
>>> --
>>> -----
>>> Regards
>>>
>>> S?awek "sZiBis" Skowron
>>
>>
>>
>> --
>> -----
>> Pozdrawiam
>>
>> S?awek "sZiBis" Skowron
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2012-08-24 16:39 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-08-23 13:52 Ceph remap/recovery stuck Sławomir Skowron
2012-08-24 11:54 ` Sławomir Skowron
2012-08-24 16:35 ` Sage Weil
2012-08-24 16:39 ` Sławomir Skowron
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.