* mon memory leak @ 2013-03-13 10:44 Stefan Priebe - Profihost AG 2013-03-13 11:02 ` Sébastien Han 0 siblings, 1 reply; 6+ messages in thread From: Stefan Priebe - Profihost AG @ 2013-03-13 10:44 UTC (permalink / raw) To: ceph-devel@vger.kernel.org Hi, are there any known ceph-mon memory leaks in bobtail? Today i've seen a ceph-mon process consuming 50GB Memory. Stefan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mon memory leak 2013-03-13 10:44 mon memory leak Stefan Priebe - Profihost AG @ 2013-03-13 11:02 ` Sébastien Han 2013-03-13 13:42 ` Travis Rhoden 0 siblings, 1 reply; 6+ messages in thread From: Sébastien Han @ 2013-03-13 11:02 UTC (permalink / raw) To: Stefan Priebe - Profihost AG; +Cc: ceph-devel@vger.kernel.org Hi, I do have some ceph-mon memory leak as well but it's on Argonaut. See the memory consumption here: time %MEM VSZ RSS 09:45:01 20.1 5688736 4567072 09:50:02 0.0 133324 11804 => restart mon process 09:55:01 0.0 133324 11804 10:00:01 0.0 134380 12872 10:05:01 0.0 137460 14108 10:10:01 0.0 138484 15164 10:15:01 0.0 138484 15144 10:20:01 0.0 139508 15184 10:25:01 0.0 139508 15380 10:30:01 0.0 139636 15788 10:35:01 0.0 139636 16052 10:40:01 0.0 140660 17108 10:45:01 0.0 140660 17108 10:50:01 0.0 140660 17012 10:55:01 0.0 140660 17052 11:00:01 0.0 140660 16992 11:05:01 0.0 141684 17200 11:10:01 0.0 141684 17368 11:15:01 0.0 141684 17828 11:20:01 0.0 141684 18148 11:25:01 0.0 142708 19284 11:30:01 0.0 142708 19336 11:35:01 0.0 142708 19252 11:40:01 0.0 142708 19252 11:45:01 0.0 142708 19252 11:50:01 0.0 143732 19312 11:55:01 0.0 143732 20140 12:00:01 0.0 143732 20140 12:05:01 0.0 143732 20092 12:10:01 0.0 144756 21156 12:15:01 0.0 144756 21156 12:20:01 0.0 144756 21392 12:25:01 0.0 144756 21448 12:30:01 0.0 145780 21508 12:35:01 0.0 145780 21508 12:40:01 0.0 145780 22352 12:45:01 0.0 145780 22520 12:50:01 0.0 145780 22520 12:55:01 0.1 146804 23420 13:00:01 0.1 146804 23420 13:05:01 0.1 146804 23420 13:10:02 0.1 146804 23420 13:15:01 0.1 147828 24476 13:20:01 0.1 147828 24388 13:25:01 0.1 147828 24388 13:30:01 0.1 147828 24316 13:35:01 0.1 147828 24492 13:40:01 0.1 148852 25500 13:45:01 0.1 148852 25448 13:50:01 0.1 148852 25360 13:55:01 0.1 148852 25348 14:00:01 0.1 149876 26384 14:05:01 0.1 149876 26384 14:10:01 0.1 149876 26384 14:15:01 0.1 149876 26304 14:20:01 0.1 150900 27240 14:25:01 0.1 150900 27400 14:30:01 0.1 150900 27400 14:35:01 0.1 151028 27272 14:40:01 0.1 151028 27392 14:45:01 0.1 152052 28288 14:50:01 0.1 152052 28288 14:55:01 0.1 152052 28368 15:00:01 0.1 152052 28236 15:05:01 0.1 153108 28236 15:10:01 0.1 153108 29324 15:15:01 0.1 153108 29476 15:20:01 0.1 153108 29360 15:25:01 0.1 153108 29360 15:30:01 0.1 154132 30416 15:35:01 0.1 154132 30308 15:40:01 0.1 154132 30308 15:45:01 0.1 154132 30436 15:50:01 0.1 154132 30468 15:55:01 0.1 155156 30468 16:00:01 0.1 155156 31216 16:05:01 0.1 155156 31784 16:10:01 0.1 155156 31784 16:15:01 0.1 155156 31640 16:20:01 0.1 156180 31792 16:25:01 0.1 156180 32056 16:30:01 0.1 156180 32700 16:35:01 0.1 156180 32700 16:40:01 0.1 156180 32556 16:45:01 0.1 156180 32556 16:50:01 0.1 157204 33740 16:55:01 0.1 157204 33740 17:00:01 0.1 157204 33740 17:05:01 0.1 157204 33672 17:10:01 0.1 158228 33596 17:15:01 0.1 158228 34568 17:20:01 0.1 158228 34568 17:25:01 0.1 158228 34516 17:30:01 0.1 158228 34600 17:35:01 0.1 159252 35464 17:40:01 0.1 159252 35464 17:45:01 0.1 159252 35400 17:50:01 0.1 159252 35532 17:55:01 0.1 160276 35860 18:00:01 0.1 160276 36244 18:05:01 0.1 160276 36316 18:10:01 0.1 160276 36580 18:15:01 0.1 160276 36876 18:20:01 0.1 161300 37000 18:25:01 0.1 161300 37264 18:30:01 0.1 161300 37520 18:35:01 0.1 161300 37680 18:40:01 0.1 162324 38792 18:45:01 0.1 162324 38792 18:50:01 0.1 162452 38760 18:55:01 0.1 162452 38688 19:00:01 0.1 163476 39696 19:05:01 0.1 163476 39780 19:10:01 0.1 163476 39780 19:15:01 0.1 163476 39688 19:20:01 0.1 163476 39680 19:25:01 0.1 164500 39900 19:30:01 0.1 164500 39888 19:35:01 0.1 164500 40676 19:40:01 0.1 164500 40628 19:45:01 0.1 164500 40856 19:50:01 0.1 164500 40856 19:55:01 0.1 165524 40748 20:00:01 0.1 165524 41036 20:05:01 0.1 165524 41172 20:10:01 0.1 165524 41640 20:15:01 0.1 165524 41860 20:20:02 0.1 166548 41860 20:25:01 0.1 166548 42124 20:30:01 0.1 166548 42204 20:35:01 0.1 166548 42540 20:40:01 0.1 167572 42904 20:45:01 0.1 167572 43740 20:50:01 0.1 167572 43740 20:55:01 0.1 167572 43660 21:00:01 0.1 167572 43696 21:05:01 0.1 167572 43696 21:10:01 0.1 168596 44752 21:15:01 0.1 168596 44752 21:20:01 0.1 168596 44740 21:25:01 0.1 168596 44696 21:30:01 0.1 168596 44768 21:35:01 0.2 169656 45824 21:40:01 0.2 169656 45744 21:45:01 0.2 169656 45744 21:50:01 0.2 169656 45884 21:55:01 0.2 169656 45828 22:00:01 0.2 169656 45828 22:05:01 0.2 170680 46884 22:10:01 0.2 170680 47444 22:15:01 0.2 170680 47444 22:20:01 0.2 170680 47556 22:25:01 0.2 170680 47500 22:30:01 0.2 170680 47500 22:35:01 0.2 171704 48560 22:40:01 0.2 171704 48560 22:45:01 0.2 171704 48560 22:50:01 0.2 171704 48496 22:55:01 0.2 172728 49332 23:00:01 0.2 172728 49332 23:05:01 0.2 172728 49396 23:10:01 0.2 172728 49524 23:15:01 0.2 172728 49308 23:20:01 0.2 173752 49472 23:25:01 0.2 173880 49736 23:30:01 0.2 173880 49888 23:35:01 0.2 173880 49996 23:40:01 0.2 173880 50332 23:45:01 0.2 174904 51388 23:50:01 0.2 174904 51388 23:55:01 0.2 174904 51184 00:00:01 0.2 174904 51512 00:05:01 0.2 174904 51376 00:10:01 0.2 174904 51392 00:15:01 0.2 175928 52448 00:20:01 0.2 175928 52596 00:25:01 0.2 175928 52484 00:30:01 0.2 175928 52588 00:35:01 0.2 176952 53644 00:40:01 0.2 176952 53644 00:45:01 0.2 176952 53624 00:50:01 0.2 176952 53692 00:55:01 0.2 177976 53752 01:00:01 0.2 177976 54784 01:05:01 0.2 177976 54524 01:10:01 0.2 177976 54524 01:15:01 0.2 177976 54684 01:20:01 0.2 179000 55588 01:25:01 0.2 179000 55624 01:30:01 0.2 179000 55624 01:35:01 0.2 179000 55508 01:40:01 0.2 179000 55500 01:45:01 0.2 179000 55696 01:50:01 0.2 180024 55696 01:55:01 0.2 180024 56684 02:00:01 0.2 180024 56684 02:05:01 0.2 180024 56656 02:10:01 0.2 180024 56648 02:15:01 0.2 181048 57748 02:20:01 0.2 181048 57524 02:25:01 0.2 181048 57524 02:30:01 0.2 181048 57556 02:35:01 0.2 181048 57556 02:40:01 0.2 182072 58668 02:45:01 0.2 182072 58560 02:50:01 0.2 182072 58692 02:55:01 0.2 182072 58544 03:00:01 0.2 182072 58712 03:05:01 0.2 183096 58712 03:10:01 0.2 183096 59568 03:15:01 0.2 183096 59652 03:20:01 0.2 183096 59672 03:25:01 0.2 184120 60508 03:30:02 0.2 184120 60508 03:35:01 0.2 184120 60476 03:40:01 0.2 184120 60652 03:45:01 0.2 184248 60652 03:50:01 0.2 185272 61496 03:55:01 0.2 185272 61484 04:00:01 0.2 185272 61484 04:05:01 0.2 185272 61636 04:10:01 0.2 185272 61552 04:15:01 0.2 186328 62656 04:20:01 0.2 186328 62656 04:25:01 0.2 186328 62756 04:30:01 0.2 186328 62756 04:35:01 0.2 186328 62644 04:40:01 0.2 187352 63700 04:45:01 0.2 187352 63680 04:50:01 0.2 187352 63868 04:55:01 0.2 187352 63972 05:00:01 0.2 188376 64988 05:05:01 0.2 188376 65088 05:10:01 0.2 188376 65088 05:15:01 0.2 188376 64960 05:20:01 0.2 188376 64960 05:25:01 0.2 189400 65956 05:30:01 0.2 189400 65956 05:35:01 0.2 189400 65936 05:40:01 0.2 189400 65992 05:45:01 0.2 189400 66012 05:50:01 0.2 190424 67068 05:55:01 0.2 190424 67068 06:00:01 0.2 190424 66896 06:05:01 0.2 190424 67044 06:10:01 0.2 190424 66980 06:15:01 0.3 191448 67900 06:20:01 0.2 191448 67844 06:25:01 0.3 191448 68012 06:30:01 0.3 191448 68012 06:35:01 0.3 191448 67888 06:40:01 0.3 192472 68912 06:45:01 0.3 192472 68912 06:50:01 0.3 192472 68824 06:55:01 0.3 192472 68788 07:00:01 0.3 192472 68748 07:05:01 0.3 193496 69804 07:10:01 0.3 193496 69804 07:15:01 0.3 193496 69700 07:20:01 0.3 193496 69896 07:25:01 0.3 193496 69736 07:30:01 0.3 194520 70864 07:35:01 0.3 194520 70680 07:40:01 0.3 194520 70680 07:45:01 0.3 194520 71256 07:50:01 0.3 194520 71184 07:55:01 0.3 195544 72240 08:00:01 0.3 195544 72240 08:05:01 0.3 195544 72244 08:10:01 0.3 195672 72084 08:15:01 0.3 195672 72116 08:20:01 0.3 196696 73172 08:25:01 0.3 196696 73216 08:30:01 0.3 196696 73216 08:35:01 0.3 196696 72936 08:40:01 0.3 196696 73124 08:45:01 0.3 197720 73916 08:50:01 0.3 197720 74120 08:55:01 0.3 197720 74112 09:00:01 0.3 197720 73960 09:05:01 0.3 198744 73960 09:10:01 0.3 198744 75080 09:15:01 0.3 198744 75188 09:20:01 0.3 198744 75132 09:25:01 0.3 199768 75132 09:30:01 0.3 199768 75452 09:35:01 0.3 199768 76116 09:40:01 0.3 199768 76116 09:45:01 0.3 200792 76116 09:50:01 0.3 200792 76380 09:55:01 0.3 200792 76540 10:00:01 0.3 200792 76844 10:05:01 0.3 200792 77100 10:10:01 0.3 201848 77308 10:15:01 0.3 201848 77308 10:20:01 0.3 201848 78144 10:25:01 0.3 201848 78156 10:30:01 0.3 201848 78156 10:35:01 0.3 202872 79248 10:40:01 0.3 202872 79248 10:45:01 0.3 202872 79248 10:50:01 0.3 202872 79304 10:55:01 0.3 202872 79112 11:00:01 0.3 203896 80436 11:05:01 0.3 203896 80436 11:10:01 0.3 203896 80520 11:15:01 0.3 203896 80396 11:20:01 0.3 203896 80584 11:25:01 0.3 204920 81640 11:30:01 0.3 204920 81640 11:35:01 0.3 204920 81752 11:40:01 0.3 204920 81708 11:45:01 0.3 205944 82500 11:50:01 0.3 205944 82688 11:55:01 0.3 205944 82584 12:00:01 0.3 205944 82628 12:05:01 0.3 206968 82756 12:10:01 0.3 206968 83616 12:15:01 0.3 207096 83616 12:20:01 0.3 207096 83432 12:25:01 0.3 207096 83432 12:30:01 0.3 208120 84568 12:35:01 0.3 208120 84768 12:40:01 0.3 208120 84856 12:45:02 0.3 208120 84668 12:50:01 0.3 209144 84668 12:55:01 0.3 209144 85256 13:00:01 0.3 209144 85800 13:05:01 0.3 209144 85800 13:10:01 0.3 210168 85936 13:15:01 0.3 210168 85936 13:20:01 0.3 210168 86780 13:25:01 0.3 210168 86780 13:30:01 0.3 211192 86536 13:35:01 0.3 211192 86800 13:40:01 0.3 211192 86800 13:45:01 0.3 211192 87696 13:50:01 0.3 211192 87748 13:55:01 0.3 211192 87748 14:00:01 0.3 212216 88812 14:05:01 0.3 212216 88812 14:10:01 0.3 212216 88796 14:15:01 0.3 212216 88728 14:20:01 0.3 213240 89784 14:25:01 0.3 213240 89784 14:30:01 0.3 213240 89660 14:35:01 0.3 213240 89836 14:40:01 0.3 213240 89724 14:45:01 0.4 214264 90848 14:50:01 0.4 214264 90848 14:55:01 0.4 214264 90956 15:00:01 0.4 214264 90956 15:05:01 0.4 215288 91780 15:10:01 0.4 215288 91780 15:15:01 0.4 215288 91712 15:20:01 0.4 215288 91768 15:25:01 0.4 216312 92824 15:30:01 0.4 216312 92824 15:35:01 0.4 216312 92736 15:40:01 0.4 216312 92964 15:45:01 0.4 216312 92688 15:50:01 0.4 217336 93048 15:55:01 0.4 217336 93312 16:00:01 0.4 217336 93288 16:05:01 0.4 217336 93744 16:10:01 0.4 218392 94908 16:15:01 0.4 218520 94908 16:20:01 0.4 218520 94992 16:25:01 0.4 218520 94940 16:30:01 0.4 219544 95996 16:35:01 0.4 219544 96076 16:40:01 0.4 219544 96048 16:45:01 0.4 219544 95948 16:50:01 0.4 219544 95948 16:55:01 0.4 220568 96968 17:00:01 0.4 220568 96968 17:05:01 0.4 220568 96936 17:10:01 0.4 220568 96976 17:15:01 0.4 220568 97048 17:20:01 0.4 221592 98368 17:25:01 0.4 221592 98224 17:30:01 0.4 221592 98320 17:35:01 0.4 221592 98096 17:40:01 0.4 222620 98364 17:45:01 0.4 222620 99332 17:50:01 0.4 222620 99332 17:55:01 0.4 222620 99064 18:00:01 0.4 223644 100120 18:05:01 0.4 223644 100104 18:10:01 0.4 223644 100084 18:15:01 0.4 223644 100084 18:20:01 0.4 223644 100268 18:25:01 0.4 223644 100232 18:30:01 0.4 224668 100496 18:35:01 0.4 224668 101304 18:40:01 0.4 224668 101168 18:45:01 0.4 224668 101220 18:50:01 0.4 225692 101220 18:55:01 0.4 225692 101484 19:00:01 0.4 225692 102080 19:05:01 0.4 225692 102200 19:10:01 0.4 226716 102200 19:15:01 0.4 226716 102464 19:20:01 0.4 226716 103184 19:25:01 0.4 226716 103232 19:30:01 0.4 226716 103004 19:35:01 0.4 227740 104212 19:40:01 0.4 227740 104212 19:45:01 0.4 227740 104056 19:50:01 0.4 227740 104056 19:55:01 0.4 228764 105252 20:00:01 0.4 228764 105344 20:05:01 0.4 229788 105356 20:10:02 0.4 229788 106240 20:15:01 0.4 229916 106240 20:20:01 0.4 229916 106364 20:25:01 0.4 229916 106364 20:30:01 0.4 230940 107420 20:35:01 0.4 230940 107404 20:40:01 0.4 230940 107344 20:45:01 0.4 230940 107204 20:50:01 0.4 230940 107564 20:55:01 0.4 231964 108620 21:00:01 0.4 231964 108620 21:05:01 0.4 231964 108524 21:10:01 0.4 231964 108372 21:15:01 0.4 232988 109384 21:20:01 0.4 232988 109384 21:25:01 0.4 232988 109432 21:30:01 0.4 232988 109432 21:35:01 0.4 234012 110304 21:40:01 0.4 234012 110568 21:45:01 0.4 234012 110568 21:50:01 0.4 234012 110384 21:55:01 0.4 234012 110260 22:00:01 0.4 234012 110260 22:05:01 0.4 235068 111424 22:10:01 0.4 235068 111424 22:15:01 0.4 235068 111392 22:20:01 0.4 235068 111392 22:25:01 0.4 235068 111324 22:30:01 0.4 236092 112440 22:35:01 0.4 236092 112612 22:40:01 0.4 236092 112652 22:45:01 0.4 236092 112520 22:50:01 0.4 236092 112608 22:55:01 0.5 237116 113496 23:00:01 0.5 237116 113496 23:05:01 0.5 237116 113616 23:10:01 0.5 237116 113616 23:15:01 0.5 237116 113560 23:20:01 0.5 238140 113800 23:25:01 0.5 238140 114064 23:30:01 0.5 238140 114416 23:35:01 0.5 238140 114700 23:40:01 0.5 239164 114700 23:45:01 0.5 239164 114964 23:50:02 0.5 239164 115740 23:55:01 0.5 239164 115740 00:00:01 0.5 239164 115492 00:05:01 0.5 239164 115492 00:10:01 0.5 240188 116504 00:15:01 0.5 240188 116504 00:20:01 0.5 240188 116720 00:25:01 0.5 240188 116524 00:30:01 0.5 240188 116808 00:35:01 0.5 241340 117864 00:40:01 0.5 241340 117936 00:45:01 0.5 241340 118048 00:50:01 0.5 241340 118048 00:55:01 0.5 242364 118860 01:00:01 0.5 242364 118848 01:05:01 0.5 242364 118956 01:10:01 0.5 242364 118900 01:15:01 0.5 242364 118900 01:20:01 0.5 242364 119044 01:25:01 0.5 243388 119172 01:30:01 0.5 243388 119792 01:35:01 0.5 243388 119792 01:40:01 0.5 243388 120076 01:45:01 0.5 243388 120084 01:50:01 0.5 244412 120084 01:55:01 0.5 244412 120452 02:00:01 0.5 244412 120716 02:05:01 0.5 244412 120956 02:10:01 0.5 244412 120956 02:15:01 0.5 245436 121220 02:20:01 0.5 245436 121364 02:25:01 0.5 245436 121628 02:30:01 0.5 245436 122040 02:35:01 0.5 245436 122008 02:40:01 0.5 246460 123000 02:45:01 0.5 246460 123000 02:50:01 0.5 246460 122908 02:55:01 0.5 246460 122908 03:00:01 0.5 246460 122908 03:05:01 0.5 247484 124000 03:10:01 0.5 247484 124000 03:15:01 0.5 247484 123924 03:20:01 0.5 247484 123980 03:25:01 0.5 247484 123980 03:30:01 0.5 248508 125036 03:35:01 0.5 248508 125012 03:40:01 0.5 248508 124952 03:45:01 0.5 248508 125336 03:50:01 0.5 248508 125128 03:55:01 0.5 249532 126212 04:00:01 0.5 249532 126212 04:05:01 0.5 249532 126372 04:10:01 0.5 249532 126348 04:15:01 0.5 250556 127148 04:20:01 0.5 250556 127412 04:25:01 0.5 250556 127244 04:30:01 0.5 250556 127164 04:35:01 0.5 250556 127196 04:40:01 0.5 251612 128224 04:45:01 0.5 251612 128128 04:50:01 0.5 251612 128128 04:55:01 0.5 251740 128080 05:00:01 0.5 251740 128236 05:05:01 0.5 252764 129212 05:10:01 0.5 252764 129212 05:15:01 0.5 252764 129208 05:20:01 0.5 252764 129108 05:25:01 0.5 252764 129020 05:30:01 0.5 253788 129404 05:35:01 0.5 253788 129600 05:40:01 0.5 253788 130572 05:45:01 0.5 253788 130572 05:50:01 0.5 253788 130572 05:55:01 0.5 254812 130572 06:00:01 0.5 254812 131428 06:05:01 0.5 254812 131428 06:10:01 0.5 254812 131468 06:15:01 0.5 254812 131468 06:20:01 0.5 255836 131468 06:25:01 0.5 255836 132212 06:30:01 0.5 255836 132328 06:35:01 0.5 255836 132224 06:40:01 0.5 255836 132120 06:45:01 0.5 256860 132384 06:50:01 0.5 256860 132648 06:55:01 0.5 256860 133044 07:00:01 0.5 256860 133232 07:05:01 0.5 256860 133232 07:10:01 0.5 257884 133360 07:15:01 0.5 257884 134336 07:20:01 0.5 257884 134336 07:25:01 0.5 257884 134220 07:30:01 0.5 257884 134220 07:35:01 0.5 258908 134220 07:40:01 0.5 258908 134348 07:45:01 0.5 258908 134612 07:50:01 0.5 258908 135368 07:55:01 0.5 258908 135284 08:00:01 0.5 258908 135484 08:05:01 0.5 259932 135484 08:10:01 0.6 259932 135748 08:15:01 0.6 259932 136616 08:20:01 0.6 259932 136616 08:25:01 0.6 259932 136616 08:30:01 0.6 259932 136616 08:35:01 0.6 260956 136568 08:40:01 0.6 260956 136832 08:45:01 0.6 260956 137396 08:50:01 0.6 260956 137660 08:55:01 0.6 260956 137660 09:00:01 0.6 260956 137380 09:05:01 0.6 260956 137508 09:10:01 0.6 261980 137624 09:15:01 0.6 261980 137888 09:20:01 0.6 261980 138428 09:25:01 0.6 261980 138416 09:30:01 0.6 261980 138416 09:35:01 0.6 263004 138588 09:40:01 0.6 263004 138852 09:45:01 0.6 263132 139116 09:50:01 0.6 263132 139672 09:55:01 0.6 263132 139596 10:00:01 0.6 264156 140932 10:05:01 0.6 264156 140932 10:10:01 0.6 264156 140932 10:15:01 0.6 264156 140732 10:20:01 0.6 264156 140700 10:25:01 0.6 265180 140872 10:30:01 0.6 265180 141764 10:35:01 0.6 265180 141764 10:40:01 0.6 265180 141856 10:45:01 0.6 265180 141652 10:50:01 0.6 265180 141748 10:55:01 0.6 265180 141588 11:00:01 0.6 266204 141588 11:05:01 0.6 266204 141852 11:10:01 0.6 266204 141852 11:15:01 0.6 266204 142520 11:20:01 0.6 266204 142652 11:25:01 0.6 267228 142652 11:30:01 0.6 267228 142916 11:35:01 0.6 267228 142956 11:40:01 0.6 267228 143296 11:45:01 0.6 267228 143392 11:50:01 0.6 267228 143984 11:55:01 0.6 268284 144924 12:00:01 0.6 268284 144924 12:05:01 0.6 268284 144784 12:10:01 0.6 268284 144784 12:15:01 0.6 268284 145008 12:20:01 0.6 268284 144904 12:25:01 0.6 269308 145960 12:30:01 0.6 269308 145960 12:35:01 0.6 269308 145752 12:40:01 0.6 269308 145968 12:45:01 0.6 269308 145852 12:50:01 0.6 269308 145852 12:55:01 0.6 270332 146908 13:00:01 0.6 271356 146660 13:05:01 0.6 271356 147976 13:10:01 0.6 271356 147976 13:15:01 0.6 271356 147972 13:20:01 0.6 272380 149016 13:25:01 0.6 272380 149016 13:30:01 0.6 272380 149216 13:35:01 0.6 272380 149216 13:40:01 0.6 272380 148988 13:45:01 0.6 272380 149148 13:50:01 0.6 273404 149116 13:55:01 0.6 273404 149380 14:00:01 0.6 273404 149956 14:05:01 0.6 273404 150148 14:10:01 0.6 274428 151012 14:15:01 0.6 274428 151012 14:20:01 0.6 274556 151012 14:25:01 0.6 274556 151020 14:30:01 0.6 274556 151112 14:35:01 0.6 274556 151112 14:40:01 0.6 275584 152292 14:45:02 0.6 275584 152292 14:50:01 0.6 275584 152292 14:55:01 0.6 275584 152292 15:00:01 0.6 275584 152292 15:05:01 0.6 276608 153088 15:10:01 0.6 276608 153088 15:15:01 0.6 276608 153184 15:20:01 0.6 276608 153184 15:25:01 0.6 276608 153048 15:30:01 0.6 277632 154088 15:35:01 0.6 277632 154088 15:40:01 0.6 277632 154096 15:45:01 0.6 277632 154224 15:50:01 0.6 277632 154144 15:55:01 0.6 277632 154264 16:00:01 0.6 278656 155200 16:05:01 0.6 278656 155088 16:10:01 0.6 278656 155088 16:15:01 0.6 278656 155048 16:20:01 0.6 278656 155004 16:25:01 0.6 278656 155244 16:30:01 0.6 278656 155244 16:35:01 0.6 279680 156036 16:40:01 0.6 279680 156036 16:45:01 0.6 279680 156216 16:50:01 0.6 279680 156132 16:55:01 0.6 280704 157188 17:00:01 0.6 280704 157128 17:05:01 0.6 280704 157160 17:10:01 0.6 280704 157140 17:15:01 0.6 280704 157140 17:20:01 0.6 281728 158260 17:25:01 0.6 281728 158132 17:30:01 0.6 281728 158276 17:35:01 0.7 281728 158360 17:40:01 0.7 282752 159320 17:45:01 0.7 282752 159320 17:50:01 0.7 282752 159204 17:55:01 0.7 282752 159272 18:00:01 0.7 282752 159272 18:05:01 0.7 283808 160444 18:10:02 0.7 283808 160444 18:15:01 0.7 283808 160252 18:20:01 0.7 283808 160408 18:25:01 0.7 284832 161580 18:30:01 0.7 284832 161580 18:35:01 0.7 284832 161392 18:40:01 0.7 284832 161388 18:45:01 0.7 285984 162568 18:50:01 0.7 285984 162568 18:55:01 0.7 285984 162548 19:00:01 0.7 285984 162548 19:05:01 0.7 285984 162516 19:10:01 0.7 287008 163444 19:15:01 0.7 287008 163444 19:20:01 0.7 287008 163448 19:25:01 0.7 287008 163332 19:30:01 0.7 287008 163548 19:35:01 0.7 288032 164644 19:40:01 0.7 288032 164644 19:45:01 0.7 288032 164620 19:50:01 0.7 288032 164512 19:55:01 0.7 289056 165528 20:00:01 0.7 289056 165608 20:05:01 0.7 289056 165500 20:10:01 0.7 289056 165528 20:15:01 0.7 290080 165660 20:20:01 0.7 290080 165956 20:25:01 0.7 290080 166432 20:30:01 0.7 290080 166588 20:35:01 0.7 290080 166588 20:40:01 0.7 290080 166588 20:45:01 0.7 291104 167492 20:50:01 0.7 291104 167492 20:55:01 0.7 291104 167508 21:00:01 0.7 291104 167508 21:05:01 0.7 292128 167508 21:10:01 0.7 292128 168576 21:15:01 0.7 292128 168420 21:20:01 0.7 292128 168396 21:25:01 0.7 293152 169716 21:30:01 0.7 293152 169752 21:35:01 0.7 293152 169756 21:40:01 0.7 293152 169604 21:45:01 0.7 293152 169604 21:50:02 0.7 294176 170804 21:55:01 0.7 294176 170804 22:00:01 0.7 294176 170576 22:05:01 0.7 294176 170764 22:10:01 0.7 294176 170744 22:15:01 0.7 295200 171632 22:20:01 0.7 295200 171632 22:25:01 0.7 295200 171632 22:30:01 0.7 295200 171820 22:35:01 0.7 296224 172988 22:40:01 0.7 296224 172988 22:45:01 0.7 296224 172804 22:50:01 0.7 296352 172872 22:55:01 0.7 296352 172812 23:00:01 0.7 297376 173076 23:05:01 0.7 297376 173164 23:10:01 0.7 297376 173416 23:15:01 0.7 297376 173664 23:20:01 0.7 298400 174656 23:25:01 0.7 298400 174656 23:30:01 0.7 298400 174656 23:35:01 0.7 298400 174656 23:40:01 0.7 298400 174660 23:45:01 0.7 299424 174612 23:50:01 0.7 299424 175720 23:55:01 0.7 299424 175764 00:00:01 0.7 299424 176088 00:05:01 0.7 300480 176088 00:10:01 0.7 300480 176352 00:15:01 0.7 300480 176480 00:20:01 0.7 300480 177200 00:25:01 0.7 300480 177352 00:30:01 0.7 300480 177276 00:35:01 0.7 301504 178404 00:40:01 0.7 301504 178336 00:45:01 0.7 301504 178336 00:50:01 0.7 301504 178320 00:55:01 0.7 301504 178236 01:00:01 0.7 302528 179240 01:05:01 0.7 302528 179240 01:10:01 0.7 302528 179292 01:15:01 0.7 302528 179352 01:20:01 0.7 302528 179352 01:25:02 0.7 302528 179176 01:30:01 0.7 302528 179060 Sorry for the long post. -- Regards, Sébastien Han. On Wed, Mar 13, 2013 at 11:44 AM, Stefan Priebe - Profihost AG <s.priebe@profihost.ag> wrote: > Hi, > > are there any known ceph-mon memory leaks in bobtail? Today i've seen a > ceph-mon process consuming 50GB Memory. > > Stefan > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mon memory leak 2013-03-13 11:02 ` Sébastien Han @ 2013-03-13 13:42 ` Travis Rhoden 2013-03-13 13:58 ` Joao Eduardo Luis 0 siblings, 1 reply; 6+ messages in thread From: Travis Rhoden @ 2013-03-13 13:42 UTC (permalink / raw) To: Stefan Priebe - Profihost AG Cc: Sébastien Han, ceph-devel@vger.kernel.org Hi Stefan, I have seen a mon grow quite high on Bobtail. Joao had some theories as for why, and I was able to provide him with a memory dump of the running process. No word yet on whether it revealed anything, but I know it is on his stack. Background here: http://thread.gmane.org/gmane.comp.file-systems.ceph.user/14 - Travis On Wed, Mar 13, 2013 at 7:02 AM, Sébastien Han <han.sebastien@gmail.com> wrote: > > Hi, > > I do have some ceph-mon memory leak as well but it's on Argonaut. > > See the memory consumption here: > > time %MEM VSZ RSS > > 09:45:01 20.1 5688736 4567072 > 09:50:02 0.0 133324 11804 => restart mon process > 09:55:01 0.0 133324 11804 > 10:00:01 0.0 134380 12872 > 10:05:01 0.0 137460 14108 > 10:10:01 0.0 138484 15164 > 10:15:01 0.0 138484 15144 > 10:20:01 0.0 139508 15184 > 10:25:01 0.0 139508 15380 > 10:30:01 0.0 139636 15788 > 10:35:01 0.0 139636 16052 > 10:40:01 0.0 140660 17108 > 10:45:01 0.0 140660 17108 > 10:50:01 0.0 140660 17012 > 10:55:01 0.0 140660 17052 > 11:00:01 0.0 140660 16992 > 11:05:01 0.0 141684 17200 > 11:10:01 0.0 141684 17368 > 11:15:01 0.0 141684 17828 > 11:20:01 0.0 141684 18148 > 11:25:01 0.0 142708 19284 > 11:30:01 0.0 142708 19336 > 11:35:01 0.0 142708 19252 > 11:40:01 0.0 142708 19252 > 11:45:01 0.0 142708 19252 > 11:50:01 0.0 143732 19312 > 11:55:01 0.0 143732 20140 > 12:00:01 0.0 143732 20140 > 12:05:01 0.0 143732 20092 > 12:10:01 0.0 144756 21156 > 12:15:01 0.0 144756 21156 > 12:20:01 0.0 144756 21392 > 12:25:01 0.0 144756 21448 > 12:30:01 0.0 145780 21508 > 12:35:01 0.0 145780 21508 > 12:40:01 0.0 145780 22352 > 12:45:01 0.0 145780 22520 > 12:50:01 0.0 145780 22520 > 12:55:01 0.1 146804 23420 > 13:00:01 0.1 146804 23420 > 13:05:01 0.1 146804 23420 > 13:10:02 0.1 146804 23420 > 13:15:01 0.1 147828 24476 > 13:20:01 0.1 147828 24388 > 13:25:01 0.1 147828 24388 > 13:30:01 0.1 147828 24316 > 13:35:01 0.1 147828 24492 > 13:40:01 0.1 148852 25500 > 13:45:01 0.1 148852 25448 > 13:50:01 0.1 148852 25360 > 13:55:01 0.1 148852 25348 > 14:00:01 0.1 149876 26384 > 14:05:01 0.1 149876 26384 > 14:10:01 0.1 149876 26384 > 14:15:01 0.1 149876 26304 > 14:20:01 0.1 150900 27240 > 14:25:01 0.1 150900 27400 > 14:30:01 0.1 150900 27400 > 14:35:01 0.1 151028 27272 > 14:40:01 0.1 151028 27392 > 14:45:01 0.1 152052 28288 > 14:50:01 0.1 152052 28288 > 14:55:01 0.1 152052 28368 > 15:00:01 0.1 152052 28236 > 15:05:01 0.1 153108 28236 > 15:10:01 0.1 153108 29324 > 15:15:01 0.1 153108 29476 > 15:20:01 0.1 153108 29360 > 15:25:01 0.1 153108 29360 > 15:30:01 0.1 154132 30416 > 15:35:01 0.1 154132 30308 > 15:40:01 0.1 154132 30308 > 15:45:01 0.1 154132 30436 > 15:50:01 0.1 154132 30468 > 15:55:01 0.1 155156 30468 > 16:00:01 0.1 155156 31216 > 16:05:01 0.1 155156 31784 > 16:10:01 0.1 155156 31784 > 16:15:01 0.1 155156 31640 > 16:20:01 0.1 156180 31792 > 16:25:01 0.1 156180 32056 > 16:30:01 0.1 156180 32700 > 16:35:01 0.1 156180 32700 > 16:40:01 0.1 156180 32556 > 16:45:01 0.1 156180 32556 > 16:50:01 0.1 157204 33740 > 16:55:01 0.1 157204 33740 > 17:00:01 0.1 157204 33740 > 17:05:01 0.1 157204 33672 > 17:10:01 0.1 158228 33596 > 17:15:01 0.1 158228 34568 > 17:20:01 0.1 158228 34568 > 17:25:01 0.1 158228 34516 > 17:30:01 0.1 158228 34600 > 17:35:01 0.1 159252 35464 > 17:40:01 0.1 159252 35464 > 17:45:01 0.1 159252 35400 > 17:50:01 0.1 159252 35532 > 17:55:01 0.1 160276 35860 > 18:00:01 0.1 160276 36244 > 18:05:01 0.1 160276 36316 > 18:10:01 0.1 160276 36580 > 18:15:01 0.1 160276 36876 > 18:20:01 0.1 161300 37000 > 18:25:01 0.1 161300 37264 > 18:30:01 0.1 161300 37520 > 18:35:01 0.1 161300 37680 > 18:40:01 0.1 162324 38792 > 18:45:01 0.1 162324 38792 > 18:50:01 0.1 162452 38760 > 18:55:01 0.1 162452 38688 > 19:00:01 0.1 163476 39696 > 19:05:01 0.1 163476 39780 > 19:10:01 0.1 163476 39780 > 19:15:01 0.1 163476 39688 > 19:20:01 0.1 163476 39680 > 19:25:01 0.1 164500 39900 > 19:30:01 0.1 164500 39888 > 19:35:01 0.1 164500 40676 > 19:40:01 0.1 164500 40628 > 19:45:01 0.1 164500 40856 > 19:50:01 0.1 164500 40856 > 19:55:01 0.1 165524 40748 > 20:00:01 0.1 165524 41036 > 20:05:01 0.1 165524 41172 > 20:10:01 0.1 165524 41640 > 20:15:01 0.1 165524 41860 > 20:20:02 0.1 166548 41860 > 20:25:01 0.1 166548 42124 > 20:30:01 0.1 166548 42204 > 20:35:01 0.1 166548 42540 > 20:40:01 0.1 167572 42904 > 20:45:01 0.1 167572 43740 > 20:50:01 0.1 167572 43740 > 20:55:01 0.1 167572 43660 > 21:00:01 0.1 167572 43696 > 21:05:01 0.1 167572 43696 > 21:10:01 0.1 168596 44752 > 21:15:01 0.1 168596 44752 > 21:20:01 0.1 168596 44740 > 21:25:01 0.1 168596 44696 > 21:30:01 0.1 168596 44768 > 21:35:01 0.2 169656 45824 > 21:40:01 0.2 169656 45744 > 21:45:01 0.2 169656 45744 > 21:50:01 0.2 169656 45884 > 21:55:01 0.2 169656 45828 > 22:00:01 0.2 169656 45828 > 22:05:01 0.2 170680 46884 > 22:10:01 0.2 170680 47444 > 22:15:01 0.2 170680 47444 > 22:20:01 0.2 170680 47556 > 22:25:01 0.2 170680 47500 > 22:30:01 0.2 170680 47500 > 22:35:01 0.2 171704 48560 > 22:40:01 0.2 171704 48560 > 22:45:01 0.2 171704 48560 > 22:50:01 0.2 171704 48496 > 22:55:01 0.2 172728 49332 > 23:00:01 0.2 172728 49332 > 23:05:01 0.2 172728 49396 > 23:10:01 0.2 172728 49524 > 23:15:01 0.2 172728 49308 > 23:20:01 0.2 173752 49472 > 23:25:01 0.2 173880 49736 > 23:30:01 0.2 173880 49888 > 23:35:01 0.2 173880 49996 > 23:40:01 0.2 173880 50332 > 23:45:01 0.2 174904 51388 > 23:50:01 0.2 174904 51388 > 23:55:01 0.2 174904 51184 > 00:00:01 0.2 174904 51512 > 00:05:01 0.2 174904 51376 > 00:10:01 0.2 174904 51392 > 00:15:01 0.2 175928 52448 > 00:20:01 0.2 175928 52596 > 00:25:01 0.2 175928 52484 > 00:30:01 0.2 175928 52588 > 00:35:01 0.2 176952 53644 > 00:40:01 0.2 176952 53644 > 00:45:01 0.2 176952 53624 > 00:50:01 0.2 176952 53692 > 00:55:01 0.2 177976 53752 > 01:00:01 0.2 177976 54784 > 01:05:01 0.2 177976 54524 > 01:10:01 0.2 177976 54524 > 01:15:01 0.2 177976 54684 > 01:20:01 0.2 179000 55588 > 01:25:01 0.2 179000 55624 > 01:30:01 0.2 179000 55624 > 01:35:01 0.2 179000 55508 > 01:40:01 0.2 179000 55500 > 01:45:01 0.2 179000 55696 > 01:50:01 0.2 180024 55696 > 01:55:01 0.2 180024 56684 > 02:00:01 0.2 180024 56684 > 02:05:01 0.2 180024 56656 > 02:10:01 0.2 180024 56648 > 02:15:01 0.2 181048 57748 > 02:20:01 0.2 181048 57524 > 02:25:01 0.2 181048 57524 > 02:30:01 0.2 181048 57556 > 02:35:01 0.2 181048 57556 > 02:40:01 0.2 182072 58668 > 02:45:01 0.2 182072 58560 > 02:50:01 0.2 182072 58692 > 02:55:01 0.2 182072 58544 > 03:00:01 0.2 182072 58712 > 03:05:01 0.2 183096 58712 > 03:10:01 0.2 183096 59568 > 03:15:01 0.2 183096 59652 > 03:20:01 0.2 183096 59672 > 03:25:01 0.2 184120 60508 > 03:30:02 0.2 184120 60508 > 03:35:01 0.2 184120 60476 > 03:40:01 0.2 184120 60652 > 03:45:01 0.2 184248 60652 > 03:50:01 0.2 185272 61496 > 03:55:01 0.2 185272 61484 > 04:00:01 0.2 185272 61484 > 04:05:01 0.2 185272 61636 > 04:10:01 0.2 185272 61552 > 04:15:01 0.2 186328 62656 > 04:20:01 0.2 186328 62656 > 04:25:01 0.2 186328 62756 > 04:30:01 0.2 186328 62756 > 04:35:01 0.2 186328 62644 > 04:40:01 0.2 187352 63700 > 04:45:01 0.2 187352 63680 > 04:50:01 0.2 187352 63868 > 04:55:01 0.2 187352 63972 > 05:00:01 0.2 188376 64988 > 05:05:01 0.2 188376 65088 > 05:10:01 0.2 188376 65088 > 05:15:01 0.2 188376 64960 > 05:20:01 0.2 188376 64960 > 05:25:01 0.2 189400 65956 > 05:30:01 0.2 189400 65956 > 05:35:01 0.2 189400 65936 > 05:40:01 0.2 189400 65992 > 05:45:01 0.2 189400 66012 > 05:50:01 0.2 190424 67068 > 05:55:01 0.2 190424 67068 > 06:00:01 0.2 190424 66896 > 06:05:01 0.2 190424 67044 > 06:10:01 0.2 190424 66980 > 06:15:01 0.3 191448 67900 > 06:20:01 0.2 191448 67844 > 06:25:01 0.3 191448 68012 > 06:30:01 0.3 191448 68012 > 06:35:01 0.3 191448 67888 > 06:40:01 0.3 192472 68912 > 06:45:01 0.3 192472 68912 > 06:50:01 0.3 192472 68824 > 06:55:01 0.3 192472 68788 > 07:00:01 0.3 192472 68748 > 07:05:01 0.3 193496 69804 > 07:10:01 0.3 193496 69804 > 07:15:01 0.3 193496 69700 > 07:20:01 0.3 193496 69896 > 07:25:01 0.3 193496 69736 > 07:30:01 0.3 194520 70864 > 07:35:01 0.3 194520 70680 > 07:40:01 0.3 194520 70680 > 07:45:01 0.3 194520 71256 > 07:50:01 0.3 194520 71184 > 07:55:01 0.3 195544 72240 > 08:00:01 0.3 195544 72240 > 08:05:01 0.3 195544 72244 > 08:10:01 0.3 195672 72084 > 08:15:01 0.3 195672 72116 > 08:20:01 0.3 196696 73172 > 08:25:01 0.3 196696 73216 > 08:30:01 0.3 196696 73216 > 08:35:01 0.3 196696 72936 > 08:40:01 0.3 196696 73124 > 08:45:01 0.3 197720 73916 > 08:50:01 0.3 197720 74120 > 08:55:01 0.3 197720 74112 > 09:00:01 0.3 197720 73960 > 09:05:01 0.3 198744 73960 > 09:10:01 0.3 198744 75080 > 09:15:01 0.3 198744 75188 > 09:20:01 0.3 198744 75132 > 09:25:01 0.3 199768 75132 > 09:30:01 0.3 199768 75452 > 09:35:01 0.3 199768 76116 > 09:40:01 0.3 199768 76116 > 09:45:01 0.3 200792 76116 > 09:50:01 0.3 200792 76380 > 09:55:01 0.3 200792 76540 > 10:00:01 0.3 200792 76844 > 10:05:01 0.3 200792 77100 > 10:10:01 0.3 201848 77308 > 10:15:01 0.3 201848 77308 > 10:20:01 0.3 201848 78144 > 10:25:01 0.3 201848 78156 > 10:30:01 0.3 201848 78156 > 10:35:01 0.3 202872 79248 > 10:40:01 0.3 202872 79248 > 10:45:01 0.3 202872 79248 > 10:50:01 0.3 202872 79304 > 10:55:01 0.3 202872 79112 > 11:00:01 0.3 203896 80436 > 11:05:01 0.3 203896 80436 > 11:10:01 0.3 203896 80520 > 11:15:01 0.3 203896 80396 > 11:20:01 0.3 203896 80584 > 11:25:01 0.3 204920 81640 > 11:30:01 0.3 204920 81640 > 11:35:01 0.3 204920 81752 > 11:40:01 0.3 204920 81708 > 11:45:01 0.3 205944 82500 > 11:50:01 0.3 205944 82688 > 11:55:01 0.3 205944 82584 > 12:00:01 0.3 205944 82628 > 12:05:01 0.3 206968 82756 > 12:10:01 0.3 206968 83616 > 12:15:01 0.3 207096 83616 > 12:20:01 0.3 207096 83432 > 12:25:01 0.3 207096 83432 > 12:30:01 0.3 208120 84568 > 12:35:01 0.3 208120 84768 > 12:40:01 0.3 208120 84856 > 12:45:02 0.3 208120 84668 > 12:50:01 0.3 209144 84668 > 12:55:01 0.3 209144 85256 > 13:00:01 0.3 209144 85800 > 13:05:01 0.3 209144 85800 > 13:10:01 0.3 210168 85936 > 13:15:01 0.3 210168 85936 > 13:20:01 0.3 210168 86780 > 13:25:01 0.3 210168 86780 > 13:30:01 0.3 211192 86536 > 13:35:01 0.3 211192 86800 > 13:40:01 0.3 211192 86800 > 13:45:01 0.3 211192 87696 > 13:50:01 0.3 211192 87748 > 13:55:01 0.3 211192 87748 > 14:00:01 0.3 212216 88812 > 14:05:01 0.3 212216 88812 > 14:10:01 0.3 212216 88796 > 14:15:01 0.3 212216 88728 > 14:20:01 0.3 213240 89784 > 14:25:01 0.3 213240 89784 > 14:30:01 0.3 213240 89660 > 14:35:01 0.3 213240 89836 > 14:40:01 0.3 213240 89724 > 14:45:01 0.4 214264 90848 > 14:50:01 0.4 214264 90848 > 14:55:01 0.4 214264 90956 > 15:00:01 0.4 214264 90956 > 15:05:01 0.4 215288 91780 > 15:10:01 0.4 215288 91780 > 15:15:01 0.4 215288 91712 > 15:20:01 0.4 215288 91768 > 15:25:01 0.4 216312 92824 > 15:30:01 0.4 216312 92824 > 15:35:01 0.4 216312 92736 > 15:40:01 0.4 216312 92964 > 15:45:01 0.4 216312 92688 > 15:50:01 0.4 217336 93048 > 15:55:01 0.4 217336 93312 > 16:00:01 0.4 217336 93288 > 16:05:01 0.4 217336 93744 > 16:10:01 0.4 218392 94908 > 16:15:01 0.4 218520 94908 > 16:20:01 0.4 218520 94992 > 16:25:01 0.4 218520 94940 > 16:30:01 0.4 219544 95996 > 16:35:01 0.4 219544 96076 > 16:40:01 0.4 219544 96048 > 16:45:01 0.4 219544 95948 > 16:50:01 0.4 219544 95948 > 16:55:01 0.4 220568 96968 > 17:00:01 0.4 220568 96968 > 17:05:01 0.4 220568 96936 > 17:10:01 0.4 220568 96976 > 17:15:01 0.4 220568 97048 > 17:20:01 0.4 221592 98368 > 17:25:01 0.4 221592 98224 > 17:30:01 0.4 221592 98320 > 17:35:01 0.4 221592 98096 > 17:40:01 0.4 222620 98364 > 17:45:01 0.4 222620 99332 > 17:50:01 0.4 222620 99332 > 17:55:01 0.4 222620 99064 > 18:00:01 0.4 223644 100120 > 18:05:01 0.4 223644 100104 > 18:10:01 0.4 223644 100084 > 18:15:01 0.4 223644 100084 > 18:20:01 0.4 223644 100268 > 18:25:01 0.4 223644 100232 > 18:30:01 0.4 224668 100496 > 18:35:01 0.4 224668 101304 > 18:40:01 0.4 224668 101168 > 18:45:01 0.4 224668 101220 > 18:50:01 0.4 225692 101220 > 18:55:01 0.4 225692 101484 > 19:00:01 0.4 225692 102080 > 19:05:01 0.4 225692 102200 > 19:10:01 0.4 226716 102200 > 19:15:01 0.4 226716 102464 > 19:20:01 0.4 226716 103184 > 19:25:01 0.4 226716 103232 > 19:30:01 0.4 226716 103004 > 19:35:01 0.4 227740 104212 > 19:40:01 0.4 227740 104212 > 19:45:01 0.4 227740 104056 > 19:50:01 0.4 227740 104056 > 19:55:01 0.4 228764 105252 > 20:00:01 0.4 228764 105344 > 20:05:01 0.4 229788 105356 > 20:10:02 0.4 229788 106240 > 20:15:01 0.4 229916 106240 > 20:20:01 0.4 229916 106364 > 20:25:01 0.4 229916 106364 > 20:30:01 0.4 230940 107420 > 20:35:01 0.4 230940 107404 > 20:40:01 0.4 230940 107344 > 20:45:01 0.4 230940 107204 > 20:50:01 0.4 230940 107564 > 20:55:01 0.4 231964 108620 > 21:00:01 0.4 231964 108620 > 21:05:01 0.4 231964 108524 > 21:10:01 0.4 231964 108372 > 21:15:01 0.4 232988 109384 > 21:20:01 0.4 232988 109384 > 21:25:01 0.4 232988 109432 > 21:30:01 0.4 232988 109432 > 21:35:01 0.4 234012 110304 > 21:40:01 0.4 234012 110568 > 21:45:01 0.4 234012 110568 > 21:50:01 0.4 234012 110384 > 21:55:01 0.4 234012 110260 > 22:00:01 0.4 234012 110260 > 22:05:01 0.4 235068 111424 > 22:10:01 0.4 235068 111424 > 22:15:01 0.4 235068 111392 > 22:20:01 0.4 235068 111392 > 22:25:01 0.4 235068 111324 > 22:30:01 0.4 236092 112440 > 22:35:01 0.4 236092 112612 > 22:40:01 0.4 236092 112652 > 22:45:01 0.4 236092 112520 > 22:50:01 0.4 236092 112608 > 22:55:01 0.5 237116 113496 > 23:00:01 0.5 237116 113496 > 23:05:01 0.5 237116 113616 > 23:10:01 0.5 237116 113616 > 23:15:01 0.5 237116 113560 > 23:20:01 0.5 238140 113800 > 23:25:01 0.5 238140 114064 > 23:30:01 0.5 238140 114416 > 23:35:01 0.5 238140 114700 > 23:40:01 0.5 239164 114700 > 23:45:01 0.5 239164 114964 > 23:50:02 0.5 239164 115740 > 23:55:01 0.5 239164 115740 > 00:00:01 0.5 239164 115492 > 00:05:01 0.5 239164 115492 > 00:10:01 0.5 240188 116504 > 00:15:01 0.5 240188 116504 > 00:20:01 0.5 240188 116720 > 00:25:01 0.5 240188 116524 > 00:30:01 0.5 240188 116808 > 00:35:01 0.5 241340 117864 > 00:40:01 0.5 241340 117936 > 00:45:01 0.5 241340 118048 > 00:50:01 0.5 241340 118048 > 00:55:01 0.5 242364 118860 > 01:00:01 0.5 242364 118848 > 01:05:01 0.5 242364 118956 > 01:10:01 0.5 242364 118900 > 01:15:01 0.5 242364 118900 > 01:20:01 0.5 242364 119044 > 01:25:01 0.5 243388 119172 > 01:30:01 0.5 243388 119792 > 01:35:01 0.5 243388 119792 > 01:40:01 0.5 243388 120076 > 01:45:01 0.5 243388 120084 > 01:50:01 0.5 244412 120084 > 01:55:01 0.5 244412 120452 > 02:00:01 0.5 244412 120716 > 02:05:01 0.5 244412 120956 > 02:10:01 0.5 244412 120956 > 02:15:01 0.5 245436 121220 > 02:20:01 0.5 245436 121364 > 02:25:01 0.5 245436 121628 > 02:30:01 0.5 245436 122040 > 02:35:01 0.5 245436 122008 > 02:40:01 0.5 246460 123000 > 02:45:01 0.5 246460 123000 > 02:50:01 0.5 246460 122908 > 02:55:01 0.5 246460 122908 > 03:00:01 0.5 246460 122908 > 03:05:01 0.5 247484 124000 > 03:10:01 0.5 247484 124000 > 03:15:01 0.5 247484 123924 > 03:20:01 0.5 247484 123980 > 03:25:01 0.5 247484 123980 > 03:30:01 0.5 248508 125036 > 03:35:01 0.5 248508 125012 > 03:40:01 0.5 248508 124952 > 03:45:01 0.5 248508 125336 > 03:50:01 0.5 248508 125128 > 03:55:01 0.5 249532 126212 > 04:00:01 0.5 249532 126212 > 04:05:01 0.5 249532 126372 > 04:10:01 0.5 249532 126348 > 04:15:01 0.5 250556 127148 > 04:20:01 0.5 250556 127412 > 04:25:01 0.5 250556 127244 > 04:30:01 0.5 250556 127164 > 04:35:01 0.5 250556 127196 > 04:40:01 0.5 251612 128224 > 04:45:01 0.5 251612 128128 > 04:50:01 0.5 251612 128128 > 04:55:01 0.5 251740 128080 > 05:00:01 0.5 251740 128236 > 05:05:01 0.5 252764 129212 > 05:10:01 0.5 252764 129212 > 05:15:01 0.5 252764 129208 > 05:20:01 0.5 252764 129108 > 05:25:01 0.5 252764 129020 > 05:30:01 0.5 253788 129404 > 05:35:01 0.5 253788 129600 > 05:40:01 0.5 253788 130572 > 05:45:01 0.5 253788 130572 > 05:50:01 0.5 253788 130572 > 05:55:01 0.5 254812 130572 > 06:00:01 0.5 254812 131428 > 06:05:01 0.5 254812 131428 > 06:10:01 0.5 254812 131468 > 06:15:01 0.5 254812 131468 > 06:20:01 0.5 255836 131468 > 06:25:01 0.5 255836 132212 > 06:30:01 0.5 255836 132328 > 06:35:01 0.5 255836 132224 > 06:40:01 0.5 255836 132120 > 06:45:01 0.5 256860 132384 > 06:50:01 0.5 256860 132648 > 06:55:01 0.5 256860 133044 > 07:00:01 0.5 256860 133232 > 07:05:01 0.5 256860 133232 > 07:10:01 0.5 257884 133360 > 07:15:01 0.5 257884 134336 > 07:20:01 0.5 257884 134336 > 07:25:01 0.5 257884 134220 > 07:30:01 0.5 257884 134220 > 07:35:01 0.5 258908 134220 > 07:40:01 0.5 258908 134348 > 07:45:01 0.5 258908 134612 > 07:50:01 0.5 258908 135368 > 07:55:01 0.5 258908 135284 > 08:00:01 0.5 258908 135484 > 08:05:01 0.5 259932 135484 > 08:10:01 0.6 259932 135748 > 08:15:01 0.6 259932 136616 > 08:20:01 0.6 259932 136616 > 08:25:01 0.6 259932 136616 > 08:30:01 0.6 259932 136616 > 08:35:01 0.6 260956 136568 > 08:40:01 0.6 260956 136832 > 08:45:01 0.6 260956 137396 > 08:50:01 0.6 260956 137660 > 08:55:01 0.6 260956 137660 > 09:00:01 0.6 260956 137380 > 09:05:01 0.6 260956 137508 > 09:10:01 0.6 261980 137624 > 09:15:01 0.6 261980 137888 > 09:20:01 0.6 261980 138428 > 09:25:01 0.6 261980 138416 > 09:30:01 0.6 261980 138416 > 09:35:01 0.6 263004 138588 > 09:40:01 0.6 263004 138852 > 09:45:01 0.6 263132 139116 > 09:50:01 0.6 263132 139672 > 09:55:01 0.6 263132 139596 > 10:00:01 0.6 264156 140932 > 10:05:01 0.6 264156 140932 > 10:10:01 0.6 264156 140932 > 10:15:01 0.6 264156 140732 > 10:20:01 0.6 264156 140700 > 10:25:01 0.6 265180 140872 > 10:30:01 0.6 265180 141764 > 10:35:01 0.6 265180 141764 > 10:40:01 0.6 265180 141856 > 10:45:01 0.6 265180 141652 > 10:50:01 0.6 265180 141748 > 10:55:01 0.6 265180 141588 > 11:00:01 0.6 266204 141588 > 11:05:01 0.6 266204 141852 > 11:10:01 0.6 266204 141852 > 11:15:01 0.6 266204 142520 > 11:20:01 0.6 266204 142652 > 11:25:01 0.6 267228 142652 > 11:30:01 0.6 267228 142916 > 11:35:01 0.6 267228 142956 > 11:40:01 0.6 267228 143296 > 11:45:01 0.6 267228 143392 > 11:50:01 0.6 267228 143984 > 11:55:01 0.6 268284 144924 > 12:00:01 0.6 268284 144924 > 12:05:01 0.6 268284 144784 > 12:10:01 0.6 268284 144784 > 12:15:01 0.6 268284 145008 > 12:20:01 0.6 268284 144904 > 12:25:01 0.6 269308 145960 > 12:30:01 0.6 269308 145960 > 12:35:01 0.6 269308 145752 > 12:40:01 0.6 269308 145968 > 12:45:01 0.6 269308 145852 > 12:50:01 0.6 269308 145852 > 12:55:01 0.6 270332 146908 > 13:00:01 0.6 271356 146660 > 13:05:01 0.6 271356 147976 > 13:10:01 0.6 271356 147976 > 13:15:01 0.6 271356 147972 > 13:20:01 0.6 272380 149016 > 13:25:01 0.6 272380 149016 > 13:30:01 0.6 272380 149216 > 13:35:01 0.6 272380 149216 > 13:40:01 0.6 272380 148988 > 13:45:01 0.6 272380 149148 > 13:50:01 0.6 273404 149116 > 13:55:01 0.6 273404 149380 > 14:00:01 0.6 273404 149956 > 14:05:01 0.6 273404 150148 > 14:10:01 0.6 274428 151012 > 14:15:01 0.6 274428 151012 > 14:20:01 0.6 274556 151012 > 14:25:01 0.6 274556 151020 > 14:30:01 0.6 274556 151112 > 14:35:01 0.6 274556 151112 > 14:40:01 0.6 275584 152292 > 14:45:02 0.6 275584 152292 > 14:50:01 0.6 275584 152292 > 14:55:01 0.6 275584 152292 > 15:00:01 0.6 275584 152292 > 15:05:01 0.6 276608 153088 > 15:10:01 0.6 276608 153088 > 15:15:01 0.6 276608 153184 > 15:20:01 0.6 276608 153184 > 15:25:01 0.6 276608 153048 > 15:30:01 0.6 277632 154088 > 15:35:01 0.6 277632 154088 > 15:40:01 0.6 277632 154096 > 15:45:01 0.6 277632 154224 > 15:50:01 0.6 277632 154144 > 15:55:01 0.6 277632 154264 > 16:00:01 0.6 278656 155200 > 16:05:01 0.6 278656 155088 > 16:10:01 0.6 278656 155088 > 16:15:01 0.6 278656 155048 > 16:20:01 0.6 278656 155004 > 16:25:01 0.6 278656 155244 > 16:30:01 0.6 278656 155244 > 16:35:01 0.6 279680 156036 > 16:40:01 0.6 279680 156036 > 16:45:01 0.6 279680 156216 > 16:50:01 0.6 279680 156132 > 16:55:01 0.6 280704 157188 > 17:00:01 0.6 280704 157128 > 17:05:01 0.6 280704 157160 > 17:10:01 0.6 280704 157140 > 17:15:01 0.6 280704 157140 > 17:20:01 0.6 281728 158260 > 17:25:01 0.6 281728 158132 > 17:30:01 0.6 281728 158276 > 17:35:01 0.7 281728 158360 > 17:40:01 0.7 282752 159320 > 17:45:01 0.7 282752 159320 > 17:50:01 0.7 282752 159204 > 17:55:01 0.7 282752 159272 > 18:00:01 0.7 282752 159272 > 18:05:01 0.7 283808 160444 > 18:10:02 0.7 283808 160444 > 18:15:01 0.7 283808 160252 > 18:20:01 0.7 283808 160408 > 18:25:01 0.7 284832 161580 > 18:30:01 0.7 284832 161580 > 18:35:01 0.7 284832 161392 > 18:40:01 0.7 284832 161388 > 18:45:01 0.7 285984 162568 > 18:50:01 0.7 285984 162568 > 18:55:01 0.7 285984 162548 > 19:00:01 0.7 285984 162548 > 19:05:01 0.7 285984 162516 > 19:10:01 0.7 287008 163444 > 19:15:01 0.7 287008 163444 > 19:20:01 0.7 287008 163448 > 19:25:01 0.7 287008 163332 > 19:30:01 0.7 287008 163548 > 19:35:01 0.7 288032 164644 > 19:40:01 0.7 288032 164644 > 19:45:01 0.7 288032 164620 > 19:50:01 0.7 288032 164512 > 19:55:01 0.7 289056 165528 > 20:00:01 0.7 289056 165608 > 20:05:01 0.7 289056 165500 > 20:10:01 0.7 289056 165528 > 20:15:01 0.7 290080 165660 > 20:20:01 0.7 290080 165956 > 20:25:01 0.7 290080 166432 > 20:30:01 0.7 290080 166588 > 20:35:01 0.7 290080 166588 > 20:40:01 0.7 290080 166588 > 20:45:01 0.7 291104 167492 > 20:50:01 0.7 291104 167492 > 20:55:01 0.7 291104 167508 > 21:00:01 0.7 291104 167508 > 21:05:01 0.7 292128 167508 > 21:10:01 0.7 292128 168576 > 21:15:01 0.7 292128 168420 > 21:20:01 0.7 292128 168396 > 21:25:01 0.7 293152 169716 > 21:30:01 0.7 293152 169752 > 21:35:01 0.7 293152 169756 > 21:40:01 0.7 293152 169604 > 21:45:01 0.7 293152 169604 > 21:50:02 0.7 294176 170804 > 21:55:01 0.7 294176 170804 > 22:00:01 0.7 294176 170576 > 22:05:01 0.7 294176 170764 > 22:10:01 0.7 294176 170744 > 22:15:01 0.7 295200 171632 > 22:20:01 0.7 295200 171632 > 22:25:01 0.7 295200 171632 > 22:30:01 0.7 295200 171820 > 22:35:01 0.7 296224 172988 > 22:40:01 0.7 296224 172988 > 22:45:01 0.7 296224 172804 > 22:50:01 0.7 296352 172872 > 22:55:01 0.7 296352 172812 > 23:00:01 0.7 297376 173076 > 23:05:01 0.7 297376 173164 > 23:10:01 0.7 297376 173416 > 23:15:01 0.7 297376 173664 > 23:20:01 0.7 298400 174656 > 23:25:01 0.7 298400 174656 > 23:30:01 0.7 298400 174656 > 23:35:01 0.7 298400 174656 > 23:40:01 0.7 298400 174660 > 23:45:01 0.7 299424 174612 > 23:50:01 0.7 299424 175720 > 23:55:01 0.7 299424 175764 > 00:00:01 0.7 299424 176088 > 00:05:01 0.7 300480 176088 > 00:10:01 0.7 300480 176352 > 00:15:01 0.7 300480 176480 > 00:20:01 0.7 300480 177200 > 00:25:01 0.7 300480 177352 > 00:30:01 0.7 300480 177276 > 00:35:01 0.7 301504 178404 > 00:40:01 0.7 301504 178336 > 00:45:01 0.7 301504 178336 > 00:50:01 0.7 301504 178320 > 00:55:01 0.7 301504 178236 > 01:00:01 0.7 302528 179240 > 01:05:01 0.7 302528 179240 > 01:10:01 0.7 302528 179292 > 01:15:01 0.7 302528 179352 > 01:20:01 0.7 302528 179352 > 01:25:02 0.7 302528 179176 > 01:30:01 0.7 302528 179060 > > > Sorry for the long post. > -- > Regards, > Sébastien Han. > > > On Wed, Mar 13, 2013 at 11:44 AM, Stefan Priebe - Profihost AG > <s.priebe@profihost.ag> wrote: > > Hi, > > > > are there any known ceph-mon memory leaks in bobtail? Today i've seen a > > ceph-mon process consuming 50GB Memory. > > > > Stefan > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mon memory leak 2013-03-13 13:42 ` Travis Rhoden @ 2013-03-13 13:58 ` Joao Eduardo Luis 2013-03-13 14:06 ` Stefan Priebe - Profihost AG 0 siblings, 1 reply; 6+ messages in thread From: Joao Eduardo Luis @ 2013-03-13 13:58 UTC (permalink / raw) To: Travis Rhoden Cc: Stefan Priebe - Profihost AG, Sébastien Han, ceph-devel@vger.kernel.org On 03/13/2013 01:42 PM, Travis Rhoden wrote: > Hi Stefan, > > I have seen a mon grow quite high on Bobtail. Joao had some theories > as for why, and I was able to provide him with a memory dump of the > running process. No word yet on whether it revealed anything, but I > know it is on his stack. Background here: > http://thread.gmane.org/gmane.comp.file-systems.ceph.user/14 > > - Travis Hi all, As Travis points out, this has been seen before. I've spent a fair amount of time chasing this down now, but haven't got to any useful conclusions yet. Although this is concerning, chasing down this kind of behaviour tends to be a time sink and there are other more pressing issues that have been requiring most of my attention. This hasn't been forgotten and I've been allocating my time to it as possible. On Sébastien's issue, that may very well be caused by some of the memleaks that have been fixed some time since argonaut, prior to bobtail. I'm expecting to be able to find the time Real Soon Now to put a doc together with ways, for anyone willing, to provide us further insight on what's happening. The monitor is supposed to be able to dump a heap profile (using gperftools) on-the-fly, but I recall having some issues with that not so long ago, so that's probably one thing to look into asap. In any case, Stefan, do you by chance still have that monitor going? If so, are you able to tell us what's that monitor rank? 'ceph mon_status' should help you assessing that. Furthermore, assuming you are using a version prior to v0.58, any chance you can run a 'du -chs /var/lib/ceph/mon/ceph-foo', with foo being the mem-hogging monitor's id, and then, if you notice an abnormal disk consumption on one of the directories, dive in to check where said consumption is happening? -Joao > > On Wed, Mar 13, 2013 at 7:02 AM, Sébastien Han <han.sebastien@gmail.com> wrote: >> >> Hi, >> >> I do have some ceph-mon memory leak as well but it's on Argonaut. >> >> See the memory consumption here: >> [snipped] >> >> Sorry for the long post. >> -- >> Regards, >> Sébastien Han. >> >> >> On Wed, Mar 13, 2013 at 11:44 AM, Stefan Priebe - Profihost AG >> <s.priebe@profihost.ag> wrote: >>> Hi, >>> >>> are there any known ceph-mon memory leaks in bobtail? Today i've seen a >>> ceph-mon process consuming 50GB Memory. >>> >>> Stefan >>> -- >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >>> the body of a message to majordomo@vger.kernel.org >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> -- >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mon memory leak 2013-03-13 13:58 ` Joao Eduardo Luis @ 2013-03-13 14:06 ` Stefan Priebe - Profihost AG 2013-03-13 14:16 ` Joao Eduardo Luis 0 siblings, 1 reply; 6+ messages in thread From: Stefan Priebe - Profihost AG @ 2013-03-13 14:06 UTC (permalink / raw) To: Joao Eduardo Luis Cc: Travis Rhoden, Sébastien Han, ceph-devel@vger.kernel.org Hi Joao, Am 13.03.2013 14:58, schrieb Joao Eduardo Luis: > I'm expecting to be able to find the time Real Soon Now to put a doc > together with ways, for anyone willing, to provide us further insight on > what's happening. The monitor is supposed to be able to dump a heap > profile (using gperftools) on-the-fly, but I recall having some issues > with that not so long ago, so that's probably one thing to look into asap. Thanks - that would be great i'm willing ;-) > In any case, Stefan, do you by chance still have that monitor going? Sadly not i needed the memory ;-) > Furthermore, assuming you are using a > version prior to v0.58, Yes current Bobtail 0.56.3 > any chance you can run a 'du -chs > /var/lib/ceph/mon/ceph-foo', No idea what you expect to see but all mon servers consume 180MB of disk space. [: ~]# du -chs /ceph/mon 180M /ceph/mon 180M total Greets, Stefan ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: mon memory leak 2013-03-13 14:06 ` Stefan Priebe - Profihost AG @ 2013-03-13 14:16 ` Joao Eduardo Luis 0 siblings, 0 replies; 6+ messages in thread From: Joao Eduardo Luis @ 2013-03-13 14:16 UTC (permalink / raw) To: Stefan Priebe - Profihost AG Cc: Travis Rhoden, Sébastien Han, ceph-devel@vger.kernel.org On 03/13/2013 02:06 PM, Stefan Priebe - Profihost AG wrote: > Hi Joao, > Am 13.03.2013 14:58, schrieb Joao Eduardo Luis: > >> I'm expecting to be able to find the time Real Soon Now to put a doc >> together with ways, for anyone willing, to provide us further insight on >> what's happening. The monitor is supposed to be able to dump a heap >> profile (using gperftools) on-the-fly, but I recall having some issues >> with that not so long ago, so that's probably one thing to look into asap. > > Thanks - that would be great i'm willing ;-) Cool, good to know. I'll post something to the ml as soon as I have something worth sharing. Most likely next week. It's unlikely I'll get around to do it still this week. > >> In any case, Stefan, do you by chance still have that monitor going? > Sadly not i needed the memory ;-) > >> Furthermore, assuming you are using a >> version prior to v0.58, > Yes current Bobtail 0.56.3 > >> any chance you can run a 'du -chs >> /var/lib/ceph/mon/ceph-foo', > No idea what you expect to see but all mon servers consume 180MB of disk > space. eh, just trying to figure out if said memory consumption is due to some large map being kept in memory. Given we always write to the store when maps are updated, and only after that do we load them to memory, I was hoping to see some incredibly high disk consumption (while the monitor was running) in order to assert whether the mem consumption would be tied to some specific map. But given that the store is periodically trimmed this was a long shot anyway. Still worth a try though ;) -Joao > > [: ~]# du -chs /ceph/mon > 180M /ceph/mon > 180M total > > Greets, > Stefan > ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-03-13 14:17 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-03-13 10:44 mon memory leak Stefan Priebe - Profihost AG 2013-03-13 11:02 ` Sébastien Han 2013-03-13 13:42 ` Travis Rhoden 2013-03-13 13:58 ` Joao Eduardo Luis 2013-03-13 14:06 ` Stefan Priebe - Profihost AG 2013-03-13 14:16 ` Joao Eduardo Luis
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.