From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dave Spano Subject: OSD Crash Date: Mon, 4 Mar 2013 13:02:03 -0500 (EST) Message-ID: <33027299.339.1362420122090.JavaMail.dspano@it1> References: <8566685.312.1362419807745.JavaMail.dspano@it1> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_336_6488584.1362420122087" Return-path: Received: from rrcs-24-103-221-203.nys.biz.rr.com ([24.103.221.203]:54526 "EHLO mail.optogenics.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758229Ab3CDSCF (ORCPT ); Mon, 4 Mar 2013 13:02:05 -0500 In-Reply-To: <8566685.312.1362419807745.JavaMail.dspano@it1> Sender: ceph-devel-owner@vger.kernel.org List-ID: To: ceph-devel ------=_Part_336_6488584.1362420122087 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit I had one of my OSDs crash yesterday. I'm using ceph version 0.56.3 (6eb7e15a4783b122e9b0c85ea9ba064145958aa5). The part of the log file where the crash happened is attached. Not really sure what lead up to it, but I did get an alert from my server monitor telling me my swap space got really low around the time it crashed. The OSD reconnected after restarting the service. Currently, I'm waiting patiently as 1 of my 400 pgs gets out of active+clean+scrubbing status. Dave Spano Optogenics Systems Administrator ------=_Part_336_6488584.1362420122087 Content-Type: text/x-log; name=osd.0.log Content-Disposition: attachment; filename=osd.0.log Content-Transfer-Encoding: base64 ICAgLTE3PiAyMDEzLTAzLTAzIDEzOjAyOjEzLjQ3ODE1MiA3ZjVkNWE5YjU3MDAgIDUgLS1PU0Q6 OnRyYWNrZXItLSByZXFpZDogY2xpZW50LjEzMDM5LjA6Njg2MDM1OSwgc2VxOiA1MzkzMjIyLCB0 aW1lOiAyMDEzLTAzLTAzIDEzOjAyOjEzLjQ3ODEzNCwgZXZlbnQ6IHdyaXRlX3RocmVhZF9pbl9q b3VybmFsX2J1ZmZlciwgcmVxdWVzdDogb3NkX3N1Yl9vcChjbGllbnQuMTMwMzkuMDo2ODYwMzU5 IDMuMCBhMTBjMTdjOC9yYi4wLjJkZDcuMTZkMjhjNGYuMDAwMDAwMDAwMDJmL2hlYWQvLzMgW10g diA0MTEnMTk4MDA3NCBzbmFwc2V0PTA9W106W10gc25hcGM9MD1bXSkgdjcKICAgLTE2PiAyMDEz LTAzLTAzIDEzOjAyOjEzLjQ3ODE1MyA3ZjVkNTU5YWI3MDAgIDEgLS0gMTkyLjE2OC4zLjExOjY4 MDEvNDUwMCAtLT4gb3NkLjEgMTkyLjE2OC4zLjEyOjY4MDIvMjQ2NyAtLSBvc2Rfc3ViX29wX3Jl cGx5KGNsaWVudC4xNDAwMC4xOjU3MDcwMCAwLjE2IDVlMDFhOTYvMTAwMDAzNzk3ZjIuMDAwMDAw MDAvaGVhZC8vMCBbXSBvbmRpc2ssIHJlc3VsdCA9IDApIHYxIC0tID8rMCAweGM0NWNjODAKICAg LTE1PiAyMDEzLTAzLTAzIDEzOjAyOjEzLjQ3ODE4NCA3ZjVkNWE5YjU3MDAgIDUgLS1PU0Q6OnRy YWNrZXItLSByZXFpZDogY2xpZW50LjE0MDAwLjE6NTcwNzAxLCBzZXE6IDUzOTMyMjMsIHRpbWU6 IDIwMTMtMDMtMDMgMTM6MDI6MTMuNDc4MTg0LCBldmVudDogd3JpdGVfdGhyZWFkX2luX2pvdXJu YWxfYnVmZmVyLCByZXF1ZXN0OiBvc2Rfc3ViX29wKGNsaWVudC4xNDAwMC4xOjU3MDcwMSAwLjIy IDQwZGNjY2EyLzEwMDAwMTE2NGNhLjAwMDAwMDAyL2hlYWQvLzAgW10gdiA0MTEnNDQ3MzY5IHNu YXBzZXQ9MD1bXTpbXSBzbmFwYz0wPVtdKSB2NwogICAtMTQ+IDIwMTMtMDMtMDMgMTM6MDI6MTMu NDc4MjA5IDdmNWQ1YTliNTcwMCAgNSAtLU9TRDo6dHJhY2tlci0tIHJlcWlkOiBjbGllbnQuMTE3 NTUuMDoyNjI1NjU4LCBzZXE6IDUzOTMyMjUsIHRpbWU6IDIwMTMtMDMtMDMgMTM6MDI6MTMuNDc4 MjA5LCBldmVudDogd3JpdGVfdGhyZWFkX2luX2pvdXJuYWxfYnVmZmVyLCByZXF1ZXN0OiBvc2Rf c3ViX29wKGNsaWVudC4xMTc1NS4wOjI2MjU2NTggMy43IDJjYjAwNmE3L3JiLjAuMmVhNC42MTRj Mjc3Zi4wMDAwMDAwMDEwM2QvaGVhZC8vMyBbXSB2IDQxMSc2MDk1NTI5IHNuYXBzZXQ9MD1bXTpb XSBzbmFwYz0wPVtdKSB2NwogICAtMTM+IDIwMTMtMDMtMDMgMTM6MDI6MTMuNDc4MjM0IDdmNWQ1 YTliNTcwMCAgNSAtLU9TRDo6dHJhY2tlci0tIHJlcWlkOiBjbGllbnQuMTE3NTUuMDoyNjI1NjU5 LCBzZXE6IDUzOTMyMjYsIHRpbWU6IDIwMTMtMDMtMDMgMTM6MDI6MTMuNDc4MjM0LCBldmVudDog d3JpdGVfdGhyZWFkX2luX2pvdXJuYWxfYnVmZmVyLCByZXF1ZXN0OiBvc2Rfc3ViX29wKGNsaWVu dC4xMTc1NS4wOjI2MjU2NTkgMy43IDJjYjAwNmE3L3JiLjAuMmVhNC42MTRjMjc3Zi4wMDAwMDAw MDEwM2QvaGVhZC8vMyBbXSB2IDQxMSc2MDk1NTMwIHNuYXBzZXQ9MD1bXTpbXSBzbmFwYz0wPVtd KSB2NwogICAtMTI+IDIwMTMtMDMtMDMgMTM6MDI6MTMuNDg0Njk2IDdmNWQ1NDlhOTcwMCAgMSAt LSAxOTIuMTY4LjMuMTE6NjgwMC80NTAwIDw9PSBjbGllbnQuMTE3NTUgMTkyLjE2OC4xLjY0OjAv MTA2MjQxMSA5MDEyOCA9PT09IHBpbmcgdjEgPT09PSAwKzArMCAoMCAwIDApIDB4ZmY0ZTAwMCBj b24gMHgzMDdhNmUwCiAgIC0xMT4gMjAxMy0wMy0wMyAxMzowMjoxMy40ODk0NTcgN2Y1ZDRmOTlm NzAwICA1IC0tT1NEOjp0cmFja2VyLS0gcmVxaWQ6IGNsaWVudC4xMTc1NS4wOjI2MjU2NjAsIHNl cTogNTM5MzIyNywgdGltZTogMjAxMy0wMy0wMyAxMzowMjoxMy40ODk0NTcsIGV2ZW50OiBzdGFy dGVkLCByZXF1ZXN0OiBvc2Rfc3ViX29wKGNsaWVudC4xMTc1NS4wOjI2MjU2NjAgMy43IDJjYjAw NmE3L3JiLjAuMmVhNC42MTRjMjc3Zi4wMDAwMDAwMDEwM2QvaGVhZC8vMyBbXSB2IDQxMSc2MDk1 NTMxIHNuYXBzZXQ9MD1bXTpbXSBzbmFwYz0wPVtdKSB2NwogICAtMTA+IDIwMTMtMDMtMDMgMTM6 MDI6MTMuNDg5NTAzIDdmNWQ0Zjk5ZjcwMCAgNSAtLU9TRDo6dHJhY2tlci0tIHJlcWlkOiBjbGll bnQuMTE3NTUuMDoyNjI1NjYwLCBzZXE6IDUzOTMyMjcsIHRpbWU6IDIwMTMtMDMtMDMgMTM6MDI6 MTMuNDg5NTAzLCBldmVudDogY29tbWl0X3F1ZXVlZF9mb3Jfam91cm5hbF93cml0ZSwgcmVxdWVz dDogb3NkX3N1Yl9vcChjbGllbnQuMTE3NTUuMDoyNjI1NjYwIDMuNyAyY2IwMDZhNy9yYi4wLjJl YTQuNjE0YzI3N2YuMDAwMDAwMDAxMDNkL2hlYWQvLzMgW10gdiA0MTEnNjA5NTUzMSBzbmFwc2V0 PTA9W106W10gc25hcGM9MD1bXSkgdjcKICAgIC05PiAyMDEzLTAzLTAzIDEzOjAyOjEzLjU3MTYz MiA3ZjVkNTAxYTA3MDAgIDUgLS1PU0Q6OnRyYWNrZXItLSByZXFpZDogY2xpZW50LjExNzU1LjA6 MjYyNTY1Nywgc2VxOiA1MzkzMjI0LCB0aW1lOiAyMDEzLTAzLTAzIDEzOjAyOjEzLjU3MTYzMSwg ZXZlbnQ6IHN0YXJ0ZWQsIHJlcXVlc3Q6IG9zZF9vcChjbGllbnQuMTE3NTUuMDoyNjI1NjU3IHJi LjAuMmVhNC42MTRjMjc3Zi4wMDAwMDAwMDAwM2QgW3dyaXRlIDEyNTMzNzZ+NDA5Nl0gMy5jN2Jk NmZmMSkgdjQKICAgIC04PiAyMDEzLTAzLTAzIDEzOjAyOjEzLjU3MTY2MSA3ZjVkNTAxYTA3MDAg IDUgLS1PU0Q6OnRyYWNrZXItLSByZXFpZDogY2xpZW50LjExNzU1LjA6MjYyNTY1Nywgc2VxOiA1 MzkzMjI0LCB0aW1lOiAyMDEzLTAzLTAzIDEzOjAyOjEzLjU3MTY2MSwgZXZlbnQ6IHN0YXJ0ZWQs IHJlcXVlc3Q6IG9zZF9vcChjbGllbnQuMTE3NTUuMDoyNjI1NjU3IHJiLjAuMmVhNC42MTRjMjc3 Zi4wMDAwMDAwMDAwM2QgW3dyaXRlIDEyNTMzNzZ+NDA5Nl0gMy5jN2JkNmZmMSkgdjQKICAgIC03 PiAyMDEzLTAzLTAzIDEzOjAyOjEzLjU3MTczMyA3ZjVkNTAxYTA3MDAgIDUgLS1PU0Q6OnRyYWNr ZXItLSByZXFpZDogY2xpZW50LjExNzU1LjA6MjYyNTY1Nywgc2VxOiA1MzkzMjI0LCB0aW1lOiAy MDEzLTAzLTAzIDEzOjAyOjEzLjU3MTczMywgZXZlbnQ6IHdhaXRpbmcgZm9yIHN1Ym9wcyBmcm9t IFsxXSwgcmVxdWVzdDogb3NkX29wKGNsaWVudC4xMTc1NS4wOjI2MjU2NTcgcmIuMC4yZWE0LjYx NGMyNzdmLjAwMDAwMDAwMDAzZCBbd3JpdGUgMTI1MzM3Nn40MDk2XSAzLmM3YmQ2ZmYxKSB2NAog ICAgLTY+IDIwMTMtMDMtMDMgMTM6MDI6MTMuNTk4MDI4IDdmNWQ1YTliNTcwMCAgNSAtLU9TRDo6 dHJhY2tlci0tIHJlcWlkOiBjbGllbnQuMTMwMzkuMDo2ODYwMzU5LCBzZXE6IDUzOTMyMjIsIHRp bWU6IDIwMTMtMDMtMDMgMTM6MDI6MTMuNTk4MDI3LCBldmVudDogam91cm5hbGVkX2NvbXBsZXRp b25fcXVldWVkLCByZXF1ZXN0OiBvc2Rfc3ViX29wKGNsaWVudC4xMzAzOS4wOjY4NjAzNTkgMy4w IGExMGMxN2M4L3JiLjAuMmRkNy4xNmQyOGM0Zi4wMDAwMDAwMDAwMmYvaGVhZC8vMyBbXSB2IDQx MScxOTgwMDc0IHNuYXBzZXQ9MD1bXTpbXSBzbmFwYz0wPVtdKSB2NwogICAgLTU+IDIwMTMtMDMt MDMgMTM6MDI6MTMuNTk4MDYxIDdmNWQ1YTliNTcwMCAgNSAtLU9TRDo6dHJhY2tlci0tIHJlcWlk OiBjbGllbnQuMTQwMDAuMTo1NzA3MDEsIHNlcTogNTM5MzIyMywgdGltZTogMjAxMy0wMy0wMyAx MzowMjoxMy41OTgwNjEsIGV2ZW50OiBqb3VybmFsZWRfY29tcGxldGlvbl9xdWV1ZWQsIHJlcXVl c3Q6IG9zZF9zdWJfb3AoY2xpZW50LjE0MDAwLjE6NTcwNzAxIDAuMjIgNDBkY2NjYTIvMTAwMDAx MTY0Y2EuMDAwMDAwMDIvaGVhZC8vMCBbXSB2IDQxMSc0NDczNjkgc25hcHNldD0wPVtdOltdIHNu YXBjPTA9W10pIHY3CiAgICAtND4gMjAxMy0wMy0wMyAxMzowMjoxMy41OTgwODEgN2Y1ZDVhOWI1 NzAwICA1IC0tT1NEOjp0cmFja2VyLS0gcmVxaWQ6IGNsaWVudC4xMTc1NS4wOjI2MjU2NTgsIHNl cTogNTM5MzIyNSwgdGltZTogMjAxMy0wMy0wMyAxMzowMjoxMy41OTgwODEsIGV2ZW50OiBqb3Vy bmFsZWRfY29tcGxldGlvbl9xdWV1ZWQsIHJlcXVlc3Q6IG9zZF9zdWJfb3AoY2xpZW50LjExNzU1 LjA6MjYyNTY1OCAzLjcgMmNiMDA2YTcvcmIuMC4yZWE0LjYxNGMyNzdmLjAwMDAwMDAwMTAzZC9o ZWFkLy8zIFtdIHYgNDExJzYwOTU1Mjkgc25hcHNldD0wPVtdOltdIHNuYXBjPTA9W10pIHY3CiAg ICAtMz4gMjAxMy0wMy0wMyAxMzowMjoxMy41OTgwOTggN2Y1ZDVhOWI1NzAwICA1IC0tT1NEOjp0 cmFja2VyLS0gcmVxaWQ6IGNsaWVudC4xMTc1NS4wOjI2MjU2NTksIHNlcTogNTM5MzIyNiwgdGlt ZTogMjAxMy0wMy0wMyAxMzowMjoxMy41OTgwOTgsIGV2ZW50OiBqb3VybmFsZWRfY29tcGxldGlv bl9xdWV1ZWQsIHJlcXVlc3Q6IG9zZF9zdWJfb3AoY2xpZW50LjExNzU1LjA6MjYyNTY1OSAzLjcg MmNiMDA2YTcvcmIuMC4yZWE0LjYxNGMyNzdmLjAwMDAwMDAwMTAzZC9oZWFkLy8zIFtdIHYgNDEx JzYwOTU1MzAgc25hcHNldD0wPVtdOltdIHNuYXBjPTA9W10pIHY3CiAgICAtMj4gMjAxMy0wMy0w MyAxMzowMjoxMy41OTgxMzQgN2Y1ZDVhOWI1NzAwICA1IC0tT1NEOjp0cmFja2VyLS0gcmVxaWQ6 IGNsaWVudC4xMTc1NS4wOjI2MjU2NjAsIHNlcTogNTM5MzIyNywgdGltZTogMjAxMy0wMy0wMyAx MzowMjoxMy41OTgxMzQsIGV2ZW50OiB3cml0ZV90aHJlYWRfaW5fam91cm5hbF9idWZmZXIsIHJl cXVlc3Q6IG9zZF9zdWJfb3AoY2xpZW50LjExNzU1LjA6MjYyNTY2MCAzLjcgMmNiMDA2YTcvcmIu MC4yZWE0LjYxNGMyNzdmLjAwMDAwMDAwMTAzZC9oZWFkLy8zIFtdIHYgNDExJzYwOTU1MzEgc25h cHNldD0wPVtdOltdIHNuYXBjPTA9W10pIHY3CiAgICAtMT4gMjAxMy0wMy0wMyAxMzowMjoxMy41 OTgyNTcgN2Y1ZDVhOWI1NzAwICA1IC0tT1NEOjp0cmFja2VyLS0gcmVxaWQ6IGNsaWVudC4xMTc1 NS4wOjI2MjU2NjAsIHNlcTogNTM5MzIyNywgdGltZTogMjAxMy0wMy0wMyAxMzowMjoxMy41OTgy NTcsIGV2ZW50OiBqb3VybmFsZWRfY29tcGxldGlvbl9xdWV1ZWQsIHJlcXVlc3Q6IG9zZF9zdWJf b3AoY2xpZW50LjExNzU1LjA6MjYyNTY2MCAzLjcgMmNiMDA2YTcvcmIuMC4yZWE0LjYxNGMyNzdm LjAwMDAwMDAwMTAzZC9oZWFkLy8zIFtdIHYgNDExJzYwOTU1MzEgc25hcHNldD0wPVtdOltdIHNu YXBjPTA9W10pIHY3CiAgICAgMD4gMjAxMy0wMy0wMyAxMzowMjoxMy43NTMwNjQgN2Y1ZDRjMDk3 NzAwIC0xICoqKiBDYXVnaHQgc2lnbmFsIChBYm9ydGVkKSAqKgogaW4gdGhyZWFkIDdmNWQ0YzA5 NzcwMAoKIGNlcGggdmVyc2lvbiAwLjU2LjMgKDZlYjdlMTVhNDc4M2IxMjJlOWIwYzg1ZWE5YmEw NjQxNDU5NThhYTUpCiAxOiAvdXNyL2Jpbi9jZXBoLW9zZCgpIFsweDc4NDMwYV0KIDI6ICgoKSsw eGZjYjApIFsweDdmNWQ2MGZjM2NiMF0KIDM6IChnc2lnbmFsKCkrMHgzNSkgWzB4N2Y1ZDVmOTgy NDI1XQogNDogKGFib3J0KCkrMHgxN2IpIFsweDdmNWQ1Zjk4NWI4Yl0KIDU6IChfX2dudV9jeHg6 Ol9fdmVyYm9zZV90ZXJtaW5hdGVfaGFuZGxlcigpKzB4MTFkKSBbMHg3ZjVkNjAyZDQ2OWRdCiA2 OiAoKCkrMHhiNTg0NikgWzB4N2Y1ZDYwMmQyODQ2XQogNzogKCgpKzB4YjU4NzMpIFsweDdmNWQ2 MDJkMjg3M10KIDg6ICgoKSsweGI1OTZlKSBbMHg3ZjVkNjAyZDI5NmVdCiA5OiAoY2VwaDo6YnVm ZmVyOjpjcmVhdGVfcGFnZV9hbGlnbmVkKHVuc2lnbmVkIGludCkrMHg5NSkgWzB4ODJlZjI1XQog MTA6IChQaXBlOjpyZWFkX21lc3NhZ2UoTWVzc2FnZSoqKSsweDI0MjEpIFsweDhkMzU5MV0KIDEx OiAoUGlwZTo6cmVhZGVyKCkrMHg4YzIpIFsweDhlM2RiMl0KIDEyOiAoUGlwZTo6UmVhZGVyOjpl bnRyeSgpKzB4ZCkgWzB4OGU2NjhkXQogMTM6ICgoKSsweDdlOWEpIFsweDdmNWQ2MGZiYmU5YV0K IDE0OiAoY2xvbmUoKSsweDZkKSBbMHg3ZjVkNWZhM2ZjYmRdCiBOT1RFOiBhIGNvcHkgb2YgdGhl IGV4ZWN1dGFibGUsIG9yIGBvYmpkdW1wIC1yZFMgPGV4ZWN1dGFibGU+YCBpcyBuZWVkZWQgdG8g aW50ZXJwcmV0IHRoaXMuCgotLS0gbG9nZ2luZyBsZXZlbHMgLS0tCiAgIDAvIDUgbm9uZQogICAw LyAxIGxvY2tkZXAKICAgMC8gMSBjb250ZXh0CiAgIDEvIDEgY3J1c2gKICAgMS8gNSBtZHMKICAg MS8gNSBtZHNfYmFsYW5jZXIKICAgMS8gNSBtZHNfbG9ja2VyCiAgIDEvIDUgbWRzX2xvZwogICAx LyA1IG1kc19sb2dfZXhwaXJlCiAgIDEvIDUgbWRzX21pZ3JhdG9yCiAgIDAvIDEgYnVmZmVyCiAg IDAvIDEgdGltZXIKICAgMC8gMSBmaWxlcgogICAwLyAxIHN0cmlwZXIKICAgMC8gMSBvYmplY3Rl cgogICAwLyA1IHJhZG9zCiAgIDAvIDUgcmJkCiAgIDAvIDUgam91cm5hbGVyCiAgIDAvIDUgb2Jq ZWN0Y2FjaGVyCiAgIDAvIDUgY2xpZW50CiAgIDAvIDUgb3NkCiAgIDAvIDUgb3B0cmFja2VyCiAg IDAvIDUgb2JqY2xhc3MKICAgMS8gMyBmaWxlc3RvcmUKICAgMS8gMyBqb3VybmFsCiAgIDAvIDUg bXMKICAgMS8gNSBtb24KICAgMC8xMCBtb25jCiAgIDAvIDUgcGF4b3MKICAgMC8gNSB0cAogICAx LyA1IGF1dGgKICAgMS8gNSBjcnlwdG8KICAgMS8gMSBmaW5pc2hlcgogICAxLyA1IGhlYXJ0YmVh dG1hcAogICAxLyA1IHBlcmZjb3VudGVyCiAgIDEvIDUgcmd3CiAgIDEvIDUgaGFkb29wCiAgIDEv IDUgamF2YWNsaWVudAogICAxLyA1IGFzb2sKICAgMS8gMSB0aHJvdHRsZQogIC0yLy0yIChzeXNs b2cgdGhyZXNob2xkKQogIC0xLy0xIChzdGRlcnIgdGhyZXNob2xkKQogIG1heF9yZWNlbnQgICAg MTAwMDAwCiAgbWF4X25ldyAgICAgICAgIDEwMDAKICBsb2dfZmlsZSAvdmFyL2xvZy9jZXBoL29z ZC4wLmxvZwotLS0gZW5kIGR1bXAgb2YgcmVjZW50IGV2ZW50cyAtLS0Kcm9vdEBoYTE6L3Zhci9s b2cvY2VwaCMgCgo= ------=_Part_336_6488584.1362420122087--