0.40 OSD - Address family not supported by protocol

0.40 OSD - Address family not supported by protocol

[Josh Pieper]

I just upgraded our test cluster to 0.40, and immediately after
starting up get asserts in all the OSDs.  I've inlined a relevant
backtrace below, is there anything else that would be useful for
debugging?

Our test cluster is 3 ubuntu 11.10 amd64 machines, each with a mon and
osd.

Looking at an strace, it is pretty clearly asking for an invalid
address family, although I'm not sure where it is coming from.

[pid 30648] socket(PF_UNSPEC, SOCK_STREAM, 0 <unfinished ...>
[pid 30648] <... socket resumed> )      = -1 EAFNOSUPPORT (Address family not supported by protocol)

-Josh

-------
2012-01-14 09:31:03.395266 7f67edf08700 -- 10.1.10.71:6801/27529 >> 10.1.10.73:6801/8127 pipe(0x14e0780 sd=19 pgs=0 cs=0 l=0).connect claims to be 10.1.10.73:6801/24029 not 10.1.10.73:6801/8127 - wrong node!
2012-01-14 09:31:03.395579 7f67ede07700 -- :/27530 >> :/0 pipe(0x14e0500 sd=-1 pgs=0 cs=0 l=0).connect couldn't created socket Address family not supported by protocol
msg/SimpleMessenger.cc: In function 'int SimpleMessenger::Pipe::connect()', in thread '7f67ede07700'
msg/SimpleMessenger.cc: 1038: FAILED assert(0)
 ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
 1: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
 2: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
 3: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
 4: (()+0x7efc) [0x7f67ffdf4efc]
 5: (clone()+0x6d) [0x7f67fe42589d]
 ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
 1: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
 2: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
 3: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
 4: (()+0x7efc) [0x7f67ffdf4efc]
 5: (clone()+0x6d) [0x7f67fe42589d]
*** Caught signal (Aborted) **
 in thread 7f67ede07700
 ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
 1: /usr/bin/ceph-osd() [0x5fd926]
 2: (()+0x10060) [0x7f67ffdfd060]
 3: (gsignal()+0x35) [0x7f67fe37a3a5]
 4: (abort()+0x17b) [0x7f67fe37db0b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f67fec38d7d]
 6: (()+0xb9f26) [0x7f67fec36f26]
 7: (()+0xb9f53) [0x7f67fec36f53]
 8: (()+0xba04e) [0x7f67fec3704e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x193) [0x5cfd33]
 10: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
 11: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
 12: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
 13: (()+0x7efc) [0x7f67ffdf4efc]
 14: (clone()+0x6d) [0x7f67fe42589d]

Re: 0.40 OSD - Address family not supported by protocol

[Sage Weil]

Hi Josh,

On Sat, 14 Jan 2012, Josh Pieper wrote:
> I just upgraded our test cluster to 0.40, and immediately after
> starting up get asserts in all the OSDs.  I've inlined a relevant
> backtrace below, is there anything else that would be useful for
> debugging?

Are you coming from 0.39 or something older?

You might try reverting 4728f4f8e09878c583c65cd882e031d37f8d903e and see 
if that does it..

Can you reproduce it with --debug-osd 10 and --debug-ms 10?

sage

> 
> Our test cluster is 3 ubuntu 11.10 amd64 machines, each with a mon and
> osd.
> 
> Looking at an strace, it is pretty clearly asking for an invalid
> address family, although I'm not sure where it is coming from.
> 
> [pid 30648] socket(PF_UNSPEC, SOCK_STREAM, 0 <unfinished ...>
> [pid 30648] <... socket resumed> )      = -1 EAFNOSUPPORT (Address family not supported by protocol)
> 
> -Josh
> 
> -------
> 2012-01-14 09:31:03.395266 7f67edf08700 -- 10.1.10.71:6801/27529 >> 10.1.10.73:6801/8127 pipe(0x14e0780 sd=19 pgs=0 cs=0 l=0).connect claims to be 10.1.10.73:6801/24029 not 10.1.10.73:6801/8127 - wrong node!
> 2012-01-14 09:31:03.395579 7f67ede07700 -- :/27530 >> :/0 pipe(0x14e0500 sd=-1 pgs=0 cs=0 l=0).connect couldn't created socket Address family not supported by protocol
> msg/SimpleMessenger.cc: In function 'int SimpleMessenger::Pipe::connect()', in thread '7f67ede07700'
> msg/SimpleMessenger.cc: 1038: FAILED assert(0)
>  ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
>  1: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
>  2: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
>  3: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
>  4: (()+0x7efc) [0x7f67ffdf4efc]
>  5: (clone()+0x6d) [0x7f67fe42589d]
>  ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
>  1: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
>  2: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
>  3: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
>  4: (()+0x7efc) [0x7f67ffdf4efc]
>  5: (clone()+0x6d) [0x7f67fe42589d]
> *** Caught signal (Aborted) **
>  in thread 7f67ede07700
>  ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
>  1: /usr/bin/ceph-osd() [0x5fd926]
>  2: (()+0x10060) [0x7f67ffdfd060]
>  3: (gsignal()+0x35) [0x7f67fe37a3a5]
>  4: (abort()+0x17b) [0x7f67fe37db0b]
>  5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f67fec38d7d]
>  6: (()+0xb9f26) [0x7f67fec36f26]
>  7: (()+0xb9f53) [0x7f67fec36f53]
>  8: (()+0xba04e) [0x7f67fec3704e]
>  9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x193) [0x5cfd33]
>  10: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
>  11: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
>  12: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
>  13: (()+0x7efc) [0x7f67ffdf4efc]
>  14: (clone()+0x6d) [0x7f67fe42589d]
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: 0.40 OSD - Address family not supported by protocol

[Josh Pieper]

Sage Weil wrote:
> Hi Josh,
> 
> On Sat, 14 Jan 2012, Josh Pieper wrote:
> > I just upgraded our test cluster to 0.40, and immediately after
> > starting up get asserts in all the OSDs.  I've inlined a relevant
> > backtrace below, is there anything else that would be useful for
> > debugging?
> 
> Are you coming from 0.39 or something older?

I was upgrading from 0.39.

> You might try reverting 4728f4f8e09878c583c65cd882e031d37f8d903e and see 
> if that does it..
> 
> Can you reproduce it with --debug-osd 10 and --debug-ms 10?

Unfortunately, I cannot appear to reproduce the problem any more.
Re-upgrading to 0.40 now shows no problem, I've tried to explore the
range of things I may have done, but with no luck. I had to trash my
journals in order to downgrade, so there is some amount of state that
was lost which may be related to my inability to reproduce now?

For what it is worth, I believe the problem may have been caused by
something the 0.40 versions were sending.  As I was downgrading back
to 0.39, the downgraded 0.39 version kept dying with the same error as
long as one of the 0.40 versions was still up.

I did not know of the ms debugging when I was first investigating, but
looking through my old data, I have a trace with OSD debug set to 20
of the 0.39 version dying of the fault:

http://joshp.no-ip.com:8080/20120114-osd-family-error.log.bz2

-Josh

Re: 0.40 OSD - Address family not supported by protocol

[Sage Weil]

Hi Josh,

Can you attach one of your OSDmaps with the poison entries?  Between

	ceph osd getmap 149 -o /tmp/149
	ceph osd getmap 155 -o /tmp/155

I should see one of them.

Thanks!
sage


On Sat, 14 Jan 2012, Josh Pieper wrote:

> Sage Weil wrote:
> > Hi Josh,
> > 
> > On Sat, 14 Jan 2012, Josh Pieper wrote:
> > > I just upgraded our test cluster to 0.40, and immediately after
> > > starting up get asserts in all the OSDs.  I've inlined a relevant
> > > backtrace below, is there anything else that would be useful for
> > > debugging?
> > 
> > Are you coming from 0.39 or something older?
> 
> I was upgrading from 0.39.
> 
> > You might try reverting 4728f4f8e09878c583c65cd882e031d37f8d903e and see 
> > if that does it..
> > 
> > Can you reproduce it with --debug-osd 10 and --debug-ms 10?
> 
> Unfortunately, I cannot appear to reproduce the problem any more.
> Re-upgrading to 0.40 now shows no problem, I've tried to explore the
> range of things I may have done, but with no luck. I had to trash my
> journals in order to downgrade, so there is some amount of state that
> was lost which may be related to my inability to reproduce now?
> 
> For what it is worth, I believe the problem may have been caused by
> something the 0.40 versions were sending.  As I was downgrading back
> to 0.39, the downgraded 0.39 version kept dying with the same error as
> long as one of the 0.40 versions was still up.
> 
> I did not know of the ms debugging when I was first investigating, but
> looking through my old data, I have a trace with OSD debug set to 20
> of the 0.39 version dying of the fault:
> 
> http://joshp.no-ip.com:8080/20120114-osd-family-error.log.bz2
> 
> -Josh
> 
> 

Re: 0.40 OSD - Address family not supported by protocol

[Josh Pieper]

[-- Attachment #1: Type: text/plain, Size: 265 bytes --]

Sage,

Those two are attached.

-Josh

Sage Weil wrote:
> Hi Josh,
> 
> Can you attach one of your OSDmaps with the poison entries?  Between
> 
> 	ceph osd getmap 149 -o /tmp/149
> 	ceph osd getmap 155 -o /tmp/155
> 
> I should see one of them.
> 
> Thanks!
> sage

[-- Attachment #2: osdmaps.tar.gz --]
[-- Type: application/octet-stream, Size: 689 bytes --]

Re: 0.40 OSD - Address family not supported by protocol

[Sage Weil]

Hi Josh,

I just sorted this out.  The problem was that the encoding for 
OSDSuperblock was changed, and that struct was embedded in the MOSDBoot 
message.  Some of your OSDs restarted befor the monitors, so the old 
monitors saw the new structure and misdecoded the message with garbage 
(well, zeros) for the heartbeat address.  This made it into the OSDMap, 
and a very impolite assert in the messenger code made the process crash 
when it got an error from socket(2).

The assert and error handling is cleaned up.  There isn't a nice way to 
fix the behavior of the old code, though, so for everyone else: 
upgrade/restart the monitors before the osds to avoid triggering this.  If 
you do, restarting the OSDs (possibly a couple of times) will clear it up.  
Once all of the ':/0' values disappear from 'ceph osd dump' you're in the 
clear.

sage


http://tracker.newdream.net/issues/1942

On Sat, 14 Jan 2012, Josh Pieper wrote:

> I just upgraded our test cluster to 0.40, and immediately after
> starting up get asserts in all the OSDs.  I've inlined a relevant
> backtrace below, is there anything else that would be useful for
> debugging?
> 
> Our test cluster is 3 ubuntu 11.10 amd64 machines, each with a mon and
> osd.
> 
> Looking at an strace, it is pretty clearly asking for an invalid
> address family, although I'm not sure where it is coming from.
> 
> [pid 30648] socket(PF_UNSPEC, SOCK_STREAM, 0 <unfinished ...>
> [pid 30648] <... socket resumed> )      = -1 EAFNOSUPPORT (Address family not supported by protocol)
> 
> -Josh
> 
> -------
> 2012-01-14 09:31:03.395266 7f67edf08700 -- 10.1.10.71:6801/27529 >> 10.1.10.73:6801/8127 pipe(0x14e0780 sd=19 pgs=0 cs=0 l=0).connect claims to be 10.1.10.73:6801/24029 not 10.1.10.73:6801/8127 - wrong node!
> 2012-01-14 09:31:03.395579 7f67ede07700 -- :/27530 >> :/0 pipe(0x14e0500 sd=-1 pgs=0 cs=0 l=0).connect couldn't created socket Address family not supported by protocol
> msg/SimpleMessenger.cc: In function 'int SimpleMessenger::Pipe::connect()', in thread '7f67ede07700'
> msg/SimpleMessenger.cc: 1038: FAILED assert(0)
>  ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
>  1: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
>  2: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
>  3: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
>  4: (()+0x7efc) [0x7f67ffdf4efc]
>  5: (clone()+0x6d) [0x7f67fe42589d]
>  ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
>  1: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
>  2: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
>  3: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
>  4: (()+0x7efc) [0x7f67ffdf4efc]
>  5: (clone()+0x6d) [0x7f67fe42589d]
> *** Caught signal (Aborted) **
>  in thread 7f67ede07700
>  ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
>  1: /usr/bin/ceph-osd() [0x5fd926]
>  2: (()+0x10060) [0x7f67ffdfd060]
>  3: (gsignal()+0x35) [0x7f67fe37a3a5]
>  4: (abort()+0x17b) [0x7f67fe37db0b]
>  5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f67fec38d7d]
>  6: (()+0xb9f26) [0x7f67fec36f26]
>  7: (()+0xb9f53) [0x7f67fec36f53]
>  8: (()+0xba04e) [0x7f67fec3704e]
>  9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x193) [0x5cfd33]
>  10: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
>  11: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
>  12: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
>  13: (()+0x7efc) [0x7f67ffdf4efc]
>  14: (clone()+0x6d) [0x7f67fe42589d]
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: 0.40 OSD - Address family not supported by protocol

[Josh Pieper]

Sage,

Thanks for sorting out the root cause!

-Josh

Sage Weil wrote:
> Hi Josh,
> 
> I just sorted this out.  The problem was that the encoding for 
> OSDSuperblock was changed, and that struct was embedded in the MOSDBoot 
> message.  Some of your OSDs restarted befor the monitors, so the old 
> monitors saw the new structure and misdecoded the message with garbage 
> (well, zeros) for the heartbeat address.  This made it into the OSDMap, 
> and a very impolite assert in the messenger code made the process crash 
> when it got an error from socket(2).
> 
> The assert and error handling is cleaned up.  There isn't a nice way to 
> fix the behavior of the old code, though, so for everyone else: 
> upgrade/restart the monitors before the osds to avoid triggering this.  If 
> you do, restarting the OSDs (possibly a couple of times) will clear it up.  
> Once all of the ':/0' values disappear from 'ceph osd dump' you're in the 
> clear.
> 
> sage
> 
> 
> http://tracker.newdream.net/issues/1942
> 
> On Sat, 14 Jan 2012, Josh Pieper wrote:
> 
> > I just upgraded our test cluster to 0.40, and immediately after
> > starting up get asserts in all the OSDs.  I've inlined a relevant
> > backtrace below, is there anything else that would be useful for
> > debugging?
> > 
> > Our test cluster is 3 ubuntu 11.10 amd64 machines, each with a mon and
> > osd.
> > 
> > Looking at an strace, it is pretty clearly asking for an invalid
> > address family, although I'm not sure where it is coming from.
> > 
> > [pid 30648] socket(PF_UNSPEC, SOCK_STREAM, 0 <unfinished ...>
> > [pid 30648] <... socket resumed> )      = -1 EAFNOSUPPORT (Address family not supported by protocol)
> > 
> > -Josh
> > 
> > -------
> > 2012-01-14 09:31:03.395266 7f67edf08700 -- 10.1.10.71:6801/27529 >> 10.1.10.73:6801/8127 pipe(0x14e0780 sd=19 pgs=0 cs=0 l=0).connect claims to be 10.1.10.73:6801/24029 not 10.1.10.73:6801/8127 - wrong node!
> > 2012-01-14 09:31:03.395579 7f67ede07700 -- :/27530 >> :/0 pipe(0x14e0500 sd=-1 pgs=0 cs=0 l=0).connect couldn't created socket Address family not supported by protocol
> > msg/SimpleMessenger.cc: In function 'int SimpleMessenger::Pipe::connect()', in thread '7f67ede07700'
> > msg/SimpleMessenger.cc: 1038: FAILED assert(0)
> >  ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
> >  1: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
> >  2: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
> >  3: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
> >  4: (()+0x7efc) [0x7f67ffdf4efc]
> >  5: (clone()+0x6d) [0x7f67fe42589d]
> >  ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
> >  1: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
> >  2: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
> >  3: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
> >  4: (()+0x7efc) [0x7f67ffdf4efc]
> >  5: (clone()+0x6d) [0x7f67fe42589d]
> > *** Caught signal (Aborted) **
> >  in thread 7f67ede07700
> >  ceph version 0.40 (commit:7eea40ea37fb3a68a2042a2218c9b8c9c40a843e)
> >  1: /usr/bin/ceph-osd() [0x5fd926]
> >  2: (()+0x10060) [0x7f67ffdfd060]
> >  3: (gsignal()+0x35) [0x7f67fe37a3a5]
> >  4: (abort()+0x17b) [0x7f67fe37db0b]
> >  5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f67fec38d7d]
> >  6: (()+0xb9f26) [0x7f67fec36f26]
> >  7: (()+0xb9f53) [0x7f67fec36f53]
> >  8: (()+0xba04e) [0x7f67fec3704e]
> >  9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x193) [0x5cfd33]
> >  10: (SimpleMessenger::Pipe::connect()+0x87c) [0x5be81c]
> >  11: (SimpleMessenger::Pipe::writer()+0x456) [0x5c1536]
> >  12: (SimpleMessenger::Pipe::Writer::entry()+0xd) [0x4b228d]
> >  13: (()+0x7efc) [0x7f67ffdf4efc]
> >  14: (clone()+0x6d) [0x7f67fe42589d]
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 

-- 
Shaw's Principle:
	Build a system that even a fool can use, and only a fool will
	want to use it.