* Bug #1047 reproduced
@ 2011-12-01 8:37 Amon Ott
2011-12-02 11:17 ` Amon Ott
0 siblings, 1 reply; 14+ messages in thread
From: Amon Ott @ 2011-12-01 8:37 UTC (permalink / raw)
To: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 608 bytes --]
On all four nodes of my test cluster, MDS crashes with a trace like that in
bug #1047. Example and ceph.conf attached. Ceph server side is from git
master, last commit ce6572273943ffdca4b7dc5344152d6c35106a2d.
MDS does not start on any node here, it reliably crashes with that assert.
Amon Ott
--
Dr. Amon Ott
m-privacy GmbH Tel: +49 30 24342334
Am Köllnischen Park 1 Fax: +49 30 24342336
10179 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer:
Dipl.-Kfm. Holger Maczkowsky,
Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
[-- Attachment #2: ceph.conf --]
[-- Type: text/plain, Size: 1054 bytes --]
[global]
pid file = /var/run/ceph/$name.pid
debug ms = 1
keyring = /etc/ceph/keyring
cluster_network = 192.168.111.0/24
[mon]
mon data = /var/lib/ceph/mon
; Use odd number of monitors, three is good, five or more on big clusters
[mon.0]
host = tgpro1
mon addr = 192.168.111.1
[mon.1]
host = tgpro2
mon addr = 192.168.111.2
[mon.2]
host = tgpro3
mon addr = 192.168.111.3
[mds]
max mds = 2
[mds.0]
host = tgpro1
mds addr = 192.168.111.1
mds standby replay = true
[mds.1]
host = tgpro2
mds addr = 192.168.111.2
mds standby replay = true
[mds.2]
host = tgpro3
mds addr = 192.168.111.3
mds standby replay = true
[mds.3]
host = tgpro4
mds addr = 192.168.111.4
mds standby replay = true
[osd]
sudo = true
osd data = /ceph/data
; osd journal = /ceph/journal
; osd journal size = 512
osd journal = /dev/sda7
filestore journal = writeahead
[osd.0]
host = tgpro1
osd addr = 192.168.111.1
[osd.1]
host = tgpro2
osd addr = 192.168.111.2
[osd.2]
host = tgpro3
osd addr = 192.168.111.3
[osd.3]
host = tgpro4
osd addr = 192.168.111.4
[-- Attachment #3: mds-crash-anchor_map.log --]
[-- Type: text/x-log, Size: 1747 bytes --]
2011-12-01 09:24:48.852444 486c1b70 -- 192.168.111.4:6802/25235 <== mds.0 192.168.111.4:6802/25235 0 ==== mds_table_request(anchortable query 8 bytes) v1 ==== 0+0+0 (0 0 0) 0x113a5240 con 0x110c0000
mds/AnchorServer.cc: In function 'virtual void AnchorServer::handle_query(MMDSTableRequest*)', in thread '486c1b70'
mds/AnchorServer.cc: 249: FAILED assert(anchor_map.count(curino) == 1)
ceph version (commit:)
1: (AnchorServer::handle_query(MMDSTableRequest*)+0x1c2) [0x10dfd272]
2: (MDSTableServer::handle_request(MMDSTableRequest*)+0xd4) [0x10dfbb54]
3: (MDS::handle_deferrable_message(Message*)+0xe01) [0x10b97611]
4: (MDS::_dispatch(Message*)+0x1ae2) [0x10baf402]
5: (MDS::ms_dispatch(Message*)+0xa5) [0x10bafac5]
6: (SimpleMessenger::dispatch_entry()+0x7c9) [0x10ec3fd9]
7: (SimpleMessenger::DispatchThread::entry()+0x3b) [0x10b83f7b]
8: (Thread::_entry_func(void*)+0x1c) [0x10e7c3bc]
9: (()+0x5905) [0x4adfe905]
10: (clone()+0x5e) [0x4a7968ce]
ceph version (commit:)
1: (AnchorServer::handle_query(MMDSTableRequest*)+0x1c2) [0x10dfd272]
2: (MDSTableServer::handle_request(MMDSTableRequest*)+0xd4) [0x10dfbb54]
3: (MDS::handle_deferrable_message(Message*)+0xe01) [0x10b97611]
4: (MDS::_dispatch(Message*)+0x1ae2) [0x10baf402]
5: (MDS::ms_dispatch(Message*)+0xa5) [0x10bafac5]
6: (SimpleMessenger::dispatch_entry()+0x7c9) [0x10ec3fd9]
7: (SimpleMessenger::DispatchThread::entry()+0x3b) [0x10b83f7b]
8: (Thread::_entry_func(void*)+0x1c) [0x10e7c3bc]
9: (()+0x5905) [0x4adfe905]
10: (clone()+0x5e) [0x4a7968ce]
*** Caught signal (Segmentation fault) **
in thread 486c1b70
ceph version (commit:)
1: (()+0x4703a3) [0x10f613a3]
2: [0x4ae40400]
3: (abort()+0xea) [0x4a6f653a]
reraise_fatal: failed to re-raise signal 11
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2011-12-01 8:37 Bug #1047 reproduced Amon Ott
@ 2011-12-02 11:17 ` Amon Ott
2011-12-02 16:57 ` Sage Weil
0 siblings, 1 reply; 14+ messages in thread
From: Amon Ott @ 2011-12-02 11:17 UTC (permalink / raw)
To: ceph-devel
On Thursday 01 December 2011 you wrote:
> On all four nodes of my test cluster, MDS crashes with a trace like that in
> bug #1047. Example and ceph.conf attached. Ceph server side is from git
> master, last commit ce6572273943ffdca4b7dc5344152d6c35106a2d.
>
> MDS does not start on any node here, it reliably crashes with that assert.
Does it makes sense for you to keep the cluster in that broken state, so that
we can reproduce that bug or test a potential fix? Otherwise, I would
recreate the Ceph filesystem to make more tests. I also have a full log of
one mds from start to crash here.
Amon Ott
--
Dr. Amon Ott
m-privacy GmbH Tel: +49 30 24342334
Am Köllnischen Park 1 Fax: +49 30 24342336
10179 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer:
Dipl.-Kfm. Holger Maczkowsky,
Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2011-12-02 11:17 ` Amon Ott
@ 2011-12-02 16:57 ` Sage Weil
2011-12-05 10:21 ` Amon Ott
2011-12-21 12:37 ` Amon Ott
0 siblings, 2 replies; 14+ messages in thread
From: Sage Weil @ 2011-12-02 16:57 UTC (permalink / raw)
To: Amon Ott; +Cc: ceph-devel
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1514 bytes --]
On Fri, 2 Dec 2011, Amon Ott wrote:
> On Thursday 01 December 2011 you wrote:
> > On all four nodes of my test cluster, MDS crashes with a trace like that in
> > bug #1047. Example and ceph.conf attached. Ceph server side is from git
> > master, last commit ce6572273943ffdca4b7dc5344152d6c35106a2d.
> >
> > MDS does not start on any node here, it reliably crashes with that assert.
>
> Does it makes sense for you to keep the cluster in that broken state, so that
> we can reproduce that bug or test a potential fix? Otherwise, I would
> recreate the Ceph filesystem to make more tests. I also have a full log of
> one mds from start to crash here.
Can you attach the log to #1047 for posterity? I'll take a quick look and
see if there is any further info to gain from the log. I'm guessing the
actual bug occured before the crash, when the anchor table wasn't updated
properly, but there may be clues here.
Thanks!
sage
>
> Amon Ott
> --
> Dr. Amon Ott
> m-privacy GmbH Tel: +49 30 24342334
> Am Köllnischen Park 1 Fax: +49 30 24342336
> 10179 Berlin http://www.m-privacy.de
>
> Amtsgericht Charlottenburg, HRB 84946
>
> Geschäftsführer:
> Dipl.-Kfm. Holger Maczkowsky,
> Roman Maczkowsky
>
> GnuPG-Key-ID: 0x2DD3A649
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2011-12-02 16:57 ` Sage Weil
@ 2011-12-05 10:21 ` Amon Ott
2011-12-06 13:20 ` Amon Ott
2011-12-21 12:37 ` Amon Ott
1 sibling, 1 reply; 14+ messages in thread
From: Amon Ott @ 2011-12-05 10:21 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
On Friday 02 December 2011 wrote Sage Weil:
> On Fri, 2 Dec 2011, Amon Ott wrote:
> > On Thursday 01 December 2011 you wrote:
> > > On all four nodes of my test cluster, MDS crashes with a trace like
> > > that in bug #1047. Example and ceph.conf attached. Ceph server side is
> > > from git master, last commit ce6572273943ffdca4b7dc5344152d6c35106a2d.
> > >
> > > MDS does not start on any node here, it reliably crashes with that
> > > assert.
> >
> > Does it makes sense for you to keep the cluster in that broken state, so
> > that we can reproduce that bug or test a potential fix? Otherwise, I
> > would recreate the Ceph filesystem to make more tests. I also have a full
> > log of one mds from start to crash here.
>
> Can you attach the log to #1047 for posterity? I'll take a quick look and
> see if there is any further info to gain from the log. I'm guessing the
> actual bug occured before the crash, when the anchor table wasn't updated
> properly, but there may be clues here.
Log attached to the bug on Saturday. Will retry with Ceph 0.39, too.
Amon Ott
--
Dr. Amon Ott
m-privacy GmbH Tel: +49 30 24342334
Am Köllnischen Park 1 Fax: +49 30 24342336
10179 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer:
Dipl.-Kfm. Holger Maczkowsky,
Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2011-12-05 10:21 ` Amon Ott
@ 2011-12-06 13:20 ` Amon Ott
0 siblings, 0 replies; 14+ messages in thread
From: Amon Ott @ 2011-12-06 13:20 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
On Monday 05 December 2011 wrote Amon Ott:
> On Friday 02 December 2011 wrote Sage Weil:
> > On Fri, 2 Dec 2011, Amon Ott wrote:
> > > On Thursday 01 December 2011 you wrote:
> > > > On all four nodes of my test cluster, MDS crashes with a trace like
> > > > that in bug #1047. Example and ceph.conf attached. Ceph server side
> > > > is from git master, last commit
> > > > ce6572273943ffdca4b7dc5344152d6c35106a2d.
> > > >
> > > > MDS does not start on any node here, it reliably crashes with that
> > > > assert.
> > >
> > > Does it makes sense for you to keep the cluster in that broken state,
> > > so that we can reproduce that bug or test a potential fix? Otherwise, I
> > > would recreate the Ceph filesystem to make more tests. I also have a
> > > full log of one mds from start to crash here.
> >
> > Can you attach the log to #1047 for posterity? I'll take a quick look
> > and see if there is any further info to gain from the log. I'm guessing
> > the actual bug occured before the crash, when the anchor table wasn't
> > updated properly, but there may be clues here.
>
> Log attached to the bug on Saturday. Will retry with Ceph 0.39, too.
Seems to be fixed with 0.39.
Amon Ott
--
Dr. Amon Ott
m-privacy GmbH Tel: +49 30 24342334
Am Köllnischen Park 1 Fax: +49 30 24342336
10179 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer:
Dipl.-Kfm. Holger Maczkowsky,
Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2011-12-02 16:57 ` Sage Weil
2011-12-05 10:21 ` Amon Ott
@ 2011-12-21 12:37 ` Amon Ott
2011-12-21 16:18 ` Gregory Farnum
1 sibling, 1 reply; 14+ messages in thread
From: Amon Ott @ 2011-12-21 12:37 UTC (permalink / raw)
To: Sage Weil; +Cc: ceph-devel
On Friday 02 December 2011 wrote Sage Weil:
> On Fri, 2 Dec 2011, Amon Ott wrote:
> > On Thursday 01 December 2011 you wrote:
> > > On all four nodes of my test cluster, MDS crashes with a trace like
> > > that in bug #1047. Example and ceph.conf attached. Ceph server side is
> > > from git master, last commit ce6572273943ffdca4b7dc5344152d6c35106a2d.
> > >
> > > MDS does not start on any node here, it reliably crashes with that
> > > assert.
> >
> > Does it makes sense for you to keep the cluster in that broken state, so
> > that we can reproduce that bug or test a potential fix? Otherwise, I
> > would recreate the Ceph filesystem to make more tests. I also have a full
> > log of one mds from start to crash here.
>
> Can you attach the log to #1047 for posterity? I'll take a quick look and
> see if there is any further info to gain from the log. I'm guessing the
> actual bug occured before the crash, when the anchor table wasn't updated
> properly, but there may be clues here.
Did you find some time to look into this? The bug makes Ceph unusable for us
even with moderate load. All mds instances die with the same assert, the only
way to recover in that state is to recreate the complete ceph fs and restore
backups.
Amon Ott
--
Dr. Amon Ott
m-privacy GmbH Tel: +49 30 24342334
Am Köllnischen Park 1 Fax: +49 30 24342336
10179 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer:
Dipl.-Kfm. Holger Maczkowsky,
Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2011-12-21 12:37 ` Amon Ott
@ 2011-12-21 16:18 ` Gregory Farnum
2011-12-21 16:36 ` Amon Ott
0 siblings, 1 reply; 14+ messages in thread
From: Gregory Farnum @ 2011-12-21 16:18 UTC (permalink / raw)
To: Amon Ott; +Cc: Sage Weil, ceph-devel, Alexandre Oliva
On Wed, Dec 21, 2011 at 4:37 AM, Amon Ott <a.ott@m-privacy.de> wrote:
> On Friday 02 December 2011 wrote Sage Weil:
>> On Fri, 2 Dec 2011, Amon Ott wrote:
>> > On Thursday 01 December 2011 you wrote:
>> > > On all four nodes of my test cluster, MDS crashes with a trace like
>> > > that in bug #1047. Example and ceph.conf attached. Ceph server side is
>> > > from git master, last commit ce6572273943ffdca4b7dc5344152d6c35106a2d.
>> > >
>> > > MDS does not start on any node here, it reliably crashes with that
>> > > assert.
>> >
>> > Does it makes sense for you to keep the cluster in that broken state, so
>> > that we can reproduce that bug or test a potential fix? Otherwise, I
>> > would recreate the Ceph filesystem to make more tests. I also have a full
>> > log of one mds from start to crash here.
>>
>> Can you attach the log to #1047 for posterity? I'll take a quick look and
>> see if there is any further info to gain from the log. I'm guessing the
>> actual bug occured before the crash, when the anchor table wasn't updated
>> properly, but there may be clues here.
>
> Did you find some time to look into this? The bug makes Ceph unusable for us
> even with moderate load. All mds instances die with the same assert, the only
> way to recover in that state is to recreate the complete ceph fs and restore
> backups.
Sage is gone on vacation right now (unless he decides not to be for a
while), but we've been focusing our efforts on the OSDs lately so I
don't think he's looked at it. I'll see if I can carve out some time
tomorrow or Friday, but I can't promise anything.
Alexandre, can you check this bug and make sure it looks like the same
one you reported as #1850?
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2011-12-21 16:18 ` Gregory Farnum
@ 2011-12-21 16:36 ` Amon Ott
2011-12-23 0:27 ` Gregory Farnum
0 siblings, 1 reply; 14+ messages in thread
From: Amon Ott @ 2011-12-21 16:36 UTC (permalink / raw)
To: Gregory Farnum; +Cc: Sage Weil, ceph-devel, Alexandre Oliva
On Wednesday 21 December 2011 wrote Gregory Farnum:
> On Wed, Dec 21, 2011 at 4:37 AM, Amon Ott <a.ott@m-privacy.de> wrote:
> > On Friday 02 December 2011 wrote Sage Weil:
> >> On Fri, 2 Dec 2011, Amon Ott wrote:
> >> > On Thursday 01 December 2011 you wrote:
> >> > > On all four nodes of my test cluster, MDS crashes with a trace like
> >> > > that in bug #1047. Example and ceph.conf attached. Ceph server side
> >> > > is from git master, last commit
> >> > > ce6572273943ffdca4b7dc5344152d6c35106a2d.
> >> > >
> >> > > MDS does not start on any node here, it reliably crashes with that
> >> > > assert.
> >> >
> >> > Does it makes sense for you to keep the cluster in that broken state,
> >> > so that we can reproduce that bug or test a potential fix? Otherwise,
> >> > I would recreate the Ceph filesystem to make more tests. I also have a
> >> > full log of one mds from start to crash here.
> >>
> >> Can you attach the log to #1047 for posterity? I'll take a quick look
> >> and see if there is any further info to gain from the log. I'm guessing
> >> the actual bug occured before the crash, when the anchor table wasn't
> >> updated properly, but there may be clues here.
> >
> > Did you find some time to look into this? The bug makes Ceph unusable for
> > us even with moderate load. All mds instances die with the same assert,
> > the only way to recover in that state is to recreate the complete ceph fs
> > and restore backups.
>
> Sage is gone on vacation right now (unless he decides not to be for a
> while), but we've been focusing our efforts on the OSDs lately so I
> don't think he's looked at it. I'll see if I can carve out some time
> tomorrow or Friday, but I can't promise anything.
>
> Alexandre, can you check this bug and make sure it looks like the same
> one you reported as #1850?
Thank you for looking into it. The behaviour in #1850 looks quite similar to
our bug, apart from the hardlinks. We copy many files here in our tests, too.
Last time I hit the bug I had really restarted the master mds.
Amon Ott
--
Dr. Amon Ott
m-privacy GmbH Tel: +49 30 24342334
Am Köllnischen Park 1 Fax: +49 30 24342336
10179 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer:
Dipl.-Kfm. Holger Maczkowsky,
Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2011-12-21 16:36 ` Amon Ott
@ 2011-12-23 0:27 ` Gregory Farnum
2011-12-23 9:58 ` Amon Ott
2011-12-29 11:30 ` Amon Ott
0 siblings, 2 replies; 14+ messages in thread
From: Gregory Farnum @ 2011-12-23 0:27 UTC (permalink / raw)
To: Amon Ott; +Cc: ceph-devel
On Wed, Dec 21, 2011 at 8:36 AM, Amon Ott <a.ott@m-privacy.de> wrote:
> On Wednesday 21 December 2011 wrote Gregory Farnum:
>> On Wed, Dec 21, 2011 at 4:37 AM, Amon Ott <a.ott@m-privacy.de> wrote:
>> > On Friday 02 December 2011 wrote Sage Weil:
>> >> On Fri, 2 Dec 2011, Amon Ott wrote:
>> >> > On Thursday 01 December 2011 you wrote:
>> >> > > On all four nodes of my test cluster, MDS crashes with a trace like
>> >> > > that in bug #1047. Example and ceph.conf attached. Ceph server side
>> >> > > is from git master, last commit
>> >> > > ce6572273943ffdca4b7dc5344152d6c35106a2d.
>> >> > >
>> >> > > MDS does not start on any node here, it reliably crashes with that
>> >> > > assert.
>> >> >
>> >> > Does it makes sense for you to keep the cluster in that broken state,
>> >> > so that we can reproduce that bug or test a potential fix? Otherwise,
>> >> > I would recreate the Ceph filesystem to make more tests. I also have a
>> >> > full log of one mds from start to crash here.
>> >>
>> >> Can you attach the log to #1047 for posterity? I'll take a quick look
>> >> and see if there is any further info to gain from the log. I'm guessing
>> >> the actual bug occured before the crash, when the anchor table wasn't
>> >> updated properly, but there may be clues here.
>> >
>> > Did you find some time to look into this? The bug makes Ceph unusable for
>> > us even with moderate load. All mds instances die with the same assert,
>> > the only way to recover in that state is to recreate the complete ceph fs
>> > and restore backups.
>>
>> Sage is gone on vacation right now (unless he decides not to be for a
>> while), but we've been focusing our efforts on the OSDs lately so I
>> don't think he's looked at it. I'll see if I can carve out some time
>> tomorrow or Friday, but I can't promise anything.
>>
>> Alexandre, can you check this bug and make sure it looks like the same
>> one you reported as #1850?
>
> Thank you for looking into it. The behaviour in #1850 looks quite similar to
> our bug, apart from the hardlinks. We copy many files here in our tests, too.
> Last time I hit the bug I had really restarted the master mds.
Unfortunately there's not enough info in this log either. If you can
reproduce it with "mds debug = 20" and put that log somewhere, it
ought to be enough to work out what's going on, though. Sorry. :(
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2011-12-23 0:27 ` Gregory Farnum
@ 2011-12-23 9:58 ` Amon Ott
2011-12-29 11:30 ` Amon Ott
1 sibling, 0 replies; 14+ messages in thread
From: Amon Ott @ 2011-12-23 9:58 UTC (permalink / raw)
To: Gregory Farnum; +Cc: ceph-devel
On Friday 23 December 2011 wrote Gregory Farnum:
> Unfortunately there's not enough info in this log either. If you can
> reproduce it with "mds debug = 20" and put that log somewhere, it
> ought to be enough to work out what's going on, though. Sorry. :(
Will do that as soon as I can.
Amon Ott
--
Dr. Amon Ott
m-privacy GmbH Tel: +49 30 24342334
Am Köllnischen Park 1 Fax: +49 30 24342336
10179 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer:
Dipl.-Kfm. Holger Maczkowsky,
Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2011-12-23 0:27 ` Gregory Farnum
2011-12-23 9:58 ` Amon Ott
@ 2011-12-29 11:30 ` Amon Ott
2012-01-27 14:23 ` Amon Ott
1 sibling, 1 reply; 14+ messages in thread
From: Amon Ott @ 2011-12-29 11:30 UTC (permalink / raw)
To: Gregory Farnum; +Cc: ceph-devel
[-- Attachment #1: Type: text/plain, Size: 868 bytes --]
Hi Greg,
I finally got the test cluster freed up for more Ceph testing.
On Friday 23 December 2011 wrote Gregory Farnum:
> Unfortunately there's not enough info in this log either. If you can
> reproduce it with "mds debug = 20" and put that log somewhere, it
> ought to be enough to work out what's going on, though. Sorry. :(
> -Greg
Here is what MDS logs with debug 20. No idea if it really helps. The cluster
is still in the broken state, should I try to reproduce with a recreated ceph
fs and debug 20? This could be GBs of logs.
Amon Ott
--
Dr. Amon Ott
m-privacy GmbH Tel: +49 30 24342334
Am Köllnischen Park 1 Fax: +49 30 24342336
10179 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer:
Dipl.-Kfm. Holger Maczkowsky,
Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
[-- Attachment #2: mds-crash.log --]
[-- Type: text/x-log, Size: 11270 bytes --]
2011-12-29 12:21:16.289250 4ce1e710 ceph version (commit:), process ceph-mds, pid 31315
2011-12-29 12:21:16.292369 4ae19b70 mds.-1.0 ms_handle_connect on 192.168.111.2:6789/0
2011-12-29 12:21:20.424101 4ae19b70 mds.-1.0 handle_mds_map standby
2011-12-29 12:21:30.369706 4ae19b70 mds.0.13 handle_mds_map i am now mds.0.13
2011-12-29 12:21:30.369778 4ae19b70 mds.0.13 handle_mds_map state change up:standby --> up:replay
2011-12-29 12:21:30.369809 4ae19b70 mds.0.13 replay_start
2011-12-29 12:21:30.369840 4ae19b70 mds.0.13 recovery set is
2011-12-29 12:21:30.369868 4ae19b70 mds.0.13 need osdmap epoch 252, have 251
2011-12-29 12:21:30.369890 4ae19b70 mds.0.13 waiting for osdmap 252 (which blacklists prior instance)
2011-12-29 12:21:30.369960 4ae19b70 mds.0.cache handle_mds_failure mds.0 : recovery peers are
2011-12-29 12:21:30.455693 4ae19b70 mds.0.13 ms_handle_connect on 192.168.111.2:6801/4849
2011-12-29 12:21:30.455803 4ae19b70 mds.0.13 ms_handle_connect on 192.168.111.4:6801/5054
2011-12-29 12:21:30.456156 4ae19b70 mds.0.13 ms_handle_connect on 192.168.111.3:6801/5187
2011-12-29 12:21:30.616751 4ae19b70 mds.0.cache creating system inode with ino:100
2011-12-29 12:21:30.616984 4ae19b70 mds.0.cache creating system inode with ino:1
2011-12-29 12:21:33.246212 4860db70 mds.0.13 replay_done
2011-12-29 12:21:33.246270 4860db70 mds.0.13 making mds journal writeable
2011-12-29 12:21:33.391477 4ae19b70 mds.0.13 handle_mds_map i am now mds.0.13
2011-12-29 12:21:33.391531 4ae19b70 mds.0.13 handle_mds_map state change up:replay --> up:reconnect
2011-12-29 12:21:33.391556 4ae19b70 mds.0.13 reconnect_start
2011-12-29 12:21:33.391579 4ae19b70 mds.0.13 reopen_log
2011-12-29 12:21:33.391637 4ae19b70 mds.0.server reconnect_clients -- 7 sessions
2011-12-29 12:22:21.294278 49414b70 mds.0.server reconnect gave up on client.4531 192.168.111.4:0/257053315
2011-12-29 12:22:21.294334 49414b70 mds.0.server reconnect gave up on client.5000 192.168.111.2:0/1865366646
2011-12-29 12:22:21.294363 49414b70 mds.0.server reconnect gave up on client.5113 192.168.111.1:0/4105687565
2011-12-29 12:22:21.294391 49414b70 mds.0.server reconnect gave up on client.5202 192.168.111.3:0/705815956
2011-12-29 12:22:21.294420 49414b70 mds.0.server reconnect gave up on client.5414 192.168.111.2:0/1736181750
2011-12-29 12:22:21.294447 49414b70 mds.0.server reconnect gave up on client.5417 192.168.111.3:0/3526419788
2011-12-29 12:22:21.294474 49414b70 mds.0.server reconnect gave up on client.5420 192.168.111.4:0/1587151157
2011-12-29 12:22:21.294509 49414b70 mds.0.13 reconnect_done
2011-12-29 12:22:21.419600 4ae19b70 mds.0.13 handle_mds_map i am now mds.0.13
2011-12-29 12:22:21.419647 4ae19b70 mds.0.13 handle_mds_map state change up:reconnect --> up:rejoin
2011-12-29 12:22:21.419673 4ae19b70 mds.0.13 rejoin_joint_start
2011-12-29 12:22:21.443395 4ae19b70 mds.0.13 rejoin_done
2011-12-29 12:22:21.554730 4ae19b70 mds.0.13 handle_mds_map i am now mds.0.13
2011-12-29 12:22:21.554772 4ae19b70 mds.0.13 handle_mds_map state change up:rejoin --> up:active
2011-12-29 12:22:21.554797 4ae19b70 mds.0.13 recovery_done -- successful recovery!
2011-12-29 12:22:21.557761 4ae19b70 mds.0.13 active_start
2011-12-29 12:22:21.561821 4ae19b70 mds.0.13 cluster recovered.
2011-12-29 12:22:21.569550 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80832 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.569608 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80833 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.569643 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80835 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.569675 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80836 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.569705 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80839 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.569739 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80840 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.569770 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80841 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.569801 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80843 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.569831 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80845 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.569863 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80848 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.569981 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80849 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.570403 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80851 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.571011 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80852 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.576171 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80853 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.576414 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80859 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.576573 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80861 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.577048 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80864 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.577701 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80865 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.577988 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80866 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.578247 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80867 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.578435 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80868 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.578493 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80870 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.578655 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80872 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.578813 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80874 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.578870 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80879 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.578895 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80881 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579090 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80883 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579148 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80884 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579174 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80887 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579193 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80888 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579210 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80890 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579228 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80891 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579246 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80893 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579264 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80895 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579281 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80897 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579298 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80899 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579315 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80901 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579332 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80902 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579349 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80903 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579367 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80904 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579383 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80905 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579413 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80906 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579432 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80909 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579450 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80910 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579513 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80911 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579537 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80912 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579575 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80913 <= 81240, already committed, sending ack.
2011-12-29 12:22:21.579600 4ae19b70 mds.0.tableserver(anchortable) got commit for tid 80914 <= 81240, already committed, sending ack.
mds/AnchorServer.cc: In function 'void AnchorServer::dec(inodeno_t)', in thread '4ae19b70'
mds/AnchorServer.cc: 98: FAILED assert(anchor_map.count(ino))
ceph version (commit:)
1: (AnchorServer::dec(inodeno_t)+0x33a) [0x137f0f2a]
2: (AnchorServer::_commit(unsigned long long)+0x2cc) [0x137f3d3c]
3: (MDSTableServer::handle_commit(MMDSTableRequest*)+0x1fb) [0x137ef8cb]
4: (MDS::handle_deferrable_message(Message*)+0xe01) [0x1358b7a1]
5: (MDS::_dispatch(Message*)+0x1ae2) [0x135a3592]
6: (MDS::ms_dispatch(Message*)+0xa5) [0x135a3c55]
7: (SimpleMessenger::dispatch_entry()+0x7c9) [0x138b9219]
8: (SimpleMessenger::DispatchThread::entry()+0x3b) [0x135780db]
9: (Thread::_entry_func(void*)+0x1c) [0x1387119c]
10: (()+0x5905) [0x4d556905]
11: (clone()+0x5e) [0x4ceee8ce]
ceph version (commit:)
1: (AnchorServer::dec(inodeno_t)+0x33a) [0x137f0f2a]
2: (AnchorServer::_commit(unsigned long long)+0x2cc) [0x137f3d3c]
3: (MDSTableServer::handle_commit(MMDSTableRequest*)+0x1fb) [0x137ef8cb]
4: (MDS::handle_deferrable_message(Message*)+0xe01) [0x1358b7a1]
5: (MDS::_dispatch(Message*)+0x1ae2) [0x135a3592]
6: (MDS::ms_dispatch(Message*)+0xa5) [0x135a3c55]
7: (SimpleMessenger::dispatch_entry()+0x7c9) [0x138b9219]
8: (SimpleMessenger::DispatchThread::entry()+0x3b) [0x135780db]
9: (Thread::_entry_func(void*)+0x1c) [0x1387119c]
10: (()+0x5905) [0x4d556905]
11: (clone()+0x5e) [0x4ceee8ce]
*** Caught signal (Aborted) **
in thread 4ae19b70
ceph version (commit:)
1: (()+0x471833) [0x13956833]
2: [0x4d599400]
3: [0x4d599422]
4: (gsignal()+0x4f) [0x4ce4b02f]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2011-12-29 11:30 ` Amon Ott
@ 2012-01-27 14:23 ` Amon Ott
2012-01-27 14:50 ` Sage Weil
0 siblings, 1 reply; 14+ messages in thread
From: Amon Ott @ 2012-01-27 14:23 UTC (permalink / raw)
To: Gregory Farnum; +Cc: ceph-devel
On Thursday 29 December 2011 wrote Amon Ott:
> I finally got the test cluster freed up for more Ceph testing.
>
> On Friday 23 December 2011 wrote Gregory Farnum:
> > Unfortunately there's not enough info in this log either. If you can
> > reproduce it with "mds debug = 20" and put that log somewhere, it
> > ought to be enough to work out what's going on, though. Sorry. :(
> > -Greg
>
> Here is what MDS logs with debug 20. No idea if it really helps. The
> cluster is still in the broken state, should I try to reproduce with a
> recreated ceph fs and debug 20? This could be GBs of logs.
Update: I recreated the Ceph FS with release 0.40. It broke only because of a
btrfs bug hitting two of the four nodes (after ca. one day of heavy load) and
recovered without problems when the nodes were back. Then I recreated with
ext4 as osd storage area and have not managed to break it within four days,
two of these under heavy load.
This means that this bug is probably fixed. It might be related to the
automatic reconnect of mds, which avoids meta data inconsistencies. :)
Amon Ott
--
Dr. Amon Ott
m-privacy GmbH Tel: +49 30 24342334
Am Köllnischen Park 1 Fax: +49 30 24342336
10179 Berlin http://www.m-privacy.de
Amtsgericht Charlottenburg, HRB 84946
Geschäftsführer:
Dipl.-Kfm. Holger Maczkowsky,
Roman Maczkowsky
GnuPG-Key-ID: 0x2DD3A649
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2012-01-27 14:23 ` Amon Ott
@ 2012-01-27 14:50 ` Sage Weil
2012-01-27 18:54 ` Gregory Farnum
0 siblings, 1 reply; 14+ messages in thread
From: Sage Weil @ 2012-01-27 14:50 UTC (permalink / raw)
To: Amon Ott; +Cc: Gregory Farnum, ceph-devel
On Fri, 27 Jan 2012, Amon Ott wrote:
> On Thursday 29 December 2011 wrote Amon Ott:
> > I finally got the test cluster freed up for more Ceph testing.
> >
> > On Friday 23 December 2011 wrote Gregory Farnum:
> > > Unfortunately there's not enough info in this log either. If you can
> > > reproduce it with "mds debug = 20" and put that log somewhere, it
> > > ought to be enough to work out what's going on, though. Sorry. :(
> > > -Greg
> >
> > Here is what MDS logs with debug 20. No idea if it really helps. The
> > cluster is still in the broken state, should I try to reproduce with a
> > recreated ceph fs and debug 20? This could be GBs of logs.
>
> Update: I recreated the Ceph FS with release 0.40. It broke only because of a
> btrfs bug hitting two of the four nodes (after ca. one day of heavy load) and
> recovered without problems when the nodes were back. Then I recreated with
> ext4 as osd storage area and have not managed to break it within four days,
> two of these under heavy load.
>
> This means that this bug is probably fixed. It might be related to the
> automatic reconnect of mds, which avoids meta data inconsistencies. :)
Yeah, I suspect that the problem is related to the MDS journal replay and
the two-phase-commit stuff going on with the anchor table updates. I
think we should keep this open until we can do MDS restart thrashing
against a heavy link workload.
Unless there was something you found/fixed before, Greg?
Thanks for keeping an eye on this, Amon!
sage
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: Bug #1047 reproduced
2012-01-27 14:50 ` Sage Weil
@ 2012-01-27 18:54 ` Gregory Farnum
0 siblings, 0 replies; 14+ messages in thread
From: Gregory Farnum @ 2012-01-27 18:54 UTC (permalink / raw)
To: Sage Weil; +Cc: Amon Ott, ceph-devel
On Fri, Jan 27, 2012 at 6:50 AM, Sage Weil <sage@newdream.net> wrote:
> On Fri, 27 Jan 2012, Amon Ott wrote:
>> On Thursday 29 December 2011 wrote Amon Ott:
>> > I finally got the test cluster freed up for more Ceph testing.
>> >
>> > On Friday 23 December 2011 wrote Gregory Farnum:
>> > > Unfortunately there's not enough info in this log either. If you can
>> > > reproduce it with "mds debug = 20" and put that log somewhere, it
>> > > ought to be enough to work out what's going on, though. Sorry. :(
>> > > -Greg
>> >
>> > Here is what MDS logs with debug 20. No idea if it really helps. The
>> > cluster is still in the broken state, should I try to reproduce with a
>> > recreated ceph fs and debug 20? This could be GBs of logs.
>>
>> Update: I recreated the Ceph FS with release 0.40. It broke only because of a
>> btrfs bug hitting two of the four nodes (after ca. one day of heavy load) and
>> recovered without problems when the nodes were back. Then I recreated with
>> ext4 as osd storage area and have not managed to break it within four days,
>> two of these under heavy load.
>>
>> This means that this bug is probably fixed. It might be related to the
>> automatic reconnect of mds, which avoids meta data inconsistencies. :)
>
> Yeah, I suspect that the problem is related to the MDS journal replay and
> the two-phase-commit stuff going on with the anchor table updates. I
> think we should keep this open until we can do MDS restart thrashing
> against a heavy link workload.
>
> Unless there was something you found/fixed before, Greg?
Unfortunately not — like I reported in the bug, there are definitely
multiple anchor destroy updates getting into the journal somehow, but
I was unable to figure out how it might have happened. I'm not sure I
considered synchronization bugs in MDS restart or client replay,
though...
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2012-01-27 18:54 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-12-01 8:37 Bug #1047 reproduced Amon Ott
2011-12-02 11:17 ` Amon Ott
2011-12-02 16:57 ` Sage Weil
2011-12-05 10:21 ` Amon Ott
2011-12-06 13:20 ` Amon Ott
2011-12-21 12:37 ` Amon Ott
2011-12-21 16:18 ` Gregory Farnum
2011-12-21 16:36 ` Amon Ott
2011-12-23 0:27 ` Gregory Farnum
2011-12-23 9:58 ` Amon Ott
2011-12-29 11:30 ` Amon Ott
2012-01-27 14:23 ` Amon Ott
2012-01-27 14:50 ` Sage Weil
2012-01-27 18:54 ` Gregory Farnum
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.