From: Ted Ts'o <tytso@mit.edu>
To: Sage Weil <sage@newdream.net>
Cc: ceph-devel@vger.kernel.org
Subject: Re: osd sequence number mismatches and timeout's
Date: Tue, 2 Nov 2010 11:29:16 -0400 [thread overview]
Message-ID: <20101102152916.GC25614@thunk.org> (raw)
In-Reply-To: <Pine.LNX.4.64.1011011549520.26372@cobra.newdream.net>
On Mon, Nov 01, 2010 at 03:57:55PM -0700, Sage Weil wrote:
> Is there something in dmesg before the osd22 seq number errors pop up?
Yup, you were quite right. There was a bad crc that probably caused
the seq's to get out of sync.
Nov 1 10:12:50 bdio20 kernel: [233439.052725] ceph: osd22 10.138.138.13:6804 bad crc
Nov 1 10:12:51 bdio20 kernel: [233440.672738] ceph: skipping osd22 192.168.168.13:6804 seq 1, expected 2
Nov 1 10:12:51 bdio20 kernel: [233440.672958] ceph: skipping osd22 192.168.168.13:6804 seq 2, expected 3
Nov 1 10:12:51 bdio20 kernel: [233440.675705] ceph: skipping osd22 192.168.168.13:6804 seq 3, expected 4
> Something originally caused the seq's to get out of sync. I suspect it
> was a transient network error that made the TCP session drop and
> reconnect, and it's not skipping already-received messages. There was a
> bug in the skip code (so they stayed out of sync and osd22 eventually
> timed out). I pushed a fix for that to the ceph-client.git master branch
> (df9f86fa).
BTW, it looks like something may be unhappy? I tried doing a clone of
ceph-client.git, and I'm getting a failure:
% git clone git://ceph.newdream.net/git/ceph-client.git ceph-client
Cloning into ceph-client...
fatal: I don't handle protocol '/usr/local/google/git'
I downloaded df9f86fa and will try it out. Thanks for pushing out the
patch so quickly!
- Ted
prev parent reply other threads:[~2010-11-02 15:29 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-11-01 22:14 osd sequence number mismatches and timeout's Theodore Ts'o
2010-11-01 22:57 ` Sage Weil
2010-11-02 15:29 ` Ted Ts'o [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20101102152916.GC25614@thunk.org \
--to=tytso@mit.edu \
--cc=ceph-devel@vger.kernel.org \
--cc=sage@newdream.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.