From: Dan Mick <dan.mick@inktank.com>
To: Christian Huang <ythuang@gmail.com>
Cc: ceph-devel@vger.kernel.org, sage@inktank.com
Subject: Re: CEPH RBD client kernel panic when OSD connection is lost on kernel 3.2, 3.5, 3.5.4
Date: Mon, 24 Sep 2012 11:50:16 -0700 [thread overview]
Message-ID: <5060AB68.8090803@inktank.com> (raw)
In-Reply-To: <CAP5wSLcy90qhsTnNxM=t+_SB1kpfNvchrSqdAG91P6tCe0RPRA@mail.gmail.com>
We're looking into this, Christian.
On 09/24/2012 03:23 AM, Christian Huang wrote:
> Hi,
> we met the following issue while testing ceph cluster HA.
> Appreciate if anyone can shed some light.
> could this be related to the configuration ? (ie, 2 OSD nodes only)
>
> Issue description:
> ceph rbd client will kernel panic if an OSD server loses it's
> network connectivity.
> so far, we can reproduce it with certainty.
> we have tried with the following kernels
> a. Stock kernel from 12.04 (3.2 series)
> 3.5 series, as suggested in a previous mail by Sage
> b. 3.5.0-15 from quantal repo,
> git://kernel.ubuntu.com/ubuntu/ubuntu-quantal.git, Ubuntu-3.5.0-15.22
> tag
> c. v3.5.4-quantal,
> http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.5.4-quantal/
>
> Environment:
> OS: Ubuntu 12.04 precise pangolin
> Ceph configuration:
> OSD nodes: 2 x 12 drives , 1 os drive, 11 are mapped to OSD
> 0-10, 10GbE link
> Monitor nodes: 3 x KVM virtual machines on ubuntu host.
> test client: fresh install of Ubuntu 12.04.1
> Ceph version used: 0.48, 0.48.1, 0.48.2, 0.51
> all nodes have the same kernel version.
>
> steps to reproduce:
> on the test client,
> 1. load rbd modules
> 2. create rbd device
> 3. map rbd device
> 4. use fio tool to create work load on the device, 8 threads is
> used for workload
> we have also tried with iometer, 8 workers, 32k 50/50, same results.
>
> on one of the OSD nodes,
> 1. sudo ifconfig eth0 down #where eth0 is the primary interface
> configured for ceph.
> 2. within 30 seconds, the test client will panic.
>
> this happens when there is IO activity on the RBD device, and one
> of the OSD nodes loses connectivity.
>
> The netconsole output is available available from the following
> dropbox link,
> zip: goo.gl/LHytr
>
> Best Regards
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
next prev parent reply other threads:[~2012-09-24 18:50 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-09-24 10:23 CEPH RBD client kernel panic when OSD connection is lost on kernel 3.2, 3.5, 3.5.4 Christian Huang
2012-09-24 18:50 ` Dan Mick [this message]
2012-09-24 22:59 ` Alex Elder
[not found] ` <CAP5wSLeuLhJxrqmiaz8-4SXo_Sfzs8xY23UMO2O7vivnBMNDBg@mail.gmail.com>
2012-09-25 3:38 ` Alex Elder
2012-09-25 4:33 ` Christian Huang
2012-09-25 6:09 ` Christian Huang
2012-09-25 9:26 ` Damien Churchill
2012-09-25 12:14 ` Alex Elder
2012-09-25 14:38 ` Christian Huang
2012-09-25 15:05 ` Alex Elder
2012-09-25 15:49 ` Alex Elder
2012-09-27 1:34 ` Christian Huang
2012-09-27 3:13 ` Alex Elder
2012-09-25 9:31 ` Christian Huang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5060AB68.8090803@inktank.com \
--to=dan.mick@inktank.com \
--cc=ceph-devel@vger.kernel.org \
--cc=sage@inktank.com \
--cc=ythuang@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.