From: "Jim Schutt" <jaschut-4OHPYypu0djtX7QSmKvirg@public.gmane.org>
To: "Chen, Xiaoxi" <xiaoxi.chen-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Cc: "ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
<ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
"ceph-users-Qp0mS5GaXlQ@public.gmane.org"
<ceph-users-Qp0mS5GaXlQ@public.gmane.org>
Subject: Re: OSD state flipping when cluster-network in high utilization
Date: Wed, 15 May 2013 09:05:51 -0600 [thread overview]
Message-ID: <5193A44F.1060509@sandia.gov> (raw)
In-Reply-To: <6F3FA899187F0043BA1827A69DA2F7CC0145F746-0J0gbvR4kTiiAffOGbnezLfspsVTdybXVpNB7YpNyf8@public.gmane.org>
On 05/14/2013 09:23 PM, Chen, Xiaoxi wrote:
>> How responsive generally is the machine under load? Is there available CPU?
> The machine works well, and the issued OSDs are likely the same, seems because they have relative slower disk( disk type are the same but the latency is a bit higher ,8ms -> 10ms).
>
> Top show no idle % but still have 30+% of io_wait, my colleague educate me that io_wait can be treated as free.
>
> Another information is offload the heartbeat to 1Gb nic doesn't solve the problem, what's more, when we doing random write test, we can still see this flipping happen. So I would like to say it may related with CPU scheduler ? The heartbeat thread (in busy OSD ) failed to get enough cpu cycle.
>
FWIW, also take a close look at your monitor daemons, and
whether they show any signs of being overloaded.
I frequently see OSDs wrongly marked down when my
mons cannot keep up with their workload.
-- Jim
next prev parent reply other threads:[~2013-05-15 15:05 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-05-14 8:56 OSD state flipping when cluster-network in high utilization Chen, Xiaoxi
2013-05-14 15:30 ` [ceph-users] " Sage Weil
2013-05-14 15:35 ` Mark Nelson
[not found] ` <519259C5.3000109-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>
2013-05-14 23:13 ` Chen, Xiaoxi
[not found] ` <6AC1548D-DE2C-4E71-AC73-3903B57C76C6-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
2013-05-14 23:22 ` Sage Weil
[not found] ` <alpine.DEB.2.00.1305141620410.12954-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org>
2013-05-15 3:23 ` Chen, Xiaoxi
[not found] ` <6F3FA899187F0043BA1827A69DA2F7CC0145F746-0J0gbvR4kTiiAffOGbnezLfspsVTdybXVpNB7YpNyf8@public.gmane.org>
2013-05-15 3:40 ` Sage Weil
[not found] ` <alpine.DEB.2.00.1305142039520.23131-vIokxiIdD2AQNTJnQDzGJqxOck334EZe@public.gmane.org>
2013-05-15 12:53 ` Chen, Xiaoxi
[not found] ` <alpine.DEB.2.00.1305150851190.14301@cobra.newdream.net>
2013-05-23 2:08 ` [ceph-users] " Chen, Xiaoxi
2013-05-15 15:05 ` Jim Schutt [this message]
2013-05-15 15:12 ` Chen, Xiaoxi
2013-05-15 15:38 ` Jim Schutt
2013-05-15 0:52 ` Wales Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5193A44F.1060509@sandia.gov \
--to=jaschut-4ohpyypu0djtx7qsmkvirg@public.gmane.org \
--cc=ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
--cc=ceph-users-Qp0mS5GaXlQ@public.gmane.org \
--cc=xiaoxi.chen-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.