From: Stefan Priebe - Profihost AG <s.priebe@profihost.ag>
To: Sage Weil <sage@inktank.com>
Cc: "ceph-devel@vger.kernel.org" <ceph-devel@vger.kernel.org>
Subject: Re: endless flying slow requests
Date: Thu, 15 Nov 2012 09:09:53 +0100 [thread overview]
Message-ID: <50A4A351.8020506@profihost.ag> (raw)
In-Reply-To: <alpine.DEB.2.00.1211140656590.6512@cobra.newdream.net>
Am 14.11.2012 15:59, schrieb Sage Weil:
> Hi Stefan,
>
> I would be nice to confirm that no clients are waiting on replies for
> these requests; currently we suspect that the OSD request tracking is the
> buggy part. If you query the OSD admin socket you should be able to dump
> requests and see the client IP, and then query the client.
>
> Is it librbd? In that case you likely need to change the config so that
> it is listening on an admin socket ('admin socket = path').
Yes it is. So i have to specify admin socket at the KVM host? How do i
query the admin socket for requests?
Stefan
> On Wed, 14 Nov 2012, Stefan Priebe - Profihost AG wrote:
>
>> Hello list,
>>
>> i see this several times. Endless flying slow requests. And they never stop
>> until i restart the mentioned osd.
>>
>> 2012-11-14 10:11:57.513395 osd.24 [WRN] 1 slow requests, 1 included below;
>> oldest blocked for > 31789.858457 secs
>> 2012-11-14 10:11:57.513399 osd.24 [WRN] slow request 31789.858457 seconds old,
>> received at 2012-11-14 01:22:07.654922: osd_op(client.30286.0:6719
>> rbd_data.75c55bf2fdd7.0000000000001399 [write 282624~4096] 3.3f6d2373) v4
>> currently delayed
>> 2012-11-14 10:11:58.513584 osd.24 [WRN] 1 slow requests, 1 included below;
>> oldest blocked for > 31790.858646 secs
>> 2012-11-14 10:11:58.513586 osd.24 [WRN] slow request 31790.858646 seconds old,
>> received at 2012-11-14 01:22:07.654922: osd_op(client.30286.0:6719
>> rbd_data.75c55bf2fdd7.0000000000001399 [write 282624~4096] 3.3f6d2373) v4
>> currently delayed
>> 2012-11-14 10:11:59.513766 osd.24 [WRN] 1 slow requests, 1 included below;
>> oldest blocked for > 31791.858827 secs
>> 2012-11-14 10:11:59.513768 osd.24 [WRN] slow request 31791.858827 seconds old,
>> received at 2012-11-14 01:22:07.654922: osd_op(client.30286.0:6719
>> rbd_data.75c55bf2fdd7.0000000000001399 [write 282624~4096] 3.3f6d2373) v4
>> currently delayed
>> 2012-11-14 10:12:00.513909 osd.24 [WRN] 1 slow requests, 1 included below;
>> oldest blocked for > 31792.858971 secs
>> 2012-11-14 10:12:00.513916 osd.24 [WRN] slow request 31792.858971 seconds old,
>> received at 2012-11-14 01:22:07.654922: osd_op(client.30286.0:6719
>> rbd_data.75c55bf2fdd7.0000000000001399 [write 282624~4096] 3.3f6d2373) v4
>> currently delayed
>> 2012-11-14 10:12:01.514061 osd.24 [WRN] 1 slow requests, 1 included below;
>> oldest blocked for > 31793.859124 secs
>> 2012-11-14 10:12:01.514063 osd.24 [WRN] slow request 31793.859124 seconds old,
>> received at 2012-11-14 01:22:07.654922: osd_op(client.30286.0:6719
>> rbd_data.75c55bf2fdd7.0000000000001399 [write 282624~4096] 3.3f6d2373) v4
>> currently delayed
>>
>> When i now restart osd 24 they go away and everything is fine again.
>>
>> Stefan
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
next prev parent reply other threads:[~2012-11-15 8:10 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-11-14 9:12 endless flying slow requests Stefan Priebe - Profihost AG
2012-11-14 14:59 ` Sage Weil
2012-11-15 8:09 ` Stefan Priebe - Profihost AG [this message]
2012-11-15 8:18 ` Josh Durgin
2012-11-15 14:58 ` Sage Weil
2012-11-28 1:51 ` Sage Weil
2012-11-28 7:23 ` Stefan Priebe - Profihost AG
2012-11-28 7:47 ` Andrey Korolyov
2012-11-28 21:11 ` Samuel Just
2012-11-28 21:12 ` Samuel Just
2012-11-28 21:52 ` Moore, Shawn M
2012-11-28 22:04 ` Andrey Korolyov
2012-12-01 5:06 ` Samuel Just
2012-12-02 3:18 ` Sage Weil
2012-12-02 22:02 ` Samuel Just
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=50A4A351.8020506@profihost.ag \
--to=s.priebe@profihost.ag \
--cc=ceph-devel@vger.kernel.org \
--cc=sage@inktank.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.