All of lore.kernel.org
 help / color / mirror / Atom feed
From: Stefan Priebe <s.priebe-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
To: Kent Overstreet <kmo-PEzghdH756F8UrSeD/g0lQ@public.gmane.org>
Cc: Jens Axboe <axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org>,
	linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org,
	linux-bcache-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: Re: bcache: Fix a writeback performance regression
Date: Thu, 22 Aug 2013 07:25:18 +0200	[thread overview]
Message-ID: <5215A0BE.1040903@profihost.ag> (raw)
In-Reply-To: <20130821234715.GA4630-jC9Py7bek1znysI04z7BkA@public.gmane.org>

Am 22.08.2013 01:47, schrieb Kent Overstreet:
> On Tue, Aug 20, 2013 at 10:07:45AM +0200, Stefan Priebe - Profihost AG wrote:
>> Am 20.08.2013 10:01, schrieb Stefan Priebe - Profihost AG:
>>> Am 20.08.2013 00:27, schrieb Kent Overstreet:
>>>> On Mon, Aug 19, 2013 at 12:09:24AM +0200, Stefan Priebe wrote:
>>>>>
>>>>> Vanilla 3.10.7 + bcache: Fix a writeback performance regression
>>>>>
>>>>> http://pastebin.com/raw.php?i=LXZk4cMH
>>>>
>>>> Whoops, at first I thought this was the same bug as one I'd already been
>>>> chasing down that had been a harmless bug - turns out I didn't look
>>>> closely enough at the backtrace.
>>>>
>>>> What happened is background writeback is deadlocking, because for some
>>>> reason the workqueue it's running out of is a singlethreaded workqueue,
>>>> so as soon as it decides to queue enough writeback bios that it has to
>>>> sleep on that semaphore (which often won't happen due to the PD
>>>> controller based ratelimiting) - boom, deadlock.
>>>>
>>>> Here's the fixup patch I just tested and am applying:
>>
>> Oh i'm now seeing very high CPU spikes of kworker... i don't see if i
>> remove bcache: Fix a writeback performance regression.
>
> Are you able to reproduce it? I'm not having any luck reproducing it...

Hi,

yes but only on production system it seems it happens when there is a 
big bunch in dirty and having heavy 4K random I/O.

Stefan

WARNING: multiple messages have this Message-ID (diff)
From: Stefan Priebe <s.priebe@profihost.ag>
To: Kent Overstreet <kmo@daterainc.com>
Cc: Jens Axboe <axboe@kernel.dk>,
	linux-kernel@vger.kernel.org, linux-bcache@vger.kernel.org
Subject: Re: bcache: Fix a writeback performance regression
Date: Thu, 22 Aug 2013 07:25:18 +0200	[thread overview]
Message-ID: <5215A0BE.1040903@profihost.ag> (raw)
In-Reply-To: <20130821234715.GA4630@moria.home.lan>

Am 22.08.2013 01:47, schrieb Kent Overstreet:
> On Tue, Aug 20, 2013 at 10:07:45AM +0200, Stefan Priebe - Profihost AG wrote:
>> Am 20.08.2013 10:01, schrieb Stefan Priebe - Profihost AG:
>>> Am 20.08.2013 00:27, schrieb Kent Overstreet:
>>>> On Mon, Aug 19, 2013 at 12:09:24AM +0200, Stefan Priebe wrote:
>>>>>
>>>>> Vanilla 3.10.7 + bcache: Fix a writeback performance regression
>>>>>
>>>>> http://pastebin.com/raw.php?i=LXZk4cMH
>>>>
>>>> Whoops, at first I thought this was the same bug as one I'd already been
>>>> chasing down that had been a harmless bug - turns out I didn't look
>>>> closely enough at the backtrace.
>>>>
>>>> What happened is background writeback is deadlocking, because for some
>>>> reason the workqueue it's running out of is a singlethreaded workqueue,
>>>> so as soon as it decides to queue enough writeback bios that it has to
>>>> sleep on that semaphore (which often won't happen due to the PD
>>>> controller based ratelimiting) - boom, deadlock.
>>>>
>>>> Here's the fixup patch I just tested and am applying:
>>
>> Oh i'm now seeing very high CPU spikes of kworker... i don't see if i
>> remove bcache: Fix a writeback performance regression.
>
> Are you able to reproduce it? I'm not having any luck reproducing it...

Hi,

yes but only on production system it seems it happens when there is a 
big bunch in dirty and having heavy 4K random I/O.

Stefan

  parent reply	other threads:[~2013-08-22  5:25 UTC|newest]

Thread overview: 50+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-08-14 22:59 [GIT PULL] bcache fixes for 3.11 Kent Overstreet
2013-08-14 22:59 ` Kent Overstreet
2013-08-15  6:43 ` Stefan Priebe - Profihost AG
2013-08-15  6:43   ` Stefan Priebe - Profihost AG
2013-08-15  7:07   ` Kent Overstreet
     [not found]   ` <520C788A.1060208-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
2013-08-16 10:11     ` Stefan Priebe - Profihost AG
2013-08-16 10:11       ` Stefan Priebe - Profihost AG
     [not found]       ` <520DFAB4.1050402-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
2013-08-18 22:09         ` bcache: Fix a writeback performance regression Stefan Priebe
2013-08-18 22:09           ` Stefan Priebe
2013-08-19 22:27           ` Kent Overstreet
2013-08-19 22:27             ` Kent Overstreet
2013-08-20  8:01             ` Stefan Priebe - Profihost AG
2013-08-20  8:01               ` Stefan Priebe - Profihost AG
     [not found]               ` <52132243.8050803-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
2013-08-20  8:07                 ` Stefan Priebe - Profihost AG
2013-08-20  8:07                   ` Stefan Priebe - Profihost AG
2013-08-21 21:10                   ` Kent Overstreet
     [not found]                   ` <521323D1.1070503-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
2013-08-21 23:47                     ` Kent Overstreet
2013-08-21 23:47                       ` Kent Overstreet
     [not found]                       ` <20130821234715.GA4630-jC9Py7bek1znysI04z7BkA@public.gmane.org>
2013-08-22  5:25                         ` Stefan Priebe [this message]
2013-08-22  5:25                           ` Stefan Priebe
2013-08-22  5:27                         ` Stefan Priebe
2013-08-22  5:27                           ` Stefan Priebe
2013-08-22  5:43                           ` Kent Overstreet
2013-08-22  5:59                             ` Stefan Priebe
     [not found]                               ` <5215A8A8.90900-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
2013-08-22  6:02                                 ` Kent Overstreet
2013-08-22  6:02                                   ` Kent Overstreet
2013-08-22  7:32                                   ` Stefan Priebe - Profihost AG
2013-08-22  7:32                                     ` Stefan Priebe - Profihost AG
     [not found]                                     ` <5215BEA4.6040101-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
2013-08-22  7:42                                       ` Kent Overstreet
2013-08-22  7:42                                         ` Kent Overstreet
2013-08-26 19:21                                       ` Stefan Priebe
2013-08-26 19:21                                         ` Stefan Priebe
     [not found]                                         ` <521BAACB.4070003-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
2013-08-28  3:37                                           ` kernel neophyte
2013-08-28  3:37                                             ` kernel neophyte
     [not found]                                             ` <CAFkUHxeAUvv-nhSZSDRVwdm8SoNhHD8oSBKqDYnEO3bNAKLG6w-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28  6:05                                               ` Kent Overstreet
2013-08-28  6:05                                                 ` Kent Overstreet
2013-08-28 18:12                                                 ` kernel neophyte
     [not found]                                                   ` <CAFkUHxeryKu2VdgBTBWBHTnzzZ=aapUgLd22+7cQnFOXF4r_2g-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 18:38                                                     ` Stefan Priebe - Profihost AG
2013-08-28 18:38                                                       ` Stefan Priebe - Profihost AG
2013-08-28 18:47                                                       ` kernel neophyte
     [not found]                                                         ` <CAFkUHxcu+0Qv64_4gEGbMk5YaBkm1r7pdw4-JTamiCqy=SMUPg-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 19:20                                                           ` Stefan Priebe
2013-08-28 19:20                                                             ` Stefan Priebe
     [not found]                                                             ` <521E4D87.8020705-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
2013-08-28 20:12                                                               ` kernel neophyte
2013-08-28 20:12                                                                 ` kernel neophyte
     [not found]                                                                 ` <CAFkUHxfn8fn09d8xnhVyySOTkn_8r_5O8aqCCf=GmMr6m7nOmA-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org>
2013-08-28 20:15                                                                   ` Stefan Priebe
2013-08-28 20:15                                                                     ` Stefan Priebe
     [not found]                                                                     ` <521E5A69.30408-2Lf/h1ldwEHR5kwTpVNS9A@public.gmane.org>
2013-08-28 22:45                                                                       ` kernel neophyte
2013-08-28 22:45                                                                         ` kernel neophyte
2013-08-29 18:52                                                                       ` Stefan Priebe
2013-08-29 18:52                                                                         ` Stefan Priebe

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5215A0BE.1040903@profihost.ag \
    --to=s.priebe-2lf/h1ldwehr5kwtpvns9a@public.gmane.org \
    --cc=axboe-tSWWG44O7X1aa/9Udqfwiw@public.gmane.org \
    --cc=kmo-PEzghdH756F8UrSeD/g0lQ@public.gmane.org \
    --cc=linux-bcache-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    --cc=linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.