linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Tejun Heo <tj@kernel.org>
To: Vivek Goyal <vgoyal@redhat.com>
Cc: Josh Hunt <joshhunt00@gmail.com>, Jens Axboe <axboe@kernel.dk>,
	linux-kernel@vger.kernel.org
Subject: Re: multi-second application stall in open()
Date: Thu, 21 Jun 2012 13:36:15 -0700	[thread overview]
Message-ID: <20120621203615.GE4642@google.com> (raw)
In-Reply-To: <20120621203217.GC14095@redhat.com>

Hey, Vivek.

On Thu, Jun 21, 2012 at 04:32:17PM -0400, Vivek Goyal wrote:
> Here we deleted queue 20720 and did nothing for .6 seconds and from
> previous logs it is visible that writes are pending and queued.
> 
> For some reason cfq_schedule_dispatch() did not lead to kicking queue
> or queue was kicked but somehow write queue was not selected for
> dispatch (A case of corrupt data structures?).
> 
> Are you able to reproduce this issue on latest kernels (3.5-rc2?). I would
> say put some logs in select_queue() and see where did it bail out. That
> will confirm that select queue was called and can also give some details 
> why we did not select async queue for dispatch. (Note: select_queue is called
> multiple times so putting trace point there makes logs very verbose).

Some people are putting in watchdog timers in block layer to kick cfq
when it stalls with pending requests.  The cfq code there has diverged
quite a bit from upstream so I have no idea whether it's caused by the
same issue.  The symptom sounds exactly the same tho.  So, yeah, I
think it isn't too unlikely that we have a cfq logic bug leading to
stalls.  :(

-- 
tejun

  reply	other threads:[~2012-06-21 20:36 UTC|newest]

Thread overview: 46+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-03-06 21:56 multi-second application stall in open() Josh Hunt
2012-03-07 13:43 ` Josh Hunt
2012-03-07 16:28   ` Vivek Goyal
2012-03-07 18:56     ` Jens Axboe
2012-03-07 19:56       ` Vivek Goyal
2012-03-07 21:08       ` Josh Hunt
2012-03-08 22:22         ` Josh Hunt
2012-03-08 23:40           ` Vivek Goyal
     [not found]             ` <CAKA=qzbsL9UVYLZ3=hoT-1jfp=v=_Sr=h+YeHu0qAA=Ko_7P6w@mail.gmail.com>
2012-06-21 19:26               ` Josh Hunt
2012-06-21 20:32                 ` Vivek Goyal
2012-06-21 20:36                   ` Tejun Heo [this message]
2012-06-21 21:28                     ` Josh Hunt
2012-06-21 21:32                       ` Tejun Heo
2012-06-21 21:48                         ` Rakesh Iyer
     [not found]                         ` <CAOT6A4-a49wLHcQepUxJCDxOxfnSTEWa72OweLsmrea85OyrCg@mail.gmail.com>
2012-06-22 14:15                           ` Vivek Goyal
2012-06-21 21:11                   ` Josh Hunt
2012-06-22 14:12                     ` Vivek Goyal
2012-06-22 20:05                       ` Josh Hunt
2012-06-22 20:22                         ` Josh Hunt
2012-06-22 20:42                           ` Vivek Goyal
2012-06-22 20:53                             ` Josh Hunt
2012-06-22 20:57                               ` Josh Hunt
2012-06-22 21:34                                 ` Josh Hunt
2012-06-25 13:30                                   ` Vivek Goyal
2012-06-25 16:22                                     ` Josh Hunt
2012-06-25 21:18                                       ` Vivek Goyal
2012-06-25 23:05                                         ` Josh Hunt
2012-06-26  4:01                                           ` Josh Hunt
2012-06-26 12:59                                             ` Vivek Goyal
2012-06-26 15:18                                               ` Josh Hunt
2012-06-26 15:53                                                 ` Vivek Goyal
2012-06-26 20:37                                                   ` Josh Hunt
2012-06-26 20:56                                                     ` Tejun Heo
     [not found]                                                       ` <CAKA=qzbBtteDjHiPogCvN5jOSiPrDxx=vn96p02bXUy=6=jAgA@mail.gmail.com>
2012-06-26 23:44                                                         ` Josh Hunt
2012-06-27 17:21                                                           ` Josh Hunt
2012-06-27 17:38                                                             ` Tejun Heo
2012-06-27 17:44                                                               ` Josh Hunt
2012-06-27 17:54                                                                 ` Tejun Heo
2012-06-27 17:59                                                                   ` Josh Hunt
2012-06-29 23:02                                                                     ` Tejun Heo
2012-06-30  0:37                                                                       ` Josh Hunt
2012-07-04  1:12                                                                         ` Tejun Heo
2012-07-18 17:48                                                                           ` Tejun Heo
2012-06-26 20:43                                                 ` Tejun Heo
2012-06-25 17:26                                     ` Tejun Heo
2012-03-07 19:47     ` Josh Hunt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20120621203615.GE4642@google.com \
    --to=tj@kernel.org \
    --cc=axboe@kernel.dk \
    --cc=joshhunt00@gmail.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=vgoyal@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).