From: Jan Kara <jack@suse.cz>
To: Denys Fedorysychenko <nuclearcat@nuclearcat.com>
Cc: Alexander Viro <viro@zeniv.linux.org.uk>,
linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: endless sync on bdi_sched_wait()? 2.6.33.1
Date: Thu, 8 Apr 2010 11:28:50 +0200 [thread overview]
Message-ID: <20100408092850.GA20488@quack.suse.cz> (raw)
In-Reply-To: <201003311907.31342.nuclearcat@nuclearcat.com>
Hi,
On Wed 31-03-10 19:07:31, Denys Fedorysychenko wrote:
> I have a proxy server with "loaded" squid. On some moment i did sync, and
> expecting it to finish in reasonable time. Waited more than 30 minutes, still
> "sync". Can be reproduced easily.
>
> Here is some stats and info:
>
> Linux SUPERPROXY 2.6.33.1-build-0051 #16 SMP Wed Mar 31 17:23:28 EEST 2010
> i686 GNU/Linux
>
> SUPERPROXY ~ # iostat -k -x -d 30
> Linux 2.6.33.1-build-0051 (SUPERPROXY) 03/31/10 _i686_ (4 CPU)
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
> avgqu-sz await svctm %util
> sda 0.16 0.01 0.08 0.03 3.62 1.33 88.94
> 0.15 1389.89 59.15 0.66
> sdb 4.14 61.25 6.22 25.55 44.52 347.21 24.66
> 2.24 70.60 2.36 7.49
> sdc 4.37 421.28 9.95 98.31 318.27 2081.95 44.34
> 20.93 193.21 2.31 24.96
> sdd 2.34 339.90 3.97 117.47 95.48 1829.52 31.70
> 1.73 14.23 8.09 98.20
> sde 2.29 71.40 2.34 27.97 22.56 397.81 27.74
> 2.34 77.34 1.66 5.04
> dm-0 0.00 0.00 0.19 0.02 3.48 0.02 32.96
> 0.05 252.11 28.05 0.60
>
> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
> avgqu-sz await svctm %util
> sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
> sdb 0.00 54.67 2.93 26.87 12.27 326.13 22.71
> 2.19 73.49 1.91 5.68
> sdc 0.00 420.50 3.43 110.53 126.40 2127.73 39.56
> 23.82 209.00 2.06 23.44
> sdd 0.00 319.63 2.30 122.03 121.87 1765.87 30.37
> 1.72 13.83 7.99 99.37
> sde 0.00 71.67 0.83 30.63 6.93 409.33 26.46
> 2.66 84.68 1.51 4.76
> dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 0.00
>
>
> CPU: 8.4% usr 7.7% sys 0.0% nic 50.7% idle 27.7% io 0.6% irq 4.7% sirq
> Load average: 5.57 4.82 4.46 2/243 2032
> PID PPID USER STAT VSZ %MEM CPU %CPU COMMAND
> 1769 1552 squid R 668m 8.3 3 11.7 /usr/sbin/squid -N
> 1546 1545 root R 10800 0.1 2 6.0 /config/globax
> /config/globax.conf
> 1549 1548 root S 43264 0.5 2 1.5 /config/globax /config/globax-
> dld.conf
> 1531 2 root DW 0 0.0 0 0.3 [jbd2/sdd1-8]
> 1418 1 root S 2500 0.0 3 0.0 /sbin/syslogd -R 80.83.17.2
> 1524 2 root SW 0 0.0 0 0.0 [flush-8:32]
> 1525 2 root SW 0 0.0 1 0.0 [jbd2/sdc1-8]
> 1604 2 root DW 0 0.0 0 0.0 [flush-8:48]
> 1537 2 root SW 0 0.0 1 0.0 [jbd2/sde1-8]
> 18 2 root SW 0 0.0 3 0.0 [events/3]
> 1545 1 root S 3576 0.0 1 0.0 /config/globax
> /config/globax.conf
> 1548 1 root S 3576 0.0 0 0.0 /config/globax /config/globax-
> dld.conf
> 1918 1 ntp S 3316 0.0 3 0.0 /usr/sbin/ntpd -s
> 1919 1 root S 3268 0.0 3 0.0 /usr/sbin/ntpd -s
> 1 0 root S 2504 0.0 0 0.0 /bin/sh /init trynew trynew
> trynew trynew
> 1923 1257 root S 2504 0.0 1 0.0 /sbin/getty 38400 tty1
> 1924 1257 root S 2504 0.0 0 0.0 /sbin/getty 38400 tty2
> 1927 1257 root S 2504 0.0 0 0.0 /sbin/getty 38400 tty3
> 2015 2014 root S 2504 0.0 1 0.0 -ash
> 2032 2015 root R 2504 0.0 3 0.0 top
> 1584 1 root S 2500 0.0 1 0.0 /usr/bin/ifplugd -i eth0 -a -r
> /etc/startup/rc.ifup -t 1 -u 1 -d 1
> 1592 1 root S 2500 0.0 1 0.0 /usr/bin/ifplugd -i eth2 -a -r
> /etc/startup/rc.ifup -t 1 -u 1 -d 1
> 1587 1 root S 2500 0.0 1 0.0 /usr/bin/ifplugd -i eth1 -a -r
> /etc/startup/rc.ifup -t 1 -u 1 -d 1
> 1595 1 root S 2500 0.0 1 0.0 /usr/bin/ifplugd -i eth3 -a -r
> /etc/startup/rc.ifup -t 1 -u 1 -d 1
> 1257 1 root S 2500 0.0 0 0.0 init
> 1420 1 root S 2500 0.0 1 0.0 /sbin/klogd
> 1432 1 root S 2500 0.0 3 0.0 /usr/sbin/telnetd -f
> /etc/issue.telnet
> 1552 1 root S 2500 0.0 1 0.0 /bin/sh /bin/squidloop
> 1743 1742 root S 2500 0.0 3 0.0 ash -c gs newkernel
> 1744 1743 root S 2500 0.0 0 0.0 /bin/sh /bin/gs newkernel
> 1753 1744 root D 2368 0.0 0 0.0 sync
>
>
> SUPERPROXY ~ # cat /proc/1753/stack
> [<c019a93c>] bdi_sched_wait+0x8/0xc
> [<c019a807>] wait_on_bit+0x20/0x2c
> [<c019a9af>] sync_inodes_sb+0x6f/0x10a
> [<c019dd53>] __sync_filesystem+0x28/0x49
> [<c019ddf3>] sync_filesystems+0x7f/0xc0
> [<c019de7a>] sys_sync+0x1b/0x2d
> [<c02f7a25>] syscall_call+0x7/0xb
> [<ffffffff>] 0xffffffff
Hmm, I guess you are observing the problem reported in
https://bugzilla.kernel.org/show_bug.cgi?id=14830
There seem to be several issues in the per-bdi writeback code that
cause sync on a busy filesystem to last almost forever. To that bug are
attached two patches that fix two issues but apparently it's not all.
I'm still looking into it...
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
next prev parent reply other threads:[~2010-04-08 9:28 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-03-31 16:07 endless sync on bdi_sched_wait()? 2.6.33.1 Denys Fedorysychenko
2010-03-31 22:12 ` Dave Chinner
2010-04-01 10:42 ` Denys Fedorysychenko
2010-04-01 11:13 ` Dave Chinner
2010-04-01 20:14 ` Jeff Moyer
2010-04-08 9:28 ` Jan Kara [this message]
2010-04-08 10:12 ` Denys Fedorysychenko
2010-04-12 0:47 ` Dave Chinner
2010-04-19 1:37 ` Dave Chinner
2010-04-19 7:04 ` Dave Chinner
2010-04-19 7:23 ` Dave Chinner
2010-04-21 0:33 ` Jan Kara
2010-04-21 1:54 ` Dave Chinner
2010-04-21 13:27 ` Jan Kara
2010-04-22 0:06 ` Dave Chinner
2010-04-22 12:48 ` Jan Kara
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100408092850.GA20488@quack.suse.cz \
--to=jack@suse.cz \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=nuclearcat@nuclearcat.com \
--cc=viro@zeniv.linux.org.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.