From: Simon Kirby <sim@hostway.ca>
To: linux-nfs@vger.kernel.org
Subject: System CPU increasing on idle 2.6.36
Date: Wed, 8 Dec 2010 13:25:05 -0800 [thread overview]
Message-ID: <20101208212505.GA18192@hostway.ca> (raw)
Possibly related to the flush-processes-taking-CPU issues I saw
previously, I thought this was interesting. I found a log-crunching box
that does all of its work via NFS and spends most of the day sleeping.
It has been using a linearly-increasing amount of system time during the
time where is sleeping. munin graph:
http://0x.ca/sim/ref/2.6.36/cpu_logcrunch_nfs.png
top shows:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4767 root 20 0 0 0 0 S 16 0.0 2413:29 flush-0:69
22385 root 20 0 0 0 0 S 10 0.0 1320:23 flush-0:70
(rest idle)
perf top shows:
-------------------------------------------------------------------------------------------
PerfTop: 252 irqs/sec kernel:97.2% exact: 0.0% [1000Hz cycles], (all, 2 CPUs)
-------------------------------------------------------------------------------------------
samples pcnt function DSO
_______ _____ _______________________________ ___________
912.00 21.2% _raw_spin_lock [kernel]
592.00 13.8% nfs_writepages [kernel]
474.00 11.0% queue_io [kernel]
428.00 10.0% writeback_single_inode [kernel]
379.00 8.8% writeback_sb_inodes [kernel]
237.00 5.5% bit_waitqueue [kernel]
184.00 4.3% do_writepages [kernel]
169.00 3.9% __iget [kernel]
132.00 3.1% redirty_tail [kernel]
85.00 2.0% iput [kernel]
72.00 1.7% _atomic_dec_and_lock [kernel]
67.00 1.6% write_cache_pages [kernel]
63.00 1.5% port_inb [ipmi_si]
62.00 1.4% __mark_inode_dirty [kernel]
48.00 1.1% __wake_up_bit [kernel]
32.00 0.7% nfs_write_inode [kernel]
26.00 0.6% native_read_tsc [kernel]
25.00 0.6% radix_tree_gang_lookup_tag_slot [kernel]
25.00 0.6% schedule [kernel]
24.00 0.6% wake_up_bit [kernel]
20.00 0.5% apic_timer_interrupt [kernel]
19.00 0.4% ia32_syscall [kernel]
16.00 0.4% find_get_pages_tag [kernel]
15.00 0.3% _raw_spin_lock_irqsave [kernel]
15.00 0.3% load_balance [kernel]
13.00 0.3% ktime_get [kernel]
12.00 0.3% shrink_icache_memory [kernel]
11.00 0.3% semop libc-2.7.so
9.00 0.2% nfs_pageio_doio [kernel]
8.00 0.2% dso__find_symbol perf
8.00 0.2% down_read [kernel]
7.00 0.2% __switch_to [kernel]
7.00 0.2% __phys_addr [kernel]
7.00 0.2% nfs_pageio_init [kernel]
7.00 0.2% pagevec_lookup_tag [kernel]
7.00 0.2% _cond_resched [kernel]
7.00 0.2% up_read [kernel]
6.00 0.1% sched_clock_local [kernel]
5.00 0.1% rb_erase [kernel]
5.00 0.1% page_fault [kernel]
"tcpdump -i any -n port 2049" and "nfsstat" shows this work does not actually result
in any NFS packets.
Known 2.6.36 issue? This did not occur on 2.6.35.4, according to the
munin graphs. I'll try 2.6.37-rc an see if it changes.
Simon-
next reply other threads:[~2010-12-08 21:25 UTC|newest]
Thread overview: 40+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-12-08 21:25 Simon Kirby [this message]
2010-12-08 21:53 ` System CPU increasing on idle 2.6.36 Trond Myklebust
2010-12-08 22:36 ` Simon Kirby
2010-12-09 4:37 ` Trond Myklebust
2010-12-14 23:38 ` Simon Kirby
2010-12-15 1:10 ` Simon Kirby
2010-12-15 1:56 ` Simon Kirby
2010-12-15 18:08 ` J. Bruce Fields
2010-12-15 18:22 ` Trond Myklebust
2010-12-15 18:38 ` J. Bruce Fields
2010-12-15 19:33 ` Trond Myklebust
2010-12-15 19:49 ` J. Bruce Fields
2010-12-15 19:57 ` Trond Myklebust
2010-12-15 20:19 ` J. Bruce Fields
2010-12-15 20:32 ` Trond Myklebust
2010-12-15 21:48 ` J. Bruce Fields
2010-12-15 22:15 ` Trond Myklebust
2010-12-15 22:29 ` J. Bruce Fields
2010-12-15 22:55 ` J. Bruce Fields
2010-12-15 23:58 ` Trond Myklebust
2010-12-16 0:36 ` J. Bruce Fields
2011-09-27 0:39 ` NFS client growing system CPU Simon Kirby
2011-09-27 11:42 ` Trond Myklebust
2011-09-27 16:49 ` Simon Kirby
2011-09-27 17:04 ` Trond Myklebust
2011-09-28 19:58 ` Simon Kirby
2011-09-30 0:58 ` Simon Kirby
2011-09-30 1:11 ` Myklebust, Trond
2011-10-05 23:07 ` Simon Kirby
2010-12-18 1:08 ` System CPU increasing on idle 2.6.36 Simon Kirby
2010-12-21 20:31 ` Mark Moseley
2010-12-29 22:03 ` Simon Kirby
2011-01-04 17:42 ` Mark Moseley
2011-01-04 21:40 ` Simon Kirby
2011-01-05 19:43 ` Mark Moseley
2011-01-07 18:05 ` Mark Moseley
2011-01-07 18:12 ` Mark Moseley
2011-01-07 19:33 ` Mark Moseley
2011-01-08 0:52 ` Simon Kirby
2011-01-08 1:30 ` Mark Moseley
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20101208212505.GA18192@hostway.ca \
--to=sim@hostway.ca \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).