* NFS regression? Odd delays and lockups accessing an NFS export.
@ 2008-08-18 2:02 Grant Coady
2008-08-18 18:50 ` Athanasius
2008-08-18 19:20 ` Trond Myklebust
0 siblings, 2 replies; 64+ messages in thread
From: Grant Coady @ 2008-08-18 2:02 UTC (permalink / raw)
To: linux-kernel; +Cc: neilb, bfields, linux-nfs
Hi there,
I've been using NFS here for years, lately there's something odd going on
since about a month or so. Previously reported last month:
http://www.gossamer-threads.com/lists/linux/kernel/951419?page=last
Now with 2.6.27-rc3 on one of the client boxes I get a complete stall
at odd times when accessing the server's exported directory, cannot
see a pattern to it. Eventually recovers after a Ctrl-C. Nothing in
the server or client log files. Not easy to reproduce either.
The server runs 2.6.26.2 at the moment.
Server config, etc: http://bugsplatter.id.au/kernel/boxen/deltree/
Client config, etc: http://bugsplatter.id.au/kernel/boxen/pooh/
Grant.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-18 2:02 NFS regression? Odd delays and lockups accessing an NFS export Grant Coady
@ 2008-08-18 18:50 ` Athanasius
2008-08-18 19:19 ` Trond Myklebust
2008-08-18 19:37 ` J. K. Cliburn
2008-08-18 19:20 ` Trond Myklebust
1 sibling, 2 replies; 64+ messages in thread
From: Athanasius @ 2008-08-18 18:50 UTC (permalink / raw)
To: Grant Coady, linux-kernel; +Cc: neilb, bfields, linux-nfs
[-- Attachment #1.1: Type: text/plain, Size: 14015 bytes --]
On Mon, Aug 18, 2008 at 12:02:20PM +1000, Grant Coady wrote:
> I've been using NFS here for years, lately there's something odd going on
> since about a month or so. Previously reported last month:
> http://www.gossamer-threads.com/lists/linux/kernel/951419?page=last
>
> Now with 2.6.27-rc3 on one of the client boxes I get a complete stall
> at odd times when accessing the server's exported directory, cannot
> see a pattern to it. Eventually recovers after a Ctrl-C. Nothing in
> the server or client log files. Not easy to reproduce either.
I wonder if this is what I've been seeing. I've been otherwise too
busy to properly report it, thinking that *someone* else must also be
seeing it and it's being worked on, else it's a subtle configuration
problem my end.
I first started seeing this with 2.6.26 on the client end, 2.6.25.10 on
the server end. Things last worked fine with that same server version
and 2.6.25.10 on the client end. The affected export is:
/home/users 192.168.1.161(rw,no_root_squash,sync,no_subtree_check)
using the in-kernel NFS server, not the userspace one.
With the mount on the client being:
192.168.1.162:/home/users /home/users nfs defaults,rw,nfsvers=3,rsize=8192,wsize=8192,nosuid,nodev,soft,intr 0 0
It used to have a 'nolock' in it, but I took that out and saw no
difference. The one possible solid clue as to what's happening is this,
after I turned on all the lock etc checking in a 2.6.26.2 kernel:
Aug 12 16:10:28 emelia kernel: [ 361.851316] INFO: task firefox-bin:5716 blocked for more than 120 seconds.
Aug 12 16:10:28 emelia kernel: [ 361.851326] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 12 16:10:28 emelia kernel: [ 361.851332] firefox-bin D f7e0a018 5556 5716 5696
Aug 12 16:10:28 emelia kernel: [ 361.851348] e24f3ddc 00000046 c2cc2e40 f7e0a018 c0640030 c0642e40 c0642e40 c0642e40
Aug 12 16:10:28 emelia kernel: [ 361.851382] e2ec1018 e2ec1270 c2cc2e40 00000001 c2cc2e40 e24f3db8 00000046 c2c0b680
Aug 12 16:10:28 emelia kernel: [ 361.851406] e2e40da0 c2c0b680 e2ec1270 fffef1e4 00000001 00000046 c2c0b670 c2c0b670
Aug 12 16:10:28 emelia kernel: [ 361.851435] Call Trace:
Aug 12 16:10:28 emelia kernel: [ 361.851449] [<c01e9d71>] nfs_wait_bit_killable+0x2a/0x2e
Aug 12 16:10:28 emelia kernel: [ 361.851462] [<c041ee67>] __wait_on_bit+0x36/0x5d
Aug 12 16:10:28 emelia kernel: [ 361.851475] [<c01e9d47>] ? nfs_wait_bit_killable+0x0/0x2e
Aug 12 16:10:28 emelia kernel: [ 361.851586] [<c041ef3a>] out_of_line_wait_on_bit+0xac/0xb4
Aug 12 16:10:28 emelia kernel: [ 361.851599] [<c01e9d47>] ? nfs_wait_bit_killable+0x0/0x2e
Aug 12 16:10:28 emelia kernel: [ 361.851613] [<c0135286>] ? wake_bit_function+0x0/0x43
Aug 12 16:10:28 emelia kernel: [ 361.851632] [<c01e9d40>] nfs_wait_on_request+0x1f/0x26
Aug 12 16:10:28 emelia kernel: [ 361.851642] [<c01ed9cd>] nfs_sync_mapping_wait+0xde/0x27b
Aug 12 16:10:28 emelia kernel: [ 361.851653] [<c01e9c3d>] ? nfs_pageio_complete+0x8/0xa
Aug 12 16:10:28 emelia kernel: [ 361.851673] [<c01ee372>] __nfs_write_mapping+0x27/0x45
Aug 12 16:10:28 emelia kernel: [ 361.851688] [<c01ee3c9>] nfs_write_mapping+0x39/0x57
Aug 12 16:10:28 emelia kernel: [ 361.851701] [<c01ee40c>] nfs_wb_all+0x10/0x12
Aug 12 16:10:28 emelia kernel: [ 361.851713] [<c01e3169>] nfs_file_flush+0x8b/0xc8
Aug 12 16:10:28 emelia kernel: [ 361.851723] [<c0178505>] filp_close+0x31/0x5a
Aug 12 16:10:28 emelia kernel: [ 361.851735] [<c0126583>] put_files_struct+0x68/0xaa
Aug 12 16:10:28 emelia kernel: [ 361.851747] [<c01265fc>] exit_files+0x37/0x3c
Aug 12 16:10:28 emelia kernel: [ 361.851759] [<c0127988>] do_exit+0x21b/0x632
Aug 12 16:10:28 emelia kernel: [ 361.851771] [<c0127dfd>] do_group_exit+0x5e/0x85
Aug 12 16:10:28 emelia kernel: [ 361.851786] [<c0127e37>] sys_exit_group+0x13/0x15
Aug 12 16:10:28 emelia kernel: [ 361.851799] [<c0102e1d>] sysenter_past_esp+0x6a/0xb1
Aug 12 16:10:28 emelia kernel: [ 361.851816] =======================
Aug 12 16:10:28 emelia kernel: [ 361.851821] INFO: lockdep is turned off.
Aug 12 16:10:30 emelia kernel: [ 363.999492] INFO: task famd:2760 blocked for more than 120 seconds.
Aug 12 16:10:30 emelia kernel: [ 363.999501] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 12 16:10:30 emelia kernel: [ 363.999507] famd D c093dd28 5684 2760 1
Aug 12 16:10:30 emelia kernel: [ 363.999521] f570dbe8 00000046 00000000 c093dd28 c0640030 c0642e40 c0642e40 c0642e40
Aug 12 16:10:30 emelia kernel: [ 363.999697] f6dcc018 f6dcc270 c2cc2e40 00000001 c2cc2e40 f570dbc4 00000046 c2c027b0
Aug 12 16:10:30 emelia kernel: [ 363.999767] f6516da0 c2c027b0 f6dcc270 fffefe66 00000001 00000046 c2c027a0 c2c027a0
Aug 12 16:10:30 emelia kernel: [ 363.999783] Call Trace:
Aug 12 16:10:30 emelia kernel: [ 363.999796] [<c0400fbd>] rpc_wait_bit_killable+0x2a/0x2e
Aug 12 16:10:30 emelia kernel: [ 363.999808] [<c041ee67>] __wait_on_bit+0x36/0x5d
Aug 12 16:10:30 emelia kernel: [ 363.999817] [<c0400f93>] ? rpc_wait_bit_killable+0x0/0x2e
Aug 12 16:10:30 emelia kernel: [ 363.999828] [<c041ef3a>] out_of_line_wait_on_bit+0xac/0xb4
Aug 12 16:10:30 emelia kernel: [ 363.999836] [<c0400f93>] ? rpc_wait_bit_killable+0x0/0x2e
Aug 12 16:10:30 emelia kernel: [ 363.999846] [<c0135286>] ? wake_bit_function+0x0/0x43
Aug 12 16:10:30 emelia kernel: [ 363.999857] [<c040157a>] __rpc_execute+0xcb/0x1e1
Aug 12 16:10:30 emelia kernel: [ 363.999865] [<c04016ab>] rpc_execute+0x1b/0x1e
Aug 12 16:10:30 emelia kernel: [ 363.999872] [<c03fc3d3>] rpc_run_task+0x43/0x49
Aug 12 16:10:30 emelia kernel: [ 363.999881] [<c03fc4de>] rpc_call_sync+0x44/0x5f
Aug 12 16:10:30 emelia kernel: [ 363.999891] [<c01eeeb8>] nfs3_rpc_wrapper+0x17/0x4d
Aug 12 16:10:30 emelia kernel: [ 363.999900] [<c01ef309>] nfs3_proc_access+0xc9/0x121
Aug 12 16:10:30 emelia kernel: [ 363.999919] [<c01e261a>] nfs_do_access+0x129/0x27c
Aug 12 16:10:30 emelia kernel: [ 363.999931] [<c01e2848>] nfs_permission+0xdb/0x13f
Aug 12 16:10:30 emelia kernel: [ 363.999938] [<c01e276d>] ? nfs_permission+0x0/0x13f
Aug 12 16:10:30 emelia kernel: [ 363.999947] [<c0180451>] permission+0x91/0xd0
Aug 12 16:10:30 emelia kernel: [ 363.999956] [<c01804b2>] vfs_permission+0x10/0x12
Aug 12 16:10:30 emelia kernel: [ 363.999963] [<c0181d56>] __link_path_walk+0x106/0xc1d
Aug 12 16:10:30 emelia kernel: [ 363.999972] [<c011764e>] ? kernel_map_pages+0xff/0x116
Aug 12 16:10:30 emelia kernel: [ 363.999986] [<c01828b9>] path_walk+0x4c/0x9b
Aug 12 16:10:30 emelia kernel: [ 363.999994] [<c0182c27>] do_path_lookup+0x198/0x1e1
Aug 12 16:10:30 emelia kernel: [ 364.000003] [<c018355d>] __user_walk_fd+0x2f/0x43
Aug 12 16:10:30 emelia kernel: [ 364.000012] [<c017ce43>] vfs_lstat_fd+0x16/0x3d
Aug 12 16:10:30 emelia kernel: [ 364.000022] [<c0130542>] ? groups_free+0x34/0x38
Aug 12 16:10:30 emelia kernel: [ 364.000034] [<c017cea8>] vfs_lstat+0x11/0x13
Aug 12 16:10:30 emelia kernel: [ 364.000040] [<c017cebe>] sys_lstat64+0x14/0x28
Aug 12 16:10:30 emelia kernel: [ 364.000047] [<c0130542>] ? groups_free+0x34/0x38
Aug 12 16:10:30 emelia kernel: [ 364.000056] [<c0130685>] ? set_current_groups+0x13f/0x149
Aug 12 16:10:30 emelia kernel: [ 364.000067] [<c01311d6>] ? sys_setresuid+0x153/0x165
Aug 12 16:10:30 emelia kernel: [ 364.000077] [<c0102e58>] ? sysenter_past_esp+0xa5/0xb1
Aug 12 16:10:30 emelia kernel: [ 364.000086] [<c0102e1d>] sysenter_past_esp+0x6a/0xb1
Aug 12 16:10:30 emelia kernel: [ 364.000097] =======================
Aug 12 16:10:30 emelia kernel: [ 364.000100] INFO: lockdep is turned off.
Aug 12 16:10:33 emelia kernel: [ 366.146658] INFO: task bash:5640 blocked for more than 120 seconds.
Aug 12 16:10:33 emelia kernel: [ 366.146667] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 12 16:10:33 emelia kernel: [ 366.146672] bash D 00000000 6472 5640 5627
Aug 12 16:10:33 emelia kernel: [ 366.146682] e2d4bad4 00000046 00000000 00000000 c0640030 c0642e40 c0642e40 c0642e40
Aug 12 16:10:33 emelia kernel: [ 366.146699] e2cee018 e2cee270 c2c70e40 00000000 c2c70e40 28d8cec5 00000037 c2c02b70
Aug 12 16:10:33 emelia kernel: [ 366.146713] e2e43da0 c2c02b70 e2cee270 00000046 00000001 00000046 c2c02b60 c2c02b60
Aug 12 16:10:33 emelia kernel: [ 366.146728] Call Trace:
Aug 12 16:10:33 emelia kernel: [ 366.146739] [<c04208e1>] ? _spin_unlock_irqrestore+0x42/0x58
Aug 12 16:10:33 emelia kernel: [ 366.146992] [<c0400fbd>] rpc_wait_bit_killable+0x2a/0x2e
Aug 12 16:10:33 emelia kernel: [ 366.147001] [<c041ee67>] __wait_on_bit+0x36/0x5d
Aug 12 16:10:33 emelia kernel: [ 366.147010] [<c0400f93>] ? rpc_wait_bit_killable+0x0/0x2e
Aug 12 16:10:33 emelia kernel: [ 366.147020] [<c041ef3a>] out_of_line_wait_on_bit+0xac/0xb4
Aug 12 16:10:33 emelia kernel: [ 366.147028] [<c0400f93>] ? rpc_wait_bit_killable+0x0/0x2e
Aug 12 16:10:33 emelia kernel: [ 366.147038] [<c0135286>] ? wake_bit_function+0x0/0x43
Aug 12 16:10:33 emelia kernel: [ 366.147050] [<c040157a>] __rpc_execute+0xcb/0x1e1
Aug 12 16:10:33 emelia kernel: [ 366.147058] [<c04016ab>] rpc_execute+0x1b/0x1e
Aug 12 16:10:33 emelia kernel: [ 366.147065] [<c03fc3d3>] rpc_run_task+0x43/0x49
Aug 12 16:10:33 emelia kernel: [ 366.147074] [<c03fc4de>] rpc_call_sync+0x44/0x5f
Aug 12 16:10:33 emelia kernel: [ 366.147083] [<c01eeeb8>] nfs3_rpc_wrapper+0x17/0x4d
Aug 12 16:10:33 emelia kernel: [ 366.147093] [<c01ef4d4>] nfs3_proc_getattr+0x58/0x7a
Aug 12 16:10:33 emelia kernel: [ 366.147102] [<c01e4b31>] __nfs_revalidate_inode+0x122/0x26d
Aug 12 16:10:33 emelia kernel: [ 366.147116] [<c01e482b>] ? nfs_attribute_timeout+0x0/0xae
Aug 12 16:10:33 emelia kernel: [ 366.147126] [<c01e488c>] ? nfs_attribute_timeout+0x61/0xae
Aug 12 16:10:33 emelia kernel: [ 366.147137] [<c01e048b>] nfs_lookup_revalidate+0x245/0x471
Aug 12 16:10:33 emelia kernel: [ 366.147160] [<c0189b15>] ? __d_lookup+0x12a/0x146
Aug 12 16:10:33 emelia kernel: [ 366.147171] [<c01808ce>] do_lookup+0x115/0x154
Aug 12 16:10:33 emelia kernel: [ 366.147180] [<c0182466>] __link_path_walk+0x816/0xc1d
Aug 12 16:10:33 emelia kernel: [ 366.147190] [<c011764e>] ? kernel_map_pages+0xff/0x116
Aug 12 16:10:33 emelia kernel: [ 366.147203] [<c01828b9>] path_walk+0x4c/0x9b
Aug 12 16:10:33 emelia kernel: [ 366.147212] [<c0182c27>] do_path_lookup+0x198/0x1e1
Aug 12 16:10:33 emelia kernel: [ 366.147220] [<c0182ed0>] __path_lookup_intent_open+0x45/0x76
Aug 12 16:10:33 emelia kernel: [ 366.147230] [<c0182f11>] path_lookup_open+0x10/0x12
Aug 12 16:10:33 emelia kernel: [ 366.147238] [<c01838c9>] do_filp_open+0x9e/0x6cc
Aug 12 16:10:33 emelia kernel: [ 366.147252] [<c0178695>] ? get_unused_fd_flags+0xc3/0xcd
Aug 12 16:10:33 emelia kernel: [ 366.147263] [<c01786df>] do_sys_open+0x40/0xb6
Aug 12 16:10:33 emelia kernel: [ 366.147271] [<c0178797>] sys_open+0x1e/0x26
Aug 12 16:10:33 emelia kernel: [ 366.147279] [<c0102e1d>] sysenter_past_esp+0x6a/0xb1
Aug 12 16:10:33 emelia kernel: [ 366.147290] =======================
Aug 12 16:10:33 emelia kernel: [ 366.147294] INFO: lockdep is turned off.
Aug 12 16:10:41 emelia kernel: [ 374.736634] INFO: task gconfd-2:5492 blocked for more than 120 seconds.
Aug 12 16:10:41 emelia kernel: [ 374.736644] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 12 16:10:41 emelia kernel: [ 374.736648] gconfd-2 D e31a5dec 5132 5492 1
Aug 12 16:10:41 emelia kernel: [ 374.736658] e31a5e48 00000046 f62d3b28 e31a5dec c0640030 c0642e40 c0642e40 c0642e40
Aug 12 16:10:41 emelia kernel: [ 374.736674] e44e9018 e44e9270 c2c70e40 00000000 c2c70e40 e31a5e24 00000046 c2c05280
Aug 12 16:10:41 emelia kernel: [ 374.736689] f40a9da0 c2c05280 e44e9270 ffff24ad 00000001 00000046 c2c05270 c2c05270
Aug 12 16:10:41 emelia kernel: [ 374.736704] Call Trace:
Aug 12 16:10:41 emelia kernel: [ 374.736716] [<c01e9d71>] nfs_wait_bit_killable+0x2a/0x2e
Aug 12 16:10:41 emelia kernel: [ 374.736728] [<c041ee67>] __wait_on_bit+0x36/0x5d
Aug 12 16:10:41 emelia kernel: [ 374.736737] [<c01e9d47>] ? nfs_wait_bit_killable+0x0/0x2e
Aug 12 16:10:41 emelia kernel: [ 374.736747] [<c041ef3a>] out_of_line_wait_on_bit+0xac/0xb4
Aug 12 16:10:41 emelia kernel: [ 374.736755] [<c01e9d47>] ? nfs_wait_bit_killable+0x0/0x2e
Aug 12 16:10:41 emelia kernel: [ 374.736765] [<c0135286>] ? wake_bit_function+0x0/0x43
Aug 12 16:10:41 emelia kernel: [ 374.736777] [<c01e9d40>] nfs_wait_on_request+0x1f/0x26
Aug 12 16:10:41 emelia kernel: [ 374.736784] [<c01ed9cd>] nfs_sync_mapping_wait+0xde/0x27b
Aug 12 16:10:41 emelia kernel: [ 374.736791] [<c01e9c3d>] ? nfs_pageio_complete+0x8/0xa
Aug 12 16:10:41 emelia kernel: [ 374.736805] [<c01ee372>] __nfs_write_mapping+0x27/0x45
Aug 12 16:10:41 emelia kernel: [ 374.736813] [<c01ee3c9>] nfs_write_mapping+0x39/0x57
Aug 12 16:10:41 emelia kernel: [ 374.736910] [<c01ee40c>] nfs_wb_all+0x10/0x12
Aug 12 16:10:41 emelia kernel: [ 374.736917] [<c01e3169>] nfs_file_flush+0x8b/0xc8
Aug 12 16:10:41 emelia kernel: [ 374.736926] [<c0178505>] filp_close+0x31/0x5a
Aug 12 16:10:41 emelia kernel: [ 374.736934] [<c0178598>] sys_close+0x6a/0xa4
Aug 12 16:10:41 emelia kernel: [ 374.736942] [<c0102e1d>] sysenter_past_esp+0x6a/0xb1
Aug 12 16:10:41 emelia kernel: [ 374.736954] =======================
Aug 12 16:10:41 emelia kernel: [ 374.736958] INFO: lockdep is turned off.
Server and client kernel configs attached.
--
- Athanasius = Athanasius(at)miggy.org / http://www.miggy.org/
Finger athan(at)fysh.org for PGP key
"And it's me who is my enemy. Me who beats me up.
Me who makes the monsters. Me who strips my confidence." Paula Cole - ME
[-- Attachment #1.2: server-config --]
[-- Type: text/plain, Size: 62093 bytes --]
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.26.2
# Thu Aug 7 15:30:25 2008
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
# CONFIG_X86_64 is not set
CONFIG_X86=y
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
# CONFIG_GENERIC_LOCKBREAK is not set
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
# CONFIG_GENERIC_GPIO is not set
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
# CONFIG_ARCH_HAS_ILOG2_U32 is not set
# CONFIG_ARCH_HAS_ILOG2_U64 is not set
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
# CONFIG_GENERIC_TIME_VSYSCALL is not set
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
# CONFIG_HAVE_SETUP_PER_CPU_AREA is not set
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ZONE_DMA32 is not set
CONFIG_ARCH_POPULATES_NODE_MAP=y
# CONFIG_AUDIT_ARCH is not set
CONFIG_ARCH_SUPPORTS_AOUT=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_KTIME_SCALAR=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION="amd64"
CONFIG_LOCALVERSION_AUTO=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_TREE=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
# CONFIG_CGROUPS is not set
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_GROUP_SCHED is not set
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
# CONFIG_COMPAT_BRK is not set
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
# CONFIG_MARKERS is not set
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
# CONFIG_HAVE_DMA_ATTRS is not set
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_KMOD=y
CONFIG_BLOCK=y
# CONFIG_LBD is not set
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_LSF=y
CONFIG_BLK_DEV_BSG=y
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
CONFIG_DEFAULT_DEADLINE=y
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="deadline"
CONFIG_CLASSIC_RCU=y
#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
# CONFIG_SMP is not set
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_X86_RDC321X is not set
# CONFIG_X86_VSMP is not set
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
# CONFIG_PARAVIRT_GUEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
CONFIG_MK8=y
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_GENERIC_CPU is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CPU=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_MINIMUM_CPU_FAMILY=4
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
# CONFIG_IOMMU_HELPER is not set
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
# CONFIG_X86_MCE_P4THERMAL is not set
CONFIG_VM86=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
CONFIG_X86_REBOOTFIXUPS=y
# CONFIG_MICROCODE is not set
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
# CONFIG_SPARSEMEM_VMEMMAP_ENABLE is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_RESOURCES_64BIT is not set
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
# CONFIG_HIGHPTE is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_X86_PAT=y
# CONFIG_EFI is not set
CONFIG_SECCOMP=y
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x100000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x100000
# CONFIG_COMPAT_VDSO is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
#
# Power management options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
CONFIG_ACPI=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
# CONFIG_ACPI_BUTTON is not set
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
# CONFIG_ACPI_BAY is not set
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_WMI=y
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_TOSHIBA is not set
CONFIG_ACPI_CUSTOM_DSDT_FILE=""
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
# CONFIG_ACPI_CONTAINER is not set
# CONFIG_ACPI_SBS is not set
#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=y
# CONFIG_X86_POWERNOW_K6 is not set
# CONFIG_X86_POWERNOW_K7 is not set
CONFIG_X86_POWERNOW_K8=y
CONFIG_X86_POWERNOW_K8_ACPI=y
# CONFIG_X86_GX_SUSPMOD is not set
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
# CONFIG_X86_SPEEDSTEP_ICH is not set
# CONFIG_X86_SPEEDSTEP_SMI is not set
# CONFIG_X86_P4_CLOCKMOD is not set
# CONFIG_X86_CPUFREQ_NFORCE2 is not set
# CONFIG_X86_LONGRUN is not set
# CONFIG_X86_LONGHAUL is not set
# CONFIG_X86_E_POWERSAVER is not set
#
# shared options
#
CONFIG_X86_ACPI_CPUFREQ_PROC_INTF=y
# CONFIG_X86_SPEEDSTEP_LIB is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
# CONFIG_PCI_GOOLPC is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_ARCH_SUPPORTS_MSI=y
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY=y
# CONFIG_PCI_DEBUG is not set
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set
# CONFIG_OLPC is not set
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m
#
# Networking
#
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=y
CONFIG_XFRM_SUB_POLICY=y
CONFIG_XFRM_MIGRATE=y
# CONFIG_XFRM_STATISTICS is not set
CONFIG_NET_KEY=y
CONFIG_NET_KEY_MIGRATE=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=y
CONFIG_NET_IPGRE=y
# CONFIG_NET_IPGRE_BROADCAST is not set
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=y
CONFIG_INET_ESP=y
CONFIG_INET_IPCOMP=y
CONFIG_INET_XFRM_TUNNEL=y
CONFIG_INET_TUNNEL=y
CONFIG_INET_XFRM_MODE_TRANSPORT=y
CONFIG_INET_XFRM_MODE_TUNNEL=y
CONFIG_INET_XFRM_MODE_BEET=y
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=m
CONFIG_TCP_CONG_CUBIC=y
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_TCP_CONG_HSTCP=m
CONFIG_TCP_CONG_HYBLA=m
CONFIG_TCP_CONG_VEGAS=m
CONFIG_TCP_CONG_SCALABLE=m
CONFIG_TCP_CONG_LP=m
CONFIG_TCP_CONG_VENO=m
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
# CONFIG_DEFAULT_BIC is not set
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_HTCP is not set
# CONFIG_DEFAULT_VEGAS is not set
# CONFIG_DEFAULT_WESTWOOD is not set
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_TCP_MD5SIG=y
# CONFIG_IP_VS is not set
CONFIG_IPV6=y
CONFIG_IPV6_PRIVACY=y
# CONFIG_IPV6_ROUTER_PREF is not set
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
CONFIG_INET6_AH=y
CONFIG_INET6_ESP=y
CONFIG_INET6_IPCOMP=y
# CONFIG_IPV6_MIP6 is not set
CONFIG_INET6_XFRM_TUNNEL=y
CONFIG_INET6_TUNNEL=y
CONFIG_INET6_XFRM_MODE_TRANSPORT=y
CONFIG_INET6_XFRM_MODE_TUNNEL=y
CONFIG_INET6_XFRM_MODE_BEET=y
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=y
CONFIG_IPV6_SIT=y
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=y
CONFIG_IPV6_MULTIPLE_TABLES=y
# CONFIG_IPV6_SUBTREES is not set
CONFIG_IPV6_MROUTE=y
CONFIG_IPV6_PIMSM_V2=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_QUEUE=y
CONFIG_NETFILTER_NETLINK_LOG=y
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CT_ACCT=y
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CT_PROTO_DCCP=m
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=y
CONFIG_NF_CT_PROTO_UDPLITE=y
CONFIG_NF_CONNTRACK_AMANDA=y
CONFIG_NF_CONNTRACK_FTP=y
CONFIG_NF_CONNTRACK_H323=y
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=y
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=y
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=y
CONFIG_NETFILTER_XT_TARGET_CONNMARK=y
CONFIG_NETFILTER_XT_TARGET_DSCP=y
CONFIG_NETFILTER_XT_TARGET_MARK=y
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=y
CONFIG_NETFILTER_XT_TARGET_NFLOG=y
CONFIG_NETFILTER_XT_TARGET_NOTRACK=y
CONFIG_NETFILTER_XT_TARGET_RATEEST=y
CONFIG_NETFILTER_XT_TARGET_TRACE=y
CONFIG_NETFILTER_XT_TARGET_SECMARK=y
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=y
CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=y
CONFIG_NETFILTER_XT_MATCH_COMMENT=y
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=y
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=y
CONFIG_NETFILTER_XT_MATCH_CONNMARK=y
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
CONFIG_NETFILTER_XT_MATCH_DCCP=y
CONFIG_NETFILTER_XT_MATCH_DSCP=y
CONFIG_NETFILTER_XT_MATCH_ESP=y
CONFIG_NETFILTER_XT_MATCH_HELPER=y
CONFIG_NETFILTER_XT_MATCH_IPRANGE=y
CONFIG_NETFILTER_XT_MATCH_LENGTH=y
CONFIG_NETFILTER_XT_MATCH_LIMIT=y
CONFIG_NETFILTER_XT_MATCH_MAC=y
CONFIG_NETFILTER_XT_MATCH_MARK=y
CONFIG_NETFILTER_XT_MATCH_OWNER=y
CONFIG_NETFILTER_XT_MATCH_POLICY=y
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=y
CONFIG_NETFILTER_XT_MATCH_QUOTA=y
CONFIG_NETFILTER_XT_MATCH_RATEEST=y
CONFIG_NETFILTER_XT_MATCH_REALM=y
CONFIG_NETFILTER_XT_MATCH_SCTP=y
CONFIG_NETFILTER_XT_MATCH_STATE=y
CONFIG_NETFILTER_XT_MATCH_STATISTIC=y
CONFIG_NETFILTER_XT_MATCH_STRING=y
CONFIG_NETFILTER_XT_MATCH_TCPMSS=y
CONFIG_NETFILTER_XT_MATCH_TIME=y
CONFIG_NETFILTER_XT_MATCH_U32=y
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=y
#
# IP: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV4=y
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_RECENT=y
CONFIG_IP_NF_MATCH_ECN=y
CONFIG_IP_NF_MATCH_AH=y
CONFIG_IP_NF_MATCH_TTL=y
CONFIG_IP_NF_MATCH_ADDRTYPE=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_NF_NAT=y
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_IP_NF_TARGET_REDIRECT=y
CONFIG_IP_NF_TARGET_NETMAP=y
CONFIG_NF_NAT_SNMP_BASIC=y
CONFIG_NF_NAT_PROTO_DCCP=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_UDPLITE=y
CONFIG_NF_NAT_PROTO_SCTP=y
CONFIG_NF_NAT_FTP=y
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=y
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=y
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=y
CONFIG_IP_NF_TARGET_ECN=y
CONFIG_IP_NF_TARGET_TTL=y
CONFIG_IP_NF_TARGET_CLUSTERIP=y
CONFIG_IP_NF_RAW=y
CONFIG_IP_NF_ARPTABLES=y
CONFIG_IP_NF_ARPFILTER=y
CONFIG_IP_NF_ARP_MANGLE=y
#
# IPv6: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV6=y
# CONFIG_IP6_NF_QUEUE is not set
CONFIG_IP6_NF_IPTABLES=y
CONFIG_IP6_NF_MATCH_RT=y
CONFIG_IP6_NF_MATCH_OPTS=y
CONFIG_IP6_NF_MATCH_FRAG=y
CONFIG_IP6_NF_MATCH_HL=y
CONFIG_IP6_NF_MATCH_IPV6HEADER=y
CONFIG_IP6_NF_MATCH_AH=y
CONFIG_IP6_NF_MATCH_MH=y
CONFIG_IP6_NF_MATCH_EUI64=y
CONFIG_IP6_NF_FILTER=y
CONFIG_IP6_NF_TARGET_LOG=y
CONFIG_IP6_NF_TARGET_REJECT=y
CONFIG_IP6_NF_MANGLE=y
CONFIG_IP6_NF_TARGET_HL=y
CONFIG_IP6_NF_RAW=y
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m
CONFIG_IP_DCCP_ACKVEC=y
#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
CONFIG_IP_DCCP_CCID2=m
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=m
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_CCID3_RTO=100
CONFIG_IP_DCCP_TFRC_LIB=m
#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
CONFIG_IP_SCTP=m
CONFIG_SCTP_DBG_MSG=y
CONFIG_SCTP_DBG_OBJCNT=y
# CONFIG_SCTP_HMAC_NONE is not set
CONFIG_SCTP_HMAC_SHA1=y
# CONFIG_SCTP_HMAC_MD5 is not set
CONFIG_TIPC=m
# CONFIG_TIPC_ADVANCED is not set
# CONFIG_TIPC_DEBUG is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
CONFIG_NET_SCHED=y
#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=y
CONFIG_NET_SCH_HTB=y
CONFIG_NET_SCH_HFSC=y
CONFIG_NET_SCH_PRIO=y
CONFIG_NET_SCH_RED=y
CONFIG_NET_SCH_SFQ=y
CONFIG_NET_SCH_TEQL=y
CONFIG_NET_SCH_TBF=y
CONFIG_NET_SCH_GRED=y
CONFIG_NET_SCH_DSMARK=y
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_INGRESS=y
#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=y
CONFIG_NET_CLS_TCINDEX=y
CONFIG_NET_CLS_ROUTE4=y
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=y
CONFIG_NET_CLS_U32=y
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=y
CONFIG_NET_CLS_RSVP6=y
CONFIG_NET_CLS_FLOW=y
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=y
CONFIG_NET_EMATCH_NBYTE=y
CONFIG_NET_EMATCH_U32=y
CONFIG_NET_EMATCH_META=y
CONFIG_NET_EMATCH_TEXT=y
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=y
CONFIG_NET_ACT_GACT=y
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=y
CONFIG_NET_ACT_IPT=y
# CONFIG_NET_ACT_NAT is not set
CONFIG_NET_ACT_PEDIT=y
CONFIG_NET_ACT_SIMP=y
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
CONFIG_BT=m
CONFIG_BT_L2CAP=m
CONFIG_BT_SCO=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_HIDP=m
#
# Bluetooth device drivers
#
CONFIG_BT_HCIUSB=m
CONFIG_BT_HCIUSB_SCO=y
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
CONFIG_BT_HCIUART_LL=y
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBPA10X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIVHCI=m
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y
#
# Wireless
#
# CONFIG_CFG80211 is not set
# CONFIG_WIRELESS_EXT is not set
# CONFIG_MAC80211 is not set
# CONFIG_IEEE80211 is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
# CONFIG_STANDALONE is not set
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=m
# CONFIG_MTD is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
# CONFIG_PARPORT_SERIAL is not set
CONFIG_PARPORT_PC_FIFO=y
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
# CONFIG_PARPORT_1284 is not set
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set
#
# Protocols
#
CONFIG_ISAPNP=y
# CONFIG_PNPBIOS is not set
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_DEV_XD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=y
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
# CONFIG_ATA_OVER_ETH is not set
CONFIG_MISC_DEVICES=y
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_SGI_IOC4 is not set
CONFIG_TIFM_CORE=m
CONFIG_TIFM_7XX1=m
# CONFIG_TC1100_WMI is not set
# CONFIG_SONY_LAPTOP is not set
# CONFIG_THINKPAD_ACPI is not set
# CONFIG_INTEL_MENLOW is not set
# CONFIG_ENCLOSURE_SERVICES is not set
CONFIG_HAVE_IDE=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
#
# Please see Documentation/ide/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
CONFIG_BLK_DEV_IDEDISK=y
# CONFIG_IDEDISK_MULTI_MODE is not set
CONFIG_BLK_DEV_IDECD=m
CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
CONFIG_BLK_DEV_IDESCSI=m
CONFIG_BLK_DEV_IDEACPI=y
# CONFIG_IDE_TASK_IOCTL is not set
CONFIG_IDE_PROC_FS=y
#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_PLATFORM=m
# CONFIG_BLK_DEV_CMD640 is not set
# CONFIG_BLK_DEV_IDEPNP is not set
CONFIG_BLK_DEV_IDEDMA_SFF=y
#
# PCI IDE chipsets support
#
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_PCIBUS_ORDER=y
# CONFIG_BLK_DEV_OFFBOARD is not set
# CONFIG_BLK_DEV_GENERIC is not set
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_CS5535 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_JMICRON is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_IT8213 is not set
# CONFIG_BLK_DEV_IT821X is not set
# CONFIG_BLK_DEV_NS87415 is not set
CONFIG_BLK_DEV_PDC202XX_OLD=m
CONFIG_BLK_DEV_PDC202XX_NEW=m
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
CONFIG_BLK_DEV_VIA82CXXX=y
# CONFIG_BLK_DEV_TC86C001 is not set
#
# Other IDE chipsets support
#
#
# Note: most of these also require special kernel boot parameters
#
# CONFIG_BLK_DEV_4DRIVES is not set
# CONFIG_BLK_DEV_ALI14XX is not set
# CONFIG_BLK_DEV_DTC2278 is not set
# CONFIG_BLK_DEV_HT6560B is not set
# CONFIG_BLK_DEV_QD65XX is not set
# CONFIG_BLK_DEV_UMC8672 is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_BLK_DEV_HD_ONLY is not set
# CONFIG_BLK_DEV_HD is not set
#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=y
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set
#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m
#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
# CONFIG_SCSI_FC_TGT_ATTRS is not set
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_NCR53C406A is not set
CONFIG_SCSI_STEX=m
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_SRP is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y
CONFIG_SATA_AHCI=m
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
# CONFIG_SATA_SVW is not set
# CONFIG_ATA_PIIX is not set
# CONFIG_SATA_MV is not set
# CONFIG_SATA_NV is not set
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
CONFIG_SATA_PROMISE=m
CONFIG_SATA_SX4=m
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
CONFIG_SATA_VIA=y
# CONFIG_SATA_VITESSE is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_PATA_ACPI is not set
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CS5535 is not set
# CONFIG_PATA_CS5536 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
CONFIG_ATA_GENERIC=y
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_ISAPNP is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
# CONFIG_PATA_JMICRON is not set
# CONFIG_PATA_LEGACY is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
CONFIG_PATA_PDC_OLD=m
# CONFIG_PATA_QDI is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
CONFIG_PATA_PDC2027X=m
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
CONFIG_PATA_VIA=y
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_WINBOND_VLB is not set
# CONFIG_PATA_SCH is not set
# CONFIG_MD is not set
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
#
# Enable only one of the two stacks, unless you know what you are doing
#
# CONFIG_FIREWIRE is not set
CONFIG_IEEE1394=m
CONFIG_IEEE1394_OHCI1394=m
# CONFIG_IEEE1394_PCILYNX is not set
CONFIG_IEEE1394_SBP2=m
# CONFIG_IEEE1394_SBP2_PHYS_DMA is not set
CONFIG_IEEE1394_ETH1394_ROM_ENTRY=y
CONFIG_IEEE1394_ETH1394=m
CONFIG_IEEE1394_RAWIO=m
CONFIG_IEEE1394_VIDEO1394=m
CONFIG_IEEE1394_DV1394=m
# CONFIG_IEEE1394_VERBOSEDEBUG is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
# CONFIG_NETDEVICES_MULTIQUEUE is not set
# CONFIG_IFB is not set
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=m
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
CONFIG_PHYLIB=y
#
# MII PHY device drivers
#
# CONFIG_MARVELL_PHY is not set
# CONFIG_DAVICOM_PHY is not set
# CONFIG_QSEMI_PHY is not set
# CONFIG_LXT_PHY is not set
# CONFIG_CICADA_PHY is not set
# CONFIG_VITESSE_PHY is not set
# CONFIG_SMSC_PHY is not set
# CONFIG_BROADCOM_PHY is not set
# CONFIG_ICPLUS_PHY is not set
CONFIG_REALTEK_PHY=m
# CONFIG_FIXED_PHY is not set
# CONFIG_MDIO_BITBANG is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_NET_VENDOR_RACAL is not set
# CONFIG_NET_TULIP is not set
# CONFIG_AT1700 is not set
# CONFIG_DEPCA is not set
# CONFIG_HP100 is not set
# CONFIG_NET_ISA is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_CS89x0 is not set
CONFIG_EEPRO100=m
# CONFIG_E100 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
CONFIG_NE2K_PCI=m
CONFIG_8139CP=m
CONFIG_8139TOO=m
# CONFIG_8139TOO_PIO is not set
CONFIG_8139TOO_TUNE_TWISTER=y
CONFIG_8139TOO_8129=y
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_R6040 is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_SC92031 is not set
# CONFIG_NET_POCKET is not set
CONFIG_NETDEV_1000=y
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_E1000E is not set
# CONFIG_E1000E_ENABLED is not set
# CONFIG_IP1000 is not set
# CONFIG_IGB is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
CONFIG_SKGE=m
# CONFIG_SKGE_DEBUG is not set
CONFIG_SKY2=m
CONFIG_SKY2_DEBUG=y
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set
# CONFIG_QLA3XXX is not set
# CONFIG_ATL1 is not set
CONFIG_NETDEV_10000=y
# CONFIG_CHELSIO_T1 is not set
# CONFIG_CHELSIO_T3 is not set
# CONFIG_IXGBE is not set
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set
# CONFIG_MYRI10GE is not set
# CONFIG_NETXEN_NIC is not set
# CONFIG_NIU is not set
# CONFIG_MLX4_CORE is not set
# CONFIG_TEHUTI is not set
# CONFIG_BNX2X is not set
# CONFIG_SFC is not set
# CONFIG_TR is not set
#
# Wireless LAN
#
CONFIG_WLAN_PRE80211=y
# CONFIG_STRIP is not set
# CONFIG_ARLAN is not set
# CONFIG_WAVELAN is not set
CONFIG_WLAN_80211=y
# CONFIG_IPW2100 is not set
# CONFIG_IPW2200 is not set
# CONFIG_LIBERTAS is not set
# CONFIG_AIRO is not set
# CONFIG_HERMES is not set
# CONFIG_ATMEL is not set
# CONFIG_PRISM54 is not set
# CONFIG_USB_ZD1201 is not set
# CONFIG_USB_NET_RNDIS_WLAN is not set
# CONFIG_IWLWIFI_LEDS is not set
# CONFIG_HOSTAP is not set
#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
CONFIG_PPP=m
# CONFIG_PPP_MULTILINK is not set
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
CONFIG_PPPOL2TP=m
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLHC=m
CONFIG_SLIP_SMART=y
CONFIG_SLIP_MODE_SLIP6=y
# CONFIG_NET_FC is not set
CONFIG_NETCONSOLE=y
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set
#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
CONFIG_INPUT_POLLDEV=m
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1280
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=1024
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=m
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_INPORT is not set
# CONFIG_MOUSE_LOGIBM is not set
# CONFIG_MOUSE_PC110PAD is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_WISTRON_BTNS is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
CONFIG_INPUT_UINPUT=m
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set
#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_DEVKMEM=y
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set
#
# Serial drivers
#
CONFIG_SERIAL_8250=m
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=m
CONFIG_SERIAL_8250_PNP=m
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
# CONFIG_SERIAL_8250_MANY_PORTS is not set
CONFIG_SERIAL_8250_SHARE_IRQ=y
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
# CONFIG_SERIAL_8250_RSA is not set
#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=m
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_PRINTER=m
# CONFIG_LP_CONSOLE is not set
# CONFIG_PPDEV is not set
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=m
CONFIG_HW_RANDOM_INTEL=m
CONFIG_HW_RANDOM_AMD=m
CONFIG_HW_RANDOM_GEODE=m
CONFIG_HW_RANDOM_VIA=m
CONFIG_NVRAM=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_NSC_GPIO is not set
# CONFIG_CS5535_GPIO is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=y
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_ALGOBIT=m
#
# I2C Hardware Bus support
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_ELEKTOR is not set
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_I810 is not set
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PARPORT is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_PROSAVAGE is not set
# CONFIG_I2C_SAVAGE4 is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_SCx200_ACB is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_TINY_USB is not set
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
# CONFIG_I2C_VOODOO3 is not set
# CONFIG_I2C_PCA_ISA is not set
# CONFIG_I2C_PCA_PLATFORM is not set
#
# Miscellaneous I2C Chip support
#
# CONFIG_DS1682 is not set
CONFIG_SENSORS_EEPROM=m
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_PCF8575 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_MAX6875 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set
# CONFIG_SPI is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
CONFIG_HWMON=m
CONFIG_HWMON_VID=m
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7418 is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7473 is not set
CONFIG_SENSORS_K8TEMP=m
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_I5K_AMB is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_FSCPOS is not set
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_CORETEMP is not set
# CONFIG_SENSORS_IBMAEM is not set
# CONFIG_SENSORS_IBMPEX is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_LM63 is not set
CONFIG_SENSORS_LM75=m
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_PC87427 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_DME1737 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83L786NG is not set
CONFIG_SENSORS_W83627HF=m
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_HDAPS is not set
# CONFIG_SENSORS_APPLESMC is not set
# CONFIG_HWMON_DEBUG_CHIP is not set
CONFIG_THERMAL=y
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set
#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=y
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
# CONFIG_ALIM1535_WDT is not set
# CONFIG_ALIM7101_WDT is not set
# CONFIG_SC520_WDT is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
# CONFIG_IBMASR is not set
# CONFIG_WAFER_WDT is not set
# CONFIG_I6300ESB_WDT is not set
# CONFIG_ITCO_WDT is not set
# CONFIG_IT8712F_WDT is not set
# CONFIG_HP_WATCHDOG is not set
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
# CONFIG_60XX_WDT is not set
# CONFIG_SBC8360_WDT is not set
# CONFIG_SBC7240_WDT is not set
# CONFIG_CPU5_WDT is not set
# CONFIG_SMSC37B787_WDT is not set
CONFIG_W83627HF_WDT=m
CONFIG_W83697HF_WDT=m
# CONFIG_W83877F_WDT is not set
# CONFIG_W83977F_WDT is not set
# CONFIG_MACHZ_WDT is not set
# CONFIG_SBC_EPX_C3_WATCHDOG is not set
#
# ISA-based Watchdog Cards
#
# CONFIG_PCWATCHDOG is not set
# CONFIG_MIXCOMWD is not set
# CONFIG_WDT is not set
#
# PCI-based Watchdog Cards
#
# CONFIG_PCIPCWATCHDOG is not set
# CONFIG_WDTPCI is not set
#
# USB-based Watchdog Cards
#
# CONFIG_USBPCWATCHDOG is not set
#
# Sonics Silicon Backplane
#
CONFIG_SSB_POSSIBLE=y
# CONFIG_SSB is not set
#
# Multifunction device drivers
#
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
#
# Multimedia devices
#
#
# Multimedia core support
#
# CONFIG_VIDEO_DEV is not set
# CONFIG_DVB_CORE is not set
# CONFIG_VIDEO_MEDIA is not set
#
# Multimedia drivers
#
# CONFIG_DAB is not set
#
# Graphics support
#
CONFIG_AGP=y
# CONFIG_AGP_ALI is not set
# CONFIG_AGP_ATI is not set
# CONFIG_AGP_AMD is not set
# CONFIG_AGP_AMD64 is not set
# CONFIG_AGP_INTEL is not set
# CONFIG_AGP_NVIDIA is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_SWORKS is not set
CONFIG_AGP_VIA=y
# CONFIG_AGP_EFFICEON is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_DRM_VIA is not set
# CONFIG_DRM_SAVAGE is not set
CONFIG_VGASTATE=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=m
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_DDC=m
CONFIG_FB_CFB_FILLRECT=m
CONFIG_FB_CFB_COPYAREA=m
CONFIG_FB_CFB_IMAGEBLIT=m
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
# CONFIG_FB_SYS_FILLRECT is not set
# CONFIG_FB_SYS_COPYAREA is not set
# CONFIG_FB_SYS_IMAGEBLIT is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_SYS_FOPS is not set
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
CONFIG_FB_NVIDIA=m
CONFIG_FB_NVIDIA_I2C=y
# CONFIG_FB_NVIDIA_DEBUG is not set
# CONFIG_FB_NVIDIA_BACKLIGHT is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I810 is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
CONFIG_FB_3DFX=m
# CONFIG_FB_3DFX_ACCEL is not set
CONFIG_FB_VOODOO1=m
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_CYBLA is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_VIRTUAL is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set
#
# Display device support
#
CONFIG_DISPLAY_SUPPORT=y
#
# Display hardware drivers
#
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_VIDEO_SELECT=y
# CONFIG_MDA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=m
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_LOGO is not set
#
# Sound
#
CONFIG_SOUND=m
#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
# CONFIG_SND_DYNAMIC_MINORS is not set
CONFIG_SND_SUPPORT_OLD_API=y
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
#
# Generic devices
#
CONFIG_SND_PCSP=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_MTS64 is not set
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
# CONFIG_SND_PORTMAN2X4 is not set
#
# ISA devices
#
# CONFIG_SND_ADLIB is not set
# CONFIG_SND_AD1816A is not set
# CONFIG_SND_AD1848 is not set
# CONFIG_SND_ALS100 is not set
# CONFIG_SND_AZT2320 is not set
# CONFIG_SND_CMI8330 is not set
# CONFIG_SND_CS4231 is not set
# CONFIG_SND_CS4232 is not set
# CONFIG_SND_CS4236 is not set
# CONFIG_SND_DT019X is not set
# CONFIG_SND_ES968 is not set
# CONFIG_SND_ES1688 is not set
# CONFIG_SND_ES18XX is not set
# CONFIG_SND_SC6000 is not set
# CONFIG_SND_GUSCLASSIC is not set
# CONFIG_SND_GUSEXTREME is not set
# CONFIG_SND_GUSMAX is not set
# CONFIG_SND_INTERWAVE is not set
# CONFIG_SND_INTERWAVE_STB is not set
# CONFIG_SND_OPL3SA2 is not set
# CONFIG_SND_OPTI92X_AD1848 is not set
# CONFIG_SND_OPTI92X_CS4231 is not set
# CONFIG_SND_OPTI93X is not set
# CONFIG_SND_MIRO is not set
# CONFIG_SND_SB8 is not set
# CONFIG_SND_SB16 is not set
# CONFIG_SND_SBAWE is not set
# CONFIG_SND_SGALAXY is not set
# CONFIG_SND_SSCAPE is not set
# CONFIG_SND_WAVEFRONT is not set
#
# PCI devices
#
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_HDA_INTEL is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_HIFIER is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
# CONFIG_SND_INTEL8X0 is not set
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SIS7019 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
CONFIG_SND_VIA82XX=m
CONFIG_SND_VIA82XX_MODEM=m
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=900
#
# USB devices
#
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set
# CONFIG_SND_USB_CAIAQ is not set
#
# System on Chip audio support
#
# CONFIG_SND_SOC is not set
#
# ALSA SoC audio for Freescale SOCs
#
#
# SoC Audio for the Texas Instruments OMAP
#
#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=m
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HID_DEBUG is not set
# CONFIG_HIDRAW is not set
#
# USB Input Devices
#
CONFIG_USB_HID=m
# CONFIG_USB_HIDINPUT_POWERBOOK is not set
# CONFIG_HID_FF is not set
CONFIG_USB_HIDDEV=y
#
# USB HID Boot Protocol drivers
#
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
CONFIG_USB_DEBUG=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
CONFIG_USB_DEVICE_CLASS=y
CONFIG_USB_DYNAMIC_MINORS=y
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
# CONFIG_USB_WDM is not set
#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#
#
# may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
CONFIG_USB_STORAGE_DEBUG=y
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
CONFIG_USB_STORAGE_USBAT=y
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y
CONFIG_USB_STORAGE_ALAUDA=y
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_LIBUSUAL is not set
#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
CONFIG_USB_MON=y
#
# USB port drivers
#
# CONFIG_USB_USS720 is not set
CONFIG_USB_SERIAL=m
CONFIG_USB_EZUSB=y
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRCABLE=m
# CONFIG_USB_SERIAL_AIRPRIME is not set
# CONFIG_USB_SERIAL_ARK3116 is not set
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_CH341 is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_CP2101 is not set
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
# CONFIG_USB_SERIAL_EMPEG is not set
# CONFIG_USB_SERIAL_FTDI_SIO is not set
# CONFIG_USB_SERIAL_FUNSOFT is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_GARMIN is not set
# CONFIG_USB_SERIAL_IPW is not set
# CONFIG_USB_SERIAL_IUU is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
# CONFIG_USB_SERIAL_MOS7720 is not set
# CONFIG_USB_SERIAL_MOS7840 is not set
# CONFIG_USB_SERIAL_MOTOROLA is not set
# CONFIG_USB_SERIAL_NAVMAN is not set
# CONFIG_USB_SERIAL_PL2303 is not set
# CONFIG_USB_SERIAL_OTI6858 is not set
# CONFIG_USB_SERIAL_SPCP8X5 is not set
# CONFIG_USB_SERIAL_HP4X is not set
# CONFIG_USB_SERIAL_SAFE is not set
# CONFIG_USB_SERIAL_SIERRAWIRELESS is not set
# CONFIG_USB_SERIAL_TI is not set
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
CONFIG_USB_SERIAL_OPTION=m
# CONFIG_USB_SERIAL_OMNINET is not set
# CONFIG_USB_SERIAL_DEBUG is not set
#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_GADGET is not set
# CONFIG_MMC is not set
CONFIG_MEMSTICK=m
# CONFIG_MEMSTICK_DEBUG is not set
#
# MemoryStick drivers
#
# CONFIG_MEMSTICK_UNSAFE_RESUME is not set
CONFIG_MSPRO_BLOCK=m
#
# MemoryStick Host Controller Drivers
#
CONFIG_MEMSTICK_TIFM_MS=m
CONFIG_MEMSTICK_JMICRON_38X=m
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
CONFIG_EDAC=y
#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_MM_EDAC=y
# CONFIG_EDAC_AMD76X is not set
# CONFIG_EDAC_E7XXX is not set
# CONFIG_EDAC_E752X is not set
# CONFIG_EDAC_I82875P is not set
# CONFIG_EDAC_I82975X is not set
# CONFIG_EDAC_I3000 is not set
# CONFIG_EDAC_I82860 is not set
# CONFIG_EDAC_R82600 is not set
# CONFIG_EDAC_I5000 is not set
CONFIG_RTC_LIB=m
CONFIG_RTC_CLASS=m
#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set
#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_DS1307 is not set
# CONFIG_RTC_DRV_DS1374 is not set
# CONFIG_RTC_DRV_DS1672 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
# CONFIG_RTC_DRV_RS5C372 is not set
# CONFIG_RTC_DRV_ISL1208 is not set
# CONFIG_RTC_DRV_X1205 is not set
# CONFIG_RTC_DRV_PCF8563 is not set
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
#
# SPI RTC drivers
#
#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=m
# CONFIG_RTC_DRV_DS1511 is not set
# CONFIG_RTC_DRV_DS1553 is not set
# CONFIG_RTC_DRV_DS1742 is not set
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_V3020 is not set
#
# on-CPU RTC drivers
#
CONFIG_DMADEVICES=y
#
# DMA Devices
#
# CONFIG_INTEL_IOATDMA is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
#
# Firmware Drivers
#
CONFIG_EDD=y
# CONFIG_EDD_OFF is not set
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
# CONFIG_ISCSI_IBFT_FIND is not set
#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
# CONFIG_EXT3_FS_POSIX_ACL is not set
# CONFIG_EXT3_FS_SECURITY is not set
# CONFIG_EXT4DEV_FS is not set
CONFIG_JBD=y
CONFIG_JBD_DEBUG=y
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
# CONFIG_XFS_FS is not set
# CONFIG_OCFS2_FS is not set
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
# CONFIG_PRINT_QUOTA_WARNING is not set
# CONFIG_QFMT_V1 is not set
# CONFIG_QFMT_V2 is not set
CONFIG_QUOTACTL=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m
# CONFIG_FUSE_FS is not set
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_CONFIGFS_FS=y
#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
CONFIG_MINIX_FS=m
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
CONFIG_SUNRPC_BIND34=y
CONFIG_RPCSEC_GSS_KRB5=y
CONFIG_RPCSEC_GSS_SPKM3=m
CONFIG_SMB_FS=m
CONFIG_SMB_NLS_DEFAULT=y
CONFIG_SMB_NLS_REMOTE="cp437"
CONFIG_CIFS=m
CONFIG_CIFS_STATS=y
# CONFIG_CIFS_STATS2 is not set
# CONFIG_CIFS_WEAK_PW_HASH is not set
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_DEBUG2 is not set
CONFIG_CIFS_EXPERIMENTAL=y
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso-8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=y
CONFIG_NLS_CODEPAGE_852=y
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_2=y
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=y
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=y
CONFIG_DLM=m
# CONFIG_DLM_DEBUG is not set
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_ENABLE_WARN_DEPRECATED=y
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_DETECT_SOFTLOCKUP=y
CONFIG_SCHED_DEBUG=y
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
CONFIG_DEBUG_MUTEXES=y
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_WRITECOUNT is not set
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
CONFIG_FRAME_POINTER=y
CONFIG_BOOT_PRINTK_DELAY=y
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_NONPROMISC_DEVMEM=y
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
# CONFIG_DEBUG_RODATA_TEST is not set
# CONFIG_DEBUG_NX_TEST is not set
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_DOUBLEFAULT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
# CONFIG_CPA_DEBUG is not set
#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_AUTHENC=y
CONFIG_CRYPTO_TEST=m
#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=m
#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=m
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=m
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=m
#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
#
# Digest
#
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
#
# Ciphers
#
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_586=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SALSA20_586=m
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_586=m
#
# Compression
#
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_LZO=m
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
CONFIG_CRYPTO_DEV_GEODE=m
CONFIG_CRYPTO_DEV_HIFN_795X=m
CONFIG_CRYPTO_DEV_HIFN_795X_RNG=y
CONFIG_HAVE_KVM=y
# CONFIG_VIRTUALIZATION is not set
#
# Library routines
#
CONFIG_BITREVERSE=m
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=m
CONFIG_CRC7=m
CONFIG_LIBCRC32C=y
CONFIG_AUDIT_GENERIC=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=m
CONFIG_LZO_DECOMPRESS=m
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=y
CONFIG_TEXTSEARCH_BM=y
CONFIG_TEXTSEARCH_FSM=y
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
[-- Attachment #1.3: client-config --]
[-- Type: text/plain, Size: 61444 bytes --]
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.26.2
# Thu Aug 7 14:02:37 2008
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
# CONFIG_X86_64 is not set
CONFIG_X86=y
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
# CONFIG_GENERIC_LOCKBREAK is not set
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
# CONFIG_GENERIC_GPIO is not set
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
# CONFIG_ARCH_HAS_ILOG2_U32 is not set
# CONFIG_ARCH_HAS_ILOG2_U64 is not set
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
# CONFIG_GENERIC_TIME_VSYSCALL is not set
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ZONE_DMA32 is not set
CONFIG_ARCH_POPULATES_NODE_MAP=y
# CONFIG_AUDIT_ARCH is not set
CONFIG_ARCH_SUPPORTS_AOUT=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_X86_32_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_KTIME_SCALAR=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
CONFIG_AUDIT_TREE=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
# CONFIG_CGROUPS is not set
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_GROUP_SCHED is not set
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
# CONFIG_COMPAT_BRK is not set
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
# CONFIG_MARKERS is not set
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
# CONFIG_HAVE_DMA_ATTRS is not set
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
# CONFIG_LBD is not set
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_LSF=y
CONFIG_BLK_DEV_BSG=y
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_CLASSIC_RCU=y
#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_X86_RDC321X is not set
# CONFIG_X86_VSMP is not set
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
# CONFIG_PARAVIRT_GUEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
CONFIG_MCORE2=y
# CONFIG_GENERIC_CPU is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CPU=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_P6_NOP=y
CONFIG_X86_TSC=y
CONFIG_X86_MINIMUM_CPU_FAMILY=6
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
# CONFIG_IOMMU_HELPER is not set
CONFIG_NR_CPUS=8
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
# CONFIG_PREEMPT_RCU is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_X86_MCE_P4THERMAL=y
CONFIG_VM86=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
CONFIG_X86_REBOOTFIXUPS=y
# CONFIG_MICROCODE is not set
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
# CONFIG_SPARSEMEM_VMEMMAP_ENABLE is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_RESOURCES_64BIT is not set
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_HIGHPTE=y
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
CONFIG_X86_PAT=y
# CONFIG_EFI is not set
CONFIG_IRQBALANCE=y
CONFIG_SECCOMP=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_SCHED_HRTICK=y
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x100000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x100000
CONFIG_HOTPLUG_CPU=y
CONFIG_COMPAT_VDSO=y
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
#
# Power management options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_SLEEP=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
# CONFIG_ACPI_BUTTON is not set
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
# CONFIG_ACPI_BAY is not set
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_THERMAL=y
CONFIG_ACPI_WMI=y
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_TOSHIBA is not set
CONFIG_ACPI_CUSTOM_DSDT_FILE=""
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
# CONFIG_ACPI_SBS is not set
CONFIG_X86_APM_BOOT=y
CONFIG_APM=y
# CONFIG_APM_IGNORE_USER_SUSPEND is not set
CONFIG_APM_DO_ENABLE=y
CONFIG_APM_CPU_IDLE=y
CONFIG_APM_DISPLAY_BLANK=y
# CONFIG_APM_ALLOW_INTS is not set
# CONFIG_APM_REAL_MODE_POWER_OFF is not set
#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=y
# CONFIG_X86_POWERNOW_K6 is not set
# CONFIG_X86_POWERNOW_K7 is not set
# CONFIG_X86_POWERNOW_K8 is not set
# CONFIG_X86_GX_SUSPMOD is not set
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
# CONFIG_X86_SPEEDSTEP_ICH is not set
# CONFIG_X86_SPEEDSTEP_SMI is not set
CONFIG_X86_P4_CLOCKMOD=y
# CONFIG_X86_CPUFREQ_NFORCE2 is not set
# CONFIG_X86_LONGRUN is not set
# CONFIG_X86_LONGHAUL is not set
# CONFIG_X86_E_POWERSAVER is not set
#
# shared options
#
CONFIG_X86_ACPI_CPUFREQ_PROC_INTF=y
CONFIG_X86_SPEEDSTEP_LIB=y
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
# CONFIG_PCI_GOOLPC is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
# CONFIG_PCIEASPM is not set
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_LEGACY=y
# CONFIG_PCI_DEBUG is not set
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
# CONFIG_EISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set
# CONFIG_OLPC is not set
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m
#
# Networking
#
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=y
CONFIG_XFRM_SUB_POLICY=y
CONFIG_XFRM_MIGRATE=y
# CONFIG_XFRM_STATISTICS is not set
CONFIG_NET_KEY=m
CONFIG_NET_KEY_MIGRATE=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=y
CONFIG_NET_IPGRE=y
# CONFIG_NET_IPGRE_BROADCAST is not set
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=y
CONFIG_INET_ESP=y
CONFIG_INET_IPCOMP=y
CONFIG_INET_XFRM_TUNNEL=y
CONFIG_INET_TUNNEL=y
CONFIG_INET_XFRM_MODE_TRANSPORT=y
CONFIG_INET_XFRM_MODE_TUNNEL=y
CONFIG_INET_XFRM_MODE_BEET=y
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IP_VS is not set
CONFIG_IPV6=y
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
CONFIG_INET6_AH=y
CONFIG_INET6_ESP=y
CONFIG_INET6_IPCOMP=y
CONFIG_IPV6_MIP6=y
CONFIG_INET6_XFRM_TUNNEL=y
CONFIG_INET6_TUNNEL=y
CONFIG_INET6_XFRM_MODE_TRANSPORT=y
CONFIG_INET6_XFRM_MODE_TUNNEL=y
CONFIG_INET6_XFRM_MODE_BEET=y
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=y
CONFIG_IPV6_SIT=y
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=y
CONFIG_IPV6_MULTIPLE_TABLES=y
# CONFIG_IPV6_SUBTREES is not set
# CONFIG_IPV6_MROUTE is not set
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_QUEUE=y
CONFIG_NETFILTER_NETLINK_LOG=y
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CT_ACCT=y
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_SECMARK=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CT_PROTO_DCCP=y
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
CONFIG_NF_CT_PROTO_UDPLITE=m
# CONFIG_NF_CONNTRACK_AMANDA is not set
CONFIG_NF_CONNTRACK_FTP=y
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=y
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=y
CONFIG_NETFILTER_XT_TARGET_CONNMARK=y
CONFIG_NETFILTER_XT_TARGET_DSCP=y
CONFIG_NETFILTER_XT_TARGET_MARK=y
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=y
CONFIG_NETFILTER_XT_TARGET_NFLOG=y
CONFIG_NETFILTER_XT_TARGET_NOTRACK=y
CONFIG_NETFILTER_XT_TARGET_RATEEST=y
CONFIG_NETFILTER_XT_TARGET_TRACE=y
CONFIG_NETFILTER_XT_TARGET_SECMARK=y
CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=y
CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=y
CONFIG_NETFILTER_XT_MATCH_COMMENT=y
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=y
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=y
CONFIG_NETFILTER_XT_MATCH_CONNMARK=y
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
CONFIG_NETFILTER_XT_MATCH_DCCP=y
CONFIG_NETFILTER_XT_MATCH_DSCP=y
CONFIG_NETFILTER_XT_MATCH_ESP=y
CONFIG_NETFILTER_XT_MATCH_HELPER=y
CONFIG_NETFILTER_XT_MATCH_IPRANGE=y
CONFIG_NETFILTER_XT_MATCH_LENGTH=y
CONFIG_NETFILTER_XT_MATCH_LIMIT=y
CONFIG_NETFILTER_XT_MATCH_MAC=y
CONFIG_NETFILTER_XT_MATCH_MARK=y
CONFIG_NETFILTER_XT_MATCH_OWNER=y
CONFIG_NETFILTER_XT_MATCH_POLICY=y
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=y
CONFIG_NETFILTER_XT_MATCH_QUOTA=y
CONFIG_NETFILTER_XT_MATCH_RATEEST=y
CONFIG_NETFILTER_XT_MATCH_REALM=y
CONFIG_NETFILTER_XT_MATCH_SCTP=y
CONFIG_NETFILTER_XT_MATCH_STATE=y
CONFIG_NETFILTER_XT_MATCH_STATISTIC=y
CONFIG_NETFILTER_XT_MATCH_STRING=y
CONFIG_NETFILTER_XT_MATCH_TCPMSS=y
CONFIG_NETFILTER_XT_MATCH_TIME=y
CONFIG_NETFILTER_XT_MATCH_U32=y
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=y
#
# IP: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV4=y
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_RECENT=y
CONFIG_IP_NF_MATCH_ECN=y
CONFIG_IP_NF_MATCH_AH=y
CONFIG_IP_NF_MATCH_TTL=y
CONFIG_IP_NF_MATCH_ADDRTYPE=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_NF_NAT=y
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_IP_NF_TARGET_REDIRECT=y
CONFIG_IP_NF_TARGET_NETMAP=y
CONFIG_NF_NAT_SNMP_BASIC=y
CONFIG_NF_NAT_PROTO_DCCP=y
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_UDPLITE=m
CONFIG_NF_NAT_PROTO_SCTP=m
CONFIG_NF_NAT_FTP=y
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
# CONFIG_NF_NAT_AMANDA is not set
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=y
CONFIG_IP_NF_TARGET_ECN=y
CONFIG_IP_NF_TARGET_TTL=y
CONFIG_IP_NF_TARGET_CLUSTERIP=y
CONFIG_IP_NF_RAW=y
CONFIG_IP_NF_ARPTABLES=y
CONFIG_IP_NF_ARPFILTER=y
CONFIG_IP_NF_ARP_MANGLE=y
#
# IPv6: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV6=y
CONFIG_IP6_NF_QUEUE=m
CONFIG_IP6_NF_IPTABLES=y
CONFIG_IP6_NF_MATCH_RT=y
CONFIG_IP6_NF_MATCH_OPTS=y
CONFIG_IP6_NF_MATCH_FRAG=y
CONFIG_IP6_NF_MATCH_HL=y
CONFIG_IP6_NF_MATCH_IPV6HEADER=y
CONFIG_IP6_NF_MATCH_AH=y
CONFIG_IP6_NF_MATCH_MH=y
CONFIG_IP6_NF_MATCH_EUI64=y
CONFIG_IP6_NF_FILTER=y
CONFIG_IP6_NF_TARGET_LOG=y
CONFIG_IP6_NF_TARGET_REJECT=y
CONFIG_IP6_NF_MANGLE=y
CONFIG_IP6_NF_TARGET_HL=y
CONFIG_IP6_NF_RAW=y
CONFIG_IP_DCCP=m
CONFIG_INET_DCCP_DIAG=m
CONFIG_IP_DCCP_ACKVEC=y
#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
CONFIG_IP_DCCP_CCID2=m
# CONFIG_IP_DCCP_CCID2_DEBUG is not set
CONFIG_IP_DCCP_CCID3=m
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_CCID3_RTO=100
CONFIG_IP_DCCP_TFRC_LIB=m
#
# DCCP Kernel Hacking
#
# CONFIG_IP_DCCP_DEBUG is not set
CONFIG_IP_SCTP=m
CONFIG_SCTP_DBG_MSG=y
CONFIG_SCTP_DBG_OBJCNT=y
# CONFIG_SCTP_HMAC_NONE is not set
CONFIG_SCTP_HMAC_SHA1=y
# CONFIG_SCTP_HMAC_MD5 is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
CONFIG_NET_SCHED=y
#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=y
CONFIG_NET_SCH_HTB=y
CONFIG_NET_SCH_HFSC=y
CONFIG_NET_SCH_PRIO=y
CONFIG_NET_SCH_RED=y
CONFIG_NET_SCH_SFQ=y
CONFIG_NET_SCH_TEQL=y
CONFIG_NET_SCH_TBF=y
CONFIG_NET_SCH_GRED=y
CONFIG_NET_SCH_DSMARK=y
CONFIG_NET_SCH_NETEM=y
CONFIG_NET_SCH_INGRESS=y
#
# Classification
#
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=y
CONFIG_NET_CLS_TCINDEX=y
CONFIG_NET_CLS_ROUTE4=y
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=y
CONFIG_NET_CLS_U32=y
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=y
CONFIG_NET_CLS_RSVP6=y
CONFIG_NET_CLS_FLOW=y
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=y
CONFIG_NET_EMATCH_NBYTE=y
CONFIG_NET_EMATCH_U32=y
CONFIG_NET_EMATCH_META=y
CONFIG_NET_EMATCH_TEXT=y
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=y
CONFIG_NET_ACT_GACT=y
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=y
CONFIG_NET_ACT_IPT=y
CONFIG_NET_ACT_NAT=y
CONFIG_NET_ACT_PEDIT=y
CONFIG_NET_ACT_SIMP=y
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y
#
# Network testing
#
CONFIG_NET_PKTGEN=m
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
CONFIG_BT=m
CONFIG_BT_L2CAP=m
CONFIG_BT_SCO=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_HIDP=m
#
# Bluetooth device drivers
#
CONFIG_BT_HCIUSB=m
CONFIG_BT_HCIUSB_SCO=y
CONFIG_BT_HCIBTSDIO=m
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
CONFIG_BT_HCIUART_LL=y
CONFIG_BT_HCIBCM203X=m
CONFIG_BT_HCIBPA10X=m
CONFIG_BT_HCIBFUSB=m
CONFIG_BT_HCIVHCI=m
CONFIG_AF_RXRPC=m
# CONFIG_AF_RXRPC_DEBUG is not set
CONFIG_RXKAD=m
CONFIG_FIB_RULES=y
#
# Wireless
#
# CONFIG_CFG80211 is not set
# CONFIG_WIRELESS_EXT is not set
# CONFIG_MAC80211 is not set
# CONFIG_IEEE80211 is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
# CONFIG_STANDALONE is not set
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=m
# CONFIG_MTD is not set
CONFIG_PARPORT=m
CONFIG_PARPORT_PC=m
CONFIG_PARPORT_PC_FIFO=y
# CONFIG_PARPORT_PC_SUPERIO is not set
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
# CONFIG_PARPORT_1284 is not set
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set
#
# Protocols
#
CONFIG_ISAPNP=y
# CONFIG_PNPBIOS is not set
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_DEV_XD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_NBD=m
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=y
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
# CONFIG_ATA_OVER_ETH is not set
CONFIG_MISC_DEVICES=y
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_SGI_IOC4 is not set
CONFIG_TIFM_CORE=m
CONFIG_TIFM_7XX1=m
# CONFIG_TC1100_WMI is not set
# CONFIG_SONY_LAPTOP is not set
# CONFIG_THINKPAD_ACPI is not set
CONFIG_INTEL_MENLOW=m
# CONFIG_ENCLOSURE_SERVICES is not set
CONFIG_HAVE_IDE=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
#
# Please see Documentation/ide/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
CONFIG_BLK_DEV_IDEDISK=y
# CONFIG_IDEDISK_MULTI_MODE is not set
CONFIG_BLK_DEV_IDECD=m
CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
CONFIG_BLK_DEV_IDESCSI=m
CONFIG_BLK_DEV_IDEACPI=y
# CONFIG_IDE_TASK_IOCTL is not set
CONFIG_IDE_PROC_FS=y
#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_PLATFORM=m
# CONFIG_BLK_DEV_CMD640 is not set
# CONFIG_BLK_DEV_IDEPNP is not set
CONFIG_BLK_DEV_IDEDMA_SFF=y
#
# PCI IDE chipsets support
#
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_PCIBUS_ORDER=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_CS5535 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
CONFIG_BLK_DEV_JMICRON=y
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_IT8213 is not set
# CONFIG_BLK_DEV_IT821X is not set
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_BLK_DEV_TC86C001 is not set
#
# Other IDE chipsets support
#
#
# Note: most of these also require special kernel boot parameters
#
# CONFIG_BLK_DEV_4DRIVES is not set
# CONFIG_BLK_DEV_ALI14XX is not set
# CONFIG_BLK_DEV_DTC2278 is not set
# CONFIG_BLK_DEV_HT6560B is not set
# CONFIG_BLK_DEV_QD65XX is not set
# CONFIG_BLK_DEV_UMC8672 is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_BLK_DEV_HD_ONLY is not set
# CONFIG_BLK_DEV_HD is not set
#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set
#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m
#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=m
CONFIG_SCSI_FC_ATTRS=m
CONFIG_SCSI_ISCSI_ATTRS=m
CONFIG_SCSI_SAS_ATTRS=m
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
# CONFIG_SCSI_IMM is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_SRP is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
CONFIG_SATA_PMP=y
# CONFIG_SATA_AHCI is not set
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
# CONFIG_SATA_SVW is not set
CONFIG_ATA_PIIX=y
# CONFIG_SATA_MV is not set
# CONFIG_SATA_NV is not set
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
# CONFIG_SATA_INIC162X is not set
CONFIG_PATA_ACPI=m
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CS5535 is not set
# CONFIG_PATA_CS5536 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
CONFIG_ATA_GENERIC=y
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_ISAPNP is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
CONFIG_PATA_JMICRON=y
# CONFIG_PATA_LEGACY is not set
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_QDI is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_WINBOND_VLB is not set
CONFIG_PATA_SCH=m
# CONFIG_MD is not set
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
#
# Enable only one of the two stacks, unless you know what you are doing
#
# CONFIG_FIREWIRE is not set
CONFIG_IEEE1394=m
CONFIG_IEEE1394_OHCI1394=m
# CONFIG_IEEE1394_PCILYNX is not set
CONFIG_IEEE1394_SBP2=m
# CONFIG_IEEE1394_SBP2_PHYS_DMA is not set
CONFIG_IEEE1394_ETH1394_ROM_ENTRY=y
CONFIG_IEEE1394_ETH1394=m
CONFIG_IEEE1394_RAWIO=m
CONFIG_IEEE1394_VIDEO1394=m
CONFIG_IEEE1394_DV1394=m
# CONFIG_IEEE1394_VERBOSEDEBUG is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
# CONFIG_NETDEVICES_MULTIQUEUE is not set
# CONFIG_IFB is not set
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=m
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
# CONFIG_NET_ETHERNET is not set
CONFIG_MII=m
CONFIG_NETDEV_1000=y
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_E1000E is not set
# CONFIG_E1000E_ENABLED is not set
# CONFIG_IP1000 is not set
# CONFIG_IGB is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set
# CONFIG_QLA3XXX is not set
CONFIG_ATL1=m
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set
#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set
# CONFIG_IWLWIFI_LEDS is not set
#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
CONFIG_PPP=m
# CONFIG_PPP_MULTILINK is not set
# CONFIG_PPP_FILTER is not set
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=m
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
# CONFIG_PPPOL2TP is not set
CONFIG_SLIP=m
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLHC=m
CONFIG_SLIP_SMART=y
CONFIG_SLIP_MODE_SLIP6=y
# CONFIG_NET_FC is not set
CONFIG_NETCONSOLE=y
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set
#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
CONFIG_INPUT_POLLDEV=m
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1280
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=1024
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=m
CONFIG_INPUT_EVBUG=m
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_INPORT is not set
# CONFIG_MOUSE_LOGIBM is not set
# CONFIG_MOUSE_PC110PAD is not set
# CONFIG_MOUSE_VSXXXAA is not set
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
# CONFIG_JOYSTICK_A3D is not set
# CONFIG_JOYSTICK_ADI is not set
# CONFIG_JOYSTICK_COBRA is not set
# CONFIG_JOYSTICK_GF2K is not set
# CONFIG_JOYSTICK_GRIP is not set
# CONFIG_JOYSTICK_GRIP_MP is not set
# CONFIG_JOYSTICK_GUILLEMOT is not set
# CONFIG_JOYSTICK_INTERACT is not set
# CONFIG_JOYSTICK_SIDEWINDER is not set
# CONFIG_JOYSTICK_TMDC is not set
# CONFIG_JOYSTICK_IFORCE is not set
# CONFIG_JOYSTICK_WARRIOR is not set
# CONFIG_JOYSTICK_MAGELLAN is not set
# CONFIG_JOYSTICK_SPACEORB is not set
# CONFIG_JOYSTICK_SPACEBALL is not set
# CONFIG_JOYSTICK_STINGER is not set
# CONFIG_JOYSTICK_TWIDJOY is not set
# CONFIG_JOYSTICK_ZHENHUA is not set
# CONFIG_JOYSTICK_DB9 is not set
# CONFIG_JOYSTICK_GAMECON is not set
# CONFIG_JOYSTICK_TURBOGRAFX is not set
CONFIG_JOYSTICK_JOYDUMP=m
# CONFIG_JOYSTICK_XPAD is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_WISTRON_BTNS is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
CONFIG_INPUT_UINPUT=m
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PARKBD is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
# CONFIG_GAMEPORT_L4 is not set
# CONFIG_GAMEPORT_EMU10K1 is not set
# CONFIG_GAMEPORT_FM801 is not set
#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_DEVKMEM=y
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set
#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set
CONFIG_FIX_EARLYCON_MEM=y
#
# Non-8250 serial port support
#
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
# CONFIG_PRINTER is not set
# CONFIG_PPDEV is not set
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
CONFIG_IPMI_POWEROFF=m
CONFIG_HW_RANDOM=m
CONFIG_HW_RANDOM_INTEL=m
# CONFIG_HW_RANDOM_AMD is not set
# CONFIG_HW_RANDOM_GEODE is not set
CONFIG_HW_RANDOM_VIA=m
CONFIG_NVRAM=y
CONFIG_RTC=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_NSC_GPIO is not set
# CONFIG_CS5535_GPIO is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
CONFIG_HPET_RTC_IRQ=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=y
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_ALGOBIT=m
#
# I2C Hardware Bus support
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
CONFIG_I2C_I801=y
# CONFIG_I2C_I810 is not set
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_PARPORT is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_PROSAVAGE is not set
# CONFIG_I2C_SAVAGE4 is not set
# CONFIG_I2C_SIMTEC is not set
# CONFIG_SCx200_ACB is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_TINY_USB is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set
# CONFIG_I2C_VOODOO3 is not set
# CONFIG_I2C_PCA_ISA is not set
# CONFIG_I2C_PCA_PLATFORM is not set
#
# Miscellaneous I2C Chip support
#
# CONFIG_DS1682 is not set
CONFIG_SENSORS_EEPROM=m
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_PCF8575 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_MAX6875 is not set
# CONFIG_SENSORS_TSL2550 is not set
CONFIG_I2C_DEBUG_CORE=y
CONFIG_I2C_DEBUG_ALGO=y
CONFIG_I2C_DEBUG_BUS=y
CONFIG_I2C_DEBUG_CHIP=y
CONFIG_SPI=y
# CONFIG_SPI_DEBUG is not set
CONFIG_SPI_MASTER=y
#
# SPI Master Controller Drivers
#
# CONFIG_SPI_BITBANG is not set
# CONFIG_SPI_BUTTERFLY is not set
# CONFIG_SPI_LM70_LLP is not set
#
# SPI Protocol Masters
#
# CONFIG_SPI_AT25 is not set
# CONFIG_SPI_SPIDEV is not set
# CONFIG_SPI_TLE62X0 is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=y
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ABITUGURU3 is not set
CONFIG_SENSORS_AD7418=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1029=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ADT7470=m
# CONFIG_SENSORS_ADT7473 is not set
# CONFIG_SENSORS_K8TEMP is not set
CONFIG_SENSORS_ASB100=y
# CONFIG_SENSORS_ATXP1 is not set
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_I5K_AMB=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_F71882FG=m
CONFIG_SENSORS_F75375S=m
CONFIG_SENSORS_FSCHER=m
CONFIG_SENSORS_FSCPOS=m
CONFIG_SENSORS_FSCHMD=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
CONFIG_SENSORS_CORETEMP=y
# CONFIG_SENSORS_IBMAEM is not set
# CONFIG_SENSORS_IBMPEX is not set
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM70=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_LM93=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_MAX6650=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_PC87427=m
# CONFIG_SENSORS_SIS5595 is not set
CONFIG_SENSORS_DME1737=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
# CONFIG_SENSORS_ADS7828 is not set
CONFIG_SENSORS_THMC50=m
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83793=m
CONFIG_SENSORS_W83L785TS=m
# CONFIG_SENSORS_W83L786NG is not set
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_HDAPS=m
# CONFIG_SENSORS_APPLESMC is not set
CONFIG_HWMON_DEBUG_CHIP=y
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set
#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=y
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
# CONFIG_ALIM1535_WDT is not set
# CONFIG_ALIM7101_WDT is not set
# CONFIG_SC520_WDT is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
# CONFIG_IBMASR is not set
# CONFIG_WAFER_WDT is not set
# CONFIG_I6300ESB_WDT is not set
# CONFIG_ITCO_WDT is not set
# CONFIG_IT8712F_WDT is not set
# CONFIG_HP_WATCHDOG is not set
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
# CONFIG_60XX_WDT is not set
# CONFIG_SBC8360_WDT is not set
# CONFIG_SBC7240_WDT is not set
# CONFIG_CPU5_WDT is not set
# CONFIG_SMSC37B787_WDT is not set
# CONFIG_W83627HF_WDT is not set
# CONFIG_W83697HF_WDT is not set
# CONFIG_W83877F_WDT is not set
# CONFIG_W83977F_WDT is not set
# CONFIG_MACHZ_WDT is not set
# CONFIG_SBC_EPX_C3_WATCHDOG is not set
#
# ISA-based Watchdog Cards
#
# CONFIG_PCWATCHDOG is not set
# CONFIG_MIXCOMWD is not set
# CONFIG_WDT is not set
#
# PCI-based Watchdog Cards
#
# CONFIG_PCIPCWATCHDOG is not set
# CONFIG_WDTPCI is not set
#
# USB-based Watchdog Cards
#
# CONFIG_USBPCWATCHDOG is not set
#
# Sonics Silicon Backplane
#
CONFIG_SSB_POSSIBLE=y
# CONFIG_SSB is not set
#
# Multifunction device drivers
#
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
#
# Multimedia devices
#
#
# Multimedia core support
#
CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2_COMMON=m
CONFIG_VIDEO_ALLOW_V4L1=y
CONFIG_VIDEO_V4L1_COMPAT=y
# CONFIG_DVB_CORE is not set
CONFIG_VIDEO_MEDIA=m
#
# Multimedia drivers
#
CONFIG_MEDIA_ATTACH=y
CONFIG_MEDIA_TUNER=m
# CONFIG_MEDIA_TUNER_CUSTOMIZE is not set
CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA8290=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_TEA5761=m
CONFIG_MEDIA_TUNER_TEA5767=m
CONFIG_MEDIA_TUNER_MT20XX=m
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=m
CONFIG_VIDEO_V4L2=m
CONFIG_VIDEO_V4L1=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEOBUF_VMALLOC=m
CONFIG_VIDEO_CAPTURE_DRIVERS=y
# CONFIG_VIDEO_ADV_DEBUG is not set
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
CONFIG_VIDEO_VIVI=m
# CONFIG_VIDEO_BT848 is not set
# CONFIG_VIDEO_PMS is not set
# CONFIG_VIDEO_BWQCAM is not set
CONFIG_VIDEO_CQCAM=m
# CONFIG_VIDEO_CPIA is not set
# CONFIG_VIDEO_CPIA2 is not set
# CONFIG_VIDEO_SAA5246A is not set
# CONFIG_VIDEO_SAA5249 is not set
# CONFIG_TUNER_3036 is not set
# CONFIG_VIDEO_STRADIS is not set
# CONFIG_VIDEO_ZORAN is not set
# CONFIG_VIDEO_SAA7134 is not set
# CONFIG_VIDEO_MXB is not set
# CONFIG_VIDEO_DPC is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
# CONFIG_VIDEO_CX88 is not set
# CONFIG_VIDEO_IVTV is not set
# CONFIG_VIDEO_CAFE_CCIC is not set
CONFIG_V4L_USB_DRIVERS=y
CONFIG_USB_VIDEO_CLASS=m
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
# CONFIG_VIDEO_PVRUSB2 is not set
# CONFIG_VIDEO_EM28XX is not set
# CONFIG_VIDEO_USBVISION is not set
# CONFIG_USB_VICAM is not set
# CONFIG_USB_IBMCAM is not set
# CONFIG_USB_KONICAWC is not set
# CONFIG_USB_QUICKCAM_MESSENGER is not set
# CONFIG_USB_ET61X251 is not set
# CONFIG_VIDEO_OVCAMCHIP is not set
# CONFIG_USB_W9968CF is not set
# CONFIG_USB_OV511 is not set
# CONFIG_USB_SE401 is not set
# CONFIG_USB_SN9C102 is not set
# CONFIG_USB_STV680 is not set
# CONFIG_USB_ZC0301 is not set
# CONFIG_USB_PWC is not set
# CONFIG_USB_ZR364XX is not set
# CONFIG_USB_STKWEBCAM is not set
CONFIG_SOC_CAMERA=m
CONFIG_SOC_CAMERA_MT9M001=m
CONFIG_SOC_CAMERA_MT9V022=m
# CONFIG_RADIO_ADAPTERS is not set
# CONFIG_DAB is not set
#
# Graphics support
#
# CONFIG_AGP is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_VIA is not set
# CONFIG_DRM_SAVAGE is not set
CONFIG_VGASTATE=m
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=m
CONFIG_FIRMWARE_EDID=y
CONFIG_FB_DDC=m
CONFIG_FB_CFB_FILLRECT=m
CONFIG_FB_CFB_COPYAREA=m
CONFIG_FB_CFB_IMAGEBLIT=m
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
# CONFIG_FB_SYS_FILLRECT is not set
# CONFIG_FB_SYS_COPYAREA is not set
# CONFIG_FB_SYS_IMAGEBLIT is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_SYS_FOPS is not set
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
CONFIG_FB_NVIDIA=m
CONFIG_FB_NVIDIA_I2C=y
# CONFIG_FB_NVIDIA_DEBUG is not set
# CONFIG_FB_NVIDIA_BACKLIGHT is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I810 is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_CYBLA is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_VIRTUAL is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set
#
# Display device support
#
CONFIG_DISPLAY_SUPPORT=y
#
# Display hardware drivers
#
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_VIDEO_SELECT=y
# CONFIG_MDA_CONSOLE is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=m
# CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY is not set
# CONFIG_FRAMEBUFFER_CONSOLE_ROTATION is not set
# CONFIG_FONTS is not set
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
# CONFIG_LOGO is not set
#
# Sound
#
CONFIG_SOUND=m
#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_RTCTIMER=m
CONFIG_SND_SEQ_RTCTIMER_DEFAULT=y
# CONFIG_SND_DYNAMIC_MINORS is not set
CONFIG_SND_SUPPORT_OLD_API=y
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
#
# Generic devices
#
CONFIG_SND_PCSP=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_MTS64 is not set
CONFIG_SND_SERIAL_U16550=m
CONFIG_SND_MPU401=m
# CONFIG_SND_PORTMAN2X4 is not set
#
# ISA devices
#
# CONFIG_SND_ADLIB is not set
# CONFIG_SND_AD1816A is not set
# CONFIG_SND_AD1848 is not set
# CONFIG_SND_ALS100 is not set
# CONFIG_SND_AZT2320 is not set
# CONFIG_SND_CMI8330 is not set
# CONFIG_SND_CS4231 is not set
# CONFIG_SND_CS4232 is not set
# CONFIG_SND_CS4236 is not set
# CONFIG_SND_DT019X is not set
# CONFIG_SND_ES968 is not set
# CONFIG_SND_ES1688 is not set
# CONFIG_SND_ES18XX is not set
# CONFIG_SND_SC6000 is not set
# CONFIG_SND_GUSCLASSIC is not set
# CONFIG_SND_GUSEXTREME is not set
# CONFIG_SND_GUSMAX is not set
# CONFIG_SND_INTERWAVE is not set
# CONFIG_SND_INTERWAVE_STB is not set
# CONFIG_SND_OPL3SA2 is not set
# CONFIG_SND_OPTI92X_AD1848 is not set
# CONFIG_SND_OPTI92X_CS4231 is not set
# CONFIG_SND_OPTI93X is not set
# CONFIG_SND_MIRO is not set
# CONFIG_SND_SB8 is not set
# CONFIG_SND_SB16 is not set
# CONFIG_SND_SBAWE is not set
# CONFIG_SND_SGALAXY is not set
# CONFIG_SND_SSCAPE is not set
# CONFIG_SND_WAVEFRONT is not set
#
# PCI devices
#
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CS5535AUDIO is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
CONFIG_SND_HDA_INTEL=m
# CONFIG_SND_HDA_HWDEP is not set
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_ATIHDMI=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_HIFIER is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
CONFIG_SND_INTEL8X0=m
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SIS7019 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
CONFIG_SND_AC97_POWER_SAVE=y
CONFIG_SND_AC97_POWER_SAVE_DEFAULT=900
#
# SPI devices
#
#
# USB devices
#
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set
# CONFIG_SND_USB_CAIAQ is not set
#
# System on Chip audio support
#
# CONFIG_SND_SOC is not set
#
# ALSA SoC audio for Freescale SOCs
#
#
# SoC Audio for the Texas Instruments OMAP
#
#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=m
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HID_DEBUG is not set
# CONFIG_HIDRAW is not set
#
# USB Input Devices
#
CONFIG_USB_HID=m
# CONFIG_USB_HIDINPUT_POWERBOOK is not set
# CONFIG_HID_FF is not set
CONFIG_USB_HIDDEV=y
#
# USB HID Boot Protocol drivers
#
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
CONFIG_USB_DEBUG=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
CONFIG_USB_DEVICE_CLASS=y
CONFIG_USB_DYNAMIC_MINORS=y
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
CONFIG_USB_WDM=m
#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#
#
# may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
CONFIG_USB_STORAGE_DEBUG=y
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
CONFIG_USB_STORAGE_USBAT=y
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y
CONFIG_USB_STORAGE_ALAUDA=y
CONFIG_USB_STORAGE_ONETOUCH=y
# CONFIG_USB_STORAGE_KARMA is not set
CONFIG_USB_STORAGE_CYPRESS_ATACB=y
# CONFIG_USB_LIBUSUAL is not set
#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
CONFIG_USB_MON=y
#
# USB port drivers
#
# CONFIG_USB_USS720 is not set
CONFIG_USB_SERIAL=m
# CONFIG_USB_EZUSB is not set
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRCABLE=m
# CONFIG_USB_SERIAL_AIRPRIME is not set
# CONFIG_USB_SERIAL_ARK3116 is not set
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_CH341 is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_CP2101 is not set
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
# CONFIG_USB_SERIAL_EMPEG is not set
# CONFIG_USB_SERIAL_FTDI_SIO is not set
# CONFIG_USB_SERIAL_FUNSOFT is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_GARMIN is not set
# CONFIG_USB_SERIAL_IPW is not set
# CONFIG_USB_SERIAL_IUU is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
# CONFIG_USB_SERIAL_MOS7720 is not set
# CONFIG_USB_SERIAL_MOS7840 is not set
CONFIG_USB_SERIAL_MOTOROLA=m
# CONFIG_USB_SERIAL_NAVMAN is not set
# CONFIG_USB_SERIAL_PL2303 is not set
# CONFIG_USB_SERIAL_OTI6858 is not set
CONFIG_USB_SERIAL_SPCP8X5=m
# CONFIG_USB_SERIAL_HP4X is not set
# CONFIG_USB_SERIAL_SAFE is not set
# CONFIG_USB_SERIAL_SIERRAWIRELESS is not set
# CONFIG_USB_SERIAL_TI is not set
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
CONFIG_USB_SERIAL_OPTION=m
# CONFIG_USB_SERIAL_OMNINET is not set
# CONFIG_USB_SERIAL_DEBUG is not set
#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
CONFIG_USB_ISIGHTFW=m
# CONFIG_USB_GADGET is not set
CONFIG_MMC=m
# CONFIG_MMC_DEBUG is not set
# CONFIG_MMC_UNSAFE_RESUME is not set
#
# MMC/SD Card Drivers
#
CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_BOUNCE=y
# CONFIG_SDIO_UART is not set
CONFIG_MMC_TEST=m
#
# MMC/SD Host Controller Drivers
#
CONFIG_MMC_SDHCI=m
CONFIG_MMC_RICOH_MMC=m
CONFIG_MMC_WBSD=m
CONFIG_MMC_TIFM_SD=m
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
CONFIG_ACCESSIBILITY=y
# CONFIG_INFINIBAND is not set
CONFIG_EDAC=y
#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_MM_EDAC=y
# CONFIG_EDAC_AMD76X is not set
# CONFIG_EDAC_E7XXX is not set
# CONFIG_EDAC_E752X is not set
# CONFIG_EDAC_I82875P is not set
# CONFIG_EDAC_I82975X is not set
# CONFIG_EDAC_I3000 is not set
# CONFIG_EDAC_I82860 is not set
# CONFIG_EDAC_R82600 is not set
# CONFIG_EDAC_I5000 is not set
# CONFIG_RTC_CLASS is not set
CONFIG_DMADEVICES=y
#
# DMA Devices
#
CONFIG_INTEL_IOATDMA=y
CONFIG_DMA_ENGINE=y
#
# DMA Clients
#
CONFIG_NET_DMA=y
CONFIG_DCA=y
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set
#
# Firmware Drivers
#
CONFIG_EDD=y
# CONFIG_EDD_OFF is not set
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
CONFIG_ISCSI_IBFT_FIND=y
CONFIG_ISCSI_IBFT=m
#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
# CONFIG_EXT3_FS_XATTR is not set
# CONFIG_EXT4DEV_FS is not set
CONFIG_JBD=y
CONFIG_JBD_DEBUG=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
# CONFIG_XFS_FS is not set
# CONFIG_OCFS2_FS is not set
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
CONFIG_PRINT_QUOTA_WARNING=y
# CONFIG_QFMT_V1 is not set
# CONFIG_QFMT_V2 is not set
CONFIG_QUOTACTL=y
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m
# CONFIG_FUSE_FS is not set
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=y
CONFIG_NTFS_DEBUG=y
CONFIG_NTFS_RW=y
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_CONFIGFS_FS=y
#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_ECRYPT_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
CONFIG_MINIX_FS=m
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
CONFIG_SUNRPC_BIND34=y
CONFIG_RPCSEC_GSS_KRB5=y
CONFIG_RPCSEC_GSS_SPKM3=m
CONFIG_SMB_FS=m
CONFIG_SMB_NLS_DEFAULT=y
CONFIG_SMB_NLS_REMOTE="cp850"
CONFIG_CIFS=m
CONFIG_CIFS_STATS=y
# CONFIG_CIFS_STATS2 is not set
CONFIG_CIFS_WEAK_PW_HASH=y
CONFIG_CIFS_XATTR=y
CONFIG_CIFS_POSIX=y
# CONFIG_CIFS_DEBUG2 is not set
CONFIG_CIFS_EXPERIMENTAL=y
CONFIG_CIFS_UPCALL=y
# CONFIG_CIFS_DFS_UPCALL is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
CONFIG_LDM_PARTITION=y
CONFIG_LDM_DEBUG=y
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso-8859-1"
CONFIG_NLS_CODEPAGE_437=y
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=y
CONFIG_NLS_CODEPAGE_852=y
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_2=y
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=y
CONFIG_DLM=m
# CONFIG_DLM_DEBUG is not set
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_ENABLE_WARN_DEPRECATED=y
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_SHIRQ=y
CONFIG_DETECT_SOFTLOCKUP=y
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y
CONFIG_DEBUG_OBJECTS=y
# CONFIG_DEBUG_OBJECTS_SELFTEST is not set
CONFIG_DEBUG_OBJECTS_FREE=y
CONFIG_DEBUG_OBJECTS_TIMERS=y
CONFIG_DEBUG_SLAB=y
CONFIG_DEBUG_SLAB_LEAK=y
CONFIG_DEBUG_PREEMPT=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_PI_LIST=y
CONFIG_RT_MUTEX_TESTER=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
CONFIG_LOCKDEP=y
CONFIG_LOCK_STAT=y
CONFIG_DEBUG_LOCKDEP=y
CONFIG_TRACE_IRQFLAGS=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
CONFIG_DEBUG_LOCKING_API_SELFTESTS=y
CONFIG_STACKTRACE=y
CONFIG_DEBUG_KOBJECT=y
CONFIG_DEBUG_HIGHMEM=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_WRITECOUNT=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_SG=y
CONFIG_FRAME_POINTER=y
CONFIG_BOOT_PRINTK_DELAY=y
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_NONPROMISC_DEVMEM=y
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_STACK_USAGE=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_X86_PTDUMP=y
CONFIG_DEBUG_RODATA=y
# CONFIG_DEBUG_RODATA_TEST is not set
# CONFIG_DEBUG_NX_TEST is not set
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_DOUBLEFAULT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
CONFIG_CPA_DEBUG=y
#
# Security options
#
CONFIG_KEYS=y
# CONFIG_KEYS_DEBUG_PROC_KEYS is not set
# CONFIG_SECURITY is not set
CONFIG_SECURITY_FILE_CAPABILITIES=y
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_CRYPTD=m
CONFIG_CRYPTO_AUTHENC=y
CONFIG_CRYPTO_TEST=m
#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=m
CONFIG_CRYPTO_GCM=m
CONFIG_CRYPTO_SEQIV=m
#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=m
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_LRW=y
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=m
#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_XCBC=m
#
# Digest
#
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=m
#
# Ciphers
#
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_586=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=m
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SALSA20_586=m
CONFIG_CRYPTO_SEED=m
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_586=m
#
# Compression
#
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_LZO=m
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
CONFIG_CRYPTO_DEV_GEODE=m
# CONFIG_CRYPTO_DEV_HIFN_795X is not set
CONFIG_HAVE_KVM=y
# CONFIG_VIRTUALIZATION is not set
#
# Library routines
#
CONFIG_BITREVERSE=m
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=m
CONFIG_CRC7=m
CONFIG_LIBCRC32C=m
CONFIG_AUDIT_GENERIC=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=m
CONFIG_LZO_DECOMPRESS=m
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=y
CONFIG_TEXTSEARCH_BM=y
CONFIG_TEXTSEARCH_FSM=y
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-18 18:50 ` Athanasius
@ 2008-08-18 19:19 ` Trond Myklebust
2008-08-18 19:37 ` J. K. Cliburn
1 sibling, 0 replies; 64+ messages in thread
From: Trond Myklebust @ 2008-08-18 19:19 UTC (permalink / raw)
To: Athanasius; +Cc: Grant Coady, linux-kernel, neilb, bfields, linux-nfs
On Mon, 2008-08-18 at 19:50 +0100, Athanasius wrote:
> On Mon, Aug 18, 2008 at 12:02:20PM +1000, Grant Coady wrote:
> > I've been using NFS here for years, lately there's something odd going on
> > since about a month or so. Previously reported last month:
> > http://www.gossamer-threads.com/lists/linux/kernel/951419?page=last
> >
> > Now with 2.6.27-rc3 on one of the client boxes I get a complete stall
> > at odd times when accessing the server's exported directory, cannot
> > see a pattern to it. Eventually recovers after a Ctrl-C. Nothing in
> > the server or client log files. Not easy to reproduce either.
>
> I wonder if this is what I've been seeing. I've been otherwise too
> busy to properly report it, thinking that *someone* else must also be
> seeing it and it's being worked on, else it's a subtle configuration
> problem my end.
Your lockdep trace basically shows that the rpc layer is blocking for
some reason. Could you please try to reproduce the problem, and then do
echo 0 >/proc/sys/sunrpc/rpc_debug
...and see what the output from 'dmesg' shows?
Cheers
Trond
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-18 2:02 NFS regression? Odd delays and lockups accessing an NFS export Grant Coady
2008-08-18 18:50 ` Athanasius
@ 2008-08-18 19:20 ` Trond Myklebust
2008-08-20 1:10 ` Grant Coady
` (2 more replies)
1 sibling, 3 replies; 64+ messages in thread
From: Trond Myklebust @ 2008-08-18 19:20 UTC (permalink / raw)
To: Grant Coady; +Cc: linux-kernel, neilb, bfields, linux-nfs
On Mon, 2008-08-18 at 12:02 +1000, Grant Coady wrote:
> Hi there,
>
> I've been using NFS here for years, lately there's something odd going on
> since about a month or so. Previously reported last month:
> http://www.gossamer-threads.com/lists/linux/kernel/951419?page=last
>
> Now with 2.6.27-rc3 on one of the client boxes I get a complete stall
> at odd times when accessing the server's exported directory, cannot
> see a pattern to it. Eventually recovers after a Ctrl-C. Nothing in
> the server or client log files. Not easy to reproduce either.
>
> The server runs 2.6.26.2 at the moment.
>
> Server config, etc: http://bugsplatter.id.au/kernel/boxen/deltree/
> Client config, etc: http://bugsplatter.id.au/kernel/boxen/pooh/
Please try to reproduce the hang, then do
echo 0 >/proc/sys/sunrpc/rpc_debug
and send the output from 'dmesg'...
Cheers
Trond
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-18 18:50 ` Athanasius
2008-08-18 19:19 ` Trond Myklebust
@ 2008-08-18 19:37 ` J. K. Cliburn
2008-08-18 23:13 ` Athanasius
2009-05-12 20:27 ` Frank Filz
1 sibling, 2 replies; 64+ messages in thread
From: J. K. Cliburn @ 2008-08-18 19:37 UTC (permalink / raw)
To: Grant Coady, linux-kernel, neilb, bfields, linux-nfs
Athanasius wrote:
> On Mon, Aug 18, 2008 at 12:02:20PM +1000, Grant Coady wrote:
>> I've been using NFS here for years, lately there's something odd going on
>> since about a month or so. Previously reported last month:
>> http://www.gossamer-threads.com/lists/linux/kernel/951419?page=last
>>
>> Now with 2.6.27-rc3 on one of the client boxes I get a complete stall
>> at odd times when accessing the server's exported directory, cannot
>> see a pattern to it. Eventually recovers after a Ctrl-C. Nothing in
>> the server or client log files. Not easy to reproduce either.
>
> I wonder if this is what I've been seeing. I've been otherwise too
> busy to properly report it, thinking that *someone* else must also be
> seeing it and it's being worked on, else it's a subtle configuration
> problem my end.
>
> I first started seeing this with 2.6.26 on the client end,
I see you're running the atl1 driver on your client. I'm chasing an
intermittent bug in that driver that seems to be affected by certain
offload parameters. Try disabling TSO and see if things improve.
ethtool -K eth0 tso off
Let me know if it helps.
Jay
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-18 19:37 ` J. K. Cliburn
@ 2008-08-18 23:13 ` Athanasius
2009-05-12 20:27 ` Frank Filz
1 sibling, 0 replies; 64+ messages in thread
From: Athanasius @ 2008-08-18 23:13 UTC (permalink / raw)
To: jacliburn; +Cc: Grant Coady, linux-kernel, neilb, bfields, linux-nfs
[-- Attachment #1: Type: text/plain, Size: 1368 bytes --]
On Mon, Aug 18, 2008 at 02:37:45PM -0500, J. K. Cliburn wrote:
> Athanasius wrote:
> > I wonder if this is what I've been seeing. I've been otherwise too
> >busy to properly report it, thinking that *someone* else must also be
> >seeing it and it's being worked on, else it's a subtle configuration
> >problem my end.
> >
> > I first started seeing this with 2.6.26 on the client end,
>
> I see you're running the atl1 driver on your client. I'm chasing an
> intermittent bug in that driver that seems to be affected by certain
> offload parameters. Try disabling TSO and see if things improve.
>
> ethtool -K eth0 tso off
>
> Let me know if it helps.
Indeed it does, and indeed tso was on by default. I booted up once
after first trying this, thinking I had things set to apply it
automatically, but hadn't and ran into the exact same problem. This 2nd
time around with the setting definitely applied things appear to be
working normally. At least I can actually start Firefox up and have it
work (lots of /home/users/... access over NFS to the server).
thanks,
-Ath
--
- Athanasius = Athanasius(at)miggy.org / http://www.miggy.org/
Finger athan(at)fysh.org for PGP key
"And it's me who is my enemy. Me who beats me up.
Me who makes the monsters. Me who strips my confidence." Paula Cole - ME
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-18 19:20 ` Trond Myklebust
@ 2008-08-20 1:10 ` Grant Coady
2008-08-20 23:17 ` Grant Coady
2008-08-22 10:23 ` Ian Campbell
2 siblings, 0 replies; 64+ messages in thread
From: Grant Coady @ 2008-08-20 1:10 UTC (permalink / raw)
To: Trond Myklebust; +Cc: Grant Coady, linux-kernel, neilb, bfields, linux-nfs
Hi Trond,
On Mon, 18 Aug 2008 15:20:58 -0400, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>On Mon, 2008-08-18 at 12:02 +1000, Grant Coady wrote:
>> Hi there,
>>
>> I've been using NFS here for years, lately there's something odd going on
>> since about a month or so. Previously reported last month:
>> http://www.gossamer-threads.com/lists/linux/kernel/951419?page=last
>>
>> Now with 2.6.27-rc3 on one of the client boxes I get a complete stall
>> at odd times when accessing the server's exported directory, cannot
>> see a pattern to it. Eventually recovers after a Ctrl-C. Nothing in
>> the server or client log files. Not easy to reproduce either.
>>
>> The server runs 2.6.26.2 at the moment.
>>
>> Server config, etc: http://bugsplatter.id.au/kernel/boxen/deltree/
>> Client config, etc: http://bugsplatter.id.au/kernel/boxen/pooh/
>
>Please try to reproduce the hang, then do
>
> echo 0 >/proc/sys/sunrpc/rpc_debug
>
>and send the output from 'dmesg'...
dmesg and logs from client machine showed nothing, dmesg from server:
-pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops--
(nothing, nada, zilch after what looks like the header line)
Situation, first use of 'df' after reboot, client machine hung for many
seconds before the /home/common export showed up, this is not repeatable.
Mounting another export and doing 'df' again had no delays.
Client is running 2.6.27-rc3-git5 (x86_64) at the moment, configs, etc
at: http://bugsplatter.id.au/kernel/boxen/pooh64/
Server as above. Am I supposed to have some debugging turned on in the
.configs?
Grant.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-18 19:20 ` Trond Myklebust
2008-08-20 1:10 ` Grant Coady
@ 2008-08-20 23:17 ` Grant Coady
2008-08-22 10:23 ` Ian Campbell
2 siblings, 0 replies; 64+ messages in thread
From: Grant Coady @ 2008-08-20 23:17 UTC (permalink / raw)
To: Trond Myklebust; +Cc: Grant Coady, linux-kernel, neilb, bfields, linux-nfs
On Mon, 18 Aug 2008 15:20:58 -0400, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>On Mon, 2008-08-18 at 12:02 +1000, Grant Coady wrote:
>> Hi there,
>>
>> I've been using NFS here for years, lately there's something odd going on
>> since about a month or so. Previously reported last month:
>> http://www.gossamer-threads.com/lists/linux/kernel/951419?page=last
>>
>> Now with 2.6.27-rc3 on one of the client boxes I get a complete stall
>> at odd times when accessing the server's exported directory, cannot
>> see a pattern to it. Eventually recovers after a Ctrl-C. Nothing in
>> the server or client log files. Not easy to reproduce either.
>>
>> The server runs 2.6.26.2 at the moment.
>>
>> Server config, etc: http://bugsplatter.id.au/kernel/boxen/deltree/
>> Client config, etc: http://bugsplatter.id.au/kernel/boxen/pooh/
>
>Please try to reproduce the hang, then do
>
> echo 0 >/proc/sys/sunrpc/rpc_debug
>
>and send the output from 'dmesg'...
It's not NFS, I just had WinXP box stall completely (no KB NumLock
response) on a PuTTY (ssh) terminal to a linux box running 2.6.26.3
for about five or ten seconds, when it came back to operation the
PuTYY toolbar icon was flashing. Dunno which terminal as I had four
open at the time, but they all to a server box running 2.6.26.3 with
multiple Intel pro/100 (e100) NICs.
I think this may be to do with the e100 network driver instead, as a
few weeks ago I started compiling it in rather than as a module like
I've been doing for years (in response to the firmware (microcode)
being dropped on the floor last month issue). This box is the server
(deltree) that all linux boxes mount a /home/common NFS export from.
All machines have the Intel pro/100 NIC, going to recompile the lot
back to modules, see if that fixes the problem.
Grant.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-18 19:20 ` Trond Myklebust
2008-08-20 1:10 ` Grant Coady
2008-08-20 23:17 ` Grant Coady
@ 2008-08-22 10:23 ` Ian Campbell
2008-08-22 18:08 ` Trond Myklebust
2 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-08-22 10:23 UTC (permalink / raw)
To: Trond Myklebust; +Cc: Grant Coady, linux-kernel, neilb, bfields, linux-nfs
[-- Attachment #1: Type: text/plain, Size: 5025 bytes --]
On Mon, 2008-08-18 at 15:20 -0400, Trond Myklebust wrote:
> Please try to reproduce the hang, then do
>
> echo 0 >/proc/sys/sunrpc/rpc_debug
>
> and send the output from 'dmesg'...
I've also been seeing some NFS related lockups, although I'm not sure if
they are the same as the one in this thread or not. Client is 2.6.26
(Debian's kernel) and Server is 2.6.25 (also a Debian kernel, but from
backports.org).
rpc_debug on the server gives nothing, on the client gives:
[144741.637997] -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops--
[144741.637997] 3439 0004 0080 -11 f3f48200 100003 f7770000 0 xprt_sending fa0ae88e fa0bddf4
[144741.637997] 3438 0001 00a0 0 f77f2a00 100003 f77700d0 15000 xprt_pending fa0ae88e fa0bddf4
There are no processes running with pid 3439 3438 (I don't think it's
that sort of pid though).
mounts points are:
hopkins:/storage/music /storage/music nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
hopkins:/storage/mythtv/recordings /var/lib/mythtv/recordings nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
hopkins:/var/lib/mythvideo /var/lib/mythvideo nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
hopkins:/storage/home/ijc /home/ijc nfs rw,vers=3,rsize=131072,wsize=131072,namlen=255,hard,nointr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
and all seem to be effected.
It hasn't happened this time (yet) but usually I get a hung task
backtrace like this:
Aug 4 06:27:28 iranon kernel: [137969.382277] INFO: task mythbackend:3161 blocked for more than 120 seconds.
Aug 4 06:27:28 iranon kernel: [137969.382287] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 4 06:27:28 iranon kernel: [137969.382291] mythbackend D 00005dfc 0 3161 1
Aug 4 06:27:28 iranon kernel: [137969.382295] f2006c70 00000082 4b05cc7e 00005dfc f2006df0 c1c0e920 00000000 f7273390
Aug 4 06:27:28 iranon kernel: [137969.382301] 0000c8b4 00000000 00000001 f7273398 00000282 00000202 f71b0ab0 f200ddf0
Aug 4 06:27:28 iranon kernel: [137969.382306] c1c012bc fa18f0a6 fa18f0bf c02b45a7 f71b0ab0 00000000 f200de0c fa18f0a6
Aug 4 06:27:28 iranon kernel: [137969.382311] Call Trace:
Aug 4 06:27:28 iranon kernel: [137969.382347] [<fa18f0a6>] nfs_wait_schedule+0x0/0x1e [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382384] [<fa18f0bf>] nfs_wait_schedule+0x19/0x1e [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382399] [<c02b45a7>] __wait_on_bit_lock+0x2a/0x52
Aug 4 06:27:28 iranon kernel: [137969.382407] [<fa18f0a6>] nfs_wait_schedule+0x0/0x1e [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382421] [<c02b462e>] out_of_line_wait_on_bit_lock+0x5f/0x67
Aug 4 06:27:28 iranon kernel: [137969.382429] [<c0133a1d>] wake_bit_function+0x0/0x3c
Aug 4 06:27:28 iranon kernel: [137969.382441] [<fa18f16e>] __nfs_revalidate_inode+0xaa/0x211 [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382458] [<c017bac0>] do_lookup+0x53/0x145
Aug 4 06:27:28 iranon kernel: [137969.382466] [<c0187609>] mntput_no_expire+0x11/0x64
Aug 4 06:27:28 iranon kernel: [137969.382472] [<c017d971>] __link_path_walk+0xa71/0xb65
Aug 4 06:27:28 iranon kernel: [137969.382477] [<c017bac0>] do_lookup+0x53/0x145
Aug 4 06:27:28 iranon kernel: [137969.382483] [<c0187609>] mntput_no_expire+0x11/0x64
Aug 4 06:27:28 iranon kernel: [137969.382492] [<c0187609>] mntput_no_expire+0x11/0x64
Aug 4 06:27:28 iranon kernel: [137969.382496] [<c017daf5>] path_walk+0x90/0x98
Aug 4 06:27:28 iranon kernel: [137969.382502] [<fa18f935>] nfs_getattr+0x8f/0xbe [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382520] [<fa18f8a6>] nfs_getattr+0x0/0xbe [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382536] [<c01785ac>] vfs_getattr+0x36/0x4d
Aug 4 06:27:28 iranon kernel: [137969.382545] [<c0178684>] vfs_lstat_fd+0x27/0x39
Aug 4 06:27:28 iranon kernel: [137969.382550] [<fa18bca6>] nfs_permission+0x0/0x129 [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382567] [<c0187609>] mntput_no_expire+0x11/0x64
Aug 4 06:27:28 iranon kernel: [137969.382572] [<c0174914>] sys_faccessat+0x11e/0x15d
Aug 4 06:27:28 iranon kernel: [137969.382582] [<c0178703>] sys_lstat64+0xf/0x23
Aug 4 06:27:28 iranon kernel: [137969.382588] [<c017603c>] vfs_read+0xe3/0x11e
Aug 4 06:27:28 iranon kernel: [137969.382598] [<c0174962>] sys_access+0xf/0x13
Aug 4 06:27:28 iranon kernel: [137969.382603] [<c01077e8>] sysenter_past_esp+0x6d/0xa5
Aug 4 06:27:28 iranon kernel: [137969.382617] =======================
Sysrq-T from dmesg (therefore truncated a bit) is attached.
Ian.
--
Ian Campbell
There are no answers, only cross-references.
-- Weiner
[-- Attachment #2: sysrq --]
[-- Type: text/plain, Size: 124790 bytes --]
d+0x3c/0x63
[148193.520887] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520887] =======================
[148193.520887] mysqld S 005fdb20 0 3378 2818
[148193.520887] f3be45c0 00000082 f3b7b200 005fdb20 00006034 f3be474c c1c0efa0 00000000
[148193.520887] 00000000 f3bf3e48 000361ad 00000000 00000000 00000001 f7339b78 00000286
[148193.520887] 7fffffff 00000000 f3bf1e5c f3bb1ac0 c02b7ea9 f78cd840 000000a0 f78cd840
[148193.520887] Call Trace:
[148193.520887] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520887] [<c0253e7d>] __alloc_skb+0x49/0xf7
[148193.520887] [<c013187f>] prepare_to_wait+0x12/0x49
[148193.520887] [<c02a7004>] unix_stream_recvmsg+0x1e3/0x480
[148193.520887] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520887] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520887] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520887] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520887] [<f889008a>] ide_intr+0x190/0x19b [ide_core]
[148193.520887] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520887] [<c0174778>] vfs_read+0x92/0x11e
[148193.520887] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520887] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520887] =======================
[148193.520887] mysqld S 8a931ce4 0 3529 2818
[148193.520887] f3424ce0 00000082 f7a639e0 8a931ce4 000086c5 f3424e6c c1c0efa0 00000000
[148193.520887] f74019a0 023420f5 00008e45 00000000 00000000 00000000 00000000 000000ff
[148193.520887] 7fffffff 00000000 f2ef3e5c f368d040 c02b7ea9 f3eda260 000008e0 f3eda260
[148193.520887] Call Trace:
[148193.520887] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520887] [<c0253e7d>] __alloc_skb+0x49/0xf7
[148193.520887] [<c013187f>] prepare_to_wait+0x12/0x49
[148193.520887] [<c02a7004>] unix_stream_recvmsg+0x1e3/0x480
[148193.520887] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520887] [<c0137502>] clocksource_get_next+0x39/0x3f
[148193.520887] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520887] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520887] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520887] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520887] [<c0174778>] vfs_read+0x92/0x11e
[148193.520887] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520887] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520887] =======================
[148193.520887] mysqld S 93aced7a 0 3530 2818
[148193.520887] f34244a0 00000082 00000001 93aced7a 000083f1 f342462c c1c0efa0 00000000
[148193.520887] 00000001 00000001 00278dd9 00000000 f2ec1e88 00000cda 00000000 00000246
[148193.520887] f2ec1eb0 00000000 c04037ac 087448e4 c013a642 00000210 00000210 00000000
[148193.520887] Call Trace:
[148193.520887] [<c013a642>] futex_wait+0x199/0x31d
[148193.520887] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520887] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520887] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520887] [<c0182157>] d_kill+0x37/0x46
[148193.520887] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520887] [<c017289d>] filp_close+0x4d/0x53
[148193.520887] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520887] =======================
[148193.520887] mysqld S bb6ed648 0 4898 2818
[148193.520887] f79220e0 00000082 00000001 bb6ed648 000083fc f792226c c1c0efa0 00000000
[148193.520887] 00000001 00000001 0001ab0e 00000000 f7947e88 000035c5 00000000 00000246
[148193.520887] f7947eb0 00000000 c04037ac 087448e4 c013a642 00000212 00000212 00000000
[148193.520887] Call Trace:
[148193.520887] [<c013a642>] futex_wait+0x199/0x31d
[148193.520887] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520887] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520887] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520887] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520887] [<c0182157>] d_kill+0x37/0x46
[148193.520887] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520887] [<c017289d>] filp_close+0x4d/0x53
[148193.520887] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520887] =======================
[148193.520887] mysqld S 69331d39 0 4899 2818
[148193.520887] f7923580 00000082 00000001 69331d39 00007742 f792370c c1c0efa0 00000000
[148193.520887] 00000001 00000001 000083ae 00000000 f2f19e88 0000467a 00000000 00000246
[148193.520887] f2f19eb0 00000000 c04037ac 087448e4 c013a642 0000020b 0000020b 00000000
[148193.520887] Call Trace:
[148193.520887] [<c013a642>] futex_wait+0x199/0x31d
[148193.520887] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520887] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520887] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520887] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520887] [<c0182157>] d_kill+0x37/0x46
[148193.520887] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520887] [<c017289d>] filp_close+0x4d/0x53
[148193.520887] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520887] =======================
[148193.520887] mysqld S 693158ed 0 4979 2818
[148193.520887] f34248c0 00000082 00000001 693158ed 00007742 f3424a4c c1c0efa0 00000000
[148193.520887] 00000001 00000001 0000781b 00000000 f2ef7e88 0000467a 00000000 00000246
[148193.520887] f2ef7eb0 00000000 c04037ac 087448e4 c013a642 0000020a 0000020a 00000000
[148193.520887] Call Trace:
[148193.520887] [<c013a642>] futex_wait+0x199/0x31d
[148193.520887] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520887] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520887] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520887] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520887] [<c0182157>] d_kill+0x37/0x46
[148193.520887] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520887] [<c017289d>] filp_close+0x4d/0x53
[148193.520887] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520887] =======================
[148193.520887] mysqld S 6941a010 0 4980 2818
[148193.520887] f3424080 00000082 00000001 6941a010 00007742 f342420c c1c0efa0 00000000
[148193.520887] 00000001 00000001 0000aeba 00000000 f2f9de88 0000467a 00000000 00000246
[148193.520888] f2f9deb0 00000000 c04037ac 087448e4 c013a642 0000020c 0000020c 00000000
[148193.520888] Call Trace:
[148193.520888] [<c013a642>] futex_wait+0x199/0x31d
[148193.520888] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520888] [<c0182157>] d_kill+0x37/0x46
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c017289d>] filp_close+0x4d/0x53
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] logger S 2f87513b 0 2859 2818
[148193.520888] f7a735e0 00000086 d662d853 2f87513b 00000007 f7a7376c c1c0efa0 00000000
[148193.520888] f3f63f04 f740eea0 0003fbbc 00000000 c0182366 f7169884 00000246 c013187f
[148193.520888] f3f75200 00001000 f3f63e84 f3f63fa0 c017967e 00000000 f7a735e0 c0131774
[148193.520888] Call Trace:
[148193.520888] [<c0182366>] dput+0x15/0xbb
[148193.520888] [<c013187f>] prepare_to_wait+0x12/0x49
[148193.520888] [<c017967e>] pipe_wait+0x51/0x6f
[148193.520888] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520888] [<c0179d50>] pipe_read+0x2bf/0x32b
[148193.520888] [<c017c34f>] path_walk+0x67/0x70
[148193.520888] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520888] [<c0176a3d>] cp_new_stat64+0xfc/0x10e
[148193.520888] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520888] [<c011d36d>] hrtick_set+0x8f/0xd8
[148193.520888] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520888] [<c0173f20>] do_sync_read+0x0/0xfe
[148193.520888] [<c0174767>] vfs_read+0x81/0x11e
[148193.520888] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] inetd S a145ef04 0 2968 1
[148193.520888] f78984e0 00000082 c034d300 a145ef04 000058d4 f789866c c1c0efa0 00000000
[148193.520888] c036e2e0 0233b2df 00056aa8 00000000 00000000 00000000 00000000 000000ff
[148193.520888] 7fffffff f77460a0 00000010 f778bf9c c02b7ea9 c01042a7 f778be5c 00000203
[148193.520888] Call Trace:
[148193.520888] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520888] [<c01042a7>] common_interrupt+0x23/0x28
[148193.520888] [<c01318c2>] remove_wait_queue+0xc/0x34
[148193.520888] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520888] [<c027b144>] tcp_poll+0x14/0x10e
[148193.520888] [<c017f09a>] do_select+0x364/0x3bd
[148193.520888] [<c017f59e>] __pollwait+0x0/0xac
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c029df63>] bictcp_cong_avoid+0x12/0x2a1
[148193.520888] [<c022016b>] mix_pool_bytes_extract+0x5c/0x165
[148193.520888] [<c027e2ad>] tcp_cong_avoid+0xd/0x1b
[148193.520888] [<c0280bd2>] tcp_ack+0x15c4/0x1757
[148193.520888] [<c029df63>] bictcp_cong_avoid+0x12/0x2a1
[148193.520888] [<c029df63>] bictcp_cong_avoid+0x12/0x2a1
[148193.520888] [<c027e2ad>] tcp_cong_avoid+0xd/0x1b
[148193.520888] [<c0280bd2>] tcp_ack+0x15c4/0x1757
[148193.520888] [<c01e0532>] number+0x13d/0x217
[148193.520888] [<c0283dbc>] tcp_current_mss+0xaa/0xc8
[148193.520888] [<c029df63>] bictcp_cong_avoid+0x12/0x2a1
[148193.520888] [<c027e2ad>] tcp_cong_avoid+0xd/0x1b
[148193.520888] [<c0280bd2>] tcp_ack+0x15c4/0x1757
[148193.520888] [<c0283dbc>] tcp_current_mss+0xaa/0xc8
[148193.520888] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520888] [<c0288920>] tcp_v4_do_rcv+0x37f/0x3e8
[148193.520888] [<c010f4b1>] flush_tlb_page+0x3e/0x62
[148193.520888] [<c011731a>] ptep_set_access_flags+0x52/0x5c
[148193.520888] [<c0161e6f>] do_wp_page+0x1da/0x49f
[148193.520888] [<f887846c>] e1000_clean_rx_irq+0x429/0x46f [e1000]
[148193.520888] [<c0163e87>] handle_mm_fault+0x623/0x690
[148193.520888] [<f887880a>] e1000_clean+0xf7/0x215 [e1000]
[148193.520888] [<c017f718>] sys_select+0xce/0x180
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] sensord S daed64ed 0 2984 1
[148193.520888] f749e0c0 00000082 c1c0c2f8 daed64ed 000086c4 f749e24c c1c0efa0 00000000
[148193.520888] f3811f48 c1c0c2f8 0021726f 00000000 d334ba31 000086d2 00000282 f3811f48
[148193.520888] f3811f48 00000001 bfddf9b4 f3810000 c02b8363 00000000 00000001 c0134240
[148193.520888] Call Trace:
[148193.520888] [<c02b8363>] do_nanosleep+0x51/0x87
[148193.520888] [<c0134240>] hrtimer_nanosleep+0x39/0x96
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c02b834a>] do_nanosleep+0x38/0x87
[148193.520888] [<c01342de>] sys_nanosleep+0x41/0x51
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] smartd S 00000000 0 3001 1
[148193.520888] f75509e0 00000086 c1c0c2f8 00000000 c0133c26 f7550b6c c1c0efa0 00000000
[148193.520888] f3817f48 023208c3 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] f3817f48 00000001 bf96f044 f3816000 c02b8363 00000000 00000001 c0134240
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c02b8363>] do_nanosleep+0x51/0x87
[148193.520888] [<c0134240>] hrtimer_nanosleep+0x39/0x96
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c02b834a>] do_nanosleep+0x38/0x87
[148193.520888] [<c01342de>] sys_nanosleep+0x41/0x51
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] uptimed S 00000000 0 3024 1
[148193.520888] f77a20e0 00000086 c1c0c2f8 00000000 c0133c26 f77a226c c1c0efa0 00000000
[148193.520888] f3fcdf48 023419cc 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] f3fcdf48 00000001 bf98edc4 f3fcc000 c02b8363 00000000 00000001 c0134240
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c02b8363>] do_nanosleep+0x51/0x87
[148193.520888] [<c0134240>] hrtimer_nanosleep+0x39/0x96
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c02b834a>] do_nanosleep+0x38/0x87
[148193.520888] [<c01342de>] sys_nanosleep+0x41/0x51
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] xfs S f3bcee20 0 3043 1
[148193.520888] f749e900 00000086 c034d300 f3bcee20 00008645 f749ea8c c1c0efa0 00000000
[148193.520888] c0129950 f3fe9b5c 00019ada 00000000 00000286 c0129a63 00000000 00000286
[148193.520888] 023453c6 023453c6 00000040 f3fe9f9c c02b7f01 fa10bf00 fa10ec70 023453c6
[148193.520888] Call Trace:
[148193.520888] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520888] [<c0129a63>] __mod_timer+0x99/0xa3
[148193.520888] [<c02b7f01>] schedule_timeout+0x6b/0x86
[148193.520888] [<c012968c>] process_timeout+0x0/0x5
[148193.520888] [<c02b7efc>] schedule_timeout+0x66/0x86
[148193.520888] [<c017f09a>] do_select+0x364/0x3bd
[148193.520888] [<c017f59e>] __pollwait+0x0/0xac
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<f888fbdf>] ide_do_request+0x5f2/0x81b [ide_core]
[148193.520888] [<f888fda3>] ide_do_request+0x7b6/0x81b [ide_core]
[148193.520888] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013435a>] ktime_get_ts+0x22/0x49
[148193.520888] [<c013438e>] ktime_get+0xd/0x21
[148193.520888] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520888] [<f88426b7>] ehci_irq+0x108/0x128 [ehci_hcd]
[148193.520888] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520888] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013435a>] ktime_get_ts+0x22/0x49
[148193.520888] [<c013438e>] ktime_get+0xd/0x21
[148193.520888] [<c0119081>] hrtick_start_fair+0xeb/0x12c
[148193.520888] [<c011fc07>] check_preempt_wakeup+0xa2/0xc2
[148193.520888] [<c011b614>] try_to_wake_up+0xe8/0xf1
[148193.520888] [<c01183ee>] __wake_up_common+0x2e/0x58
[148193.520888] [<c011a59f>] __wake_up_sync+0x2a/0x3e
[148193.520888] [<c02a8175>] unix_write_space+0x3a/0x63
[148193.520888] [<c0253396>] skb_dequeue+0x42/0x48
[148193.520888] [<c02a71b1>] unix_stream_recvmsg+0x390/0x480
[148193.520888] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520888] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520888] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520888] [<c01df80c>] rb_insert_color+0x8c/0xad
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c017f6e9>] sys_select+0x9f/0x180
[148193.520888] [<c0135f76>] do_gettimeofday+0xd/0x27
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] ntpd S c0152eff 0 3155 1
[148193.520888] f744f0e0 00200082 c034d300 c0152eff 000006c0 f744f26c c1c0efa0 00000000
[148193.520888] c0105f3f 02342b05 f3fc1be4 00000000 00000000 00000000 00000000 000000ff
[148193.520888] 7fffffff f788f8c0 00200000 f3fc1f9c c02b7ea9 f792d260 f3fc1be4 00200246
[148193.520888] Call Trace:
[148193.520888] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520888] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520888] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520888] [<c0255831>] datagram_poll+0x14/0xab
[148193.520888] [<c028cd49>] udp_poll+0x10/0xd5
[148193.520888] [<c017f09a>] do_select+0x364/0x3bd
[148193.520888] [<c017f59e>] __pollwait+0x0/0xac
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013435a>] ktime_get_ts+0x22/0x49
[148193.520888] [<c013438e>] ktime_get+0xd/0x21
[148193.520888] [<c0119081>] hrtick_start_fair+0xeb/0x12c
[148193.520888] [<c011fc07>] check_preempt_wakeup+0xa2/0xc2
[148193.520888] [<c011b614>] try_to_wake_up+0xe8/0xf1
[148193.520888] [<c01183ee>] __wake_up_common+0x2e/0x58
[148193.520888] [<c011a567>] complete+0x28/0x36
[148193.520888] [<f8858df0>] usb_hcd_giveback_urb+0x64/0x8f [usbcore]
[148193.520888] [<c016c583>] dma_pool_free+0x57/0xe9
[148193.520888] [<f883f3fd>] qh_completions+0x9a/0x304 [ehci_hcd]
[148193.520888] [<c024c850>] pci_conf1_read+0xa4/0xac
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520888] [<c0109237>] save_i387+0x123/0x139
[148193.520888] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520888] [<c0102b85>] setup_sigcontext+0x105/0x189
[148193.520888] [<c012a0aa>] recalc_sigpending+0xb/0x2c
[148193.520888] [<c01034ad>] do_notify_resume+0x565/0x649
[148193.520888] [<c0109020>] restore_i387+0x9f/0xda
[148193.520888] [<c017f718>] sys_select+0xce/0x180
[148193.520888] [<c0102d2e>] restore_sigcontext+0x113/0x168
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 1985ddd1 0 3168 1
[148193.520888] f77a2500 00000082 c034d300 1985ddd1 000086c0 f77a268c c1c0efa0 00000000
[148193.520888] c0129950 02342776 023444c1 f3807b5c 00000000 00000000 00000000 000000ff
[148193.520888] 023444c1 023444c1 00002000 f3807f9c c02b7f01 c03f258c c03f258c 023444c1
[148193.520888] Call Trace:
[148193.520888] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520888] [<c02b7f01>] schedule_timeout+0x6b/0x86
[148193.520888] [<c012968c>] process_timeout+0x0/0x5
[148193.520888] [<c02b7efc>] schedule_timeout+0x66/0x86
[148193.520888] [<c017f09a>] do_select+0x364/0x3bd
[148193.520888] [<c017f59e>] __pollwait+0x0/0xac
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c017f59e>] __pollwait+0x0/0xac
[148193.520888] [<c0257d8e>] netif_receive_skb+0x2d6/0x343
[148193.520888] [<c025a139>] process_backlog+0x74/0xce
[148193.520888] [<c0259a7e>] net_rx_action+0x9c/0x1b9
[148193.520888] [<c0126419>] __do_softirq+0x66/0xd3
[148193.520888] [<c029df63>] bictcp_cong_avoid+0x12/0x2a1
[148193.520888] [<c0275d31>] ip_finish_output+0x1c5/0x1fc
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520888] [<f88426b7>] ehci_irq+0x108/0x128 [ehci_hcd]
[148193.520888] [<c01110b8>] ack_ioapic_quirk_irq+0x34/0x85
[148193.520888] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520888] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520888] [<c012679d>] irq_exit+0x50/0x67
[148193.520888] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520888] [<c024c850>] pci_conf1_read+0xa4/0xac
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabb2ab5>] _nv008482rm+0x1c/0x20 [nvidia]
[148193.520888] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520888] [<fac9c410>] nv_kern_isr+0x46/0x78 [nvidia]
[148193.520888] [<c01110b8>] ack_ioapic_quirk_irq+0x34/0x85
[148193.520888] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520888] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520888] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520888] [<c024c850>] pci_conf1_read+0xa4/0xac
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520888] [<fabf33ea>] _nv003983rm+0x1d5/0x212 [nvidia]
[148193.520888] [<fabf1bd2>] _nv003976rm+0x3d/0x65 [nvidia]
[148193.520888] [<fabf7ed2>] rm_isr+0x9c/0xef [nvidia]
[148193.520888] [<c017f6e9>] sys_select+0x9f/0x180
[148193.520888] [<c0135f76>] do_gettimeofday+0xd/0x27
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 3229efef 0 3232 1
[148193.520888] f7858080 00000082 c1c0c2f8 3229efef 00008685 f785820c c1c0efa0 00000000
[148193.520888] f3febf48 c1c0c2f8 0000d2a1 00000000 35949909 000086cc 00000282 f3febf48
[148193.520888] f3febf48 00000001 b4747174 f3fea000 c02b8363 00000000 00000001 c0134240
[148193.520888] Call Trace:
[148193.520888] [<c02b8363>] do_nanosleep+0x51/0x87
[148193.520888] [<c0134240>] hrtimer_nanosleep+0x39/0x96
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c02b834a>] do_nanosleep+0x38/0x87
[148193.520888] [<c01342de>] sys_nanosleep+0x41/0x51
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S d999efa2 0 3262 1
[148193.520888] f75505c0 00000082 c1c0c2f8 d999efa2 000086c7 f755074c c1c0efa0 00000000
[148193.520888] f3853e5c c1c0c2f8 000026f6 00000000 1534bbe7 000086c8 00000282 f3853f98
[148193.520888] 00000000 00000000 f3853e5c 08c350d4 c013a6a5 00048517 00048517 00000000
[148193.520888] Call Trace:
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c011d36d>] hrtick_set+0x8f/0xd8
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S e9acbdfd 0 3263 1
[148193.520888] f749f980 00000082 c1c0c2f8 e9acbdfd 000086c7 f749fb0c c1c0efa0 00000000
[148193.520888] f386fe5c c1c0c2f8 00001379 00000000 01843d07 000086c8 00000282 f386ff98
[148193.520888] 00000000 00000000 f386fe5c 08c4858c c013a6a5 000b4d6b 000b4d6b 00000000
[148193.520888] Call Trace:
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c010fc69>] lapic_next_event+0xc/0x10
[148193.520888] [<c0138608>] clockevents_program_event+0xc4/0xd2
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S f52b36b1 0 3264 1
[148193.520888] f77a3160 00000082 c1c0c2f8 f52b36b1 000086c7 f77a32ec c1c0efa0 00000000
[148193.520888] f3863e5c c1c0c2f8 00000005 00000000 30c67d65 000086c8 00000282 f3863f98
[148193.520888] 00000000 00000000 f3863e5c 08c2999c c013a6a5 00048561 00048561 00000000
[148193.520888] Call Trace:
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c011d36d>] hrtick_set+0x8f/0xd8
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 00000000 0 3298 1
[148193.520888] f744e480 00000082 c1c0c2f8 00000000 c0133c26 f744e60c c1c0efa0 00000000
[148193.520888] f38cde5c 02342aab 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] 00000000 00000000 f38cde5c 08c3b674 c013a6a5 00048505 00048505 00000000
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c0119bba>] __dequeue_entity+0x1f/0x71
[148193.520888] [<c011d358>] hrtick_set+0x7a/0xd8
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 00000000 0 3299 1
[148193.520888] f78880a0 00000082 c1c0c2f8 00000000 c0133c26 f788822c c1c0efa0 00000000
[148193.520888] f38cfe5c 02342aef 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] 00000000 00000000 f38cfe5c 08c7c96c c013a6a5 000b4d67 000b4d67 00000000
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c010fc69>] lapic_next_event+0xc/0x10
[148193.520888] [<c0138608>] clockevents_program_event+0xc4/0xd2
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 00000000 0 3302 1
[148193.520888] f7551a60 00000082 c1c0c2f8 00000000 c0133c26 f7551bec c1c0efa0 00000000
[148193.520888] f38a9e5c 02342b14 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] 00000000 00000000 f38a9e5c 08c3daa4 c013a6a5 0004855d 0004855d 00000000
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c013435a>] ktime_get_ts+0x22/0x49
[148193.520888] [<c010296e>] __switch_to+0xa5/0x125
[148193.520888] [<c011d358>] hrtick_set+0x7a/0xd8
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S d999fb3b 0 3305 1
[148193.520888] f78888e0 00000082 c1c0c2f8 d999fb3b 000086c7 f7888a6c c1c0efa0 00000000
[148193.520888] f38d5e5c c1c0c2f8 0000ae39 00000000 1534cb52 000086c8 00000282 f38d5f98
[148193.520888] 00000000 00000000 f38d5e5c 08c3e834 c013a6a5 00048557 00048557 00000000
[148193.520888] Call Trace:
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c011d36d>] hrtick_set+0x8f/0xd8
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 00000000 0 3306 1
[148193.520888] f7a72560 00000082 c1c0c2f8 00000000 c0133c26 f7a726ec c1c0efa0 00000000
[148193.520888] f38d7e5c 02342ae6 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] 00000000 00000000 f38d7e5c 08c9011c c013a6a5 000b4d61 000b4d61 00000000
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520888] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520888] [<c011c59e>] rebalance_domains+0x6d/0x355
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 00000000 0 3311 1
[148193.520888] f7a73a00 00000082 c1c0c2f8 00000000 c0133c26 f7a73b8c c1c0efa0 00000000
[148193.520888] f39dbe5c 02342b1e 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] 00000000 00000000 f39dbe5c 08c3ed8c c013a6a5 0004855b 0004855b 00000000
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c0119bba>] __dequeue_entity+0x1f/0x71
[148193.520888] [<c011d358>] hrtick_set+0x7a/0xd8
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S d99a3be6 0 3316 1
[148193.520888] f7859100 00000082 c1c0c2f8 d99a3be6 000086c7 f785928c c1c0efa0 00000000
[148193.520888] f3b2be5c c1c0c2f8 0000c83c 00000000 153509cc 000086c8 00000282 f3b2bf98
[148193.520888] 00000000 00000000 f3b2be5c 08c3c204 c013a6a5 00048553 00048553 00000000
[148193.520888] Call Trace:
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c011d36d>] hrtick_set+0x8f/0xd8
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 00000000 0 3317 1
[148193.520888] f74af160 00000082 c1c0c2f8 00000000 c0133c26 f74af2ec c1c0efa0 00000000
[148193.520888] f3b2de5c 02342adf 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] 00000000 00000000 f3b2de5c 08cc42a4 c013a6a5 000b4d55 000b4d55 00000000
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c01294f8>] run_timer_softirq+0x11a/0x17c
[148193.520888] [<c011f9be>] run_rebalance_domains+0x28/0x98
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S f3367837 0 3326 1
[148193.520888] f74ae920 00000082 c1c0c2f8 f3367837 000086c7 f74aeaac c1c0efa0 00000000
[148193.520888] f3b35e5c c1c0c2f8 00000d94 00000000 2ed13c9c 000086c8 00000282 f3b35f98
[148193.520888] 00000000 00000000 f3b35e5c 08c3cb5c c013a6a5 00048557 00048557 00000000
[148193.520888] Call Trace:
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c011d36d>] hrtick_set+0x8f/0xd8
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 00000000 0 3328 1
[148193.520888] f7a62d80 00000082 c1c0c2f8 00000000 c0133c26 f7a62f0c c1c0efa0 00000000
[148193.520888] f3b57e5c 02342ab8 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] 00000000 00000000 f3b57e5c 08c289ac c013a6a5 0004730b 0004730b 00000000
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c010fc69>] lapic_next_event+0xc/0x10
[148193.520888] [<c0138608>] clockevents_program_event+0xc4/0xd2
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend D 272c2867 0 3329 1
[148193.520888] f78588c0 00000082 c034d300 272c2867 000086c1 f7858a4c c1c0efa0 00000000
[148193.520888] 00000012 02339963 000022b1 00000000 00000000 00000000 00000000 00000246
[148193.520888] f3b31d7c 00000000 f3b31d84 c1c00dd0 fa0b27a9 c02b803e fa0b2784 f3ee5488
[148193.520888] Call Trace:
[148193.520888] [<fa0b27a9>] rpc_wait_bit_killable+0x25/0x2a [sunrpc]
[148193.520888] [<c02b803e>] __wait_on_bit+0x33/0x58
[148193.520888] [<fa0b2784>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
[148193.520888] [<fa0b2784>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
[148193.520888] [<c02b80c2>] out_of_line_wait_on_bit+0x5f/0x67
[148193.520888] [<c01317a1>] wake_bit_function+0x0/0x3c
[148193.520888] [<fa0b2bea>] __rpc_execute+0xbe/0x1d9 [sunrpc]
[148193.520888] [<fa0ada72>] rpc_run_task+0x40/0x45 [sunrpc]
[148193.520888] [<fa0adb00>] rpc_call_sync+0x38/0x52 [sunrpc]
[148193.520888] [<fa12e23d>] nfs3_rpc_wrapper+0x14/0x49 [nfs]
[148193.520888] [<fa12e4fe>] nfs3_proc_getattr+0x52/0x73 [nfs]
[148193.520888] [<fa1242fd>] __nfs_revalidate_inode+0xed/0x211 [nfs]
[148193.520888] [<c017a2ac>] do_lookup+0x53/0x153
[148193.520888] [<c0186774>] mntput_no_expire+0x13/0xd9
[148193.520888] [<c017c1f4>] __link_path_walk+0xa17/0xb0b
[148193.520888] [<c0186774>] mntput_no_expire+0x13/0xd9
[148193.520888] [<c0186774>] mntput_no_expire+0x13/0xd9
[148193.520888] [<c017c34f>] path_walk+0x67/0x70
[148193.520888] [<fa124a5e>] nfs_getattr+0x8f/0xbe [nfs]
[148193.520888] [<fa1249cf>] nfs_getattr+0x0/0xbe [nfs]
[148193.520888] [<c0176e52>] vfs_getattr+0x36/0x4d
[148193.520888] [<c0176f2a>] vfs_lstat_fd+0x27/0x39
[148193.520888] [<fa120d73>] nfs_permission+0x0/0x11d [nfs]
[148193.520888] [<c01730c2>] sys_faccessat+0x14f/0x15b
[148193.520888] [<c0176fa9>] sys_lstat64+0xf/0x23
[148193.520888] [<c01747c9>] vfs_read+0xe3/0x11e
[148193.520888] [<c01730dd>] sys_access+0xf/0x13
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 8a9e1aa3 0 3333 1
[148193.520888] f7a639e0 00000082 c1c0c2f8 8a9e1aa3 000086c5 f7a63b6c c1c0efa0 00000000
[148193.520888] f3b5df48 c1c0c2f8 000aebde 00000000 82e56cda 000086d3 00000282 f3b5df48
[148193.520888] f3b5df48 00000001 acf37de4 f3b5c000 c02b8363 00000000 00000001 c0134240
[148193.520888] Call Trace:
[148193.520888] [<c02b8363>] do_nanosleep+0x51/0x87
[148193.520888] [<c0134240>] hrtimer_nanosleep+0x39/0x96
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c02b834a>] do_nanosleep+0x38/0x87
[148193.520888] [<c01342de>] sys_nanosleep+0x41/0x51
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 00000000 0 3335 1
[148193.520888] f7a635c0 00000082 c1c0c2f8 00000000 c0133c26 f7a6374c c1c0efa0 00000000
[148193.520888] f3b59e5c 02342b1e 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] 00000000 00000000 f3b59e5c 08ccc9a4 c013a6a5 00090aa9 00090aa9 00000000
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c013438e>] ktime_get+0xd/0x21
[148193.520888] [<c010296e>] __switch_to+0xa5/0x125
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 8d4f3f25 0 3336 1
[148193.520888] f7a62540 00000082 f77a2500 8d4f3f25 0000000b f7a626cc c1c0efa0 00000000
[148193.520888] 00000000 001200d2 00000bc0 00000000 00000001 00000001 00000000 00000246
[148193.520888] f3b5feb0 00000000 c04042c4 08c34194 c013a642 00000003 00000003 00000000
[148193.520888] Call Trace:
[148193.520888] [<c013a642>] futex_wait+0x199/0x31d
[148193.520888] [<c015d523>] mark_page_accessed+0x18/0x2b
[148193.520888] [<c0158670>] filemap_fault+0x1f5/0x359
[148193.520888] [<c0117c11>] kunmap_atomic+0x67/0x8a
[148193.520888] [<c0161c56>] __do_fault+0x30e/0x34d
[148193.520888] [<c011b614>] try_to_wake_up+0xe8/0xf1
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c017289d>] filp_close+0x4d/0x53
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c01029b1>] __switch_to+0xe8/0x125
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 8d4fe72c 0 3337 1
[148193.520888] f7858ce0 00000082 f77a2500 8d4fe72c 0000000b f7858e6c c1c0efa0 00000000
[148193.520888] 00000000 001200d2 000006e8 00000000 000408b4 f3b33e64 00000000 00000246
[148193.520888] f3b33eb0 00000000 c0404238 08cd9d6c c013a642 00000003 00000003 00000000
[148193.520888] Call Trace:
[148193.520888] [<c013a642>] futex_wait+0x199/0x31d
[148193.520888] [<c013438e>] ktime_get+0xd/0x21
[148193.520888] [<c01df7cc>] rb_insert_color+0x4c/0xad
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c017289d>] filp_close+0x4d/0x53
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c010296e>] __switch_to+0xa5/0x125
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 8d508441 0 3338 1
[148193.520888] f3b5ba20 00000082 f77a2500 8d508441 0000000b f3b5bbac c1c0efa0 00000000
[148193.520888] 00000000 00000000 00000631 00000000 00000000 f3b61e64 00000000 00000246
[148193.520888] f3b61eb0 00000000 c0403a40 08cd73bc c013a642 00000003 00000003 00000000
[148193.520888] Call Trace:
[148193.520888] [<c013a642>] futex_wait+0x199/0x31d
[148193.520888] [<c013438e>] ktime_get+0xd/0x21
[148193.520888] [<c01df7cc>] rb_insert_color+0x4c/0xad
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c010296e>] __switch_to+0xa5/0x125
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S f3b5b600 0 3339 1
[148193.520888] f3b5b600 00000082 c0119081 f3b5b600 f3b5b600 f3b5b78c c1c0efa0 00000000
[148193.520888] f3b7b200 02285426 00000001 c011fc07 00000000 00000000 00000000 000000ff
[148193.520888] f3b63eb0 00000000 c0403f18 08cd2f9c c013a642 00001385 00001385 00000000
[148193.520888] Call Trace:
[148193.520888] [<c0119081>] hrtick_start_fair+0xeb/0x12c
[148193.520888] [<c011fc07>] check_preempt_wakeup+0xa2/0xc2
[148193.520888] [<c013a642>] futex_wait+0x199/0x31d
[148193.520888] [<c0184bf6>] file_update_time+0x83/0xac
[148193.520888] [<c0187278>] mnt_drop_write+0x1a/0xb4
[148193.520888] [<c0179a15>] pipe_write+0x379/0x3c7
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520888] [<c013438e>] ktime_get+0xd/0x21
[148193.520888] [<c010296e>] __switch_to+0xa5/0x125
[148193.520888] [<c011d358>] hrtick_set+0x7a/0xd8
[148193.520888] [<c02b7cd7>] schedule+0x63d/0x66f
[148193.520888] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520888] [<c0174c1b>] sys_write+0x3c/0x63
[148193.520888] [<c0174c3a>] sys_write+0x5b/0x63
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend D 626f0644 0 3340 1
[148193.520888] f3b5b1e0 00000082 c314d000 626f0644 000083f3 f3b5b36c c1c0efa0 00000000
[148193.520888] 00000000 00000000 012cf0b9 00000000 f3ee22bc f3ee5ee0 f78b3840 00000246
[148193.520888] f3b65ccc 00000000 f3b65cd4 c1c01010 fa0b27a9 c02b803e fa0b2784 f3ee5f28
[148193.520888] Call Trace:
[148193.520888] [<fa0b27a9>] rpc_wait_bit_killable+0x25/0x2a [sunrpc]
[148193.520888] [<c02b803e>] __wait_on_bit+0x33/0x58
[148193.520888] [<fa0b2784>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
[148193.520888] [<fa0b2784>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
[148193.520888] [<c02b80c2>] out_of_line_wait_on_bit+0x5f/0x67
[148193.520888] [<c01317a1>] wake_bit_function+0x0/0x3c
[148193.520888] [<fa0b2bea>] __rpc_execute+0xbe/0x1d9 [sunrpc]
[148193.520888] [<fa0ada72>] rpc_run_task+0x40/0x45 [sunrpc]
[148193.520888] [<fa0adb00>] rpc_call_sync+0x38/0x52 [sunrpc]
[148193.520888] [<fa12e23d>] nfs3_rpc_wrapper+0x14/0x49 [nfs]
[148193.520888] [<fa12e76f>] nfs3_proc_access+0x101/0x166 [nfs]
[148193.520888] [<c011fc07>] check_preempt_wakeup+0xa2/0xc2
[148193.520888] [<c011b614>] try_to_wake_up+0xe8/0xf1
[148193.520888] [<c0131781>] autoremove_wake_function+0xd/0x2d
[148193.520888] [<c01183ee>] __wake_up_common+0x2e/0x58
[148193.520888] [<c011a59f>] __wake_up_sync+0x2a/0x3e
[148193.520888] [<fa120c00>] nfs_do_access+0x134/0x2a7 [nfs]
[148193.520888] [<fa0b36d8>] rpcauth_lookupcred+0x69/0x8f [sunrpc]
[148193.520888] [<fa120e30>] nfs_permission+0xbd/0x11d [nfs]
[148193.520888] [<fa120d73>] nfs_permission+0x0/0x11d [nfs]
[148193.520888] [<c017a6cd>] permission+0xa3/0xfc
[148193.520888] [<c017b8f9>] __link_path_walk+0x11c/0xb0b
[148193.520888] [<c0161e6f>] do_wp_page+0x1da/0x49f
[148193.520888] [<c017c31f>] path_walk+0x37/0x70
[148193.520888] [<c017c5ce>] do_path_lookup+0x122/0x184
[148193.520888] [<c017ce2b>] __user_walk_fd+0x29/0x3a
[148193.520888] [<c017301a>] sys_faccessat+0xa7/0x15b
[148193.520888] [<f88426b7>] ehci_irq+0x108/0x128 [ehci_hcd]
[148193.520888] [<c01110b8>] ack_ioapic_quirk_irq+0x34/0x85
[148193.520888] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520888] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520888] [<c01730dd>] sys_access+0xf/0x13
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] mythbackend S 54dac9f3 0 3379 1
[148193.520888] f3b7b200 00000082 f77a2500 54dac9f3 000083f5 f3b7b38c c1c0efa0 00000000
[148193.520888] c036e2e0 02287b2e 00006456 00000000 00000000 00000000 00000000 000000ff
[148193.520888] 7fffffff f367c800 00080000 f3bf5f9c c02b7ea9 00020000 00000011 00000207
[148193.520888] Call Trace:
[148193.520888] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520888] [<c01318c2>] remove_wait_queue+0xc/0x34
[148193.520888] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520888] [<c027b144>] tcp_poll+0x14/0x10e
[148193.520888] [<c017f09a>] do_select+0x364/0x3bd
[148193.520888] [<c017f59e>] __pollwait+0x0/0xac
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c0253723>] __kfree_skb+0x8/0x61
[148193.520888] [<c0255bd4>] sk_stream_kill_queues+0xab/0xf7
[148193.520888] [<c027961c>] inet_csk_destroy_sock+0xab/0xf9
[148193.520888] [<c0280fe0>] tcp_fin+0x77/0x15d
[148193.520888] [<c0281aad>] tcp_data_queue+0x285/0xac9
[148193.520888] [<c0129a63>] __mod_timer+0x99/0xa3
[148193.520888] [<c02831c6>] tcp_rcv_state_process+0x93d/0x9b2
[148193.520888] [<c0288920>] tcp_v4_do_rcv+0x37f/0x3e8
[148193.520888] [<c0278a6f>] __inet_lookup_established+0xc4/0x16e
[148193.520888] [<c026197c>] sk_filter+0x63/0x6b
[148193.520888] [<c028a29d>] tcp_v4_rcv+0x292/0x609
[148193.520888] [<c0272927>] ip_local_deliver_finish+0xe8/0x183
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c011d36d>] hrtick_set+0x8f/0xd8
[148193.520888] [<c02b7cd7>] schedule+0x63d/0x66f
[148193.520888] [<c0126702>] local_bh_enable+0x66/0x71
[148193.520888] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520888] [<c0184ce6>] touch_atime+0xc7/0xd1
[148193.520888] [<c0187278>] mnt_drop_write+0x1a/0xb4
[148193.520888] [<c0179dae>] pipe_read+0x31d/0x32b
[148193.520888] [<c024c850>] pci_conf1_read+0xa4/0xac
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabb2ab5>] _nv008482rm+0x1c/0x20 [nvidia]
[148193.520888] [<fabf33ea>] _nv003983rm+0x1d5/0x212 [nvidia]
[148193.520888] [<fabf1bd2>] _nv003976rm+0x3d/0x65 [nvidia]
[148193.520888] [<fabf7ed2>] rm_isr+0x9c/0xef [nvidia]
[148193.520888] [<c017f718>] sys_select+0xce/0x180
[148193.520888] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] atd S 00000000 0 3180 1
[148193.520888] f744ecc0 00000082 c1c0c2f8 00000000 c0133c26 f744ee4c c1c0efa0 00000000
[148193.520888] f3fc3f48 02320f84 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] f3fc3f48 00000001 bf920974 f3fc2000 c02b8363 00000000 00000001 c0134240
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c02b8363>] do_nanosleep+0x51/0x87
[148193.520888] [<c0134240>] hrtimer_nanosleep+0x39/0x96
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c02b834a>] do_nanosleep+0x38/0x87
[148193.520888] [<c01342de>] sys_nanosleep+0x41/0x51
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] cron S 00000000 0 3207 1
[148193.520888] f78584a0 00000082 c1c0c2f8 00000000 c0133c26 f785862c c1c0efa0 00000000
[148193.520888] f3fdff48 0233f236 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520888] f3fdff48 00000001 bff10364 f3fde000 c02b8363 00000000 00000001 c0134240
[148193.520888] Call Trace:
[148193.520888] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520888] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520888] [<c02b8363>] do_nanosleep+0x51/0x87
[148193.520888] [<c0134240>] hrtimer_nanosleep+0x39/0x96
[148193.520888] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520888] [<c02b834a>] do_nanosleep+0x38/0x87
[148193.520888] [<c01342de>] sys_nanosleep+0x41/0x51
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] kdvb-fe-0 S c1c0c2c0 0 3234 2
[148193.520888] f749e4e0 00000046 c034d300 c1c0c2c0 00000002 f749e66c c1c0efa0 00000000
[148193.520888] c0129950 023429d5 02342bc8 f384ff70 00000000 00000000 00000000 000000ff
[148193.520888] 02342bc8 02342bc8 f79379a8 000001d9 c02b7f01 c03f23ac c03f23ac 02342bc8
[148193.520888] Call Trace:
[148193.520888] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520888] [<c02b7f01>] schedule_timeout+0x6b/0x86
[148193.520888] [<c012968c>] process_timeout+0x0/0x5
[148193.520888] [<c02b7efc>] schedule_timeout+0x66/0x86
[148193.520888] [<f92cd3ab>] dvb_frontend_thread+0x12a/0x2a7 [dvb_core]
[148193.520888] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520888] [<f92cd281>] dvb_frontend_thread+0x0/0x2a7 [dvb_core]
[148193.520888] [<c01316b3>] kthread+0x38/0x5d
[148193.520888] [<c013167b>] kthread+0x0/0x5d
[148193.520888] [<c01044f3>] kernel_thread_helper+0x7/0x10
[148193.520888] =======================
[148193.520888] apache2 S e99d6d39 0 3235 1
[148193.520888] f748ed00 00200082 c034d300 e99d6d39 000086c7 f748ee8c c1c0efa0 00000000
[148193.520888] c0129950 f3831b5c 0000468d 00000000 00200286 c0129a63 00000000 00200286
[148193.520888] 02342be8 02342be8 f77d7c2c f3831f9c c02b7f01 c1c0c690 c03f24ac 02342be8
[148193.520888] Call Trace:
[148193.520888] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520888] [<c0129a63>] __mod_timer+0x99/0xa3
[148193.520888] [<c02b7f01>] schedule_timeout+0x6b/0x86
[148193.520888] [<c012968c>] process_timeout+0x0/0x5
[148193.520888] [<c02b7efc>] schedule_timeout+0x66/0x86
[148193.520888] [<c017f09a>] do_select+0x364/0x3bd
[148193.520888] [<f8876e57>] e1000_xmit_frame+0xa59/0xaa3 [e1000]
[148193.520888] [<c017f59e>] __pollwait+0x0/0xac
[148193.520888] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520888] [<c0108214>] nommu_map_single+0x2b/0x43
[148193.520888] [<fabdff58>] _nv004365rm+0x24/0x70 [nvidia]
[148193.520888] [<fabdff84>] _nv004365rm+0x50/0x70 [nvidia]
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabb28fc>] _nv008244rm+0x70/0x95 [nvidia]
[148193.520888] [<c024c850>] pci_conf1_read+0xa4/0xac
[148193.520888] [<c024d73d>] raw_pci_read+0x3d/0x45
[148193.520888] [<c024d80b>] pci_read+0x1c/0x21
[148193.520888] [<c01e4d9c>] pci_bus_read_config_word+0x59/0x64
[148193.520888] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520888] [<fabb2ab5>] _nv008482rm+0x1c/0x20 [nvidia]
[148193.520888] [<fabb2a02>] _nv008484rm+0x21/0x3b [nvidia]
[148193.520888] [<fabb2764>] _nv008168rm+0xbc/0xc6 [nvidia]
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c013435a>] ktime_get_ts+0x22/0x49
[148193.520888] [<c013438e>] ktime_get+0xd/0x21
[148193.520888] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520888] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520888] [<c010fc69>] lapic_next_event+0xc/0x10
[148193.520888] [<c0138608>] clockevents_program_event+0xc4/0xd2
[148193.520888] [<f88426b7>] ehci_irq+0x108/0x128 [ehci_hcd]
[148193.520888] [<c01110b8>] ack_ioapic_quirk_irq+0x34/0x85
[148193.520888] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520888] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520888] [<c012679d>] irq_exit+0x50/0x67
[148193.520888] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520888] [<c01042a7>] common_interrupt+0x23/0x28
[148193.520888] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520888] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520888] [<c0139378>] tick_program_event+0x2b/0x4a
[148193.520888] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520888] [<c0129a63>] __mod_timer+0x99/0xa3
[148193.520888] [<f88e11bd>] cdrom_timer_expiry+0x0/0x4b [ide_cd_mod]
[148193.520888] [<f88e2b59>] cdrom_newpc_intr+0x0/0x54d [ide_cd_mod]
[148193.520888] [<f8890a43>] ide_set_handler+0x2c/0x3e [ide_core]
[148193.520888] [<f88e11bd>] cdrom_timer_expiry+0x0/0x4b [ide_cd_mod]
[148193.520888] [<f88e301a>] cdrom_newpc_intr+0x4c1/0x54d [ide_cd_mod]
[148193.520888] [<f88e11bd>] cdrom_timer_expiry+0x0/0x4b [ide_cd_mod]
[148193.520888] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520888] [<f88e2b59>] cdrom_newpc_intr+0x0/0x54d [ide_cd_mod]
[148193.520888] [<f889008a>] ide_intr+0x190/0x19b [ide_core]
[148193.520888] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520888] [<c01318c2>] remove_wait_queue+0xc/0x34
[148193.520888] [<c0124ad9>] do_wait+0x8c4/0x969
[148193.520888] [<c017f6e9>] sys_select+0x9f/0x180
[148193.520888] [<c0124c06>] sys_wait4+0x88/0x92
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] getty S 8ca24196 0 3251 1
[148193.520888] f747f100 00000082 00000008 8ca24196 0000000a f747f28c c1c0efa0 00000000
[148193.520888] f7750810 f3865f68 00000001 00000000 00000000 00000001 00000002 c1ae3060
[148193.520888] 7fffffff f7750800 f3865f44 bfefdb3b c02b7ea9 57183067 f3eeaf7c b7fdf000
[148193.520888] Call Trace:
[148193.520888] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520888] [<c0162622>] unmap_vmas+0x31a/0x5ad
[148193.520888] [<c0134c00>] up+0x9/0x2a
[148193.520888] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520888] [<c0226370>] read_chan+0x357/0x5d3
[148193.520888] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c0221548>] tty_ldisc_deref+0x4c/0x5b
[148193.520888] [<c0226019>] read_chan+0x0/0x5d3
[148193.520888] [<c02237e0>] tty_read+0x59/0x96
[148193.520888] [<c0223787>] tty_read+0x0/0x96
[148193.520888] [<c0174767>] vfs_read+0x81/0x11e
[148193.520888] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] getty S 8ca24194 0 3253 1
[148193.520888] f77a2d40 00000082 00000008 8ca24194 0000000a f77a2ecc c1c0efa0 00000000
[148193.520888] f74cf810 f386bf68 00000001 00000000 00000000 00000001 00000002 c1ae3060
[148193.520888] 7fffffff f74cf800 f386bf44 bfa1865b c02b7ea9 57183067 f3fa4bf0 b7efc000
[148193.520888] Call Trace:
[148193.520888] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520888] [<c0162622>] unmap_vmas+0x31a/0x5ad
[148193.520888] [<c0134c00>] up+0x9/0x2a
[148193.520888] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520888] [<c0226370>] read_chan+0x357/0x5d3
[148193.520888] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c0221548>] tty_ldisc_deref+0x4c/0x5b
[148193.520888] [<c0226019>] read_chan+0x0/0x5d3
[148193.520888] [<c02237e0>] tty_read+0x59/0x96
[148193.520888] [<c0223787>] tty_read+0x0/0x96
[148193.520888] [<c0174767>] vfs_read+0x81/0x11e
[148193.520888] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] getty S 8ca2418e 0 3255 1
[148193.520888] f7550e00 00000082 00000008 8ca2418e 0000000a f7550f8c c1c0efa0 00000000
[148193.520888] f78b6810 f386df68 00000001 00000000 00000000 00000001 00000002 c1ae3060
[148193.520888] 7fffffff f78b6800 f386df44 bfc638ab c02b7ea9 57183067 f380ed1c b7f47000
[148193.520888] Call Trace:
[148193.520888] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520888] [<c0162622>] unmap_vmas+0x31a/0x5ad
[148193.520888] [<c0134c00>] up+0x9/0x2a
[148193.520888] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520888] [<c0226370>] read_chan+0x357/0x5d3
[148193.520888] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c0221548>] tty_ldisc_deref+0x4c/0x5b
[148193.520888] [<c0226019>] read_chan+0x0/0x5d3
[148193.520888] [<c02237e0>] tty_read+0x59/0x96
[148193.520888] [<c0223787>] tty_read+0x0/0x96
[148193.520888] [<c0174767>] vfs_read+0x81/0x11e
[148193.520888] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] getty S 8ca24192 0 3257 1
[148193.520888] f75501a0 00000082 00000008 8ca24192 0000000a f755032c c1c0efa0 00000000
[148193.520888] f7975810 f3851f68 00000001 00000000 00000000 00000001 00000002 c1ae3060
[148193.520888] 7fffffff f7975800 f3851f44 bfda49eb c02b7ea9 57183067 f3f65e18 b7f86000
[148193.520888] Call Trace:
[148193.520888] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520888] [<c0162622>] unmap_vmas+0x31a/0x5ad
[148193.520888] [<c0134c00>] up+0x9/0x2a
[148193.520888] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520888] [<c0226370>] read_chan+0x357/0x5d3
[148193.520888] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c0221548>] tty_ldisc_deref+0x4c/0x5b
[148193.520888] [<c0226019>] read_chan+0x0/0x5d3
[148193.520888] [<c02237e0>] tty_read+0x59/0x96
[148193.520888] [<c0223787>] tty_read+0x0/0x96
[148193.520888] [<c0174767>] vfs_read+0x81/0x11e
[148193.520888] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] getty S 8ca24190 0 3259 1
[148193.520888] f748e8e0 00000086 00000008 8ca24190 0000000a f748ea6c c1c0efa0 00000000
[148193.520888] f3f53c10 f3855f68 00000001 00000000 00000000 00000001 00000002 c1ae3060
[148193.520888] 7fffffff f3f53c00 f3855f44 bfe3927b c02b7ea9 57183067 f3ff5c74 b7f1d000
[148193.520888] Call Trace:
[148193.520888] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520888] [<c0162622>] unmap_vmas+0x31a/0x5ad
[148193.520888] [<c0134c00>] up+0x9/0x2a
[148193.520888] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520888] [<c0226370>] read_chan+0x357/0x5d3
[148193.520888] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c0221548>] tty_ldisc_deref+0x4c/0x5b
[148193.520888] [<c0226019>] read_chan+0x0/0x5d3
[148193.520888] [<c02237e0>] tty_read+0x59/0x96
[148193.520888] [<c0223787>] tty_read+0x0/0x96
[148193.520888] [<c0174767>] vfs_read+0x81/0x11e
[148193.520888] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] getty S 8d499799 0 3260 1
[148193.520888] f748f540 00000086 00000000 8d499799 0000000a f748f6cc c1c0efa0 00000000
[148193.520888] f77e7810 f3857f68 000116d1 00000000 00000000 00000001 00000002 c1ae3060
[148193.520888] 7fffffff f77e7800 f3857f44 bfced91b c02b7ea9 c0235552 c0418be0 f77e7800
[148193.520888] Call Trace:
[148193.520888] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520888] [<c0235552>] serial8250_start_tx+0x28/0xb6
[148193.520888] [<c0232726>] __uart_start+0x2e/0x30
[148193.520888] [<c0232b16>] uart_start+0x1e/0x2a
[148193.520888] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520888] [<c0226370>] read_chan+0x357/0x5d3
[148193.520888] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c0226019>] read_chan+0x0/0x5d3
[148193.520888] [<c02237e0>] tty_read+0x59/0x96
[148193.520888] [<c0223787>] tty_read+0x0/0x96
[148193.520888] [<c0174767>] vfs_read+0x81/0x11e
[148193.520888] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] su S 00000000 0 3261 1
[148193.520888] f748e4c0 00000082 c1bf9680 00000000 c1000000 f748e64c c1c0efa0 00000000
[148193.520888] f385bb7c ffff0661 00000000 00000000 00000000 00000000 00000000 000000ff
[148193.520888] ffffffea 00000001 f748e4b8 00000000 c0124aa7 00000006 00000000 00000003
[148193.520888] Call Trace:
[148193.520888] [<c0124aa7>] do_wait+0x892/0x969
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c0124bfd>] sys_wait4+0x7f/0x92
[148193.520888] [<c0124c23>] sys_waitpid+0x13/0x17
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] kdvb-fe-1 S 9b2e8f10 0 3265 2
[148193.520888] f747ece0 00000046 c034d300 9b2e8f10 000086ba f747ee6c c1c0efa0 00000000
[148193.520888] c0129950 02342a87 02342c65 f3867f70 00000000 00000000 00000000 000000ff
[148193.520888] 02342c65 02342c65 f7a68da8 000001d9 c02b7f01 c1c0c6d0 f3fadb5c 02342c65
[148193.520888] Call Trace:
[148193.520888] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520888] [<c02b7f01>] schedule_timeout+0x6b/0x86
[148193.520888] [<c012968c>] process_timeout+0x0/0x5
[148193.520888] [<c02b7efc>] schedule_timeout+0x66/0x86
[148193.520888] [<f92cd3ab>] dvb_frontend_thread+0x12a/0x2a7 [dvb_core]
[148193.520888] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520888] [<f92cd281>] dvb_frontend_thread+0x0/0x2a7 [dvb_core]
[148193.520888] [<c01316b3>] kthread+0x38/0x5d
[148193.520888] [<c013167b>] kthread+0x0/0x5d
[148193.520888] [<c01044f3>] kernel_thread_helper+0x7/0x10
[148193.520888] =======================
[148193.520888] startx S 57686065 0 3276 3261
[148193.520888] f7a731c0 00000086 00000018 57686065 c1670f0c f7a7334c c1c0efa0 00000000
[148193.520888] f3878fd8 ffff06b4 c1670f0c 57686065 00000000 00000000 00000000 000000ff
[148193.520888] ffffffea 00000001 f7a731b8 00000000 c0124aa7 00000004 00000000 00000003
[148193.520888] Call Trace:
[148193.520888] [<c0124aa7>] do_wait+0x892/0x969
[148193.520888] [<c011d36d>] hrtick_set+0x8f/0xd8
[148193.520888] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520888] [<c0124bfd>] sys_wait4+0x7f/0x92
[148193.520888] [<c0124c23>] sys_waitpid+0x13/0x17
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] apache2 S cfeb6685 0 3290 3235
[148193.520888] f7888d00 00200086 f2f21dc4 cfeb6685 000083f1 f7888e8c c1c0efa0 00000000
[148193.520888] f778f000 f38adf2c 000140af 00000000 00000040 fa10d160 f38adea8 f72bce00
[148193.520888] 7fffffff 7fffffff f38adeb8 f3f6e75c c02b7ea9 00000000 fa0fb320 c02b8f4d
[148193.520888] Call Trace:
[148193.520888] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520888] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520888] [<c02506e2>] lock_sock_nested+0x84/0x8c
[148193.520888] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520888] [<c02505e8>] release_sock+0xf/0x85
[148193.520888] [<c0279933>] inet_csk_accept+0x9b/0x1db
[148193.520888] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520888] [<c0293a3c>] inet_accept+0x1c/0x9b
[148193.520888] [<c024fad0>] sys_accept+0xe5/0x1bd
[148193.520888] [<f8875bc4>] e1000_clean_tx_irq+0xaf/0x263 [e1000]
[148193.520888] [<f887880a>] e1000_clean+0xf7/0x215 [e1000]
[148193.520888] [<c024fc50>] sys_socketcall+0xa8/0x19e
[148193.520888] [<c012679d>] irq_exit+0x50/0x67
[148193.520888] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520888] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520888] =======================
[148193.520888] xinit S 7fffffff 0 3296 3276
[148193.520888] f7898900 00000086 c0135ee4 7fffffff 7fffffff f7898a8c c1c0efa0 00000000
[148193.520889] c013435a ffff0c42 00000000 c013438e 00000000 00000000 00000000 000000ff
[148193.520889] ffffffea 00000001 f78988f8 00000000 c0124aa7 00000004 00000000 00000003
[148193.520889] Call Trace:
[148193.520889] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520889] [<c013435a>] ktime_get_ts+0x22/0x49
[148193.520889] [<c013438e>] ktime_get+0xd/0x21
[148193.520889] [<c0124aa7>] do_wait+0x892/0x969
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c0124bfd>] sys_wait4+0x7f/0x92
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] Xorg S f3bcda9d 0 3297 3296
[148193.520889] f7889960 00003082 f749e900 f3bcda9d 00008645 f7889aec c1c0efa0 00000000
[148193.520889] c036e2e0 02244e37 00000ba0 00000000 c035b684 00000000 00000001 00003082
[148193.520889] 7fffffff f7a66ee0 04000000 f38a7f9c c02b7ea9 00003282 f7a66c60 00003082
[148193.520889] Call Trace:
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c0221548>] tty_ldisc_deref+0x4c/0x5b
[148193.520889] [<c0222343>] tty_poll+0x53/0x60
[148193.520889] [<c017f09a>] do_select+0x364/0x3bd
[148193.520889] [<c017f59e>] __pollwait+0x0/0xac
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520889] [<c012679d>] irq_exit+0x50/0x67
[148193.520889] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520889] [<c01042a7>] common_interrupt+0x23/0x28
[148193.520889] [<c0253e7d>] __alloc_skb+0x49/0xf7
[148193.520889] [<c025087b>] sock_alloc_send_skb+0x68/0x192
[148193.520889] [<c02a71b1>] unix_stream_recvmsg+0x390/0x480
[148193.520889] [<c02532c3>] skb_queue_tail+0x11/0x2d
[148193.520889] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520889] [<c024e454>] sock_aio_write+0xdd/0xea
[148193.520889] [<c0173edf>] do_sync_write+0xbf/0x100
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c01340dc>] hrtimer_try_to_cancel+0x5c/0x63
[148193.520889] [<c017f718>] sys_select+0xce/0x180
[148193.520889] [<c0174c1b>] sys_write+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] kdvb-fe-2 S d8827452 0 3303 2
[148193.520889] f7859520 00000046 c034d300 d8827452 000086b7 f78596ac c1c0efa0 00000000
[148193.520889] c0129950 023429d5 02342be7 f38abf70 00000000 00000000 00000000 000000ff
[148193.520889] 02342be7 02342be7 f7a685a8 000001d9 c02b7f01 c03f24a4 c03f24a4 02342be7
[148193.520889] Call Trace:
[148193.520889] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520889] [<c02b7f01>] schedule_timeout+0x6b/0x86
[148193.520889] [<c012968c>] process_timeout+0x0/0x5
[148193.520889] [<c02b7efc>] schedule_timeout+0x66/0x86
[148193.520889] [<f92cd3ab>] dvb_frontend_thread+0x12a/0x2a7 [dvb_core]
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<f92cd281>] dvb_frontend_thread+0x0/0x2a7 [dvb_core]
[148193.520889] [<c01316b3>] kthread+0x38/0x5d
[148193.520889] [<c013167b>] kthread+0x0/0x5d
[148193.520889] [<c01044f3>] kernel_thread_helper+0x7/0x10
[148193.520889] =======================
[148193.520889] kdvb-fe-3 S b52961e3 0 3312 2
[148193.520889] f7551640 00000046 f743eca0 b52961e3 000086c7 f75517cc c1c0efa0 00000000
[148193.520889] c0129950 f38d1f70 00000002 00000000 00000286 c0129a63 00000000 00000286
[148193.520889] 02342beb 02342beb f796fda8 000001d9 c02b7f01 c03f24c4 c03f24c4 02342beb
[148193.520889] Call Trace:
[148193.520889] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520889] [<c0129a63>] __mod_timer+0x99/0xa3
[148193.520889] [<c02b7f01>] schedule_timeout+0x6b/0x86
[148193.520889] [<c012968c>] process_timeout+0x0/0x5
[148193.520889] [<c02b7efc>] schedule_timeout+0x66/0x86
[148193.520889] [<f92cd3ab>] dvb_frontend_thread+0x12a/0x2a7 [dvb_core]
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<f92cd281>] dvb_frontend_thread+0x0/0x2a7 [dvb_core]
[148193.520889] [<c01316b3>] kthread+0x38/0x5d
[148193.520889] [<c013167b>] kthread+0x0/0x5d
[148193.520889] [<c01044f3>] kernel_thread_helper+0x7/0x10
[148193.520889] =======================
[148193.520889] mythfrontend S f3b73b5c 0 3343 3296
[148193.520889] f3b5adc0 00000086 c034d300 f3b73b5c 0000007b f3b5af4c c1c0efa0 00000000
[148193.520889] c0129950 02342b44 02342b47 f3b73b5c 00000000 00000000 00000000 000000ff
[148193.520889] 02342b47 02342b47 20000000 f3b73f9c c02b7f01 c03f1fa4 c03f1fa4 02342b47
[148193.520889] Call Trace:
[148193.520889] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520889] [<c02b7f01>] schedule_timeout+0x6b/0x86
[148193.520889] [<c012968c>] process_timeout+0x0/0x5
[148193.520889] [<c02b7efc>] schedule_timeout+0x66/0x86
[148193.520889] [<c017f09a>] do_select+0x364/0x3bd
[148193.520889] [<c017f59e>] __pollwait+0x0/0xac
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c0257d8e>] netif_receive_skb+0x2d6/0x343
[148193.520889] [<c025a139>] process_backlog+0x74/0xce
[148193.520889] [<c0259a7e>] net_rx_action+0x9c/0x1b9
[148193.520889] [<c0126419>] __do_softirq+0x66/0xd3
[148193.520889] [<c0126702>] local_bh_enable+0x66/0x71
[148193.520889] [<c025a614>] dev_queue_xmit+0x287/0x2bf
[148193.520889] [<c0275d31>] ip_finish_output+0x1c5/0x1fc
[148193.520889] [<c02758bf>] ip_local_out+0x15/0x17
[148193.520889] [<c02762e1>] ip_queue_xmit+0x25b/0x29d
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<fabdff58>] _nv004365rm+0x24/0x70 [nvidia]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<fabdff84>] _nv004365rm+0x50/0x70 [nvidia]
[148193.520889] [<fabdff58>] _nv004365rm+0x24/0x70 [nvidia]
[148193.520889] [<f88426b7>] ehci_irq+0x108/0x128 [ehci_hcd]
[148193.520889] [<c01110b8>] ack_ioapic_quirk_irq+0x34/0x85
[148193.520889] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520889] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520889] [<c012679d>] irq_exit+0x50/0x67
[148193.520889] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520889] [<c0253396>] skb_dequeue+0x42/0x48
[148193.520889] [<c02a71b1>] unix_stream_recvmsg+0x390/0x480
[148193.520889] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520889] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520889] [<c021a373>] pnp_bus_suspend+0x2/0x57
[148193.520889] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520889] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c017f6e9>] sys_select+0x9f/0x180
[148193.520889] [<c0135f76>] do_gettimeofday+0xd/0x27
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] mythfrontend S 00000000 0 3362 3296
[148193.520889] f3b7a180 00000086 c1c0c2f8 00000000 c0133c26 f3b7a30c c1c0efa0 00000000
[148193.520889] f3bbfe5c 02342b1e 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520889] 00000000 00000000 f3bbfe5c 08302f2c c013a6a5 00090a81 00090a81 00000000
[148193.520889] Call Trace:
[148193.520889] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520889] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520889] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520889] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520889] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520889] [<c01110b8>] ack_ioapic_quirk_irq+0x34/0x85
[148193.520889] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520889] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520889] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] mythfrontend S 00000000 0 3363 3296
[148193.520889] f3b7a5a0 00000086 c1c0c2f8 00000000 c0133c26 f3b7a72c c1c0efa0 00000000
[148193.520889] f3b69e5c 02342b2d 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520889] 00000000 00000000 f3b69e5c b408f278 c013a6a5 00000001 00000001 00000000
[148193.520889] Call Trace:
[148193.520889] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520889] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520889] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520889] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520889] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520889] [<c011fc07>] check_preempt_wakeup+0xa2/0xc2
[148193.520889] [<c011c59e>] rebalance_domains+0x6d/0x355
[148193.520889] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520889] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] mythfrontend S f67d381c 0 3364 3296
[148193.520889] f3b7a9c0 00000086 c034d300 f67d381c 000086c7 f3b7ab4c c1c0efa0 00000000
[148193.520889] c0129950 f3b6bb5c 0000284a 00000000 00000286 c0129a63 00000000 00000286
[148193.520889] 02342c1e 02342c1e 00000200 f3b6bf9c c02b7f01 c03f1e5c c03f1e5c 02342c1e
[148193.520889] Call Trace:
[148193.520889] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520889] [<c0129a63>] __mod_timer+0x99/0xa3
[148193.520889] [<c02b7f01>] schedule_timeout+0x6b/0x86
[148193.520889] [<c012968c>] process_timeout+0x0/0x5
[148193.520889] [<c02b7efc>] schedule_timeout+0x66/0x86
[148193.520889] [<c017f09a>] do_select+0x364/0x3bd
[148193.520889] [<c017f59e>] __pollwait+0x0/0xac
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520889] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520889] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520889] [<c024c850>] pci_conf1_read+0xa4/0xac
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<c024d73d>] raw_pci_read+0x3d/0x45
[148193.520889] [<c024d80b>] pci_read+0x1c/0x21
[148193.520889] [<f88426b7>] ehci_irq+0x108/0x128 [ehci_hcd]
[148193.520889] [<c01110b8>] ack_ioapic_quirk_irq+0x34/0x85
[148193.520889] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520889] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520889] [<f8a30b0b>] ivtv_irq_handler+0xb37/0xdb8 [ivtv]
[148193.520889] [<c02506e2>] lock_sock_nested+0x84/0x8c
[148193.520889] [<c02505e8>] release_sock+0xf/0x85
[148193.520889] [<f88425c9>] ehci_irq+0x1a/0x128 [ehci_hcd]
[148193.520889] [<c01110b8>] ack_ioapic_quirk_irq+0x34/0x85
[148193.520889] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520889] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520889] [<c012679d>] irq_exit+0x50/0x67
[148193.520889] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520889] [<c01042a7>] common_interrupt+0x23/0x28
[148193.520889] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520889] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520889] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520889] [<c01042a7>] common_interrupt+0x23/0x28
[148193.520889] [<c0253816>] skb_release_data+0x2c/0x86
[148193.520889] [<c0250bcb>] sk_free+0x97/0xaa
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c028e644>] udp_ioctl+0x5a/0x5e
[148193.520889] [<c028e5ea>] udp_ioctl+0x0/0x5e
[148193.520889] [<c017f6e9>] sys_select+0x9f/0x180
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] mythfrontend S b40e9cb7 0 3370 3296
[148193.520889] f3be5a60 00000086 00000001 b40e9cb7 00005ba3 f3be5bec c1c0efa0 00000000
[148193.520889] 000001fe 003519eb 0002be4e 00000000 00351a31 00000000 00000000 00000246
[148193.520889] f3be7eb0 00000000 c0403fe0 082e6aa4 c013a642 00000011 00000011 00000000
[148193.520889] Call Trace:
[148193.520889] [<c013a642>] futex_wait+0x199/0x31d
[148193.520889] [<c01df7cc>] rb_insert_color+0x4c/0xad
[148193.520889] [<c0157c5a>] generic_file_aio_write+0x5b/0xa9
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520889] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520889] [<c0174c1b>] sys_write+0x3c/0x63
[148193.520889] [<c0174c3a>] sys_write+0x5b/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] mythfrontend S d81b2ae8 0 3380 3296
[148193.520889] f3be5220 00000086 f3be41a0 d81b2ae8 000086c5 f3be53ac c1c0efa0 00000000
[148193.520889] c036e2e0 0234187e 00028e28 00000000 00000000 00000000 00000000 000000ff
[148193.520889] 7fffffff f2eb1e80 08000000 f3bf7f9c c02b7ea9 00002000 0000000d 00000207
[148193.520889] Call Trace:
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c01318c2>] remove_wait_queue+0xc/0x34
[148193.520889] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520889] [<c027b144>] tcp_poll+0x14/0x10e
[148193.520889] [<c017f09a>] do_select+0x364/0x3bd
[148193.520889] [<c017f59e>] __pollwait+0x0/0xac
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<f8876e57>] e1000_xmit_frame+0xa59/0xaa3 [e1000]
[148193.520889] [<c018fd1f>] __getblk+0x27/0x24e
[148193.520889] [<f88f876b>] do_get_write_access+0x2f8/0x331 [jbd]
[148193.520889] [<f897a523>] __ext3_get_inode_loc+0xcf/0x26c [ext3]
[148193.520889] [<f898537b>] __ext3_journal_dirty_metadata+0x13/0x32 [ext3]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<f88425c9>] ehci_irq+0x1a/0x128 [ehci_hcd]
[148193.520889] [<f88426b7>] ehci_irq+0x108/0x128 [ehci_hcd]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<f88426b7>] ehci_irq+0x108/0x128 [ehci_hcd]
[148193.520889] [<c01110b8>] ack_ioapic_quirk_irq+0x34/0x85
[148193.520889] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520889] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520889] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520889] [<c0254f2d>] memcpy_toiovec+0x27/0x4a
[148193.520889] [<c0255428>] skb_copy_datagram_iovec+0xfe/0x1bf
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02505e8>] release_sock+0xf/0x85
[148193.520889] [<c027d0dc>] tcp_recvmsg+0x8b6/0x9b8
[148193.520889] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520889] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520889] [<c01e4d9c>] pci_bus_read_config_word+0x59/0x64
[148193.520889] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520889] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c027b579>] tcp_ioctl+0x101/0x10c
[148193.520889] [<c027b478>] tcp_ioctl+0x0/0x10c
[148193.520889] [<c0292cd3>] inet_ioctl+0xa0/0xa3
[148193.520889] [<c017f718>] sys_select+0xce/0x180
[148193.520889] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] mythfrontend S 486a6f85 0 3382 3296
[148193.520889] f3be4e00 00000086 f74aed40 486a6f85 00005d45 f3be4f8c c1c0efa0 00000000
[148193.520889] 00000000 00000000 00062a3a 00000000 c0253e7d 00000000 f74139e0 f3eced80
[148193.520889] 7fffffff 00000000 f3443e5c f3bb1040 c02b7ea9 c0135ee4 224ee1f6 f3443df8
[148193.520889] Call Trace:
[148193.520889] [<c0253e7d>] __alloc_skb+0x49/0xf7
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520889] [<c02a814a>] unix_write_space+0xf/0x63
[148193.520889] [<c013187f>] prepare_to_wait+0x12/0x49
[148193.520889] [<c02a7004>] unix_stream_recvmsg+0x1e3/0x480
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520889] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c01790f4>] pipe_ioctl+0x51/0x56
[148193.520889] [<c01790a3>] pipe_ioctl+0x0/0x56
[148193.520889] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520889] [<c0174778>] vfs_read+0x92/0x11e
[148193.520889] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] irexec S 48682d9d 0 3351 3343
[148193.520889] f3b7ba40 00000082 f7a52100 48682d9d 00005d45 f3b7bbcc c1c0efa0 00000000
[148193.520889] c30c5660 f795e7c4 0000af9e 00000000 c011b614 00000003 00000002 c3131c60
[148193.520889] 7fffffff 00000000 f3b7de5c f388d040 c02b7ea9 f752604c 37b00000 f7813638
[148193.520889] Call Trace:
[148193.520889] [<c011b614>] try_to_wake_up+0xe8/0xf1
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c02a814a>] unix_write_space+0xf/0x63
[148193.520889] [<c013187f>] prepare_to_wait+0x12/0x49
[148193.520889] [<c02a7004>] unix_stream_recvmsg+0x1e3/0x480
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<f92c97db>] dvb_dmx_swfilter+0xc5/0x105 [dvb_core]
[148193.520889] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520889] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520889] [<c0174778>] vfs_read+0x92/0x11e
[148193.520889] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] mtd S 33f0e486 0 3355 1
[148193.520889] f3b5a160 00000086 c034d300 33f0e486 000086c7 f3b5a2ec c1c0efa0 00000000
[148193.520889] c0129950 02342adf 02342bdc f3b9bb5c 00000000 00000000 00000000 000000ff
[148193.520889] 02342bdc 02342bdc 00000020 f3b9bf9c c02b7f01 f7441b5c c03f244c 02342bdc
[148193.520889] Call Trace:
[148193.520889] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520889] [<c02b7f01>] schedule_timeout+0x6b/0x86
[148193.520889] [<c012968c>] process_timeout+0x0/0x5
[148193.520889] [<c02b7efc>] schedule_timeout+0x66/0x86
[148193.520889] [<c017f09a>] do_select+0x364/0x3bd
[148193.520889] [<c017f59e>] __pollwait+0x0/0xac
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<fabdff86>] _nv004365rm+0x52/0x70 [nvidia]
[148193.520889] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520889] [<fabb28fc>] _nv008244rm+0x70/0x95 [nvidia]
[148193.520889] [<c024c850>] pci_conf1_read+0xa4/0xac
[148193.520889] [<c024d73d>] raw_pci_read+0x3d/0x45
[148193.520889] [<c024d80b>] pci_read+0x1c/0x21
[148193.520889] [<c01e4d9c>] pci_bus_read_config_word+0x59/0x64
[148193.520889] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520889] [<fabb2ab5>] _nv008482rm+0x1c/0x20 [nvidia]
[148193.520889] [<fabb2a02>] _nv008484rm+0x21/0x3b [nvidia]
[148193.520889] [<fabb2764>] _nv008168rm+0xbc/0xc6 [nvidia]
[148193.520889] [<c024c850>] pci_conf1_read+0xa4/0xac
[148193.520889] [<c024c850>] pci_conf1_read+0xa4/0xac
[148193.520889] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520889] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520889] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520889] [<c0129a63>] __mod_timer+0x99/0xa3
[148193.520889] [<f883f230>] end_unlink_async+0xb5/0xcd [ehci_hcd]
[148193.520889] [<f884262c>] ehci_irq+0x7d/0x128 [ehci_hcd]
[148193.520889] [<c01110b8>] ack_ioapic_quirk_irq+0x34/0x85
[148193.520889] [<c0151dd6>] handle_IRQ_event+0x23/0x51
[148193.520889] [<c0152eff>] handle_fasteoi_irq+0x9b/0xa4
[148193.520889] [<c012679d>] irq_exit+0x50/0x67
[148193.520889] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520889] [<c01042a7>] common_interrupt+0x23/0x28
[148193.520889] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520889] [<c0133b45>] hrtimer_reprogram+0x83/0x9b
[148193.520889] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520889] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520889] [<c024c850>] pci_conf1_read+0xa4/0xac
[148193.520889] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520889] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520889] [<fabe00db>] _nv004359rm+0x19/0x1d [nvidia]
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<fabf33ea>] _nv003983rm+0x1d5/0x212 [nvidia]
[148193.520889] [<fabf1bd2>] _nv003976rm+0x3d/0x65 [nvidia]
[148193.520889] [<fabf7ed2>] rm_isr+0x9c/0xef [nvidia]
[148193.520889] [<c017f6e9>] sys_select+0x9f/0x180
[148193.520889] [<c0135f76>] do_gettimeofday+0xd/0x27
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] mtd S 00000000 0 3358 1
[148193.520889] f3b5a580 00000086 c1c0c2f8 00000000 c0133c26 f3b5a70c c1c0efa0 00000000
[148193.520889] f3b75e5c 02342ad8 00000282 c01341ee 00000000 00000000 00000000 000000ff
[148193.520889] 00000000 00000000 f3b75e5c b53632a8 c013a6a5 00000001 00000001 00000000
[148193.520889] Call Trace:
[148193.520889] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520889] [<c01341ee>] hrtimer_start+0xf7/0x110
[148193.520889] [<c013a6a5>] futex_wait+0x1fc/0x31d
[148193.520889] [<c0133e18>] hrtimer_wakeup+0x0/0x18
[148193.520889] [<c013a689>] futex_wait+0x1e0/0x31d
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c013b4df>] do_futex+0x68/0x6ff
[148193.520889] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520889] [<c013bc47>] sys_futex+0xd1/0xe6
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] ratpoison S ced50264 0 3357 3343
[148193.520889] f3b5a9a0 00000082 f7889960 ced50264 00005d3c f3b5ab2c c1c0efa0 00000000
[148193.520889] c012679d c0105f3f 0000254f 00000000 00000008 00000003 c01042a7 f3b4cac0
[148193.520889] 7fffffff f3fbe2c0 00000008 f3b9ff9c c02b7ea9 00000246 0000007b 00000203
[148193.520889] Call Trace:
[148193.520889] [<c012679d>] irq_exit+0x50/0x67
[148193.520889] [<c0105f3f>] do_IRQ+0x52/0x63
[148193.520889] [<c01042a7>] common_interrupt+0x23/0x28
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520889] [<c02a7510>] unix_poll+0x17/0x8c
[148193.520889] [<c017f09a>] do_select+0x364/0x3bd
[148193.520889] [<c017f59e>] __pollwait+0x0/0xac
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<f8894f36>] ide_dma_exec_cmd+0x17/0x1a [ide_core]
[148193.520889] [<c0108214>] nommu_map_single+0x2b/0x43
[148193.520889] [<f8876e57>] e1000_xmit_frame+0xa59/0xaa3 [e1000]
[148193.520889] [<c0258370>] dev_hard_start_xmit+0x1eb/0x24f
[148193.520889] [<c0266456>] __qdisc_run+0xcc/0x17c
[148193.520889] [<c025a614>] dev_queue_xmit+0x287/0x2bf
[148193.520889] [<c0275d31>] ip_finish_output+0x1c5/0x1fc
[148193.520889] [<c02758bf>] ip_local_out+0x15/0x17
[148193.520889] [<c02762e1>] ip_queue_xmit+0x25b/0x29d
[148193.520889] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520889] [<c0289531>] tcp_v4_send_check+0x7a/0xb0
[148193.520889] [<c0253e7d>] __alloc_skb+0x49/0xf7
[148193.520889] [<c025087b>] sock_alloc_send_skb+0x68/0x192
[148193.520889] [<c02532c3>] skb_queue_tail+0x11/0x2d
[148193.520889] [<c0251d95>] sock_def_readable+0xf/0x58
[148193.520889] [<c02a7814>] unix_stream_sendmsg+0x1fc/0x2c7
[148193.520889] [<c02a814a>] unix_write_space+0xf/0x63
[148193.520889] [<c0253396>] skb_dequeue+0x42/0x48
[148193.520889] [<c02a71b1>] unix_stream_recvmsg+0x390/0x480
[148193.520889] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520889] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520889] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<f887880a>] e1000_clean+0xf7/0x215 [e1000]
[148193.520889] [<c017f718>] sys_select+0xce/0x180
[148193.520889] [<c0174778>] vfs_read+0x92/0x11e
[148193.520889] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] mythlcdserver S a8c418c9 0 3377 1
[148193.520889] f3be41a0 00000086 f744f920 a8c418c9 000086c7 f3be432c c1c0efa0 00000000
[148193.520889] c0129950 02342adf 02342bab f3bf3b5c 00000000 00000000 00000000 000000ff
[148193.520889] 02342bab 02342bab 00010000 f3bf3f9c c02b7f01 c03f22c4 c03f22c4 02342bab
[148193.520889] Call Trace:
[148193.520889] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520889] [<c02b7f01>] schedule_timeout+0x6b/0x86
[148193.520889] [<c012968c>] process_timeout+0x0/0x5
[148193.520889] [<c02b7efc>] schedule_timeout+0x66/0x86
[148193.520889] [<c017f09a>] do_select+0x364/0x3bd
[148193.520889] [<c017f59e>] __pollwait+0x0/0xac
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c0259a7e>] net_rx_action+0x9c/0x1b9
[148193.520889] [<c0126419>] __do_softirq+0x66/0xd3
[148193.520889] [<c0126702>] local_bh_enable+0x66/0x71
[148193.520889] [<c025a614>] dev_queue_xmit+0x287/0x2bf
[148193.520889] [<c0275d31>] ip_finish_output+0x1c5/0x1fc
[148193.520889] [<c02758bf>] ip_local_out+0x15/0x17
[148193.520889] [<c02762e1>] ip_queue_xmit+0x25b/0x29d
[148193.520889] [<c01df7cc>] rb_insert_color+0x4c/0xad
[148193.520889] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520889] [<c028485c>] tcp_transmit_skb+0x6a2/0x6d5
[148193.520889] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520889] [<c0129a63>] __mod_timer+0x99/0xa3
[148193.520889] [<c0254f2d>] memcpy_toiovec+0x27/0x4a
[148193.520889] [<c0255428>] skb_copy_datagram_iovec+0xfe/0x1bf
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02505e8>] release_sock+0xf/0x85
[148193.520889] [<c027d0dc>] tcp_recvmsg+0x8b6/0x9b8
[148193.520889] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520889] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520889] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520889] [<c0176a3d>] cp_new_stat64+0xfc/0x10e
[148193.520889] [<f8a2ffe8>] ivtv_irq_handler+0x14/0xdb8 [ivtv]
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02506e2>] lock_sock_nested+0x84/0x8c
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02505e8>] release_sock+0xf/0x85
[148193.520889] [<c027b579>] tcp_ioctl+0x101/0x10c
[148193.520889] [<c027b478>] tcp_ioctl+0x0/0x10c
[148193.520889] [<c0292cd3>] inet_ioctl+0xa0/0xa3
[148193.520889] [<c017f6e9>] sys_select+0x9f/0x180
[148193.520889] [<c0135f76>] do_gettimeofday+0xd/0x27
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] mythlcdserver S e855f4f2 0 3381 1
[148193.520889] f3be49e0 00000086 f77a2500 e855f4f2 00007750 f3be4b6c c1c0efa0 00000000
[148193.520889] 00000000 01925c5c 00003ff4 00000000 00000000 00000000 00000000 000000ff
[148193.520889] 7fffffff f3ba6840 00000040 f3be9f9c c02b7ea9 378130c0 f7877400 00000203
[148193.520889] Call Trace:
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c01318c2>] remove_wait_queue+0xc/0x34
[148193.520889] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520889] [<c0178e2b>] pipe_poll+0x24/0x7d
[148193.520889] [<c017f09a>] do_select+0x364/0x3bd
[148193.520889] [<c017f59e>] __pollwait+0x0/0xac
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<f8876e57>] e1000_xmit_frame+0xa59/0xaa3 [e1000]
[148193.520889] [<f88411df>] ehci_urb_enqueue+0x15c/0xd12 [ehci_hcd]
[148193.520889] [<c0258370>] dev_hard_start_xmit+0x1eb/0x24f
[148193.520889] [<c0266456>] __qdisc_run+0xcc/0x17c
[148193.520889] [<c025a614>] dev_queue_xmit+0x287/0x2bf
[148193.520889] [<c0275d31>] ip_finish_output+0x1c5/0x1fc
[148193.520889] [<c02758bf>] ip_local_out+0x15/0x17
[148193.520889] [<c02762e1>] ip_queue_xmit+0x25b/0x29d
[148193.520889] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520889] [<c0289531>] tcp_v4_send_check+0x7a/0xb0
[148193.520889] [<c028485c>] tcp_transmit_skb+0x6a2/0x6d5
[148193.520889] [<fa0b2aa3>] rpc_wake_up_queued_task+0x14/0x1d [sunrpc]
[148193.520889] [<c0253e7d>] __alloc_skb+0x49/0xf7
[148193.520889] [<c0135ee4>] getnstimeofday+0x37/0xbc
[148193.520889] [<c013435a>] ktime_get_ts+0x22/0x49
[148193.520889] [<c013438e>] ktime_get+0xd/0x21
[148193.520889] [<c0119081>] hrtick_start_fair+0xeb/0x12c
[148193.520889] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520889] [<c0184ce6>] touch_atime+0xc7/0xd1
[148193.520889] [<c0187278>] mnt_drop_write+0x1a/0xb4
[148193.520889] [<c0179dae>] pipe_read+0x31d/0x32b
[148193.520889] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c027b579>] tcp_ioctl+0x101/0x10c
[148193.520889] [<c027b478>] tcp_ioctl+0x0/0x10c
[148193.520889] [<c0292cd3>] inet_ioctl+0xa0/0xa3
[148193.520889] [<c017f718>] sys_select+0xce/0x180
[148193.520889] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] sshd S 7b4635e4 0 3460 2733
[148193.520889] f3425940 00000082 f3425100 7b4635e4 000000b1 f3425acc c1c0efa0 00000000
[148193.520889] 00000000 00000000 000418be 00000000 00000000 f3425940 00000000 00000000
[148193.520889] 7fffffff 00000000 f3423e5c f368dc80 c02b7ea9 f3eda560 00000060 f3423e0c
[148193.520889] Call Trace:
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c0182aea>] __d_lookup+0x98/0xd7
[148193.520889] [<c013187f>] prepare_to_wait+0x12/0x49
[148193.520889] [<c02a7004>] unix_stream_recvmsg+0x1e3/0x480
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520889] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520889] [<c0174778>] vfs_read+0x92/0x11e
[148193.520889] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] sshd S c974c621 0 3462 3460
[148193.520889] f3425520 00000086 f3425100 c974c621 000056bc f34256ac c1c0efa0 00000000
[148193.520889] 0000000a 016aa88c 00005e26 00000000 c035b684 00000000 00000001 00000082
[148193.520889] 7fffffff f3665400 00000400 f3417f9c c02b7ea9 00000282 f3665400 00000082
[148193.520889] Call Trace:
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c0221548>] tty_ldisc_deref+0x4c/0x5b
[148193.520889] [<c0222343>] tty_poll+0x53/0x60
[148193.520889] [<c017f09a>] do_select+0x364/0x3bd
[148193.520889] [<c017f59e>] __pollwait+0x0/0xac
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c0258370>] dev_hard_start_xmit+0x1eb/0x24f
[148193.520889] [<c0266456>] __qdisc_run+0xcc/0x17c
[148193.520889] [<c025a614>] dev_queue_xmit+0x287/0x2bf
[148193.520889] [<c025ea9c>] neigh_resolve_output+0x1f1/0x21b
[148193.520889] [<c0275d31>] ip_finish_output+0x1c5/0x1fc
[148193.520889] [<c02758bf>] ip_local_out+0x15/0x17
[148193.520889] [<c02762e1>] ip_queue_xmit+0x25b/0x29d
[148193.520889] [<c028485c>] tcp_transmit_skb+0x6a2/0x6d5
[148193.520889] [<c0129a63>] __mod_timer+0x99/0xa3
[148193.520889] [<c0285e67>] __tcp_push_pending_frames+0x630/0x6c9
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02505e8>] release_sock+0xf/0x85
[148193.520889] [<c027c427>] tcp_sendmsg+0x7fc/0x8fe
[148193.520889] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520889] [<c024e454>] sock_aio_write+0xdd/0xea
[148193.520889] [<c0173edf>] do_sync_write+0xbf/0x100
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c017f718>] sys_select+0xce/0x180
[148193.520889] [<c0174c1b>] sys_write+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] bash S 00000282 0 3463 3462
[148193.520889] f3425100 00000086 4b8f2065 00000282 00000000 f342528c c1c0efa0 00000000
[148193.520889] c0161e6f 016aa88c 00000001 f78c3680 00000000 00000000 00000000 000000ff
[148193.520889] 7fffffff f3ffb400 f2e9bf44 bf85611f c02b7ea9 000005f4 000005f2 f79a1f01
[148193.520889] Call Trace:
[148193.520889] [<c0161e6f>] do_wp_page+0x1da/0x49f
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<f887846c>] e1000_clean_rx_irq+0x429/0x46f [e1000]
[148193.520889] [<c013192b>] add_wait_queue+0x12/0x33
[148193.520889] [<c0226370>] read_chan+0x357/0x5d3
[148193.520889] [<c013438e>] ktime_get+0xd/0x21
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c0226019>] read_chan+0x0/0x5d3
[148193.520889] [<c02237e0>] tty_read+0x59/0x96
[148193.520889] [<c0223787>] tty_read+0x0/0x96
[148193.520889] [<c0174767>] vfs_read+0x81/0x11e
[148193.520889] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] apache2 S 9c01e3b4 0 13762 3235
[148193.520889] f2f799c0 00200086 f342a5d4 9c01e3b4 00005ea1 f2f79b4c c1c0efa0 00000000
[148193.520889] f342a5d4 00200297 00018177 00000000 f342a5d4 f354f860 f3ffb0fc f354f820
[148193.520889] 7fffffff 7fffffff f2eadeb8 f3f6e75c c02b7ea9 f342a5d4 00000000 c02b8f4d
[148193.520889] Call Trace:
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02506e2>] lock_sock_nested+0x84/0x8c
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02505e8>] release_sock+0xf/0x85
[148193.520889] [<c0279933>] inet_csk_accept+0x9b/0x1db
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c0293a3c>] inet_accept+0x1c/0x9b
[148193.520889] [<c024fad0>] sys_accept+0xe5/0x1bd
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<fa0d6738>] inet6_destroy_sock+0x22/0x3a [ipv6]
[148193.520889] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520889] [<c024fc50>] sys_socketcall+0xa8/0x19e
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] apache2 S 00000001 0 13763 3235
[148193.520889] f2f78100 00200082 c313fdc4 00000001 00000000 f2f7828c c1c0efa0 00000000
[148193.520889] f342a454 0188bf63 00000000 000000e4 00000000 00000000 00000000 000000ff
[148193.520889] 7fffffff 7fffffff f7909eb8 f3f6e75c c02b7ea9 00000000 fa0fb320 c02b8f4d
[148193.520889] Call Trace:
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02506e2>] lock_sock_nested+0x84/0x8c
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02505e8>] release_sock+0xf/0x85
[148193.520889] [<c0279933>] inet_csk_accept+0x9b/0x1db
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c0293a3c>] inet_accept+0x1c/0x9b
[148193.520889] [<c024fad0>] sys_accept+0xe5/0x1bd
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<fa0d6738>] inet6_destroy_sock+0x22/0x3a [ipv6]
[148193.520889] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520889] [<c024fc50>] sys_socketcall+0xa8/0x19e
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] apache2 S 9a19ff9b 0 13764 3235
[148193.520889] f2f78520 00200086 c0354c40 9a19ff9b 00005ea1 f2f786ac c1c0efa0 00000000
[148193.520889] 00000001 00000000 0001588b 00000000 c0354c54 00000000 00000001 00000001
[148193.520889] 7fffffff 7fffffff f2ee9eb8 f3f6e75c c02b7ea9 00000020 f3f2c130 c02b8f4d
[148193.520889] Call Trace:
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02506e2>] lock_sock_nested+0x84/0x8c
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02505e8>] release_sock+0xf/0x85
[148193.520889] [<c0279933>] inet_csk_accept+0x9b/0x1db
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c0293a3c>] inet_accept+0x1c/0x9b
[148193.520889] [<c024fad0>] sys_accept+0xe5/0x1bd
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<fa0d6738>] inet6_destroy_sock+0x22/0x3a [ipv6]
[148193.520889] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520889] [<c024fc50>] sys_socketcall+0xa8/0x19e
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] apache2 S 00000001 0 13765 3235
[148193.520889] f79239a0 00200086 f2f21384 00000001 00000000 f7923b2c c1c0efa0 00000000
[148193.520889] f778f000 02287310 00000000 00000089 00000000 00000000 00000000 000000ff
[148193.520889] 7fffffff 7fffffff f2f9beb8 f3f6e75c c02b7ea9 00000000 fa0fb320 c02b8f4d
[148193.520889] Call Trace:
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02506e2>] lock_sock_nested+0x84/0x8c
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02505e8>] release_sock+0xf/0x85
[148193.520889] [<c0279933>] inet_csk_accept+0x9b/0x1db
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c0293a3c>] inet_accept+0x1c/0x9b
[148193.520889] [<c024fad0>] sys_accept+0xe5/0x1bd
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<f887880a>] e1000_clean+0xf7/0x215 [e1000]
[148193.520889] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520889] [<c024fc50>] sys_socketcall+0xa8/0x19e
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] sshd D 00000002 0 19654 2733
[148193.520889] f7923160 00000086 c32abc50 00000002 00011200 f79232ec c1c0efa0 00000000
[148193.520889] c32abc50 0226dcfc c017163c c32abc50 00000000 00000000 00000000 000000ff
[148193.520889] c32abc74 00000000 c32abc7c c1c02390 fa0b27a9 c02b803e fa0b2784 f3ee5de8
[148193.520889] Call Trace:
[148193.520889] [<c017163c>] cache_alloc_refill+0x54/0x44d
[148193.520889] [<fa0b27a9>] rpc_wait_bit_killable+0x25/0x2a [sunrpc]
[148193.520889] [<c02b803e>] __wait_on_bit+0x33/0x58
[148193.520889] [<fa0b2784>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
[148193.520889] [<fa0b2784>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
[148193.520889] [<c02b80c2>] out_of_line_wait_on_bit+0x5f/0x67
[148193.520889] [<c01317a1>] wake_bit_function+0x0/0x3c
[148193.520889] [<fa0b2bea>] __rpc_execute+0xbe/0x1d9 [sunrpc]
[148193.520889] [<fa0ada72>] rpc_run_task+0x40/0x45 [sunrpc]
[148193.520889] [<fa0adb00>] rpc_call_sync+0x38/0x52 [sunrpc]
[148193.520889] [<fa12e23d>] nfs3_rpc_wrapper+0x14/0x49 [nfs]
[148193.520889] [<fa12e76f>] nfs3_proc_access+0x101/0x166 [nfs]
[148193.520889] [<c01d0000>] elv_merge+0x51/0x15b
[148193.520889] [<c01e0532>] number+0x13d/0x217
[148193.520889] [<c01195a6>] find_busiest_group+0x21e/0x623
[148193.520889] [<c0182aea>] __d_lookup+0x98/0xd7
[148193.520889] [<fa120c00>] nfs_do_access+0x134/0x2a7 [nfs]
[148193.520889] [<fa0b36d8>] rpcauth_lookupcred+0x69/0x8f [sunrpc]
[148193.520889] [<fa120e30>] nfs_permission+0xbd/0x11d [nfs]
[148193.520889] [<fa120d73>] nfs_permission+0x0/0x11d [nfs]
[148193.520889] [<c017a6cd>] permission+0xa3/0xfc
[148193.520889] [<c017b8f9>] __link_path_walk+0x11c/0xb0b
[148193.520889] [<f88f8170>] __journal_file_buffer+0xa2/0x123 [jbd]
[148193.520889] [<c0158670>] filemap_fault+0x1f5/0x359
[148193.520889] [<c017c31f>] path_walk+0x37/0x70
[148193.520889] [<c017c5ce>] do_path_lookup+0x122/0x184
[148193.520889] [<c017cfb0>] __path_lookup_intent_open+0x42/0x72
[148193.520889] [<c017d02f>] path_lookup_open+0xf/0x13
[148193.520889] [<c017d0ba>] do_filp_open+0x87/0x684
[148193.520889] [<c016462b>] free_pgtables+0x86/0x93
[148193.520889] [<c0172994>] do_sys_open+0x40/0xb0
[148193.520889] [<c0172a48>] sys_open+0x1e/0x23
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] sshd S dcaf0343 0 19655 19654
[148193.520889] f7922500 00000082 f7923160 dcaf0343 0000839b f792268c c1c0efa0 00000000
[148193.520889] f7923160 f7922528 0000052d 00000000 f7923160 c1c0efa0 00000001 00000002
[148193.520889] 7fffffff 00000000 f2ed5e5c f368dac0 c02b7ea9 f3ece6c0 00000140 f3ece6c0
[148193.520889] Call Trace:
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c0253e7d>] __alloc_skb+0x49/0xf7
[148193.520889] [<c013187f>] prepare_to_wait+0x12/0x49
[148193.520889] [<c02a7004>] unix_stream_recvmsg+0x1e3/0x480
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520889] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520889] [<c0174778>] vfs_read+0x92/0x11e
[148193.520889] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] sshd S c0126419 0 19656 2733
[148193.520889] f7922d40 00000082 c034d300 c0126419 00000046 f7922ecc c1c0efa0 00000000
[148193.520889] 00000046 0233819e 00000003 c035b684 00000000 00000000 00000000 000000ff
[148193.520889] 7fffffff f747b0e0 00000200 c3129f9c c02b7ea9 00000282 f747b0e0 00000082
[148193.520889] Call Trace:
[148193.520889] [<c0126419>] __do_softirq+0x66/0xd3
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c0221548>] tty_ldisc_deref+0x4c/0x5b
[148193.520889] [<c0222343>] tty_poll+0x53/0x60
[148193.520889] [<c017f09a>] do_select+0x364/0x3bd
[148193.520889] [<c017f59e>] __pollwait+0x0/0xac
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c0258370>] dev_hard_start_xmit+0x1eb/0x24f
[148193.520889] [<c0266456>] __qdisc_run+0xcc/0x17c
[148193.520889] [<c025a614>] dev_queue_xmit+0x287/0x2bf
[148193.520889] [<c025ea9c>] neigh_resolve_output+0x1f1/0x21b
[148193.520889] [<c025a614>] dev_queue_xmit+0x287/0x2bf
[148193.520889] [<f8875bc4>] e1000_clean_tx_irq+0xaf/0x263 [e1000]
[148193.520889] [<f887880a>] e1000_clean+0xf7/0x215 [e1000]
[148193.520889] [<c0259a7e>] net_rx_action+0x9c/0x1b9
[148193.520889] [<c0126419>] __do_softirq+0x66/0xd3
[148193.520889] [<c0254f2d>] memcpy_toiovec+0x27/0x4a
[148193.520889] [<c025537d>] skb_copy_datagram_iovec+0x53/0x1bf
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02505e8>] release_sock+0xf/0x85
[148193.520889] [<c027d0dc>] tcp_recvmsg+0x8b6/0x9b8
[148193.520889] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520889] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520889] [<c02244ce>] tty_ioctl+0xb97/0xbd9
[148193.520889] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520889] [<c01318c2>] remove_wait_queue+0xc/0x34
[148193.520889] [<c011a5dc>] __wake_up+0x29/0x39
[148193.520889] [<c0221548>] tty_ldisc_deref+0x4c/0x5b
[148193.520889] [<c017f718>] sys_select+0xce/0x180
[148193.520889] [<c0174c1b>] sys_write+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] bash D f640561f 0 19665 19656
[148193.520889] f7922920 00000082 c3077d46 f640561f 000086a3 f7922aac c1c0efa0 00000000
[148193.520889] fa0bdfa0 f2c7fc20 00000002 00000000 fa0b38bd f3ee5d00 f78b3840 00000246
[148193.520889] c3077d7c 00000000 c3077d84 c1c01550 fa0b27a9 c02b803e fa0b2784 f3ee5d48
[148193.520889] Call Trace:
[148193.520889] [<fa0b38bd>] rpcauth_lookup_credcache+0x42/0x180 [sunrpc]
[148193.520889] [<fa0b27a9>] rpc_wait_bit_killable+0x25/0x2a [sunrpc]
[148193.520889] [<c02b803e>] __wait_on_bit+0x33/0x58
[148193.520889] [<fa0b2784>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
[148193.520889] [<fa0b2784>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
[148193.520889] [<c02b80c2>] out_of_line_wait_on_bit+0x5f/0x67
[148193.520889] [<c01317a1>] wake_bit_function+0x0/0x3c
[148193.520889] [<fa0b2bea>] __rpc_execute+0xbe/0x1d9 [sunrpc]
[148193.520889] [<fa0ada72>] rpc_run_task+0x40/0x45 [sunrpc]
[148193.520889] [<fa0adb00>] rpc_call_sync+0x38/0x52 [sunrpc]
[148193.520889] [<fa12e23d>] nfs3_rpc_wrapper+0x14/0x49 [nfs]
[148193.520889] [<fa12e4fe>] nfs3_proc_getattr+0x52/0x73 [nfs]
[148193.520889] [<fa1242fd>] __nfs_revalidate_inode+0xed/0x211 [nfs]
[148193.520889] [<c017a2ac>] do_lookup+0x53/0x153
[148193.520889] [<c0186774>] mntput_no_expire+0x13/0xd9
[148193.520889] [<c017c1f4>] __link_path_walk+0xa17/0xb0b
[148193.520889] [<f897b2af>] ext3_getblk+0x4a/0x17d [ext3]
[148193.520889] [<f897b304>] ext3_getblk+0x9f/0x17d [ext3]
[148193.520889] [<c0186774>] mntput_no_expire+0x13/0xd9
[148193.520889] [<c017c34f>] path_walk+0x67/0x70
[148193.520889] [<fa124a5e>] nfs_getattr+0x8f/0xbe [nfs]
[148193.520889] [<fa1249cf>] nfs_getattr+0x0/0xbe [nfs]
[148193.520889] [<c0176e52>] vfs_getattr+0x36/0x4d
[148193.520889] [<c0176f2a>] vfs_lstat_fd+0x27/0x39
[148193.520889] [<c0176fa9>] sys_lstat64+0xf/0x23
[148193.520889] [<c0182366>] dput+0x15/0xbb
[148193.520889] [<c0186774>] mntput_no_expire+0x13/0xd9
[148193.520889] [<c017289d>] filp_close+0x4d/0x53
[148193.520889] [<c01739d7>] sys_close+0x5b/0x8d
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] sshd D 00000002 0 20037 2733
[148193.520889] f2f78940 00000082 c3071000 00000002 00011200 f2f78acc c1c0efa0 00000000
[148193.520889] 00000000 02319a2c 00000020 00011200 00000000 00000000 00000000 000000ff
[148193.520889] c316dc74 00000000 c316dc7c c1c01c10 fa0b27a9 c02b803e fa0b2784 f3ee5528
[148193.520889] Call Trace:
[148193.520889] [<fa0b27a9>] rpc_wait_bit_killable+0x25/0x2a [sunrpc]
[148193.520889] [<c02b803e>] __wait_on_bit+0x33/0x58
[148193.520889] [<fa0b2784>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
[148193.520889] [<fa0b2784>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
[148193.520889] [<c02b80c2>] out_of_line_wait_on_bit+0x5f/0x67
[148193.520889] [<c01317a1>] wake_bit_function+0x0/0x3c
[148193.520889] [<fa0b2bea>] __rpc_execute+0xbe/0x1d9 [sunrpc]
[148193.520889] [<fa0ada72>] rpc_run_task+0x40/0x45 [sunrpc]
[148193.520889] [<fa0adb00>] rpc_call_sync+0x38/0x52 [sunrpc]
[148193.520889] [<fa12e23d>] nfs3_rpc_wrapper+0x14/0x49 [nfs]
[148193.520889] [<fa12e76f>] nfs3_proc_access+0x101/0x166 [nfs]
[148193.520889] [<c01e0532>] number+0x13d/0x217
[148193.520889] [<c018f97f>] __find_get_block_slow+0xdd/0xe6
[148193.520889] [<c018fcee>] __find_get_block+0x176/0x180
[148193.520889] [<c0182aea>] __d_lookup+0x98/0xd7
[148193.520889] [<fa120c00>] nfs_do_access+0x134/0x2a7 [nfs]
[148193.520889] [<fa0b36d8>] rpcauth_lookupcred+0x69/0x8f [sunrpc]
[148193.520889] [<fa120e30>] nfs_permission+0xbd/0x11d [nfs]
[148193.520889] [<fa120d73>] nfs_permission+0x0/0x11d [nfs]
[148193.520889] [<c017a6cd>] permission+0xa3/0xfc
[148193.520889] [<c017b8f9>] __link_path_walk+0x11c/0xb0b
[148193.520889] [<f88f8170>] __journal_file_buffer+0xa2/0x123 [jbd]
[148193.520889] [<c0158670>] filemap_fault+0x1f5/0x359
[148193.520889] [<c017c31f>] path_walk+0x37/0x70
[148193.520889] [<c017c5ce>] do_path_lookup+0x122/0x184
[148193.520889] [<c017cfb0>] __path_lookup_intent_open+0x42/0x72
[148193.520889] [<c017d02f>] path_lookup_open+0xf/0x13
[148193.520889] [<c017d0ba>] do_filp_open+0x87/0x684
[148193.520889] [<c016462b>] free_pgtables+0x86/0x93
[148193.520889] [<c0172994>] do_sys_open+0x40/0xb0
[148193.520889] [<c0172a48>] sys_open+0x1e/0x23
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] sshd S 5fdc45c8 0 20038 20037
[148193.520889] f2f78d60 00000086 f2f78940 5fdc45c8 0000862b f2f78eec c1c0efa0 00000000
[148193.520889] f2e8e858 f2e8e640 00000599 00000000 c0129a63 00000001 00000286 f2fa3f70
[148193.520889] 7fffffff 00000000 f2fa3e5c c3036200 c02b7ea9 f3bdd840 00000140 f3bdd840
[148193.520889] Call Trace:
[148193.520889] [<c0129a63>] __mod_timer+0x99/0xa3
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c0253e7d>] __alloc_skb+0x49/0xf7
[148193.520889] [<c013187f>] prepare_to_wait+0x12/0x49
[148193.520889] [<c02a7004>] unix_stream_recvmsg+0x1e3/0x480
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c024e54b>] sock_aio_read+0xea/0xf8
[148193.520889] [<c0161c56>] __do_fault+0x30e/0x34d
[148193.520889] [<c0173fdf>] do_sync_read+0xbf/0xfe
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<f887880a>] e1000_clean+0xf7/0x215 [e1000]
[148193.520889] [<c01b9157>] security_file_permission+0xc/0xd
[148193.520889] [<c0174778>] vfs_read+0x92/0x11e
[148193.520889] [<c0174bb8>] sys_read+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] sshd S fe2ba3a2 0 20194 2733
[148193.520889] c30cf680 00000086 c30cfaa0 fe2ba3a2 000086c7 c30cf80c c1c0efa0 00000000
[148193.520889] c01042a7 02342ab8 00000001 00000000 c035b684 00000000 00000001 00000082
[148193.520889] 7fffffff c30c42c0 00000200 f2f09f9c c02b7ea9 00000282 c30c42c0 00000082
[148193.520889] Call Trace:
[148193.520889] [<c01042a7>] common_interrupt+0x23/0x28
[148193.520889] [<c02b7ea9>] schedule_timeout+0x13/0x86
[148193.520889] [<c0221548>] tty_ldisc_deref+0x4c/0x5b
[148193.520889] [<c0222343>] tty_poll+0x53/0x60
[148193.520889] [<c017f09a>] do_select+0x364/0x3bd
[148193.520889] [<c017f59e>] __pollwait+0x0/0xac
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c011b61d>] default_wake_function+0x0/0x8
[148193.520889] [<c0258370>] dev_hard_start_xmit+0x1eb/0x24f
[148193.520889] [<c0266456>] __qdisc_run+0xcc/0x17c
[148193.520889] [<c025a614>] dev_queue_xmit+0x287/0x2bf
[148193.520889] [<c025ea9c>] neigh_resolve_output+0x1f1/0x21b
[148193.520889] [<c0275d3f>] ip_finish_output+0x1d3/0x1fc
[148193.520889] [<c02758bf>] ip_local_out+0x15/0x17
[148193.520889] [<c02762e1>] ip_queue_xmit+0x25b/0x29d
[148193.520889] [<c028485c>] tcp_transmit_skb+0x6a2/0x6d5
[148193.520889] [<c0129950>] lock_timer_base+0x19/0x35
[148193.520889] [<c0129a63>] __mod_timer+0x99/0xa3
[148193.520889] [<c025070f>] sk_reset_timer+0xc/0x16
[148193.520889] [<c0285e67>] __tcp_push_pending_frames+0x630/0x6c9
[148193.520889] [<c02b8f4d>] _spin_lock_bh+0x8/0x1e
[148193.520889] [<c02505e8>] release_sock+0xf/0x85
[148193.520889] [<c027c427>] tcp_sendmsg+0x7fc/0x8fe
[148193.520889] [<c017f375>] core_sys_select+0x282/0x29f
[148193.520889] [<c024e454>] sock_aio_write+0xdd/0xea
[148193.520889] [<c0133c26>] enqueue_hrtimer+0xc9/0xd4
[148193.520889] [<c0173edf>] do_sync_write+0xbf/0x100
[148193.520889] [<c0131774>] autoremove_wake_function+0x0/0x2d
[148193.520889] [<c017f718>] sys_select+0xce/0x180
[148193.520889] [<c0174c1b>] sys_write+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] bash R running 0 20196 20194
[148193.520889] 0007f2b9 c310bef4 c03d1720 c310bef4 00000008 c01e0e95 00000000 00000000
[148193.520889] 3cef4169 00000008 ffffffff 00000021 00000400 c03d1720 01ffffff 00000010
[148193.520889] c03d1b20 00000000 ffffffff c030ef22 c310bec2 013d1720 0000000a ffffffff
[148193.520889] Call Trace:
[148193.520889] [<c01e0e95>] vsnprintf+0x411/0x452
[148193.520889] [<c0104774>] print_trace_address+0x0/0x37
[148193.520889] [<c0122d63>] printk+0x14/0x18
[148193.520889] [<c01400af>] __print_symbol+0x1c/0x25
[148193.520889] [<c013572d>] sched_clock_cpu+0x104/0x112
[148193.520889] [<c0134c00>] up+0x9/0x2a
[148193.520889] [<c01228cc>] release_console_sem+0x173/0x18c
[148193.520889] [<c01228cc>] release_console_sem+0x173/0x18c
[148193.520889] [<c0104774>] print_trace_address+0x0/0x37
[148193.520889] [<c0122d63>] printk+0x14/0x18
[148193.520889] [<c0104786>] print_trace_address+0x12/0x37
[148193.520889] [<c012fdf3>] __kernel_text_address+0x18/0x23
[148193.520889] [<c01103b4>] touch_nmi_watchdog+0x34/0x3e
[148193.520889] [<c010481e>] dump_trace+0x73/0xbd
[148193.520889] [<c010487c>] show_trace_log_lvl+0x14/0x29
[148193.520889] [<c0104921>] show_stack_log_lvl+0x90/0x98
[148193.520889] [<c0104f07>] show_stack+0x22/0x28
[148193.520889] [<c011f170>] show_state_filter+0x39/0x78
[148193.520889] [<c0230dd5>] __handle_sysrq+0x83/0xfa
[148193.520889] [<c01a6131>] write_sysrq_trigger+0x0/0x29
[148193.520889] [<c01a6154>] write_sysrq_trigger+0x23/0x29
[148193.520889] [<c01a0ee5>] proc_reg_write+0x58/0x6b
[148193.520889] [<c01a0e8d>] proc_reg_write+0x0/0x6b
[148193.520889] [<c0174649>] vfs_write+0x83/0x120
[148193.520889] [<c0174c1b>] sys_write+0x3c/0x63
[148193.520889] [<c0103853>] sysenter_past_esp+0x78/0xb1
[148193.520889] =======================
[148193.520889] Sched Debug Version: v0.07, 2.6.26-1-686 #1
[148193.520889] now at 148193560.809862 msecs
[148193.520889] .sysctl_sched_latency : 20.000000
[148193.520889] .sysctl_sched_min_granularity : 4.000000
[148193.520889] .sysctl_sched_wakeup_granularity : 10.000000
[148193.520889] .sysctl_sched_child_runs_first : 0.000001
[148193.520889] .sysctl_sched_features : 895
[148193.520889]
[148193.520889] cpu#0, 2162.720 MHz
[148193.520889] .nr_running : 2
[148193.520889] .load : 2048
[148193.520889] .nr_switches : 156718457
[148193.520889] .nr_load_updates : 17492867
[148193.520889] .nr_uninterruptible : 5
[148193.520889] .jiffies : 36973380
[148193.520889] .next_balance : 36.973380
[148193.520889] .curr->pid : 20196
[148193.520889] .clock : 148193520.886692
[148193.520889] .cpu_load[0] : 0
[148193.520889] .cpu_load[1] : 0
[148193.520889] .cpu_load[2] : 0
[148193.520889] .cpu_load[3] : 0
[148193.520889] .cpu_load[4] : 0
[148193.520889]
[148193.520889] cfs_rq[0]:
[148193.520889] .exec_clock : 0.000000
[148193.520889] .MIN_vruntime : 123570597.680058
[148193.520889] .min_vruntime : 123570617.680058
[148193.520889] .max_vruntime : 123570597.680058
[148193.520889] .spread : 0.000000
[148193.520889] .spread0 : 0.000000
[148193.520889] .nr_running : 2
[148193.520889] .load : 2048
[148193.520889] .nr_spread_over : 0
[148193.520889]
[148193.520889] runnable tasks:
[148193.520889] task PID tree-key switches prio exec-runtime sum-exec sum-sleep
[148193.520889] ----------------------------------------------------------------------------------------------------------
[148193.520889] klogd 2696 123570597.680058 7328 120 0 0 0.000000 0.000000 0.000000 /
[148193.520889] R bash 20196 123570597.680062 502 120 0 0 0.000000 0.000000 0.000000 /
[148193.520889]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-22 10:23 ` Ian Campbell
@ 2008-08-22 18:08 ` Trond Myklebust
2008-08-22 18:13 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: Trond Myklebust @ 2008-08-22 18:08 UTC (permalink / raw)
To: Ian Campbell; +Cc: Grant Coady, linux-kernel, neilb, bfields, linux-nfs
On Fri, 2008-08-22 at 11:23 +0100, Ian Campbell wrote:
> On Mon, 2008-08-18 at 15:20 -0400, Trond Myklebust wrote:
> > Please try to reproduce the hang, then do
> >
> > echo 0 >/proc/sys/sunrpc/rpc_debug
> >
> > and send the output from 'dmesg'...
>
> I've also been seeing some NFS related lockups, although I'm not sure if
> they are the same as the one in this thread or not. Client is 2.6.26
> (Debian's kernel) and Server is 2.6.25 (also a Debian kernel, but from
> backports.org).
>
> rpc_debug on the server gives nothing, on the client gives:
> [144741.637997] -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops--
> [144741.637997] 3439 0004 0080 -11 f3f48200 100003 f7770000 0 xprt_sending fa0ae88e fa0bddf4
> [144741.637997] 3438 0001 00a0 0 f77f2a00 100003 f77700d0 15000 xprt_pending fa0ae88e fa0bddf4
That's probably also a networking device driver issue candidate: your
RPC task is queued up waiting to be sent.
What networking card+device driver are you using here?
> There are no processes running with pid 3439 3438 (I don't think it's
> that sort of pid though).
The 'pid' is an internal RPC cookie that just serves to identify and
track specific RPC requests.
Cheers
Trond
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-22 18:08 ` Trond Myklebust
@ 2008-08-22 18:13 ` Ian Campbell
2008-08-22 19:33 ` John Ronciak
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-08-22 18:13 UTC (permalink / raw)
To: Trond Myklebust
Cc: Grant Coady, linux-kernel, neilb, bfields, linux-nfs,
Jeff Kirsher, Jesse Brandeburg, Bruce Allan, PJ Waskiewicz,
John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 1742 bytes --]
On Fri, 2008-08-22 at 11:08 -0700, Trond Myklebust wrote:
> On Fri, 2008-08-22 at 11:23 +0100, Ian Campbell wrote:
> > On Mon, 2008-08-18 at 15:20 -0400, Trond Myklebust wrote:
> > > Please try to reproduce the hang, then do
> > >
> > > echo 0 >/proc/sys/sunrpc/rpc_debug
> > >
> > > and send the output from 'dmesg'...
> >
> > I've also been seeing some NFS related lockups, although I'm not sure if
> > they are the same as the one in this thread or not. Client is 2.6.26
> > (Debian's kernel) and Server is 2.6.25 (also a Debian kernel, but from
> > backports.org).
> >
> > rpc_debug on the server gives nothing, on the client gives:
> > [144741.637997] -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops--
> > [144741.637997] 3439 0004 0080 -11 f3f48200 100003 f7770000 0 xprt_sending fa0ae88e fa0bddf4
> > [144741.637997] 3438 0001 00a0 0 f77f2a00 100003 f77700d0 15000 xprt_pending fa0ae88e fa0bddf4
>
> That's probably also a networking device driver issue candidate: your
> RPC task is queued up waiting to be sent.
>
> What networking card+device driver are you using here?
# ethtool -i eth0
driver: e1000
version: 7.3.20-k2-NAPI
firmware-version: N/A
bus-info: 0000:01:0a.0
Adding CC's to peeps listed in MAINTAINERS
I have to reboot the system now (or else there will be no TV to watch
during my tea ;-). It will probably repro again quite soon though.
> > There are no processes running with pid 3439 3438 (I don't think it's
> > that sort of pid though).
>
> The 'pid' is an internal RPC cookie that just serves to identify and
> track specific RPC requests.
Right, thanks.
Ian.
--
Ian Campbell
Colors may fade.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-22 18:13 ` Ian Campbell
@ 2008-08-22 19:33 ` John Ronciak
2008-08-22 20:00 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: John Ronciak @ 2008-08-22 19:33 UTC (permalink / raw)
To: Ian Campbell
Cc: Trond Myklebust, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
On Fri, Aug 22, 2008 at 11:13 AM, Ian Campbell <ijc@hellion.org.uk> wrote:
>> That's probably also a networking device driver issue candidate: your
>> RPC task is queued up waiting to be sent.
>>
>> What networking card+device driver are you using here?
>
> # ethtool -i eth0
> driver: e1000
> version: 7.3.20-k2-NAPI
> firmware-version: N/A
> bus-info: 0000:01:0a.0
There is nothing indicating that the NIC/driver is causing any sort of
problem here, at least not with what has been presented so far. When
the NFS mount isn't working is the networking still active and
working?
--
Cheers,
John
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-22 19:33 ` John Ronciak
@ 2008-08-22 20:00 ` Ian Campbell
2008-08-22 21:15 ` John Ronciak
2008-08-22 21:23 ` Trond Myklebust
0 siblings, 2 replies; 64+ messages in thread
From: Ian Campbell @ 2008-08-22 20:00 UTC (permalink / raw)
To: John Ronciak
Cc: Trond Myklebust, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]
On Fri, 2008-08-22 at 12:33 -0700, John Ronciak wrote:
> On Fri, Aug 22, 2008 at 11:13 AM, Ian Campbell <ijc@hellion.org.uk> wrote:
>
> >> That's probably also a networking device driver issue candidate: your
> >> RPC task is queued up waiting to be sent.
> >>
> >> What networking card+device driver are you using here?
> >
> > # ethtool -i eth0
> > driver: e1000
> > version: 7.3.20-k2-NAPI
> > firmware-version: N/A
> > bus-info: 0000:01:0a.0
> There is nothing indicating that the NIC/driver is causing any sort of
> problem here, at least not with what has been presented so far. When
> the NFS mount isn't working is the networking still active and
> working?
So far as I can tell, yes. I can login via ssh so long as the user
doesn't have NFS $HOME, I haven't tried much else and the box isn't
locked up at the moment, I'd bet it's fine though.
Ian.
--
Ian Campbell
"On a normal ascii line, the only safe condition to detect is a 'BREAK'
- everything else having been assigned functions by Gnu EMACS."
(By Tarl Neustaedter)
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-22 20:00 ` Ian Campbell
@ 2008-08-22 21:15 ` John Ronciak
2008-08-22 21:23 ` Trond Myklebust
1 sibling, 0 replies; 64+ messages in thread
From: John Ronciak @ 2008-08-22 21:15 UTC (permalink / raw)
To: Ian Campbell
Cc: Trond Myklebust, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
On Fri, Aug 22, 2008 at 1:00 PM, Ian Campbell <ijc@hellion.org.uk> wrote:
> So far as I can tell, yes. I can login via ssh so long as the user
> doesn't have NFS $HOME, I haven't tried much else and the box isn't
> locked up at the moment, I'd bet it's fine though.
Then it's highly unlikely that the NIC/driver is causing your problem.
We'll continue to monitor this thread waiting for any info that this
is somehow NIC/driver related.
Good luck.
--
Cheers,
John
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-22 20:00 ` Ian Campbell
2008-08-22 21:15 ` John Ronciak
@ 2008-08-22 21:23 ` Trond Myklebust
2008-08-22 21:37 ` Ian Campbell
1 sibling, 1 reply; 64+ messages in thread
From: Trond Myklebust @ 2008-08-22 21:23 UTC (permalink / raw)
To: Ian Campbell
Cc: John Ronciak, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
On Fri, 2008-08-22 at 21:00 +0100, Ian Campbell wrote:
> On Fri, 2008-08-22 at 12:33 -0700, John Ronciak wrote:
> > On Fri, Aug 22, 2008 at 11:13 AM, Ian Campbell <ijc@hellion.org.uk> wrote:
> >
> > >> That's probably also a networking device driver issue candidate: your
> > >> RPC task is queued up waiting to be sent.
> > >>
> > >> What networking card+device driver are you using here?
> > >
> > > # ethtool -i eth0
> > > driver: e1000
> > > version: 7.3.20-k2-NAPI
> > > firmware-version: N/A
> > > bus-info: 0000:01:0a.0
> > There is nothing indicating that the NIC/driver is causing any sort of
> > problem here, at least not with what has been presented so far. When
> > the NFS mount isn't working is the networking still active and
> > working?
>
> So far as I can tell, yes. I can login via ssh so long as the user
> doesn't have NFS $HOME, I haven't tried much else and the box isn't
> locked up at the moment, I'd bet it's fine though.
...and the server? Something is preventing that RPC payload from being
delivered...
Trond
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-22 21:23 ` Trond Myklebust
@ 2008-08-22 21:37 ` Ian Campbell
2008-08-22 21:56 ` Trond Myklebust
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-08-22 21:37 UTC (permalink / raw)
To: Trond Myklebust
Cc: John Ronciak, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 1671 bytes --]
On Fri, 2008-08-22 at 14:23 -0700, Trond Myklebust wrote:
> On Fri, 2008-08-22 at 21:00 +0100, Ian Campbell wrote:
> > On Fri, 2008-08-22 at 12:33 -0700, John Ronciak wrote:
> > > On Fri, Aug 22, 2008 at 11:13 AM, Ian Campbell <ijc@hellion.org.uk> wrote:
> > >
> > > >> That's probably also a networking device driver issue candidate: your
> > > >> RPC task is queued up waiting to be sent.
> > > >>
> > > >> What networking card+device driver are you using here?
> > > >
> > > > # ethtool -i eth0
> > > > driver: e1000
> > > > version: 7.3.20-k2-NAPI
> > > > firmware-version: N/A
> > > > bus-info: 0000:01:0a.0
> > > There is nothing indicating that the NIC/driver is causing any sort of
> > > problem here, at least not with what has been presented so far. When
> > > the NFS mount isn't working is the networking still active and
> > > working?
> >
> > So far as I can tell, yes. I can login via ssh so long as the user
> > doesn't have NFS $HOME, I haven't tried much else and the box isn't
> > locked up at the moment, I'd bet it's fine though.
>
> ...and the server? Something is preventing that RPC payload from being
> delivered...
I can ssh to the server fine. The same server also serves my NFS home
directory to the box I'm writing this from and I've not seen any trouble
with this box at all, it's a 2.6.18-xen box.
If I downgrade the problematic box to 2.6.24 then the hangs do not
occur. They do occur with 2.6.25. (sorry that's a critical bit of
information which I stupidly forgot to mention up til now).
Ian.
--
Ian Campbell
It is much easier to be critical than to be correct.
-- Benjamin Disraeli
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-22 21:37 ` Ian Campbell
@ 2008-08-22 21:56 ` Trond Myklebust
2008-08-22 22:41 ` Ian Campbell
` (2 more replies)
0 siblings, 3 replies; 64+ messages in thread
From: Trond Myklebust @ 2008-08-22 21:56 UTC (permalink / raw)
To: Ian Campbell
Cc: John Ronciak, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
On Fri, 2008-08-22 at 22:37 +0100, Ian Campbell wrote:
> I can ssh to the server fine. The same server also serves my NFS home
> directory to the box I'm writing this from and I've not seen any trouble
> with this box at all, it's a 2.6.18-xen box.
OK... Are you able to reproduce the problem reliably?
If so, can you provide me with a binary tcpdump or wireshark dump? If
using tcpdump, then please use something like
tcpdump -w /tmp/dump.out -s 90000 host myserver.foo.bar and port 2049
Please also try to provide a netstat dump of the current TCP connections
as soon as the hang occurs:
netstat -t
Cheers
Trond
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-22 21:56 ` Trond Myklebust
@ 2008-08-22 22:41 ` Ian Campbell
2008-08-24 18:53 ` Ian Campbell
2008-08-25 20:23 ` Grant Coady
2 siblings, 0 replies; 64+ messages in thread
From: Ian Campbell @ 2008-08-22 22:41 UTC (permalink / raw)
To: Trond Myklebust
Cc: John Ronciak, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]
On Fri, 2008-08-22 at 14:56 -0700, Trond Myklebust wrote:
> On Fri, 2008-08-22 at 22:37 +0100, Ian Campbell wrote:
> > I can ssh to the server fine. The same server also serves my NFS home
> > directory to the box I'm writing this from and I've not seen any trouble
> > with this box at all, it's a 2.6.18-xen box.
>
> OK... Are you able to reproduce the problem reliably?
It usually happens in around a day, but I can't make it happen at will
so that I can arrange to be present at the time. It has usually locked
up over night in the past.
> If so, can you provide me with a binary tcpdump or wireshark dump? If
> using tcpdump, then please use something like
>
> tcpdump -w /tmp/dump.out -s 90000 host myserver.foo.bar and port 2049
I'll try leaving this going overnight but using -C and -W to limit the
size to the disk space available.
> Please also try to provide a netstat dump of the current TCP connections
> as soon as the hang occurs:
>
> netstat -t
Will do it ASAP after it happens.
Ian.
--
Ian Campbell
revision 1.17.2.7
date: 2001/05/31 21:32:44; author: branden; state: Exp; lines: +1 -1
ARRRRGH!! GOT THE G** D*** SENSE OF A F******* TEST BACKWARDS!
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-22 21:56 ` Trond Myklebust
2008-08-22 22:41 ` Ian Campbell
@ 2008-08-24 18:53 ` Ian Campbell
2008-08-24 19:17 ` Trond Myklebust
2008-08-25 20:23 ` Grant Coady
2 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-08-24 18:53 UTC (permalink / raw)
To: Trond Myklebust
Cc: John Ronciak, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1.1: Type: text/plain, Size: 2674 bytes --]
On Fri, 2008-08-22 at 14:56 -0700, Trond Myklebust wrote:
> On Fri, 2008-08-22 at 22:37 +0100, Ian Campbell wrote:
> > I can ssh to the server fine. The same server also serves my NFS home
> > directory to the box I'm writing this from and I've not seen any trouble
> > with this box at all, it's a 2.6.18-xen box.
>
> OK... Are you able to reproduce the problem reliably?
>
> If so, can you provide me with a binary tcpdump or wireshark dump? If
> using tcpdump, then please use something like
>
> tcpdump -w /tmp/dump.out -s 90000 host myserver.foo.bar and port 2049
>
> Please also try to provide a netstat dump of the current TCP connections
> as soon as the hang occurs:
>
> netstat -t
Aug 24 18:08:59 iranon kernel: [168839.556017] nfs: server hopkins not responding, still trying
but I wasn't around until 19:38 to spot it.
netstat when I got to it was:
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost.localdo:50891 localhost.localdom:6543 ESTABLISHED
tcp 1 0 iranon.hellion.org.:ssh azathoth.hellion.:52682 CLOSE_WAIT
tcp 0 0 localhost.localdom:6543 localhost.localdo:50893 ESTABLISHED
tcp 0 0 iranon.hellion.org.:837 hopkins.hellion.org:nfs FIN_WAIT2
tcp 0 0 localhost.localdom:6543 localhost.localdo:41831 ESTABLISHED
tcp 0 0 localhost.localdo:13666 localhost.localdo:59482 ESTABLISHED
tcp 0 0 localhost.localdo:34288 localhost.localdom:6545 ESTABLISHED
tcp 0 0 iranon.hellion.org.:ssh azathoth.hellion.:48977 ESTABLISHED
tcp 0 0 iranon.hellion.org.:ssh azathoth.hellion.:52683 ESTABLISHED
tcp 0 0 localhost.localdom:6545 localhost.localdo:34288 ESTABLISHED
tcp 0 0 localhost.localdom:6543 localhost.localdo:50891 ESTABLISHED
tcp 0 0 localhost.localdo:50893 localhost.localdom:6543 ESTABLISHED
tcp 0 0 localhost.localdo:41831 localhost.localdom:6543 ESTABLISHED
tcp 0 87 localhost.localdo:59482 localhost.localdo:13666 ESTABLISHED
tcp 1 0 localhost.localdom:6543 localhost.localdo:41830 CLOSE_WAIT
(iranon is the problematic host .4, azathoth is my desktop machine .5, hopkins is the NFS server .6)
tcpdumps are pretty big. I've attached the last 100 packets captured. If
you need more I can put the full file up somewhere.
-rw-r--r-- 1 root root 1.3G Aug 24 17:57 dump.out0
-rw-r--r-- 1 root root 536M Aug 24 19:38 dump.out1
Ian.
--
Ian Campbell
Prizes are for children.
-- Charles Ives, upon being given, but refusing, the
Pulitzer prize
[-- Attachment #1.2: last100.dump.bz2 --]
[-- Type: application/x-bzip, Size: 107752 bytes --]
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-24 18:53 ` Ian Campbell
@ 2008-08-24 19:17 ` Trond Myklebust
2008-08-24 19:19 ` Trond Myklebust
0 siblings, 1 reply; 64+ messages in thread
From: Trond Myklebust @ 2008-08-24 19:17 UTC (permalink / raw)
To: Ian Campbell
Cc: John Ronciak, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
On Sun, 2008-08-24 at 19:53 +0100, Ian Campbell wrote:
> On Fri, 2008-08-22 at 14:56 -0700, Trond Myklebust wrote:
> > On Fri, 2008-08-22 at 22:37 +0100, Ian Campbell wrote:
> > > I can ssh to the server fine. The same server also serves my NFS home
> > > directory to the box I'm writing this from and I've not seen any trouble
> > > with this box at all, it's a 2.6.18-xen box.
> >
> > OK... Are you able to reproduce the problem reliably?
> >
> > If so, can you provide me with a binary tcpdump or wireshark dump? If
> > using tcpdump, then please use something like
> >
> > tcpdump -w /tmp/dump.out -s 90000 host myserver.foo.bar and port 2049
> >
> > Please also try to provide a netstat dump of the current TCP connections
> > as soon as the hang occurs:
> >
> > netstat -t
>
> Aug 24 18:08:59 iranon kernel: [168839.556017] nfs: server hopkins not responding, still trying
> but I wasn't around until 19:38 to spot it.
>
> netstat when I got to it was:
>
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 0 localhost.localdo:50891 localhost.localdom:6543 ESTABLISHED
> tcp 1 0 iranon.hellion.org.:ssh azathoth.hellion.:52682 CLOSE_WAIT
> tcp 0 0 localhost.localdom:6543 localhost.localdo:50893 ESTABLISHED
> tcp 0 0 iranon.hellion.org.:837 hopkins.hellion.org:nfs FIN_WAIT2
> tcp 0 0 localhost.localdom:6543 localhost.localdo:41831 ESTABLISHED
> tcp 0 0 localhost.localdo:13666 localhost.localdo:59482 ESTABLISHED
> tcp 0 0 localhost.localdo:34288 localhost.localdom:6545 ESTABLISHED
> tcp 0 0 iranon.hellion.org.:ssh azathoth.hellion.:48977 ESTABLISHED
> tcp 0 0 iranon.hellion.org.:ssh azathoth.hellion.:52683 ESTABLISHED
> tcp 0 0 localhost.localdom:6545 localhost.localdo:34288 ESTABLISHED
> tcp 0 0 localhost.localdom:6543 localhost.localdo:50891 ESTABLISHED
> tcp 0 0 localhost.localdo:50893 localhost.localdom:6543 ESTABLISHED
> tcp 0 0 localhost.localdo:41831 localhost.localdom:6543 ESTABLISHED
> tcp 0 87 localhost.localdo:59482 localhost.localdo:13666 ESTABLISHED
> tcp 1 0 localhost.localdom:6543 localhost.localdo:41830 CLOSE_WAIT
>
> (iranon is the problematic host .4, azathoth is my desktop machine .5, hopkins is the NFS server .6)
>
> tcpdumps are pretty big. I've attached the last 100 packets captured. If
> you need more I can put the full file up somewhere.
>
> -rw-r--r-- 1 root root 1.3G Aug 24 17:57 dump.out0
> -rw-r--r-- 1 root root 536M Aug 24 19:38 dump.out1
>
> Ian.
>From the tcpdump, it looks as if the NFS server is failing to close the
socket, when the client closes its side. You therefore end up getting
stuck in the FIN_WAIT2 state (as netstat clearly shows above).
Is the server keeping the client in this state for a very long period?
Cheers
Trond
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-24 19:17 ` Trond Myklebust
@ 2008-08-24 19:19 ` Trond Myklebust
2008-08-24 22:09 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: Trond Myklebust @ 2008-08-24 19:19 UTC (permalink / raw)
To: Ian Campbell
Cc: John Ronciak, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
On Sun, 2008-08-24 at 15:17 -0400, Trond Myklebust wrote:
> >From the tcpdump, it looks as if the NFS server is failing to close the
> socket, when the client closes its side. You therefore end up getting
> stuck in the FIN_WAIT2 state (as netstat clearly shows above).
BTW: the RPC client is closing the socket because it detected no NFS
activity for 5 minutes. Did you expect any NFS activity during this
time?
Trond
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-24 19:19 ` Trond Myklebust
@ 2008-08-24 22:09 ` Ian Campbell
2008-08-24 22:15 ` Trond Myklebust
2008-08-25 21:39 ` Roger Heflin
0 siblings, 2 replies; 64+ messages in thread
From: Ian Campbell @ 2008-08-24 22:09 UTC (permalink / raw)
To: Trond Myklebust
Cc: John Ronciak, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 1151 bytes --]
(added some quoting from previous mail to save replying twice)
On Sun, 2008-08-24 at 15:19 -0400, Trond Myklebust wrote:
> On Sun, 2008-08-24 at 15:17 -0400, Trond Myklebust wrote:
> > >From the tcpdump, it looks as if the NFS server is failing to close the
> > socket, when the client closes its side. You therefore end up getting
> > stuck in the FIN_WAIT2 state (as netstat clearly shows above).
> >
> > Is the server keeping the client in this state for a very long
> > period?
Well, it had been around an hour and a half on this occasion. Next time
it happens I can wait longer but I'm pretty sure I've come back from
time away and it's been wedged for at least a day. How long would you
expect it to remain in this state for?
> BTW: the RPC client is closing the socket because it detected no NFS
> activity for 5 minutes. Did you expect any NFS activity during this
> time?
It's a mythtv box so at times where no one is watching anything and
there isn't anything to record I expect NFS activity is pretty minimal.
Ian.
--
Ian Campbell
Can anyone remember when the times were not hard, and money not scarce?
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-24 22:09 ` Ian Campbell
@ 2008-08-24 22:15 ` Trond Myklebust
2008-08-25 9:59 ` Ian Campbell
2008-08-25 16:04 ` Tom Tucker
2008-08-25 21:39 ` Roger Heflin
1 sibling, 2 replies; 64+ messages in thread
From: Trond Myklebust @ 2008-08-24 22:15 UTC (permalink / raw)
To: Ian Campbell
Cc: John Ronciak, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
On Sun, 2008-08-24 at 23:09 +0100, Ian Campbell wrote:
> (added some quoting from previous mail to save replying twice)
>
> On Sun, 2008-08-24 at 15:19 -0400, Trond Myklebust wrote:
> > On Sun, 2008-08-24 at 15:17 -0400, Trond Myklebust wrote:
> > > >From the tcpdump, it looks as if the NFS server is failing to close the
> > > socket, when the client closes its side. You therefore end up getting
> > > stuck in the FIN_WAIT2 state (as netstat clearly shows above).
> > >
> > > Is the server keeping the client in this state for a very long
> > > period?
>
> Well, it had been around an hour and a half on this occasion. Next time
> it happens I can wait longer but I'm pretty sure I've come back from
> time away and it's been wedged for at least a day. How long would you
> expect it to remain in this state for?
The server should ideally start to close the socket as soon as it
receives the FIN from the client. I'll have a look at the code.
Trond
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-24 22:15 ` Trond Myklebust
@ 2008-08-25 9:59 ` Ian Campbell
2008-08-25 16:04 ` Tom Tucker
1 sibling, 0 replies; 64+ messages in thread
From: Ian Campbell @ 2008-08-25 9:59 UTC (permalink / raw)
To: Trond Myklebust
Cc: John Ronciak, Grant Coady, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 832 bytes --]
On Sun, 2008-08-24 at 18:15 -0400, Trond Myklebust wrote:
> The server should ideally start to close the socket as soon as it
> receives the FIN from the client. I'll have a look at the code.
The server is a 2.6.25 kernel from Debian backports.org. I was planning
to downgrade to the 2.6.24 "Etch and a half" kernel but was holding off
while this thread was in progress. If you think it would be a useful
data point I can do it sooner rather than later.
It's changing the client kernel which seems to make a difference since
running 2.6.24 against this server has given me no trouble. I guess the
newer client could expose a bug in the server though.
Ian.
--
Ian Campbell
I can't understand why a person will take a year or two to write a
novel when he can easily buy one for a few dollars.
-- Fred Allen
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-24 22:15 ` Trond Myklebust
2008-08-25 9:59 ` Ian Campbell
@ 2008-08-25 16:04 ` Tom Tucker
2008-08-25 16:54 ` Trond Myklebust
2008-08-26 19:27 ` J. Bruce Fields
1 sibling, 2 replies; 64+ messages in thread
From: Tom Tucker @ 2008-08-25 16:04 UTC (permalink / raw)
To: Trond Myklebust
Cc: Ian Campbell, John Ronciak, Grant Coady, linux-kernel, neilb,
bfields, linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
Trond Myklebust wrote:
> On Sun, 2008-08-24 at 23:09 +0100, Ian Campbell wrote:
>> (added some quoting from previous mail to save replying twice)
>>
>> On Sun, 2008-08-24 at 15:19 -0400, Trond Myklebust wrote:
>>> On Sun, 2008-08-24 at 15:17 -0400, Trond Myklebust wrote:
>>>> >From the tcpdump, it looks as if the NFS server is failing to close the
>>>> socket, when the client closes its side. You therefore end up getting
>>>> stuck in the FIN_WAIT2 state (as netstat clearly shows above).
>>>>
>>>> Is the server keeping the client in this state for a very long
>>>> period?
>> Well, it had been around an hour and a half on this occasion. Next time
>> it happens I can wait longer but I'm pretty sure I've come back from
>> time away and it's been wedged for at least a day. How long would you
>> expect it to remain in this state for?
>
> The server should ideally start to close the socket as soon as it
> receives the FIN from the client. I'll have a look at the code.
>
I don't think it should matter how long the connection stays in FIN WAIT,
the client should reconnect anyway.
Since the client seems to be the variable, I would think it might be an
issue with the client reconnect logic?
That said, 2.6.25 is when the server side transport switch logic went in.
Tom
> Trond
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-25 16:04 ` Tom Tucker
@ 2008-08-25 16:54 ` Trond Myklebust
2008-08-25 20:15 ` Tom Tucker
2008-08-26 19:27 ` J. Bruce Fields
1 sibling, 1 reply; 64+ messages in thread
From: Trond Myklebust @ 2008-08-25 16:54 UTC (permalink / raw)
To: Tom Tucker
Cc: Ian Campbell, John Ronciak, Grant Coady, linux-kernel, neilb,
bfields, linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
On Mon, 2008-08-25 at 11:04 -0500, Tom Tucker wrote:
> I don't think it should matter how long the connection stays in FIN WAIT,
> the client should reconnect anyway.
Which would break the replay cache, since we can't reconnect using the
same port.
> Since the client seems to be the variable, I would think it might be an
> issue with the client reconnect logic?
No.
Trond
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-25 16:54 ` Trond Myklebust
@ 2008-08-25 20:15 ` Tom Tucker
0 siblings, 0 replies; 64+ messages in thread
From: Tom Tucker @ 2008-08-25 20:15 UTC (permalink / raw)
To: Trond Myklebust
Cc: Ian Campbell, John Ronciak, Grant Coady, linux-kernel, neilb,
bfields, linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
Trond Myklebust wrote:
> On Mon, 2008-08-25 at 11:04 -0500, Tom Tucker wrote:
>> I don't think it should matter how long the connection stays in FIN WAIT,
>> the client should reconnect anyway.
>
> Which would break the replay cache, since we can't reconnect using the
> same port.
>
>> Since the client seems to be the variable, I would think it might be an
>> issue with the client reconnect logic?
>
> No.
>
Yeah, you're right. I got confused about which host was stuck in FIN_WAIT2.
It's the client as you pointed out earlier.
Thanks,
Tom
> Trond
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-22 21:56 ` Trond Myklebust
2008-08-22 22:41 ` Ian Campbell
2008-08-24 18:53 ` Ian Campbell
@ 2008-08-25 20:23 ` Grant Coady
2008-08-25 22:11 ` Trond Myklebust
2 siblings, 1 reply; 64+ messages in thread
From: Grant Coady @ 2008-08-25 20:23 UTC (permalink / raw)
To: Trond Myklebust
Cc: Ian Campbell, John Ronciak, Grant Coady, linux-kernel, neilb,
bfields, linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
On Fri, 22 Aug 2008 14:56:53 -0700, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>On Fri, 2008-08-22 at 22:37 +0100, Ian Campbell wrote:
>> I can ssh to the server fine. The same server also serves my NFS home
>> directory to the box I'm writing this from and I've not seen any trouble
>> with this box at all, it's a 2.6.18-xen box.
>
>OK... Are you able to reproduce the problem reliably?
>
>If so, can you provide me with a binary tcpdump or wireshark dump? If
>using tcpdump, then please use something like
>
> tcpdump -w /tmp/dump.out -s 90000 host myserver.foo.bar and port 2049
^^^^^^^^--> typo?
man tcpdump:
-s Snarf snaplen bytes of data from each packet rather than the default of
68 (with SunOS's NIT, the minimum is actually 96). 68 bytes is adequate
I've reverted the NFS server to 2.6.24.7 -- inconclusive results for me
'cos NFS stalls seem so random.
Grant.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-24 22:09 ` Ian Campbell
2008-08-24 22:15 ` Trond Myklebust
@ 2008-08-25 21:39 ` Roger Heflin
1 sibling, 0 replies; 64+ messages in thread
From: Roger Heflin @ 2008-08-25 21:39 UTC (permalink / raw)
To: Ian Campbell
Cc: Trond Myklebust, John Ronciak, Grant Coady, linux-kernel, neilb,
bfields, linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
Ian Campbell wrote:
> (added some quoting from previous mail to save replying twice)
>
> On Sun, 2008-08-24 at 15:19 -0400, Trond Myklebust wrote:
>> On Sun, 2008-08-24 at 15:17 -0400, Trond Myklebust wrote:
>>> >From the tcpdump, it looks as if the NFS server is failing to close the
>>> socket, when the client closes its side. You therefore end up getting
>>> stuck in the FIN_WAIT2 state (as netstat clearly shows above).
>>>
>>> Is the server keeping the client in this state for a very long
>>> period?
>
> Well, it had been around an hour and a half on this occasion. Next time
> it happens I can wait longer but I'm pretty sure I've come back from
> time away and it's been wedged for at least a day. How long would you
> expect it to remain in this state for?
>
>> BTW: the RPC client is closing the socket because it detected no NFS
>> activity for 5 minutes. Did you expect any NFS activity during this
>> time?
>
> It's a mythtv box so at times where no one is watching anything and
> there isn't anything to record I expect NFS activity is pretty minimal.
>
> Ian.
>
Ian,
Do you have a recording group setup on the NFS partition that mythtv is
going to be accessing?
I have seen similar funny stuff happen, it used to happened around 2.6.22*
(on each end), and quit happening around 2.6.24* and now has started happening
again with 2.6.25* on both ends.
Similar to what you have the only thing I see is "NFS server not responding" and
restarting the NFS server end (/etc/init.d/nfs restart) appears to get things to
continue on the NFS client. No other messages appear on either end that
indicate that anything is wrong, other non-nfs partitions on the client work
find, the machine is still up, and the NFS server is still up and fine, and
after a restart things will again work for a while (hours or days).
Roger
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-25 20:23 ` Grant Coady
@ 2008-08-25 22:11 ` Trond Myklebust
2008-08-26 0:29 ` Grant Coady
2008-08-26 1:06 ` Grant Coady
0 siblings, 2 replies; 64+ messages in thread
From: Trond Myklebust @ 2008-08-25 22:11 UTC (permalink / raw)
To: Grant Coady
Cc: Ian Campbell, John Ronciak, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
On Tue, 2008-08-26 at 06:23 +1000, Grant Coady wrote:
> On Fri, 22 Aug 2008 14:56:53 -0700, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>
> >On Fri, 2008-08-22 at 22:37 +0100, Ian Campbell wrote:
> >> I can ssh to the server fine. The same server also serves my NFS home
> >> directory to the box I'm writing this from and I've not seen any trouble
> >> with this box at all, it's a 2.6.18-xen box.
> >
> >OK... Are you able to reproduce the problem reliably?
> >
> >If so, can you provide me with a binary tcpdump or wireshark dump? If
> >using tcpdump, then please use something like
> >
> > tcpdump -w /tmp/dump.out -s 90000 host myserver.foo.bar and port 2049
> ^^^^^^^^--> typo?
No. The intention was to record _all_ the info in the packet for
analysis, not just random header info.
Trond
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-25 22:11 ` Trond Myklebust
@ 2008-08-26 0:29 ` Grant Coady
2008-08-26 0:59 ` Muntz, Daniel
2008-08-26 1:06 ` Grant Coady
1 sibling, 1 reply; 64+ messages in thread
From: Grant Coady @ 2008-08-26 0:29 UTC (permalink / raw)
To: Trond Myklebust
Cc: Grant Coady, Ian Campbell, John Ronciak, linux-kernel, neilb,
bfields, linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
On Mon, 25 Aug 2008 18:11:12 -0400, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>On Tue, 2008-08-26 at 06:23 +1000, Grant Coady wrote:
>> On Fri, 22 Aug 2008 14:56:53 -0700, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>>
>> >On Fri, 2008-08-22 at 22:37 +0100, Ian Campbell wrote:
>> >> I can ssh to the server fine. The same server also serves my NFS home
>> >> directory to the box I'm writing this from and I've not seen any trouble
>> >> with this box at all, it's a 2.6.18-xen box.
>> >
>> >OK... Are you able to reproduce the problem reliably?
>> >
>> >If so, can you provide me with a binary tcpdump or wireshark dump? If
>> >using tcpdump, then please use something like
>> >
>> > tcpdump -w /tmp/dump.out -s 90000 host myserver.foo.bar and port 2049
>> ^^^^^^^^--> typo?
>
>No. The intention was to record _all_ the info in the packet for
>analysis, not just random header info.
Hi Trond,
My tcpdump seems to have a 16 bit snaplen counter:
~# tcpdump -w /tmp/dump.out -s 65535 host deltree and port 2049
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
^C0 packets captured
4 packets received by filter
0 packets dropped by kernel
~# tcpdump -w /tmp/dump.out -s 65536 host deltree and port 2049
tcpdump: invalid snaplen 65536
~# tcpdump --version
tcpdump version 3.9.8
libpcap version 0.9.8
So I'm now using:
~# tcpdump -w /tmp/dump.out -s 65535 -C 10 -W 100 host deltree and port 2049
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
to get a 1GB round-robin trace buffer, I can stop the trace when problem
noticed, as it is so long between delay/stall happenings. Then I'll try
to trigger the thing.
Is this the correct style of trace you are expecting?
~$ /usr/sbin/tcpdump -r /tmp/dump.out00
reading from file /tmp/dump.out00, link-type EN10MB (Ethernet)
10:13:49.719781 IP pooh64.mire.mine.nu.2156510591 > deltree.mire.mine.nu.nfs: 116 access fh 0,1/218104576 001f
10:13:49.720215 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.2156510591: reply ok 124 access c 001f
10:13:49.720225 IP pooh64.mire.mine.nu.984 > deltree.mire.mine.nu.nfsd: . ack 1649405551 win 5840
10:13:49.720288 IP pooh64.mire.mine.nu.2173287807 > deltree.mire.mine.nu.nfs: 136 readdirplus fh 0,1/218104576 512 bytes @ 0
10:13:49.742450 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.2173287807: reply ok 1460 readdirplus
Is there some test suite I can use? Compiling kernels over NFS worked
fine yesterday, apart from the fastest box' make complaining about clock
skew. The kernel booted though, so that was okay.
Guess it's back to the interactive editing over NFS and see if the thing
manifest the delay/stalls again, I'm on the .27-rc4-git4 kernel as soon
as it compiles for the client, NFS server is 2.6.24.7 at the moment.
Grant.
^ permalink raw reply [flat|nested] 64+ messages in thread
* RE: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-26 0:29 ` Grant Coady
@ 2008-08-26 0:59 ` Muntz, Daniel
0 siblings, 0 replies; 64+ messages in thread
From: Muntz, Daniel @ 2008-08-26 0:59 UTC (permalink / raw)
To: Grant Coady, Trond Myklebust
Cc: Ian Campbell, John Ronciak, linux-kernel, neilb, bfields,
linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
Try '-s 0', from tcpdump(8): "Setting snaplen to 0 means use the
required length to catch whole packets."
-Dan
-----Original Message-----
From: Grant Coady [mailto:grant_lkml@dodo.com.au]
Sent: Monday, August 25, 2008 5:29 PM
To: Trond Myklebust
Cc: Grant Coady; Ian Campbell; John Ronciak;
linux-kernel@vger.kernel.org; neilb@suse.de; bfields@fieldses.org;
linux-nfs@vger.kernel.org; Jeff Kirsher; Jesse Brandeburg; Bruce Allan;
PJ Waskiewicz; John Ronciak; e1000-devel@lists.sourceforge.net
Subject: Re: NFS regression? Odd delays and lockups accessing an NFS
export.
On Mon, 25 Aug 2008 18:11:12 -0400, Trond Myklebust
<trond.myklebust@fys.uio.no> wrote:
>On Tue, 2008-08-26 at 06:23 +1000, Grant Coady wrote:
>> On Fri, 22 Aug 2008 14:56:53 -0700, Trond Myklebust
<trond.myklebust@fys.uio.no> wrote:
>>
>> >On Fri, 2008-08-22 at 22:37 +0100, Ian Campbell wrote:
>> >> I can ssh to the server fine. The same server also serves my NFS
>> >> home directory to the box I'm writing this from and I've not seen
>> >> any trouble with this box at all, it's a 2.6.18-xen box.
>> >
>> >OK... Are you able to reproduce the problem reliably?
>> >
>> >If so, can you provide me with a binary tcpdump or wireshark dump?
>> >If using tcpdump, then please use something like
>> >
>> > tcpdump -w /tmp/dump.out -s 90000 host myserver.foo.bar and port
>> > 2049
>> ^^^^^^^^--> typo?
>
>No. The intention was to record _all_ the info in the packet for
>analysis, not just random header info.
Hi Trond,
My tcpdump seems to have a 16 bit snaplen counter:
~# tcpdump -w /tmp/dump.out -s 65535 host deltree and port 2049
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size
65535 bytes ^C0 packets captured
4 packets received by filter
0 packets dropped by kernel
~# tcpdump -w /tmp/dump.out -s 65536 host deltree and port 2049
tcpdump: invalid snaplen 65536
~# tcpdump --version
tcpdump version 3.9.8
libpcap version 0.9.8
So I'm now using:
~# tcpdump -w /tmp/dump.out -s 65535 -C 10 -W 100 host deltree and port
2049
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size
65535 bytes
to get a 1GB round-robin trace buffer, I can stop the trace when problem
noticed, as it is so long between delay/stall happenings. Then I'll try
to trigger the thing.
Is this the correct style of trace you are expecting?
~$ /usr/sbin/tcpdump -r /tmp/dump.out00
reading from file /tmp/dump.out00, link-type EN10MB (Ethernet)
10:13:49.719781 IP pooh64.mire.mine.nu.2156510591 >
deltree.mire.mine.nu.nfs: 116 access fh 0,1/218104576 001f
10:13:49.720215 IP deltree.mire.mine.nu.nfs >
pooh64.mire.mine.nu.2156510591: reply ok 124 access c 001f
10:13:49.720225 IP pooh64.mire.mine.nu.984 > deltree.mire.mine.nu.nfsd:
. ack 1649405551 win 5840
10:13:49.720288 IP pooh64.mire.mine.nu.2173287807 >
deltree.mire.mine.nu.nfs: 136 readdirplus fh 0,1/218104576 512 bytes @ 0
10:13:49.742450 IP deltree.mire.mine.nu.nfs >
pooh64.mire.mine.nu.2173287807: reply ok 1460 readdirplus
Is there some test suite I can use? Compiling kernels over NFS worked
fine yesterday, apart from the fastest box' make complaining about clock
skew. The kernel booted though, so that was okay.
Guess it's back to the interactive editing over NFS and see if the thing
manifest the delay/stalls again, I'm on the .27-rc4-git4 kernel as soon
as it compiles for the client, NFS server is 2.6.24.7 at the moment.
Grant.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org More majordomo info
at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-25 22:11 ` Trond Myklebust
2008-08-26 0:29 ` Grant Coady
@ 2008-08-26 1:06 ` Grant Coady
1 sibling, 0 replies; 64+ messages in thread
From: Grant Coady @ 2008-08-26 1:06 UTC (permalink / raw)
To: Trond Myklebust
Cc: Grant Coady, Ian Campbell, John Ronciak, linux-kernel, neilb,
bfields, linux-nfs, Jeff Kirsher, Jesse Brandeburg, Bruce Allan,
PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 5465 bytes --]
On Mon, 25 Aug 2008 18:11:12 -0400, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>On Tue, 2008-08-26 at 06:23 +1000, Grant Coady wrote:
>> On Fri, 22 Aug 2008 14:56:53 -0700, Trond Myklebust <trond.myklebust@fys.uio.no> wrote:
>>
>> >On Fri, 2008-08-22 at 22:37 +0100, Ian Campbell wrote:
>> >> I can ssh to the server fine. The same server also serves my NFS home
>> >> directory to the box I'm writing this from and I've not seen any trouble
>> >> with this box at all, it's a 2.6.18-xen box.
>> >
>> >OK... Are you able to reproduce the problem reliably?
>> >
>> >If so, can you provide me with a binary tcpdump or wireshark dump? If
>> >using tcpdump, then please use something like
>> >
>> > tcpdump -w /tmp/dump.out -s 90000 host myserver.foo.bar and port 2049
>> ^^^^^^^^--> typo?
>
>No. The intention was to record _all_ the info in the packet for
>analysis, not just random header info.
I booted 2.6.27-rc4-git4a and started the tcpdump trace as described in
last email, got an immediate stall which cleared after 10-20 seconds, on
running a small script exported from the NFS server as user:
~$ cat /home/common/bin/boot-report
#! /bin/bash
#
version=$(uname -r)
dmesg >/home/common/dmesg-$version
# 2006-12-15 no more filter .config
#grep = /boot/config-$version >/home/common/config-$version
cp /boot/config-$version /home/common
The script had not created the file on the server when the command prompt
returned, during the stall I also opened another client terminal and ran
'df', it stalled too, and both client terminals came back to prompt after
the 10-20s delay. Then I hit Ctrl-C on the tcpdump terminal to prepare
this report.
The trace is only 27 lines, deltree is the server, pooh64 the client:
grant@pooh64:~$ /usr/sbin/tcpdump -r /tmp/dump.out00
reading from file /tmp/dump.out00, link-type EN10MB (Ethernet)
10:42:14.061410 IP pooh64.mire.mine.nu.0 > deltree.mire.mine.nu.nfs: 0 null
10:42:14.062691 IP deltree.mire.mine.nu.nfsd > pooh64.mire.mine.nu.839: S 2746115414:2746115414(0) ack 3659136038 win 5840 <mss 1460,nop,nop,sackOK>
10:42:14.062708 IP pooh64.mire.mine.nu.839 > deltree.mire.mine.nu.nfsd: . ack 1 win 5840
10:42:14.062750 IP pooh64.mire.mine.nu.1392480046 > deltree.mire.mine.nu.nfs: 112 getattr fh 0,1/218104576
10:42:14.062765 IP pooh64.mire.mine.nu.1409257262 > deltree.mire.mine.nu.nfs: 112 fsstat fh 0,1/218104576
10:42:14.062944 IP deltree.mire.mine.nu.nfsd > pooh64.mire.mine.nu.839: . ack 113 win 5840
10:42:14.062951 IP deltree.mire.mine.nu.nfsd > pooh64.mire.mine.nu.839: . ack 225 win 5840
10:42:14.063873 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.1392480046: reply ok 116 getattr DIR 40755 ids 1000/10 sz 448
10:42:14.063881 IP pooh64.mire.mine.nu.839 > deltree.mire.mine.nu.nfsd: . ack 117 win 5840
10:42:14.063890 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.1409257262: reply ok 88 fsstat tbytes 8422359040 fbytes 6605647872 abytes 6605647872
10:42:14.063893 IP pooh64.mire.mine.nu.839 > deltree.mire.mine.nu.nfsd: . ack 205 win 5840
10:42:14.064319 IP pooh64.mire.mine.nu.1426034478 > deltree.mire.mine.nu.nfs: 116 access fh 0,1/218104576 001f
10:42:14.064677 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.1426034478: reply ok 124 access c 001f
10:42:14.064732 IP pooh64.mire.mine.nu.1442811694 > deltree.mire.mine.nu.nfs: 136 readdirplus fh 0,1/218104576 512 bytes @ 0
10:42:14.067193 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.1442811694: reply ok 1460 readdirplus
10:42:14.067203 IP pooh64.mire.mine.nu.839 > deltree.mire.mine.nu.nfsd: . ack 1789 win 8760
10:42:14.067208 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.841889326: reply ok 1324
10:42:14.067328 IP pooh64.mire.mine.nu.1459588910 > deltree.mire.mine.nu.nfs: 112 getattr fh Unknown/010000000003000D0200000000000024010000060003000D02000000FE6F0000
10:42:14.068572 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.1459588910: reply ok 116 getattr DIR 40755 ids 1000/10 sz 448
10:42:14.068619 IP pooh64.mire.mine.nu.1476366126 > deltree.mire.mine.nu.nfs: 136 getattr fh Unknown/010000060003000D02000000676A000002000000000000000200000001000000
10:42:14.070615 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.1476366126: reply ok 116 getattr DIR 40755 ids 1000/10 sz 360
10:42:14.070652 IP pooh64.mire.mine.nu.1493143342 > deltree.mire.mine.nu.nfs: 140 access fh Unknown/010000060003000D02000000676A000002000000000000000200000001000000 001f
10:42:14.071576 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.1493143342: reply ok 124 access c 001f
10:42:14.071638 IP pooh64.mire.mine.nu.1509920558 > deltree.mire.mine.nu.nfs: 160 readdirplus fh Unknown/010000060003000D02000000676A000002000000000000000200000001000000 512 bytes @ 0
10:42:14.073382 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.1509920558: reply ok 1460 readdirplus
10:42:14.074096 IP deltree.mire.mine.nu.nfs > pooh64.mire.mine.nu.0: reply Unknown rpc response code=6 516
10:42:14.074102 IP pooh64.mire.mine.nu.839 > deltree.mire.mine.nu.nfsd: . ack 5445 win 17520
Sorry I forgot to do the netstat -t.
Attached the tcpdump file in case there's more info in it.
config and dmesg for client at:
http://bugsplatter.id.au/kernel/boxen/pooh64/config-2.6.27-rc4-git4a
http://bugsplatter.id.au/kernel/boxen/pooh64/dmesg-2.6.27-rc4-git4a
for server:
http://bugsplatter.id.au/kernel/boxen/deltree/config-2.6.24.7a.gz
http://bugsplatter.id.au/kernel/boxen/deltree/dmesg-2.6.24.7a.gz
Grant.
[-- Attachment #2: pooh64-2.6.27-rc4-git4a --]
[-- Type: application/octet-stream, Size: 2279 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-25 16:04 ` Tom Tucker
2008-08-25 16:54 ` Trond Myklebust
@ 2008-08-26 19:27 ` J. Bruce Fields
2008-08-27 14:43 ` Tom Tucker
1 sibling, 1 reply; 64+ messages in thread
From: J. Bruce Fields @ 2008-08-26 19:27 UTC (permalink / raw)
To: Tom Tucker
Cc: Trond Myklebust, Ian Campbell, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
On Mon, Aug 25, 2008 at 11:04:08AM -0500, Tom Tucker wrote:
> Trond Myklebust wrote:
>> On Sun, 2008-08-24 at 23:09 +0100, Ian Campbell wrote:
>>> (added some quoting from previous mail to save replying twice)
>>>
>>> On Sun, 2008-08-24 at 15:19 -0400, Trond Myklebust wrote:
>>>> On Sun, 2008-08-24 at 15:17 -0400, Trond Myklebust wrote:
>>>>> >From the tcpdump, it looks as if the NFS server is failing to close the
>>>>> socket, when the client closes its side. You therefore end up getting
>>>>> stuck in the FIN_WAIT2 state (as netstat clearly shows above).
>>>>>
>>>>> Is the server keeping the client in this state for a very long
>>>>> period?
>>> Well, it had been around an hour and a half on this occasion. Next time
>>> it happens I can wait longer but I'm pretty sure I've come back from
>>> time away and it's been wedged for at least a day. How long would you
>>> expect it to remain in this state for?
>>
>> The server should ideally start to close the socket as soon as it
>> receives the FIN from the client. I'll have a look at the code.
>>
>
> I don't think it should matter how long the connection stays in FIN WAIT,
> the client should reconnect anyway.
>
> Since the client seems to be the variable, I would think it might be an
> issue with the client reconnect logic?
>
> That said, 2.6.25 is when the server side transport switch logic went in.
Any chance you could help Trond figure out why the server might be doing
this?
If not, I'll get to it, but not as soon as I should.
--b.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-26 19:27 ` J. Bruce Fields
@ 2008-08-27 14:43 ` Tom Tucker
2008-08-30 15:47 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: Tom Tucker @ 2008-08-27 14:43 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Trond Myklebust, Ian Campbell, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
J. Bruce Fields wrote:
> On Mon, Aug 25, 2008 at 11:04:08AM -0500, Tom Tucker wrote:
>> Trond Myklebust wrote:
>>> On Sun, 2008-08-24 at 23:09 +0100, Ian Campbell wrote:
>>>> (added some quoting from previous mail to save replying twice)
>>>>
>>>> On Sun, 2008-08-24 at 15:19 -0400, Trond Myklebust wrote:
>>>>> On Sun, 2008-08-24 at 15:17 -0400, Trond Myklebust wrote:
>>>>>> >From the tcpdump, it looks as if the NFS server is failing to close the
>>>>>> socket, when the client closes its side. You therefore end up getting
>>>>>> stuck in the FIN_WAIT2 state (as netstat clearly shows above).
>>>>>>
>>>>>> Is the server keeping the client in this state for a very long
>>>>>> period?
>>>> Well, it had been around an hour and a half on this occasion. Next time
>>>> it happens I can wait longer but I'm pretty sure I've come back from
>>>> time away and it's been wedged for at least a day. How long would you
>>>> expect it to remain in this state for?
>>> The server should ideally start to close the socket as soon as it
>>> receives the FIN from the client. I'll have a look at the code.
>>>
>> I don't think it should matter how long the connection stays in FIN WAIT,
>> the client should reconnect anyway.
>>
>> Since the client seems to be the variable, I would think it might be an
>> issue with the client reconnect logic?
>>
>> That said, 2.6.25 is when the server side transport switch logic went in.
>
> Any chance you could help Trond figure out why the server might be doing
> this?
>
> If not, I'll get to it, but not as soon as I should.
>
Sure. I've actually tried to reproduce it here unsuccessfully.
As a starter, I would suggest turning on transport debugging:
# echo 256 > /proc/sys/sunrpc/rpc_debug
Here are my thoughts on how it is supposed to work. Trond if this doesn't match
your understanding, please let me know.
For the case where the client closes the connection first
because it's timeout is shorter (5min vs. 6), the sequence of events should be:
client server
close
sends FIN goes to FIN-WAIT-1
receives FIN replies with ACK
and goes to CLOSE-WAIT
receives ACK goes to FIN-WAIT-2
calls tcp_state_change callback on socket
svc_tcp_state_change sets XPT_CLOSE on
the transport and enqueues transport for
servicing by svc thread.
thread gets dequeued, calls svc_recv
svc_recv sees XPT_CLOSE and calls
svc_xprt_destroy that closes the socket
TCP sends FIN/ACK after close
receives FIN/ACK goes to TIME-WAIT
since state is TIME-WAIT and
reuse port is set, we can re-connect
There's a couple places we could be getting stuck:
- The state-change callback never gets called.
- The state-change callback gets called but svsk is null and it ignores the event
- svc_tcp_state_change enqueues the transport, but due to a race bug, the transport
doesn't actually get queued and since there is no activity it never closes
- something else
The message log with RPCDBG_SVCXPRT (256) enabled as shown above would help figure
out where it's happening.
If Ian is willing to create the log (or already has one), I'm certainly willing to
look at it.
Tom
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-27 14:43 ` Tom Tucker
@ 2008-08-30 15:47 ` Ian Campbell
2008-08-31 19:30 ` J. Bruce Fields
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-08-30 15:47 UTC (permalink / raw)
To: Tom Tucker
Cc: J. Bruce Fields, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 903 bytes --]
On Wed, 2008-08-27 at 09:43 -0500, Tom Tucker wrote:
> Sure. I've actually tried to reproduce it here unsuccessfully.
>
> As a starter, I would suggest turning on transport debugging:
>
> # echo 256 > /proc/sys/sunrpc/rpc_debug
> [...]
> If Ian is willing to create the log (or already has one), I'm
> certainly willing to look at it.
It produced only the following (is that what was expected?):
[146866.448112] -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops--
[146866.448112] 30576 0001 00a0 0 f77a1600 100003 f7903340 15000 xprt_pending fa0ba88e fa0c9df4
[146866.448112] 30577 0004 0080 -11 f77a1600 100003 f7903000 0 xprt_sending fa0ba88e fa0c9df4
Ian.
--
Ian Campbell
Life is a concentration camp. You're stuck here and there's no way
out and you can only rage impotently against your persecutors.
-- Woody Allen
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-30 15:47 ` Ian Campbell
@ 2008-08-31 19:30 ` J. Bruce Fields
2008-08-31 19:44 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: J. Bruce Fields @ 2008-08-31 19:30 UTC (permalink / raw)
To: Ian Campbell
Cc: Tom Tucker, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
On Sat, Aug 30, 2008 at 04:47:41PM +0100, Ian Campbell wrote:
> On Wed, 2008-08-27 at 09:43 -0500, Tom Tucker wrote:
> > Sure. I've actually tried to reproduce it here unsuccessfully.
> >
> > As a starter, I would suggest turning on transport debugging:
> >
> > # echo 256 > /proc/sys/sunrpc/rpc_debug
> > [...]
> > If Ian is willing to create the log (or already has one), I'm
> > certainly willing to look at it.
>
> It produced only the following (is that what was expected?):
>
> [146866.448112] -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops--
> [146866.448112] 30576 0001 00a0 0 f77a1600 100003 f7903340 15000 xprt_pending fa0ba88e fa0c9df4
> [146866.448112] 30577 0004 0080 -11 f77a1600 100003 f7903000 0 xprt_sending fa0ba88e fa0c9df4
It's normal to get something like that when you turn it on, yes (unless
someone else spots anything odd about that...) but what's really needed
is to turn this on and then reproduce the problem--it's the debugging
output that goes to the logs during the problem that'll be interesting.
--b.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-31 19:30 ` J. Bruce Fields
@ 2008-08-31 19:44 ` Ian Campbell
2008-08-31 19:46 ` J. Bruce Fields
2008-08-31 19:51 ` Tom Tucker
0 siblings, 2 replies; 64+ messages in thread
From: Ian Campbell @ 2008-08-31 19:44 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Tom Tucker, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 1636 bytes --]
On Sun, 2008-08-31 at 15:30 -0400, J. Bruce Fields wrote:
> On Sat, Aug 30, 2008 at 04:47:41PM +0100, Ian Campbell wrote:
> > On Wed, 2008-08-27 at 09:43 -0500, Tom Tucker wrote:
> > > Sure. I've actually tried to reproduce it here unsuccessfully.
> > >
> > > As a starter, I would suggest turning on transport debugging:
> > >
> > > # echo 256 > /proc/sys/sunrpc/rpc_debug
> > > [...]
> > > If Ian is willing to create the log (or already has one), I'm
> > > certainly willing to look at it.
> >
> > It produced only the following (is that what was expected?):
> >
> > [146866.448112] -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops--
> > [146866.448112] 30576 0001 00a0 0 f77a1600 100003 f7903340 15000 xprt_pending fa0ba88e fa0c9df4
> > [146866.448112] 30577 0004 0080 -11 f77a1600 100003 f7903000 0 xprt_sending fa0ba88e fa0c9df4
>
> It's normal to get something like that when you turn it on, yes (unless
> someone else spots anything odd about that...) but what's really needed
> is to turn this on and then reproduce the problem--it's the debugging
> output that goes to the logs during the problem that'll be interesting.
That's what I did. The first time I did the echo I just got the header
line, then I waited for the repro and since there had been no further
logging I ran the echo again and got the three lines above.
Sounds like you expected there to be more and ongoing logging?
Ian.
> --b.
>
--
Ian Campbell
kernel, n.:
A part of an operating system that preserves the medieval
traditions of sorcery and black art.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-31 19:44 ` Ian Campbell
@ 2008-08-31 19:46 ` J. Bruce Fields
2008-08-31 19:49 ` Ian Campbell
2008-08-31 19:51 ` Tom Tucker
1 sibling, 1 reply; 64+ messages in thread
From: J. Bruce Fields @ 2008-08-31 19:46 UTC (permalink / raw)
To: Ian Campbell
Cc: Tom Tucker, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
On Sun, Aug 31, 2008 at 08:44:02PM +0100, Ian Campbell wrote:
> On Sun, 2008-08-31 at 15:30 -0400, J. Bruce Fields wrote:
> > On Sat, Aug 30, 2008 at 04:47:41PM +0100, Ian Campbell wrote:
> > > On Wed, 2008-08-27 at 09:43 -0500, Tom Tucker wrote:
> > > > Sure. I've actually tried to reproduce it here unsuccessfully.
> > > >
> > > > As a starter, I would suggest turning on transport debugging:
> > > >
> > > > # echo 256 > /proc/sys/sunrpc/rpc_debug
> > > > [...]
> > > > If Ian is willing to create the log (or already has one), I'm
> > > > certainly willing to look at it.
> > >
> > > It produced only the following (is that what was expected?):
> > >
> > > [146866.448112] -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops--
> > > [146866.448112] 30576 0001 00a0 0 f77a1600 100003 f7903340 15000 xprt_pending fa0ba88e fa0c9df4
> > > [146866.448112] 30577 0004 0080 -11 f77a1600 100003 f7903000 0 xprt_sending fa0ba88e fa0c9df4
> >
> > It's normal to get something like that when you turn it on, yes (unless
> > someone else spots anything odd about that...) but what's really needed
> > is to turn this on and then reproduce the problem--it's the debugging
> > output that goes to the logs during the problem that'll be interesting.
>
> That's what I did. The first time I did the echo I just got the header
> line, then I waited for the repro and since there had been no further
> logging I ran the echo again and got the three lines above.
>
> Sounds like you expected there to be more and ongoing logging?
Yes. It's the server side we're interested in, by the way--are you
collecting this deubgging on the client?
--b.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-31 19:46 ` J. Bruce Fields
@ 2008-08-31 19:49 ` Ian Campbell
2008-08-31 19:51 ` Tom Tucker
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-08-31 19:49 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Tom Tucker, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 363 bytes --]
On Sun, 2008-08-31 at 15:46 -0400, J. Bruce Fields wrote:
> Yes. It's the server side we're interested in, by the way--are you
> collecting this deubgging on the client?
Er, yeah, since the client appeared to be the problem.
I'll try again on the server!
Ian.
--
Ian Campbell
The descent to Hades is the same from every place.
-- Anaxagoras
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-31 19:44 ` Ian Campbell
2008-08-31 19:46 ` J. Bruce Fields
@ 2008-08-31 19:51 ` Tom Tucker
2008-08-31 21:18 ` Ian Campbell
1 sibling, 1 reply; 64+ messages in thread
From: Tom Tucker @ 2008-08-31 19:51 UTC (permalink / raw)
To: Ian Campbell
Cc: J. Bruce Fields, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
Ian Campbell wrote:
> On Sun, 2008-08-31 at 15:30 -0400, J. Bruce Fields wrote:
>> On Sat, Aug 30, 2008 at 04:47:41PM +0100, Ian Campbell wrote:
>>> On Wed, 2008-08-27 at 09:43 -0500, Tom Tucker wrote:
>>>> Sure. I've actually tried to reproduce it here unsuccessfully.
>>>>
>>>> As a starter, I would suggest turning on transport debugging:
>>>>
>>>> # echo 256 > /proc/sys/sunrpc/rpc_debug
>>>> [...]
>>>> If Ian is willing to create the log (or already has one), I'm
>>>> certainly willing to look at it.
>>> It produced only the following (is that what was expected?):
>>>
>>> [146866.448112] -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops--
>>> [146866.448112] 30576 0001 00a0 0 f77a1600 100003 f7903340 15000 xprt_pending fa0ba88e fa0c9df4
>>> [146866.448112] 30577 0004 0080 -11 f77a1600 100003 f7903000 0 xprt_sending fa0ba88e fa0c9df4
>> It's normal to get something like that when you turn it on, yes (unless
>> someone else spots anything odd about that...) but what's really needed
>> is to turn this on and then reproduce the problem--it's the debugging
>> output that goes to the logs during the problem that'll be interesting.
>
> That's what I did. The first time I did the echo I just got the header
> line, then I waited for the repro and since there had been no further
> logging I ran the echo again and got the three lines above.
>
> Sounds like you expected there to be more and ongoing logging?
>
> Ian.
>
Looks like you ran this on the client. Sorry, Ian, I should have been more
specific. You need to modify the rpc_debug file on the server.
Tom
>> --b.
>>
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-31 19:49 ` Ian Campbell
@ 2008-08-31 19:51 ` Tom Tucker
0 siblings, 0 replies; 64+ messages in thread
From: Tom Tucker @ 2008-08-31 19:51 UTC (permalink / raw)
To: Ian Campbell
Cc: J. Bruce Fields, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
Ian Campbell wrote:
> On Sun, 2008-08-31 at 15:46 -0400, J. Bruce Fields wrote:
>> Yes. It's the server side we're interested in, by the way--are you
>> collecting this deubgging on the client?
>
> Er, yeah, since the client appeared to be the problem.
>
> I'll try again on the server!
>
Thanks Ian,
Tom
> Ian.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-31 19:51 ` Tom Tucker
@ 2008-08-31 21:18 ` Ian Campbell
2008-09-01 17:20 ` Tom Tucker
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-08-31 21:18 UTC (permalink / raw)
To: Tom Tucker
Cc: J. Bruce Fields, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 671 bytes --]
On Sun, 2008-08-31 at 14:51 -0500, Tom Tucker wrote:
>
> Looks like you ran this on the client. Sorry, Ian, I should have been
> more specific. You need to modify the rpc_debug file on the server.
I tried this on the server but it's pretty crippling (the server is
quite weedy, 300MHz K3 or something).
I'll leave it logging overnight since things should be pretty quiet (and
that's often when the problem occurs) but if there's a less aggressive
setting than 256 but which would still be useful I could leave it on
permanently until the problem happens.
Ian.
--
Ian Campbell
Marriages are made in heaven and consummated on earth.
-- John Lyly
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-31 21:18 ` Ian Campbell
@ 2008-09-01 17:20 ` Tom Tucker
2008-09-01 17:46 ` Ian Campbell
2008-09-10 8:40 ` Ian Campbell
0 siblings, 2 replies; 64+ messages in thread
From: Tom Tucker @ 2008-09-01 17:20 UTC (permalink / raw)
To: Ian Campbell
Cc: J. Bruce Fields, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
Ian Campbell wrote:
> On Sun, 2008-08-31 at 14:51 -0500, Tom Tucker wrote:
>
>> Looks like you ran this on the client. Sorry, Ian, I should have been
>> more specific. You need to modify the rpc_debug file on the server.
>>
>
> I tried this on the server but it's pretty crippling (the server is
> quite weedy, 300MHz K3 or something).
>
> I'll leave it logging overnight since things should be pretty quiet (and
> that's often when the problem occurs) but if there's a less aggressive
> setting than 256 but which would still be useful I could leave it on
> permanently until the problem happens.
>
>
Thanks Ian. Unfortunately, that's as fine grained as it gets. 256
(0x100) is the bit for transport logging.
> Ian.
>
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-01 17:20 ` Tom Tucker
@ 2008-09-01 17:46 ` Ian Campbell
2008-09-10 8:40 ` Ian Campbell
1 sibling, 0 replies; 64+ messages in thread
From: Ian Campbell @ 2008-09-01 17:46 UTC (permalink / raw)
To: Tom Tucker
Cc: J. Bruce Fields, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 456 bytes --]
On Mon, 2008-09-01 at 12:20 -0500, Tom Tucker wrote:
> Thanks Ian. Unfortunately, that's as fine grained as it gets. 256
> (0x100) is the bit for transport logging.
Doh, of course, I was thinking of 255...
No repro today. I'll let you know when I see something.
Ian.
--
Ian Campbell
History is nothing but a collection of fables and useless trifles,
cluttered up with a mass of unnecessary figures and proper names.
-- Leo Tolstoy
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
@ 2008-09-10 2:51 Benoit Plessis
0 siblings, 0 replies; 64+ messages in thread
From: Benoit Plessis @ 2008-09-10 2:51 UTC (permalink / raw)
To: linux-kernel
Hi,
I've just found this thread, and i'm relieved in some way, that i'm not
the only one
experiencing this.
NFS Server: NetApp Data On Tap v 7.2.4 (not changed
NFS Client: Dell and HP machines, using Broadcom network card
# ethtool -i eth0
driver: bnx2
version: 1.7.4
firmware-version: 4.0.3 UMP 1.1.9
bus-info: 0000:05:00.0
All machines are running on NFS Root on local LAN. Previous
configuration with kernel 2.6.22.14
was working quite fine except for some quick lost, solved within a few
second
Sep 10 01:03:54 kernel: nfs: server 10.250.255.253 not responding,
still trying
Sep 10 01:03:54 kernel: nfs: server 10.250.255.253 OK
That's already something strange that shouldn't happens
And now, after an upgrade to 2.6.25.16:
Sep 10 03:26:31 web-31-mtp-1 kernel: nfs: server 10.250.255.253 not
responding, still trying
Sep 10 03:27:40 web-31-mtp-1 kernel: INFO: task apache2:29263 blocked
for more than 120 seconds.
Sep 10 03:27:40 web-31-mtp-1 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 10 03:27:40 web-31-mtp-1 kernel: apache2 D
0000000000000000 0 29263 2288
Sep 10 03:27:40 web-31-mtp-1 kernel: ffff81005d1ad958 0000000000200086
0000000000000000 ffff810001005830
Sep 10 03:27:40 web-31-mtp-1 kernel: ffff81009f8c8080 ffffffff805734a0
ffff81009f8c82b8 0000000100970659
Sep 10 03:27:40 web-31-mtp-1 kernel: 00000000ffffffff 0000000000000000
0000000000000000 0000000000000000
Sep 10 03:27:40 web-31-mtp-1 kernel: Call Trace:
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff8046ac20>]
rpc_wait_bit_killable+0x0/0x40
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff8046ac3c>]
rpc_wait_bit_killable+0x1c/0x40
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80493f4f>]
__wait_on_bit+0x4f/0x80
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff8046ac20>]
rpc_wait_bit_killable+0x0/0x40
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80493ffa>]
out_of_line_wait_on_bit+0x7a/0xa0
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80249350>]
wake_bit_function+0x0/0x30
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff804672d3>]
xprt_connect+0x83/0x170
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff8046b141>]
__rpc_execute+0xd1/0x280
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff804644e1>]
rpc_run_task+0x51/0x70
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff8046459c>]
rpc_call_sync+0x3c/0x60
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff802edbba>]
nfs3_rpc_wrapper+0x3a/0x60
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff802ee2f1>]
nfs3_proc_access+0x141/0x240
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff8029d51f>] dput+0x1f/0xf0
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff802e0abf>]
nfs_lookup_revalidate+0x22f/0x3e0
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80249529>]
remove_wait_queue+0x19/0x60
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80298c91>]
free_poll_entry+0x11/0x20
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80298ccc>]
poll_freewait+0x2c/0x90
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80299026>]
do_sys_poll+0x2f6/0x370
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff802dee71>]
nfs_do_access+0xe1/0x340
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff802df1aa>]
nfs_permission+0xda/0x1a0
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80292d1d>]
permission+0xbd/0x150
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80294b7e>]
__link_path_walk+0x7e/0xee0
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff803e865b>]
sock_aio_read+0x16b/0x180
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80295a3a>]
path_walk+0x5a/0xc0
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80295cc3>]
do_path_lookup+0x83/0x1c0
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff802947a5>]
getname+0xe5/0x210
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff802968bb>]
__user_walk_fd+0x4b/0x80
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff8028eacf>]
vfs_stat_fd+0x2f/0x80
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff802255af>]
sys32_stat64+0x1f/0x50
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80327df1>]
__up_read+0x21/0xb0
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff804955c9>]
error_exit+0x0/0x51
Sep 10 03:27:40 web-31-mtp-1 kernel: [<ffffffff80224dd2>]
ia32_sysret+0x0/0xa
Sep 10 03:27:40 web-31-mtp-1 kernel:
Sep 10 03:27:40 web-31-mtp-1 kernel: INFO: task apache2:29339 blocked
for more than 120 seconds.
Sep 10 03:27:40 web-31-mtp-1 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
....
Sep 10 03:31:56 web-31-mtp-1 kernel: nfs: server 10.250.255.253 OK
During this time the serveur isn't available, albeit it still reply to
icmp message.
I've a bunch of identical server with this new kernel, and this doesn't
happen on every one.
Only 30% of then seem to be affected.
All machines are running the latest same BIOS/Network Firmware.
Also all machines are using a bonded interface in failover mode.
There is no particular messages on the netapp side, only this common
message:
Average size of NFS TCP packets received from host: X.X.X.X is 3858.
This is less than the MTU (9000 bytes) of the interface involved in
the data transfer.
The maximum segment size being used by TCP for this host is: 8960.
Low average size of packets received by this system might be
because of a misconfigured client system, or a poorly written
client application.
Press enter to continue
There isn't even any "retransmission timeouts" recorded during the
client side hang.
The kernel is a vanilla 2.6.25.16 compiled for amd64 architecture with
bnx2 and NFS root included in the kernel.
I'll try the latest 2.6.24 and then revert to 2.6.22 if necessary
And 'voila',
that's all i can think of at this time, don't hesitate to ask for more
informations.
Oh yes, the first problems appeared on one machine while his twin sister
doesn't have any trouble,
the only difference at this time was the value of
"net.core.netdev_max_backlog" (2500 on the machine without
trouble, 1000 on the other). But now this happens on machines with the
exact same configuration,
without many sysctl tweak, only:
# increase TCP max buffer size
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# increase Linux autotuning TCP buffer limits
# min, default, and max number of bytes to use
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# don't cache ssthresh from previous connection
net.ipv4.tcp_no_metrics_save = 1
# recommended to increase this for 1000 BT or higher
net.core.netdev_max_backlog = 2500
PS: Please CC me the reply
--
Benoit
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-01 17:20 ` Tom Tucker
2008-09-01 17:46 ` Ian Campbell
@ 2008-09-10 8:40 ` Ian Campbell
2008-09-12 22:43 ` J. Bruce Fields
1 sibling, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-09-10 8:40 UTC (permalink / raw)
To: Tom Tucker
Cc: J. Bruce Fields, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
On Mon, 2008-09-01 at 12:20 -0500, Tom Tucker wrote:
> Ian Campbell wrote:
> > On Sun, 2008-08-31 at 14:51 -0500, Tom Tucker wrote:
> >
> >> Looks like you ran this on the client. Sorry, Ian, I should have been
> >> more specific. You need to modify the rpc_debug file on the server.
> >>
> >
> > I tried this on the server but it's pretty crippling (the server is
> > quite weedy, 300MHz K3 or something).
> >
> > I'll leave it logging overnight since things should be pretty quiet (and
> > that's often when the problem occurs) but if there's a less aggressive
> > setting than 256 but which would still be useful I could leave it on
> > permanently until the problem happens.
> >
> >
> Thanks Ian. Unfortunately, that's as fine grained as it gets. 256
> (0x100) is the bit for transport logging.
Still no repro with the debugging enabled I'm afraid.
I'm wondering if the performance hit makes it unlikely to trigger since
everything slows to a crawl. I haven't been able to leave it on 24/7
since it makes all the NFS clients nearly unusable but I've had it on
overnight and when there's no one around etc, which is when the hang
generally seems to occur.
I forgot to re-enable it one evening and, perhaps inevitably, it crashed
overnight. I turned the debugging on and captured the following. It's
not much since syslog had given up for some reason, probably when the
disk filled up with logs. 192.168.1.5 is a non-problematic client so I
don't think the log will be useful since the problematic client
(192.168.1.4) isn't mentioned..
Ian.
vc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009900.094587] svc: TCP record, 140 bytes
[4009900.098604] svc: socket ee5fd600 recvfrom(e996b08c, 3956) = 140
[4009900.102587] svc: TCP complete record (140 bytes)
[4009900.106592] svc: transport ee5fd600 put into queue
[4009900.110613] svc: got len=140
[4009900.114645] svc: transport ee5fd600 busy, not enqueued
[4009900.118752] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
[4009900.122636] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009900.126588] svc: transport ee5fd600 busy, not enqueued
[4009900.130601] svc: transport ee5fd600 busy, not enqueued
[4009900.134593] svc: server eefc4000 waiting for data (to = 900000)
[4009900.162711] svc: transport ee5fd600 dequeued, inuse=1
[4009900.166598] svc: server eef4a000, pool 0, transport ee5fd600, inuse=2
[4009900.170594] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009900.174611] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009900.178592] svc: TCP record, 136 bytes
[4009900.182606] svc: socket ee5fd600 recvfrom(e975a088, 3960) = 136
[4009900.186592] svc: TCP complete record (136 bytes)
[4009900.190599] svc: transport ee5fd600 put into queue
[4009900.194605] svc: got len=136
[4009900.198632] svc: transport ee5fd600 busy, not enqueued
[4009900.202717] svc: socket ee5fd600 sendto([d6464000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
[4009900.206664] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009900.210594] svc: transport ee5fd600 busy, not enqueued
[4009900.214611] svc: transport ee5fd600 busy, not enqueued
[4009900.218599] svc: server eef4a000 waiting for data (to = 900000)
[4009900.254640] svc: transport ee5fd600 dequeued, inuse=1
[4009900.258603] svc: server ee978000, pool 0, transport ee5fd600, inuse=2
[4009900.262601] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009900.266625] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009900.270598] svc: TCP record, 128 bytes
[4009900.274618] svc: socket ee5fd600 recvfrom(dd54d080, 3968) = 128
[4009900.278598] svc: TCP complete record (128 bytes)
[4009900.282605] svc: transport ee5fd600 put into queue
[4009900.286611] svc: got len=128
[4009900.290642] svc: transport ee5fd600 busy, not enqueued
[4009900.294788] svc: socket ee5fd600 sendto([efb5c000 116... ], 116) = 116 (addr 192.168.1.5, port=910)
[4009900.298707] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009900.302600] svc: transport ee5fd600 busy, not enqueued
[4009900.306622] svc: transport ee5fd600 busy, not enqueued
[4009900.310604] svc: server ee978000 waiting for data (to = 900000)
[4009900.346682] svc: transport ee5fd600 dequeued, inuse=1
[4009900.350609] svc: server ee8fe000, pool 0, transport ee5fd600, inuse=2
[4009900.354606] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009900.358626] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009900.362603] svc: TCP record, 148 bytes
[4009900.366622] svc: socket ee5fd600 recvfrom(eec92094, 3948) = 148
[4009900.370604] svc: TCP complete record (148 bytes)
[4009900.374612] svc: transport ee5fd600 put into queue
[4009900.378610] svc: got len=148
[4009900.382652] svc: transport ee5fd600 busy, not enqueued
[4009900.386829] svc: socket ee5fd600 sendto([e98bf000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
[4009900.390688] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009900.394606] svc: transport ee5fd600 busy, not enqueued
[4009900.398621] svc: transport ee5fd600 busy, not enqueued
[4009900.402611] svc: server ee8fe000 waiting for data (to = 900000)
[4009900.438683] svc: transport ee5fd600 dequeued, inuse=1
[4009900.442614] svc: server eef12000, pool 0, transport ee5fd600, inuse=2
[4009900.446612] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009900.450634] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009900.454609] svc: TCP record, 156 bytes
[4009900.458631] svc: socket ee5fd600 recvfrom(dd78509c, 3940) = 156
[4009900.462692] svc: TCP complete record (156 bytes)
[4009900.466693] svc: transport ee5fd600 put into queue
[4009900.470610] svc: got len=156
[4009900.474657] svc: transport ee5fd600 busy, not enqueued
[4009900.478801] svc: socket ee5fd600 sendto([ee92b000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
[4009900.482696] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009900.486611] svc: transport ee5fd600 busy, not enqueued
[4009900.490630] svc: transport ee5fd600 busy, not enqueued
[4009900.494617] svc: server eef12000 waiting for data (to = 900000)
[4009900.522817] svc: transport ee5fd600 dequeued, inuse=1
[4009900.526622] svc: server ee9f2000, pool 0, transport ee5fd600, inuse=2
[4009900.530617] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009900.534639] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009900.538615] svc: TCP record, 124 bytes
[4009900.542632] svc: socket ee5fd600 recvfrom(e977007c, 3972) = 124
[4009900.546615] svc: TCP complete record (124 bytes)
[4009900.550621] svc: transport ee5fd600 put into queue
[4009900.554616] svc: got len=124
[4009900.558656] svc: transport ee5fd600 busy, not enqueued
[4009900.562766] svc: socket ee5fd600 sendto([dd5e1000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
[4009900.566724] svc: transport ee5fd600 busy, not enqueued
[4009900.570622] svc: server ee9f2000 waiting for data (to = 900000)
[4009900.607202] svc: transport ee5fd600 dequeued, inuse=1
[4009900.610629] svc: server eeeb7000, pool 0, transport ee5fd600, inuse=2
[4009900.614623] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009900.618647] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009900.622619] svc: TCP record, 124 bytes
[4009900.626641] svc: socket ee5fd600 recvfrom(e971607c, 3972) = 124
[4009900.630620] svc: TCP complete record (124 bytes)
[4009900.634630] svc: transport ee5fd600 put into queue
[4009900.638621] svc: got len=124
[4009900.642661] svc: transport ee5fd600 busy, not enqueued
[4009900.646760] svc: socket ee5fd600 sendto([e95ee000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
[4009900.650734] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009900.654622] svc: transport ee5fd600 busy, not enqueued
[4009900.658640] svc: transport ee5fd600 busy, not enqueued
[4009900.662626] svc: server eeeb7000 waiting for data (to = 900000)
[4009900.691707] svc: transport ee5fd600 dequeued, inuse=1
[4009900.694637] svc: server eef31000, pool 0, transport ee5fd600, inuse=2
[4009900.698628] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009900.702646] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009900.706625] svc: TCP record, 156 bytes
[4009900.710654] svc: socket ee5fd600 recvfrom(e99b009c, 3940) = 156
[4009900.714626] svc: TCP complete record (156 bytes)
[4009900.718632] svc: transport ee5fd600 put into queue
[4009900.722626] svc: got len=156
[4009900.725731] svc: transport ee5fd600 busy, not enqueued
[4009900.726797] svc: socket ee5fd600 sendto([e9bee000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
[4009900.730717] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009900.734627] svc: transport ee5fd600 busy, not enqueued
[4009900.738646] svc: transport ee5fd600 busy, not enqueued
[4009900.742631] svc: server eef31000 waiting for data (to = 900000)
[4009900.778691] svc: transport ee5fd600 dequeued, inuse=1
[4009900.782635] svc: server eefc4000, pool 0, transport ee5fd600, inuse=2
[4009900.786633] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009900.790649] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009900.794631] svc: TCP record, 144 bytes
[4009900.798663] svc: socket ee5fd600 recvfrom(e996b090, 3952) = 144
[4009900.802632] svc: TCP complete record (144 bytes)
[4009900.806638] svc: transport ee5fd600 put into queue
[4009900.810632] svc: got len=144
[4009900.814675] svc: transport ee5fd600 busy, not enqueued
[4009900.818858] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
[4009900.822707] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009900.826632] svc: transport ee5fd600 busy, not enqueued
[4009900.830647] svc: transport ee5fd600 busy, not enqueued
[4009900.834638] svc: server eefc4000 waiting for data (to = 900000)
[4009900.862781] svc: transport ee5fd600 dequeued, inuse=1
[4009900.866647] svc: server eef4a000, pool 0, transport ee5fd600, inuse=2
[4009900.870644] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009900.874651] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009900.878636] svc: TCP record, 136 bytes
[4009900.882649] svc: socket ee5fd600 recvfrom(e975a088, 3960) = 136
[4009900.886636] svc: TCP complete record (136 bytes)
[4009900.890642] svc: transport ee5fd600 put into queue
[4009900.894654] svc: got len=136
[4009900.898678] svc: transport ee5fd600 busy, not enqueued
[4009900.902761] svc: socket ee5fd600 sendto([d6464000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
[4009900.906714] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009900.910638] svc: transport ee5fd600 busy, not enqueued
[4009900.914654] svc: transport ee5fd600 busy, not enqueued
[4009900.918643] svc: server eef4a000 waiting for data (to = 900000)
[4009900.954826] svc: transport ee5fd600 dequeued, inuse=1
[4009900.958648] svc: server ee978000, pool 0, transport ee5fd600, inuse=2
[4009900.962645] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009900.966667] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009900.970642] svc: TCP record, 140 bytes
[4009900.974659] svc: socket ee5fd600 recvfrom(dd54d08c, 3956) = 140
[4009900.978652] svc: TCP complete record (140 bytes)
[4009900.982649] svc: transport ee5fd600 put into queue
[4009900.986642] svc: got len=140
[4009900.990686] svc: transport ee5fd600 busy, not enqueued
[4009900.994848] svc: socket ee5fd600 sendto([efb5c000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
[4009900.998725] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009901.002644] svc: transport ee5fd600 busy, not enqueued
[4009901.006662] svc: transport ee5fd600 busy, not enqueued
[4009901.010648] svc: server ee978000 waiting for data (to = 900000)
[4009901.046694] svc: transport ee5fd600 dequeued, inuse=1
[4009901.050652] svc: server ee8fe000, pool 0, transport ee5fd600, inuse=2
[4009901.054650] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009901.058666] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009901.062647] svc: TCP record, 156 bytes
[4009901.066669] svc: socket ee5fd600 recvfrom(eec9209c, 3940) = 156
[4009901.070648] svc: TCP complete record (156 bytes)
[4009901.074655] svc: transport ee5fd600 put into queue
[4009901.078648] svc: got len=156
[4009901.082688] svc: transport ee5fd600 busy, not enqueued
[4009901.086844] svc: socket ee5fd600 sendto([e98bf000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
[4009901.090750] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009901.094650] svc: transport ee5fd600 busy, not enqueued
[4009901.098671] svc: transport ee5fd600 busy, not enqueued
[4009901.102655] svc: server ee8fe000 waiting for data (to = 900000)
[4009901.138700] svc: transport ee5fd600 dequeued, inuse=1
[4009901.142658] svc: server ee9f2000, pool 0, transport ee5fd600, inuse=2
[4009901.146656] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009901.150678] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009901.154654] svc: TCP record, 124 bytes
[4009901.158671] svc: socket ee5fd600 recvfrom(e977007c, 3972) = 124
[4009901.162654] svc: TCP complete record (124 bytes)
[4009901.166659] svc: transport ee5fd600 put into queue
[4009901.170654] svc: got len=124
[4009901.174694] svc: transport ee5fd600 busy, not enqueued
[4009901.178811] svc: socket ee5fd600 sendto([dd67d000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
[4009901.182741] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009901.186655] svc: transport ee5fd600 busy, not enqueued
[4009901.190673] svc: transport ee5fd600 busy, not enqueued
[4009901.194661] svc: server ee9f2000 waiting for data (to = 900000)
[4009901.222747] svc: transport ee5fd600 dequeued, inuse=1
[4009901.226690] svc: server eef12000, pool 0, transport ee5fd600, inuse=2
[4009901.230661] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009901.234677] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009901.238658] svc: TCP record, 152 bytes
[4009901.242673] svc: socket ee5fd600 recvfrom(dd785098, 3944) = 152
[4009901.246659] svc: TCP complete record (152 bytes)
[4009901.250664] svc: transport ee5fd600 put into queue
[4009901.254659] svc: got len=152
[4009901.258700] svc: transport ee5fd600 busy, not enqueued
[4009901.262809] svc: socket ee5fd600 sendto([ee92b000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
[4009901.266734] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009901.270660] svc: transport ee5fd600 busy, not enqueued
[4009901.274677] svc: transport ee5fd600 busy, not enqueued
[4009901.278665] svc: server eef12000 waiting for data (to = 900000)
[4009901.314692] svc: transport ee5fd600 dequeued, inuse=1
[4009901.318668] svc: server eeeb7000, pool 0, transport ee5fd600, inuse=2
[4009901.322667] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009901.326682] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009901.330664] svc: TCP record, 156 bytes
[4009901.334681] svc: socket ee5fd600 recvfrom(e971609c, 3940) = 156
[4009901.338665] svc: TCP complete record (156 bytes)
[4009901.342672] svc: transport ee5fd600 put into queue
[4009901.346665] svc: got len=156
[4009901.350707] svc: transport ee5fd600 busy, not enqueued
[4009901.354849] svc: socket ee5fd600 sendto([e95ee000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
[4009901.358747] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009901.362666] svc: transport ee5fd600 busy, not enqueued
[4009901.366683] svc: transport ee5fd600 busy, not enqueued
[4009901.370672] svc: server eeeb7000 waiting for data (to = 900000)
[4009901.406725] svc: transport ee5fd600 dequeued, inuse=1
[4009901.410674] svc: server eef31000, pool 0, transport ee5fd600, inuse=2
[4009901.414672] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009901.418691] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009901.422670] svc: TCP record, 136 bytes
[4009901.426690] svc: socket ee5fd600 recvfrom(e99b0088, 3960) = 136
[4009901.430671] svc: TCP complete record (136 bytes)
[4009901.434677] svc: transport ee5fd600 put into queue
[4009901.438684] svc: got len=136
[4009901.442710] svc: transport ee5fd600 busy, not enqueued
[4009901.446878] svc: socket ee5fd600 sendto([e9bee000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
[4009901.450767] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009901.454672] svc: transport ee5fd600 busy, not enqueued
[4009901.458754] svc: transport ee5fd600 busy, not enqueued
[4009901.462690] svc: server eef31000 waiting for data (to = 900000)
[4009901.499151] svc: transport ee5fd600 dequeued, inuse=1
[4009901.502687] svc: server eefc4000, pool 0, transport ee5fd600, inuse=2
[4009901.506679] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
[4009901.510698] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
[4009901.514675] svc: TCP record, 140 bytes
[4009901.518694] svc: socket ee5fd600 recvfrom(e996b08c, 3956) = 140
[4009901.522676] svc: TCP complete record (140 bytes)
[4009901.526683] svc: transport ee5fd600 put into queue
[4009901.530676] svc: got len=140
[4009901.534716] svc: transport ee5fd600 busy, not enqueued
[4009901.542824] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
[4009901.546806] svc: socket c1888960 TCP data ready (svsk ee5fd600)
[4009901.550678] svc: transport ee5fd600 busy, not enqueued
[4009901.554700] svc: transport ee5fd600 busy, not enqueued
[4009901.558682] svc: server eefc4000 waiting for data (to = 900000)
--
Ian Campbell
GIVE: Support the helpless victims of computer error.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-10 8:40 ` Ian Campbell
@ 2008-09-12 22:43 ` J. Bruce Fields
2008-09-12 23:15 ` Tom Tucker
0 siblings, 1 reply; 64+ messages in thread
From: J. Bruce Fields @ 2008-09-12 22:43 UTC (permalink / raw)
To: Ian Campbell
Cc: Tom Tucker, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
On Wed, Sep 10, 2008 at 09:40:15AM +0100, Ian Campbell wrote:
> On Mon, 2008-09-01 at 12:20 -0500, Tom Tucker wrote:
> > Ian Campbell wrote:
> > > On Sun, 2008-08-31 at 14:51 -0500, Tom Tucker wrote:
> > >
> > >> Looks like you ran this on the client. Sorry, Ian, I should have been
> > >> more specific. You need to modify the rpc_debug file on the server.
> > >>
> > >
> > > I tried this on the server but it's pretty crippling (the server is
> > > quite weedy, 300MHz K3 or something).
> > >
> > > I'll leave it logging overnight since things should be pretty quiet (and
> > > that's often when the problem occurs) but if there's a less aggressive
> > > setting than 256 but which would still be useful I could leave it on
> > > permanently until the problem happens.
> > >
> > >
> > Thanks Ian. Unfortunately, that's as fine grained as it gets. 256
> > (0x100) is the bit for transport logging.
>
> Still no repro with the debugging enabled I'm afraid.
>
> I'm wondering if the performance hit makes it unlikely to trigger since
> everything slows to a crawl. I haven't been able to leave it on 24/7
> since it makes all the NFS clients nearly unusable but I've had it on
> overnight and when there's no one around etc, which is when the hang
> generally seems to occur.
OK, thanks for the attempt. We should figure out how to tone down those
dprintk's.
I'll try to take a closer look at this, but I'm travelling starting
tomorrow, and realistically may not find time to look at this for a
couple weeks. If someone else gets to it before me, great....
--b.
>
> I forgot to re-enable it one evening and, perhaps inevitably, it crashed
> overnight. I turned the debugging on and captured the following. It's
> not much since syslog had given up for some reason, probably when the
> disk filled up with logs. 192.168.1.5 is a non-problematic client so I
> don't think the log will be useful since the problematic client
> (192.168.1.4) isn't mentioned..
>
> Ian.
>
> vc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009900.094587] svc: TCP record, 140 bytes
> [4009900.098604] svc: socket ee5fd600 recvfrom(e996b08c, 3956) = 140
> [4009900.102587] svc: TCP complete record (140 bytes)
> [4009900.106592] svc: transport ee5fd600 put into queue
> [4009900.110613] svc: got len=140
> [4009900.114645] svc: transport ee5fd600 busy, not enqueued
> [4009900.118752] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> [4009900.122636] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009900.126588] svc: transport ee5fd600 busy, not enqueued
> [4009900.130601] svc: transport ee5fd600 busy, not enqueued
> [4009900.134593] svc: server eefc4000 waiting for data (to = 900000)
> [4009900.162711] svc: transport ee5fd600 dequeued, inuse=1
> [4009900.166598] svc: server eef4a000, pool 0, transport ee5fd600, inuse=2
> [4009900.170594] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009900.174611] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009900.178592] svc: TCP record, 136 bytes
> [4009900.182606] svc: socket ee5fd600 recvfrom(e975a088, 3960) = 136
> [4009900.186592] svc: TCP complete record (136 bytes)
> [4009900.190599] svc: transport ee5fd600 put into queue
> [4009900.194605] svc: got len=136
> [4009900.198632] svc: transport ee5fd600 busy, not enqueued
> [4009900.202717] svc: socket ee5fd600 sendto([d6464000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> [4009900.206664] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009900.210594] svc: transport ee5fd600 busy, not enqueued
> [4009900.214611] svc: transport ee5fd600 busy, not enqueued
> [4009900.218599] svc: server eef4a000 waiting for data (to = 900000)
> [4009900.254640] svc: transport ee5fd600 dequeued, inuse=1
> [4009900.258603] svc: server ee978000, pool 0, transport ee5fd600, inuse=2
> [4009900.262601] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009900.266625] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009900.270598] svc: TCP record, 128 bytes
> [4009900.274618] svc: socket ee5fd600 recvfrom(dd54d080, 3968) = 128
> [4009900.278598] svc: TCP complete record (128 bytes)
> [4009900.282605] svc: transport ee5fd600 put into queue
> [4009900.286611] svc: got len=128
> [4009900.290642] svc: transport ee5fd600 busy, not enqueued
> [4009900.294788] svc: socket ee5fd600 sendto([efb5c000 116... ], 116) = 116 (addr 192.168.1.5, port=910)
> [4009900.298707] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009900.302600] svc: transport ee5fd600 busy, not enqueued
> [4009900.306622] svc: transport ee5fd600 busy, not enqueued
> [4009900.310604] svc: server ee978000 waiting for data (to = 900000)
> [4009900.346682] svc: transport ee5fd600 dequeued, inuse=1
> [4009900.350609] svc: server ee8fe000, pool 0, transport ee5fd600, inuse=2
> [4009900.354606] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009900.358626] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009900.362603] svc: TCP record, 148 bytes
> [4009900.366622] svc: socket ee5fd600 recvfrom(eec92094, 3948) = 148
> [4009900.370604] svc: TCP complete record (148 bytes)
> [4009900.374612] svc: transport ee5fd600 put into queue
> [4009900.378610] svc: got len=148
> [4009900.382652] svc: transport ee5fd600 busy, not enqueued
> [4009900.386829] svc: socket ee5fd600 sendto([e98bf000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> [4009900.390688] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009900.394606] svc: transport ee5fd600 busy, not enqueued
> [4009900.398621] svc: transport ee5fd600 busy, not enqueued
> [4009900.402611] svc: server ee8fe000 waiting for data (to = 900000)
> [4009900.438683] svc: transport ee5fd600 dequeued, inuse=1
> [4009900.442614] svc: server eef12000, pool 0, transport ee5fd600, inuse=2
> [4009900.446612] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009900.450634] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009900.454609] svc: TCP record, 156 bytes
> [4009900.458631] svc: socket ee5fd600 recvfrom(dd78509c, 3940) = 156
> [4009900.462692] svc: TCP complete record (156 bytes)
> [4009900.466693] svc: transport ee5fd600 put into queue
> [4009900.470610] svc: got len=156
> [4009900.474657] svc: transport ee5fd600 busy, not enqueued
> [4009900.478801] svc: socket ee5fd600 sendto([ee92b000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> [4009900.482696] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009900.486611] svc: transport ee5fd600 busy, not enqueued
> [4009900.490630] svc: transport ee5fd600 busy, not enqueued
> [4009900.494617] svc: server eef12000 waiting for data (to = 900000)
> [4009900.522817] svc: transport ee5fd600 dequeued, inuse=1
> [4009900.526622] svc: server ee9f2000, pool 0, transport ee5fd600, inuse=2
> [4009900.530617] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009900.534639] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009900.538615] svc: TCP record, 124 bytes
> [4009900.542632] svc: socket ee5fd600 recvfrom(e977007c, 3972) = 124
> [4009900.546615] svc: TCP complete record (124 bytes)
> [4009900.550621] svc: transport ee5fd600 put into queue
> [4009900.554616] svc: got len=124
> [4009900.558656] svc: transport ee5fd600 busy, not enqueued
> [4009900.562766] svc: socket ee5fd600 sendto([dd5e1000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> [4009900.566724] svc: transport ee5fd600 busy, not enqueued
> [4009900.570622] svc: server ee9f2000 waiting for data (to = 900000)
> [4009900.607202] svc: transport ee5fd600 dequeued, inuse=1
> [4009900.610629] svc: server eeeb7000, pool 0, transport ee5fd600, inuse=2
> [4009900.614623] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009900.618647] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009900.622619] svc: TCP record, 124 bytes
> [4009900.626641] svc: socket ee5fd600 recvfrom(e971607c, 3972) = 124
> [4009900.630620] svc: TCP complete record (124 bytes)
> [4009900.634630] svc: transport ee5fd600 put into queue
> [4009900.638621] svc: got len=124
> [4009900.642661] svc: transport ee5fd600 busy, not enqueued
> [4009900.646760] svc: socket ee5fd600 sendto([e95ee000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> [4009900.650734] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009900.654622] svc: transport ee5fd600 busy, not enqueued
> [4009900.658640] svc: transport ee5fd600 busy, not enqueued
> [4009900.662626] svc: server eeeb7000 waiting for data (to = 900000)
> [4009900.691707] svc: transport ee5fd600 dequeued, inuse=1
> [4009900.694637] svc: server eef31000, pool 0, transport ee5fd600, inuse=2
> [4009900.698628] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009900.702646] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009900.706625] svc: TCP record, 156 bytes
> [4009900.710654] svc: socket ee5fd600 recvfrom(e99b009c, 3940) = 156
> [4009900.714626] svc: TCP complete record (156 bytes)
> [4009900.718632] svc: transport ee5fd600 put into queue
> [4009900.722626] svc: got len=156
> [4009900.725731] svc: transport ee5fd600 busy, not enqueued
> [4009900.726797] svc: socket ee5fd600 sendto([e9bee000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> [4009900.730717] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009900.734627] svc: transport ee5fd600 busy, not enqueued
> [4009900.738646] svc: transport ee5fd600 busy, not enqueued
> [4009900.742631] svc: server eef31000 waiting for data (to = 900000)
> [4009900.778691] svc: transport ee5fd600 dequeued, inuse=1
> [4009900.782635] svc: server eefc4000, pool 0, transport ee5fd600, inuse=2
> [4009900.786633] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009900.790649] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009900.794631] svc: TCP record, 144 bytes
> [4009900.798663] svc: socket ee5fd600 recvfrom(e996b090, 3952) = 144
> [4009900.802632] svc: TCP complete record (144 bytes)
> [4009900.806638] svc: transport ee5fd600 put into queue
> [4009900.810632] svc: got len=144
> [4009900.814675] svc: transport ee5fd600 busy, not enqueued
> [4009900.818858] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> [4009900.822707] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009900.826632] svc: transport ee5fd600 busy, not enqueued
> [4009900.830647] svc: transport ee5fd600 busy, not enqueued
> [4009900.834638] svc: server eefc4000 waiting for data (to = 900000)
> [4009900.862781] svc: transport ee5fd600 dequeued, inuse=1
> [4009900.866647] svc: server eef4a000, pool 0, transport ee5fd600, inuse=2
> [4009900.870644] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009900.874651] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009900.878636] svc: TCP record, 136 bytes
> [4009900.882649] svc: socket ee5fd600 recvfrom(e975a088, 3960) = 136
> [4009900.886636] svc: TCP complete record (136 bytes)
> [4009900.890642] svc: transport ee5fd600 put into queue
> [4009900.894654] svc: got len=136
> [4009900.898678] svc: transport ee5fd600 busy, not enqueued
> [4009900.902761] svc: socket ee5fd600 sendto([d6464000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> [4009900.906714] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009900.910638] svc: transport ee5fd600 busy, not enqueued
> [4009900.914654] svc: transport ee5fd600 busy, not enqueued
> [4009900.918643] svc: server eef4a000 waiting for data (to = 900000)
> [4009900.954826] svc: transport ee5fd600 dequeued, inuse=1
> [4009900.958648] svc: server ee978000, pool 0, transport ee5fd600, inuse=2
> [4009900.962645] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009900.966667] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009900.970642] svc: TCP record, 140 bytes
> [4009900.974659] svc: socket ee5fd600 recvfrom(dd54d08c, 3956) = 140
> [4009900.978652] svc: TCP complete record (140 bytes)
> [4009900.982649] svc: transport ee5fd600 put into queue
> [4009900.986642] svc: got len=140
> [4009900.990686] svc: transport ee5fd600 busy, not enqueued
> [4009900.994848] svc: socket ee5fd600 sendto([efb5c000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> [4009900.998725] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009901.002644] svc: transport ee5fd600 busy, not enqueued
> [4009901.006662] svc: transport ee5fd600 busy, not enqueued
> [4009901.010648] svc: server ee978000 waiting for data (to = 900000)
> [4009901.046694] svc: transport ee5fd600 dequeued, inuse=1
> [4009901.050652] svc: server ee8fe000, pool 0, transport ee5fd600, inuse=2
> [4009901.054650] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009901.058666] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009901.062647] svc: TCP record, 156 bytes
> [4009901.066669] svc: socket ee5fd600 recvfrom(eec9209c, 3940) = 156
> [4009901.070648] svc: TCP complete record (156 bytes)
> [4009901.074655] svc: transport ee5fd600 put into queue
> [4009901.078648] svc: got len=156
> [4009901.082688] svc: transport ee5fd600 busy, not enqueued
> [4009901.086844] svc: socket ee5fd600 sendto([e98bf000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> [4009901.090750] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009901.094650] svc: transport ee5fd600 busy, not enqueued
> [4009901.098671] svc: transport ee5fd600 busy, not enqueued
> [4009901.102655] svc: server ee8fe000 waiting for data (to = 900000)
> [4009901.138700] svc: transport ee5fd600 dequeued, inuse=1
> [4009901.142658] svc: server ee9f2000, pool 0, transport ee5fd600, inuse=2
> [4009901.146656] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009901.150678] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009901.154654] svc: TCP record, 124 bytes
> [4009901.158671] svc: socket ee5fd600 recvfrom(e977007c, 3972) = 124
> [4009901.162654] svc: TCP complete record (124 bytes)
> [4009901.166659] svc: transport ee5fd600 put into queue
> [4009901.170654] svc: got len=124
> [4009901.174694] svc: transport ee5fd600 busy, not enqueued
> [4009901.178811] svc: socket ee5fd600 sendto([dd67d000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> [4009901.182741] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009901.186655] svc: transport ee5fd600 busy, not enqueued
> [4009901.190673] svc: transport ee5fd600 busy, not enqueued
> [4009901.194661] svc: server ee9f2000 waiting for data (to = 900000)
> [4009901.222747] svc: transport ee5fd600 dequeued, inuse=1
> [4009901.226690] svc: server eef12000, pool 0, transport ee5fd600, inuse=2
> [4009901.230661] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009901.234677] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009901.238658] svc: TCP record, 152 bytes
> [4009901.242673] svc: socket ee5fd600 recvfrom(dd785098, 3944) = 152
> [4009901.246659] svc: TCP complete record (152 bytes)
> [4009901.250664] svc: transport ee5fd600 put into queue
> [4009901.254659] svc: got len=152
> [4009901.258700] svc: transport ee5fd600 busy, not enqueued
> [4009901.262809] svc: socket ee5fd600 sendto([ee92b000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> [4009901.266734] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009901.270660] svc: transport ee5fd600 busy, not enqueued
> [4009901.274677] svc: transport ee5fd600 busy, not enqueued
> [4009901.278665] svc: server eef12000 waiting for data (to = 900000)
> [4009901.314692] svc: transport ee5fd600 dequeued, inuse=1
> [4009901.318668] svc: server eeeb7000, pool 0, transport ee5fd600, inuse=2
> [4009901.322667] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009901.326682] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009901.330664] svc: TCP record, 156 bytes
> [4009901.334681] svc: socket ee5fd600 recvfrom(e971609c, 3940) = 156
> [4009901.338665] svc: TCP complete record (156 bytes)
> [4009901.342672] svc: transport ee5fd600 put into queue
> [4009901.346665] svc: got len=156
> [4009901.350707] svc: transport ee5fd600 busy, not enqueued
> [4009901.354849] svc: socket ee5fd600 sendto([e95ee000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> [4009901.358747] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009901.362666] svc: transport ee5fd600 busy, not enqueued
> [4009901.366683] svc: transport ee5fd600 busy, not enqueued
> [4009901.370672] svc: server eeeb7000 waiting for data (to = 900000)
> [4009901.406725] svc: transport ee5fd600 dequeued, inuse=1
> [4009901.410674] svc: server eef31000, pool 0, transport ee5fd600, inuse=2
> [4009901.414672] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009901.418691] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009901.422670] svc: TCP record, 136 bytes
> [4009901.426690] svc: socket ee5fd600 recvfrom(e99b0088, 3960) = 136
> [4009901.430671] svc: TCP complete record (136 bytes)
> [4009901.434677] svc: transport ee5fd600 put into queue
> [4009901.438684] svc: got len=136
> [4009901.442710] svc: transport ee5fd600 busy, not enqueued
> [4009901.446878] svc: socket ee5fd600 sendto([e9bee000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> [4009901.450767] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009901.454672] svc: transport ee5fd600 busy, not enqueued
> [4009901.458754] svc: transport ee5fd600 busy, not enqueued
> [4009901.462690] svc: server eef31000 waiting for data (to = 900000)
> [4009901.499151] svc: transport ee5fd600 dequeued, inuse=1
> [4009901.502687] svc: server eefc4000, pool 0, transport ee5fd600, inuse=2
> [4009901.506679] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> [4009901.510698] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> [4009901.514675] svc: TCP record, 140 bytes
> [4009901.518694] svc: socket ee5fd600 recvfrom(e996b08c, 3956) = 140
> [4009901.522676] svc: TCP complete record (140 bytes)
> [4009901.526683] svc: transport ee5fd600 put into queue
> [4009901.530676] svc: got len=140
> [4009901.534716] svc: transport ee5fd600 busy, not enqueued
> [4009901.542824] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> [4009901.546806] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> [4009901.550678] svc: transport ee5fd600 busy, not enqueued
> [4009901.554700] svc: transport ee5fd600 busy, not enqueued
> [4009901.558682] svc: server eefc4000 waiting for data (to = 900000)
>
>
> --
> Ian Campbell
>
> GIVE: Support the helpless victims of computer error.
>
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-12 22:43 ` J. Bruce Fields
@ 2008-09-12 23:15 ` Tom Tucker
2008-09-13 8:57 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: Tom Tucker @ 2008-09-12 23:15 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Ian Campbell, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
J. Bruce Fields wrote:
> On Wed, Sep 10, 2008 at 09:40:15AM +0100, Ian Campbell wrote:
>
>> On Mon, 2008-09-01 at 12:20 -0500, Tom Tucker wrote:
>>
>>> Ian Campbell wrote:
>>>
>>>> On Sun, 2008-08-31 at 14:51 -0500, Tom Tucker wrote:
>>>>
>>>>
>>>>> Looks like you ran this on the client. Sorry, Ian, I should have been
>>>>> more specific. You need to modify the rpc_debug file on the server.
>>>>>
>>>>>
>>>> I tried this on the server but it's pretty crippling (the server is
>>>> quite weedy, 300MHz K3 or something).
>>>>
>>>> I'll leave it logging overnight since things should be pretty quiet (and
>>>> that's often when the problem occurs) but if there's a less aggressive
>>>> setting than 256 but which would still be useful I could leave it on
>>>> permanently until the problem happens.
>>>>
>>>>
>>>>
>>> Thanks Ian. Unfortunately, that's as fine grained as it gets. 256
>>> (0x100) is the bit for transport logging.
>>>
>> Still no repro with the debugging enabled I'm afraid.
>>
>> I'm wondering if the performance hit makes it unlikely to trigger since
>> everything slows to a crawl. I haven't been able to leave it on 24/7
>> since it makes all the NFS clients nearly unusable but I've had it on
>> overnight and when there's no one around etc, which is when the hang
>> generally seems to occur.
>>
>
> OK, thanks for the attempt. We should figure out how to tone down those
> dprintk's.
>
> I'll try to take a closer look at this, but I'm travelling starting
> tomorrow, and realistically may not find time to look at this for a
> couple weeks. If someone else gets to it before me, great....
>
> --b.
>
>
>> I forgot to re-enable it one evening and, perhaps inevitably, it crashed
>> overnight. I turned the debugging on and captured the following. It's
>> not much since syslog had given up for some reason, probably when the
>> disk filled up with logs. 192.168.1.5 is a non-problematic client so I
>> don't think the log will be useful since the problematic client
>> (192.168.1.4) isn't mentioned..
>>
>> Ian.
>>
>>
Iain sadly yes. There's a thread stuck holding the BUSY bit or a thread
failed to clear the bit properly
(maybe an error path). Data continues to arrives, but the transport
never gets put back on the queue
because it's BUSY bit is set. In other words, this is a different error
than the one we've been chasing.
If I sent you a patch, could you rebuild the kernel and give it a whirl?
Also, can you give me a
kernel.org relative commit-id or tag for the kernel that you're using?
Thanks,
Tom
>> vc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009900.094587] svc: TCP record, 140 bytes
>> [4009900.098604] svc: socket ee5fd600 recvfrom(e996b08c, 3956) = 140
>> [4009900.102587] svc: TCP complete record (140 bytes)
>> [4009900.106592] svc: transport ee5fd600 put into queue
>> [4009900.110613] svc: got len=140
>> [4009900.114645] svc: transport ee5fd600 busy, not enqueued
>> [4009900.118752] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
>> [4009900.122636] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009900.126588] svc: transport ee5fd600 busy, not enqueued
>> [4009900.130601] svc: transport ee5fd600 busy, not enqueued
>> [4009900.134593] svc: server eefc4000 waiting for data (to = 900000)
>> [4009900.162711] svc: transport ee5fd600 dequeued, inuse=1
>> [4009900.166598] svc: server eef4a000, pool 0, transport ee5fd600, inuse=2
>> [4009900.170594] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009900.174611] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009900.178592] svc: TCP record, 136 bytes
>> [4009900.182606] svc: socket ee5fd600 recvfrom(e975a088, 3960) = 136
>> [4009900.186592] svc: TCP complete record (136 bytes)
>> [4009900.190599] svc: transport ee5fd600 put into queue
>> [4009900.194605] svc: got len=136
>> [4009900.198632] svc: transport ee5fd600 busy, not enqueued
>> [4009900.202717] svc: socket ee5fd600 sendto([d6464000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
>> [4009900.206664] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009900.210594] svc: transport ee5fd600 busy, not enqueued
>> [4009900.214611] svc: transport ee5fd600 busy, not enqueued
>> [4009900.218599] svc: server eef4a000 waiting for data (to = 900000)
>> [4009900.254640] svc: transport ee5fd600 dequeued, inuse=1
>> [4009900.258603] svc: server ee978000, pool 0, transport ee5fd600, inuse=2
>> [4009900.262601] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009900.266625] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009900.270598] svc: TCP record, 128 bytes
>> [4009900.274618] svc: socket ee5fd600 recvfrom(dd54d080, 3968) = 128
>> [4009900.278598] svc: TCP complete record (128 bytes)
>> [4009900.282605] svc: transport ee5fd600 put into queue
>> [4009900.286611] svc: got len=128
>> [4009900.290642] svc: transport ee5fd600 busy, not enqueued
>> [4009900.294788] svc: socket ee5fd600 sendto([efb5c000 116... ], 116) = 116 (addr 192.168.1.5, port=910)
>> [4009900.298707] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009900.302600] svc: transport ee5fd600 busy, not enqueued
>> [4009900.306622] svc: transport ee5fd600 busy, not enqueued
>> [4009900.310604] svc: server ee978000 waiting for data (to = 900000)
>> [4009900.346682] svc: transport ee5fd600 dequeued, inuse=1
>> [4009900.350609] svc: server ee8fe000, pool 0, transport ee5fd600, inuse=2
>> [4009900.354606] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009900.358626] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009900.362603] svc: TCP record, 148 bytes
>> [4009900.366622] svc: socket ee5fd600 recvfrom(eec92094, 3948) = 148
>> [4009900.370604] svc: TCP complete record (148 bytes)
>> [4009900.374612] svc: transport ee5fd600 put into queue
>> [4009900.378610] svc: got len=148
>> [4009900.382652] svc: transport ee5fd600 busy, not enqueued
>> [4009900.386829] svc: socket ee5fd600 sendto([e98bf000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
>> [4009900.390688] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009900.394606] svc: transport ee5fd600 busy, not enqueued
>> [4009900.398621] svc: transport ee5fd600 busy, not enqueued
>> [4009900.402611] svc: server ee8fe000 waiting for data (to = 900000)
>> [4009900.438683] svc: transport ee5fd600 dequeued, inuse=1
>> [4009900.442614] svc: server eef12000, pool 0, transport ee5fd600, inuse=2
>> [4009900.446612] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009900.450634] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009900.454609] svc: TCP record, 156 bytes
>> [4009900.458631] svc: socket ee5fd600 recvfrom(dd78509c, 3940) = 156
>> [4009900.462692] svc: TCP complete record (156 bytes)
>> [4009900.466693] svc: transport ee5fd600 put into queue
>> [4009900.470610] svc: got len=156
>> [4009900.474657] svc: transport ee5fd600 busy, not enqueued
>> [4009900.478801] svc: socket ee5fd600 sendto([ee92b000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
>> [4009900.482696] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009900.486611] svc: transport ee5fd600 busy, not enqueued
>> [4009900.490630] svc: transport ee5fd600 busy, not enqueued
>> [4009900.494617] svc: server eef12000 waiting for data (to = 900000)
>> [4009900.522817] svc: transport ee5fd600 dequeued, inuse=1
>> [4009900.526622] svc: server ee9f2000, pool 0, transport ee5fd600, inuse=2
>> [4009900.530617] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009900.534639] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009900.538615] svc: TCP record, 124 bytes
>> [4009900.542632] svc: socket ee5fd600 recvfrom(e977007c, 3972) = 124
>> [4009900.546615] svc: TCP complete record (124 bytes)
>> [4009900.550621] svc: transport ee5fd600 put into queue
>> [4009900.554616] svc: got len=124
>> [4009900.558656] svc: transport ee5fd600 busy, not enqueued
>> [4009900.562766] svc: socket ee5fd600 sendto([dd5e1000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
>> [4009900.566724] svc: transport ee5fd600 busy, not enqueued
>> [4009900.570622] svc: server ee9f2000 waiting for data (to = 900000)
>> [4009900.607202] svc: transport ee5fd600 dequeued, inuse=1
>> [4009900.610629] svc: server eeeb7000, pool 0, transport ee5fd600, inuse=2
>> [4009900.614623] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009900.618647] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009900.622619] svc: TCP record, 124 bytes
>> [4009900.626641] svc: socket ee5fd600 recvfrom(e971607c, 3972) = 124
>> [4009900.630620] svc: TCP complete record (124 bytes)
>> [4009900.634630] svc: transport ee5fd600 put into queue
>> [4009900.638621] svc: got len=124
>> [4009900.642661] svc: transport ee5fd600 busy, not enqueued
>> [4009900.646760] svc: socket ee5fd600 sendto([e95ee000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
>> [4009900.650734] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009900.654622] svc: transport ee5fd600 busy, not enqueued
>> [4009900.658640] svc: transport ee5fd600 busy, not enqueued
>> [4009900.662626] svc: server eeeb7000 waiting for data (to = 900000)
>> [4009900.691707] svc: transport ee5fd600 dequeued, inuse=1
>> [4009900.694637] svc: server eef31000, pool 0, transport ee5fd600, inuse=2
>> [4009900.698628] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009900.702646] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009900.706625] svc: TCP record, 156 bytes
>> [4009900.710654] svc: socket ee5fd600 recvfrom(e99b009c, 3940) = 156
>> [4009900.714626] svc: TCP complete record (156 bytes)
>> [4009900.718632] svc: transport ee5fd600 put into queue
>> [4009900.722626] svc: got len=156
>> [4009900.725731] svc: transport ee5fd600 busy, not enqueued
>> [4009900.726797] svc: socket ee5fd600 sendto([e9bee000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
>> [4009900.730717] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009900.734627] svc: transport ee5fd600 busy, not enqueued
>> [4009900.738646] svc: transport ee5fd600 busy, not enqueued
>> [4009900.742631] svc: server eef31000 waiting for data (to = 900000)
>> [4009900.778691] svc: transport ee5fd600 dequeued, inuse=1
>> [4009900.782635] svc: server eefc4000, pool 0, transport ee5fd600, inuse=2
>> [4009900.786633] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009900.790649] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009900.794631] svc: TCP record, 144 bytes
>> [4009900.798663] svc: socket ee5fd600 recvfrom(e996b090, 3952) = 144
>> [4009900.802632] svc: TCP complete record (144 bytes)
>> [4009900.806638] svc: transport ee5fd600 put into queue
>> [4009900.810632] svc: got len=144
>> [4009900.814675] svc: transport ee5fd600 busy, not enqueued
>> [4009900.818858] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
>> [4009900.822707] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009900.826632] svc: transport ee5fd600 busy, not enqueued
>> [4009900.830647] svc: transport ee5fd600 busy, not enqueued
>> [4009900.834638] svc: server eefc4000 waiting for data (to = 900000)
>> [4009900.862781] svc: transport ee5fd600 dequeued, inuse=1
>> [4009900.866647] svc: server eef4a000, pool 0, transport ee5fd600, inuse=2
>> [4009900.870644] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009900.874651] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009900.878636] svc: TCP record, 136 bytes
>> [4009900.882649] svc: socket ee5fd600 recvfrom(e975a088, 3960) = 136
>> [4009900.886636] svc: TCP complete record (136 bytes)
>> [4009900.890642] svc: transport ee5fd600 put into queue
>> [4009900.894654] svc: got len=136
>> [4009900.898678] svc: transport ee5fd600 busy, not enqueued
>> [4009900.902761] svc: socket ee5fd600 sendto([d6464000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
>> [4009900.906714] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009900.910638] svc: transport ee5fd600 busy, not enqueued
>> [4009900.914654] svc: transport ee5fd600 busy, not enqueued
>> [4009900.918643] svc: server eef4a000 waiting for data (to = 900000)
>> [4009900.954826] svc: transport ee5fd600 dequeued, inuse=1
>> [4009900.958648] svc: server ee978000, pool 0, transport ee5fd600, inuse=2
>> [4009900.962645] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009900.966667] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009900.970642] svc: TCP record, 140 bytes
>> [4009900.974659] svc: socket ee5fd600 recvfrom(dd54d08c, 3956) = 140
>> [4009900.978652] svc: TCP complete record (140 bytes)
>> [4009900.982649] svc: transport ee5fd600 put into queue
>> [4009900.986642] svc: got len=140
>> [4009900.990686] svc: transport ee5fd600 busy, not enqueued
>> [4009900.994848] svc: socket ee5fd600 sendto([efb5c000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
>> [4009900.998725] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009901.002644] svc: transport ee5fd600 busy, not enqueued
>> [4009901.006662] svc: transport ee5fd600 busy, not enqueued
>> [4009901.010648] svc: server ee978000 waiting for data (to = 900000)
>> [4009901.046694] svc: transport ee5fd600 dequeued, inuse=1
>> [4009901.050652] svc: server ee8fe000, pool 0, transport ee5fd600, inuse=2
>> [4009901.054650] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009901.058666] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009901.062647] svc: TCP record, 156 bytes
>> [4009901.066669] svc: socket ee5fd600 recvfrom(eec9209c, 3940) = 156
>> [4009901.070648] svc: TCP complete record (156 bytes)
>> [4009901.074655] svc: transport ee5fd600 put into queue
>> [4009901.078648] svc: got len=156
>> [4009901.082688] svc: transport ee5fd600 busy, not enqueued
>> [4009901.086844] svc: socket ee5fd600 sendto([e98bf000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
>> [4009901.090750] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009901.094650] svc: transport ee5fd600 busy, not enqueued
>> [4009901.098671] svc: transport ee5fd600 busy, not enqueued
>> [4009901.102655] svc: server ee8fe000 waiting for data (to = 900000)
>> [4009901.138700] svc: transport ee5fd600 dequeued, inuse=1
>> [4009901.142658] svc: server ee9f2000, pool 0, transport ee5fd600, inuse=2
>> [4009901.146656] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009901.150678] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009901.154654] svc: TCP record, 124 bytes
>> [4009901.158671] svc: socket ee5fd600 recvfrom(e977007c, 3972) = 124
>> [4009901.162654] svc: TCP complete record (124 bytes)
>> [4009901.166659] svc: transport ee5fd600 put into queue
>> [4009901.170654] svc: got len=124
>> [4009901.174694] svc: transport ee5fd600 busy, not enqueued
>> [4009901.178811] svc: socket ee5fd600 sendto([dd67d000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
>> [4009901.182741] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009901.186655] svc: transport ee5fd600 busy, not enqueued
>> [4009901.190673] svc: transport ee5fd600 busy, not enqueued
>> [4009901.194661] svc: server ee9f2000 waiting for data (to = 900000)
>> [4009901.222747] svc: transport ee5fd600 dequeued, inuse=1
>> [4009901.226690] svc: server eef12000, pool 0, transport ee5fd600, inuse=2
>> [4009901.230661] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009901.234677] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009901.238658] svc: TCP record, 152 bytes
>> [4009901.242673] svc: socket ee5fd600 recvfrom(dd785098, 3944) = 152
>> [4009901.246659] svc: TCP complete record (152 bytes)
>> [4009901.250664] svc: transport ee5fd600 put into queue
>> [4009901.254659] svc: got len=152
>> [4009901.258700] svc: transport ee5fd600 busy, not enqueued
>> [4009901.262809] svc: socket ee5fd600 sendto([ee92b000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
>> [4009901.266734] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009901.270660] svc: transport ee5fd600 busy, not enqueued
>> [4009901.274677] svc: transport ee5fd600 busy, not enqueued
>> [4009901.278665] svc: server eef12000 waiting for data (to = 900000)
>> [4009901.314692] svc: transport ee5fd600 dequeued, inuse=1
>> [4009901.318668] svc: server eeeb7000, pool 0, transport ee5fd600, inuse=2
>> [4009901.322667] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009901.326682] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009901.330664] svc: TCP record, 156 bytes
>> [4009901.334681] svc: socket ee5fd600 recvfrom(e971609c, 3940) = 156
>> [4009901.338665] svc: TCP complete record (156 bytes)
>> [4009901.342672] svc: transport ee5fd600 put into queue
>> [4009901.346665] svc: got len=156
>> [4009901.350707] svc: transport ee5fd600 busy, not enqueued
>> [4009901.354849] svc: socket ee5fd600 sendto([e95ee000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
>> [4009901.358747] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009901.362666] svc: transport ee5fd600 busy, not enqueued
>> [4009901.366683] svc: transport ee5fd600 busy, not enqueued
>> [4009901.370672] svc: server eeeb7000 waiting for data (to = 900000)
>> [4009901.406725] svc: transport ee5fd600 dequeued, inuse=1
>> [4009901.410674] svc: server eef31000, pool 0, transport ee5fd600, inuse=2
>> [4009901.414672] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009901.418691] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009901.422670] svc: TCP record, 136 bytes
>> [4009901.426690] svc: socket ee5fd600 recvfrom(e99b0088, 3960) = 136
>> [4009901.430671] svc: TCP complete record (136 bytes)
>> [4009901.434677] svc: transport ee5fd600 put into queue
>> [4009901.438684] svc: got len=136
>> [4009901.442710] svc: transport ee5fd600 busy, not enqueued
>> [4009901.446878] svc: socket ee5fd600 sendto([e9bee000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
>> [4009901.450767] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009901.454672] svc: transport ee5fd600 busy, not enqueued
>> [4009901.458754] svc: transport ee5fd600 busy, not enqueued
>> [4009901.462690] svc: server eef31000 waiting for data (to = 900000)
>> [4009901.499151] svc: transport ee5fd600 dequeued, inuse=1
>> [4009901.502687] svc: server eefc4000, pool 0, transport ee5fd600, inuse=2
>> [4009901.506679] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
>> [4009901.510698] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
>> [4009901.514675] svc: TCP record, 140 bytes
>> [4009901.518694] svc: socket ee5fd600 recvfrom(e996b08c, 3956) = 140
>> [4009901.522676] svc: TCP complete record (140 bytes)
>> [4009901.526683] svc: transport ee5fd600 put into queue
>> [4009901.530676] svc: got len=140
>> [4009901.534716] svc: transport ee5fd600 busy, not enqueued
>> [4009901.542824] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
>> [4009901.546806] svc: socket c1888960 TCP data ready (svsk ee5fd600)
>> [4009901.550678] svc: transport ee5fd600 busy, not enqueued
>> [4009901.554700] svc: transport ee5fd600 busy, not enqueued
>> [4009901.558682] svc: server eefc4000 waiting for data (to = 900000)
>>
>>
>> --
>> Ian Campbell
>>
>> GIVE: Support the helpless victims of computer error.
>>
>>
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-12 23:15 ` Tom Tucker
@ 2008-09-13 8:57 ` Ian Campbell
2008-09-16 5:48 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-09-13 8:57 UTC (permalink / raw)
To: Tom Tucker
Cc: J. Bruce Fields, Trond Myklebust, John Ronciak, Grant Coady,
linux-kernel, neilb, linux-nfs, Jeff Kirsher, Jesse Brandeburg,
Bruce Allan, PJ Waskiewicz, John Ronciak, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 18305 bytes --]
On Fri, 2008-09-12 at 18:15 -0500, Tom Tucker wrote:
> Iain sadly yes. There's a thread stuck holding the BUSY bit or a thread
> failed to clear the bit properly
> (maybe an error path). Data continues to arrives, but the transport
> never gets put back on the queue
> because it's BUSY bit is set. In other words, this is a different error
> than the one we've been chasing.
>
> If I sent you a patch, could you rebuild the kernel and give it a whirl?
> Also, can you give me a
> kernel.org relative commit-id or tag for the kernel that you're using?
I sure could. I'm using the Debian kernel which is currently at 2.6.26.3
(pkg version 2.6.26-4) although I have an update to 2.6.26.4 (via pkg
2.6.26-5) pending.
If I'm going to build my own I'll start with current git
(a551b98d5f6fce5897d497abd8bfb262efb33d2a) and repro there before trying
your patch.
Ian.
>
> Thanks,
> Tom
>
> >> vc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009900.094587] svc: TCP record, 140 bytes
> >> [4009900.098604] svc: socket ee5fd600 recvfrom(e996b08c, 3956) = 140
> >> [4009900.102587] svc: TCP complete record (140 bytes)
> >> [4009900.106592] svc: transport ee5fd600 put into queue
> >> [4009900.110613] svc: got len=140
> >> [4009900.114645] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.118752] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> >> [4009900.122636] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009900.126588] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.130601] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.134593] svc: server eefc4000 waiting for data (to = 900000)
> >> [4009900.162711] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009900.166598] svc: server eef4a000, pool 0, transport ee5fd600, inuse=2
> >> [4009900.170594] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009900.174611] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009900.178592] svc: TCP record, 136 bytes
> >> [4009900.182606] svc: socket ee5fd600 recvfrom(e975a088, 3960) = 136
> >> [4009900.186592] svc: TCP complete record (136 bytes)
> >> [4009900.190599] svc: transport ee5fd600 put into queue
> >> [4009900.194605] svc: got len=136
> >> [4009900.198632] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.202717] svc: socket ee5fd600 sendto([d6464000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> >> [4009900.206664] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009900.210594] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.214611] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.218599] svc: server eef4a000 waiting for data (to = 900000)
> >> [4009900.254640] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009900.258603] svc: server ee978000, pool 0, transport ee5fd600, inuse=2
> >> [4009900.262601] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009900.266625] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009900.270598] svc: TCP record, 128 bytes
> >> [4009900.274618] svc: socket ee5fd600 recvfrom(dd54d080, 3968) = 128
> >> [4009900.278598] svc: TCP complete record (128 bytes)
> >> [4009900.282605] svc: transport ee5fd600 put into queue
> >> [4009900.286611] svc: got len=128
> >> [4009900.290642] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.294788] svc: socket ee5fd600 sendto([efb5c000 116... ], 116) = 116 (addr 192.168.1.5, port=910)
> >> [4009900.298707] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009900.302600] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.306622] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.310604] svc: server ee978000 waiting for data (to = 900000)
> >> [4009900.346682] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009900.350609] svc: server ee8fe000, pool 0, transport ee5fd600, inuse=2
> >> [4009900.354606] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009900.358626] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009900.362603] svc: TCP record, 148 bytes
> >> [4009900.366622] svc: socket ee5fd600 recvfrom(eec92094, 3948) = 148
> >> [4009900.370604] svc: TCP complete record (148 bytes)
> >> [4009900.374612] svc: transport ee5fd600 put into queue
> >> [4009900.378610] svc: got len=148
> >> [4009900.382652] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.386829] svc: socket ee5fd600 sendto([e98bf000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> >> [4009900.390688] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009900.394606] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.398621] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.402611] svc: server ee8fe000 waiting for data (to = 900000)
> >> [4009900.438683] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009900.442614] svc: server eef12000, pool 0, transport ee5fd600, inuse=2
> >> [4009900.446612] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009900.450634] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009900.454609] svc: TCP record, 156 bytes
> >> [4009900.458631] svc: socket ee5fd600 recvfrom(dd78509c, 3940) = 156
> >> [4009900.462692] svc: TCP complete record (156 bytes)
> >> [4009900.466693] svc: transport ee5fd600 put into queue
> >> [4009900.470610] svc: got len=156
> >> [4009900.474657] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.478801] svc: socket ee5fd600 sendto([ee92b000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> >> [4009900.482696] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009900.486611] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.490630] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.494617] svc: server eef12000 waiting for data (to = 900000)
> >> [4009900.522817] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009900.526622] svc: server ee9f2000, pool 0, transport ee5fd600, inuse=2
> >> [4009900.530617] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009900.534639] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009900.538615] svc: TCP record, 124 bytes
> >> [4009900.542632] svc: socket ee5fd600 recvfrom(e977007c, 3972) = 124
> >> [4009900.546615] svc: TCP complete record (124 bytes)
> >> [4009900.550621] svc: transport ee5fd600 put into queue
> >> [4009900.554616] svc: got len=124
> >> [4009900.558656] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.562766] svc: socket ee5fd600 sendto([dd5e1000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> >> [4009900.566724] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.570622] svc: server ee9f2000 waiting for data (to = 900000)
> >> [4009900.607202] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009900.610629] svc: server eeeb7000, pool 0, transport ee5fd600, inuse=2
> >> [4009900.614623] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009900.618647] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009900.622619] svc: TCP record, 124 bytes
> >> [4009900.626641] svc: socket ee5fd600 recvfrom(e971607c, 3972) = 124
> >> [4009900.630620] svc: TCP complete record (124 bytes)
> >> [4009900.634630] svc: transport ee5fd600 put into queue
> >> [4009900.638621] svc: got len=124
> >> [4009900.642661] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.646760] svc: socket ee5fd600 sendto([e95ee000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> >> [4009900.650734] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009900.654622] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.658640] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.662626] svc: server eeeb7000 waiting for data (to = 900000)
> >> [4009900.691707] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009900.694637] svc: server eef31000, pool 0, transport ee5fd600, inuse=2
> >> [4009900.698628] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009900.702646] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009900.706625] svc: TCP record, 156 bytes
> >> [4009900.710654] svc: socket ee5fd600 recvfrom(e99b009c, 3940) = 156
> >> [4009900.714626] svc: TCP complete record (156 bytes)
> >> [4009900.718632] svc: transport ee5fd600 put into queue
> >> [4009900.722626] svc: got len=156
> >> [4009900.725731] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.726797] svc: socket ee5fd600 sendto([e9bee000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> >> [4009900.730717] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009900.734627] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.738646] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.742631] svc: server eef31000 waiting for data (to = 900000)
> >> [4009900.778691] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009900.782635] svc: server eefc4000, pool 0, transport ee5fd600, inuse=2
> >> [4009900.786633] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009900.790649] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009900.794631] svc: TCP record, 144 bytes
> >> [4009900.798663] svc: socket ee5fd600 recvfrom(e996b090, 3952) = 144
> >> [4009900.802632] svc: TCP complete record (144 bytes)
> >> [4009900.806638] svc: transport ee5fd600 put into queue
> >> [4009900.810632] svc: got len=144
> >> [4009900.814675] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.818858] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> >> [4009900.822707] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009900.826632] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.830647] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.834638] svc: server eefc4000 waiting for data (to = 900000)
> >> [4009900.862781] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009900.866647] svc: server eef4a000, pool 0, transport ee5fd600, inuse=2
> >> [4009900.870644] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009900.874651] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009900.878636] svc: TCP record, 136 bytes
> >> [4009900.882649] svc: socket ee5fd600 recvfrom(e975a088, 3960) = 136
> >> [4009900.886636] svc: TCP complete record (136 bytes)
> >> [4009900.890642] svc: transport ee5fd600 put into queue
> >> [4009900.894654] svc: got len=136
> >> [4009900.898678] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.902761] svc: socket ee5fd600 sendto([d6464000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> >> [4009900.906714] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009900.910638] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.914654] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.918643] svc: server eef4a000 waiting for data (to = 900000)
> >> [4009900.954826] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009900.958648] svc: server ee978000, pool 0, transport ee5fd600, inuse=2
> >> [4009900.962645] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009900.966667] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009900.970642] svc: TCP record, 140 bytes
> >> [4009900.974659] svc: socket ee5fd600 recvfrom(dd54d08c, 3956) = 140
> >> [4009900.978652] svc: TCP complete record (140 bytes)
> >> [4009900.982649] svc: transport ee5fd600 put into queue
> >> [4009900.986642] svc: got len=140
> >> [4009900.990686] svc: transport ee5fd600 busy, not enqueued
> >> [4009900.994848] svc: socket ee5fd600 sendto([efb5c000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> >> [4009900.998725] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009901.002644] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.006662] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.010648] svc: server ee978000 waiting for data (to = 900000)
> >> [4009901.046694] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009901.050652] svc: server ee8fe000, pool 0, transport ee5fd600, inuse=2
> >> [4009901.054650] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009901.058666] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009901.062647] svc: TCP record, 156 bytes
> >> [4009901.066669] svc: socket ee5fd600 recvfrom(eec9209c, 3940) = 156
> >> [4009901.070648] svc: TCP complete record (156 bytes)
> >> [4009901.074655] svc: transport ee5fd600 put into queue
> >> [4009901.078648] svc: got len=156
> >> [4009901.082688] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.086844] svc: socket ee5fd600 sendto([e98bf000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> >> [4009901.090750] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009901.094650] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.098671] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.102655] svc: server ee8fe000 waiting for data (to = 900000)
> >> [4009901.138700] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009901.142658] svc: server ee9f2000, pool 0, transport ee5fd600, inuse=2
> >> [4009901.146656] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009901.150678] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009901.154654] svc: TCP record, 124 bytes
> >> [4009901.158671] svc: socket ee5fd600 recvfrom(e977007c, 3972) = 124
> >> [4009901.162654] svc: TCP complete record (124 bytes)
> >> [4009901.166659] svc: transport ee5fd600 put into queue
> >> [4009901.170654] svc: got len=124
> >> [4009901.174694] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.178811] svc: socket ee5fd600 sendto([dd67d000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> >> [4009901.182741] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009901.186655] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.190673] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.194661] svc: server ee9f2000 waiting for data (to = 900000)
> >> [4009901.222747] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009901.226690] svc: server eef12000, pool 0, transport ee5fd600, inuse=2
> >> [4009901.230661] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009901.234677] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009901.238658] svc: TCP record, 152 bytes
> >> [4009901.242673] svc: socket ee5fd600 recvfrom(dd785098, 3944) = 152
> >> [4009901.246659] svc: TCP complete record (152 bytes)
> >> [4009901.250664] svc: transport ee5fd600 put into queue
> >> [4009901.254659] svc: got len=152
> >> [4009901.258700] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.262809] svc: socket ee5fd600 sendto([ee92b000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> >> [4009901.266734] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009901.270660] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.274677] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.278665] svc: server eef12000 waiting for data (to = 900000)
> >> [4009901.314692] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009901.318668] svc: server eeeb7000, pool 0, transport ee5fd600, inuse=2
> >> [4009901.322667] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009901.326682] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009901.330664] svc: TCP record, 156 bytes
> >> [4009901.334681] svc: socket ee5fd600 recvfrom(e971609c, 3940) = 156
> >> [4009901.338665] svc: TCP complete record (156 bytes)
> >> [4009901.342672] svc: transport ee5fd600 put into queue
> >> [4009901.346665] svc: got len=156
> >> [4009901.350707] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.354849] svc: socket ee5fd600 sendto([e95ee000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> >> [4009901.358747] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009901.362666] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.366683] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.370672] svc: server eeeb7000 waiting for data (to = 900000)
> >> [4009901.406725] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009901.410674] svc: server eef31000, pool 0, transport ee5fd600, inuse=2
> >> [4009901.414672] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009901.418691] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009901.422670] svc: TCP record, 136 bytes
> >> [4009901.426690] svc: socket ee5fd600 recvfrom(e99b0088, 3960) = 136
> >> [4009901.430671] svc: TCP complete record (136 bytes)
> >> [4009901.434677] svc: transport ee5fd600 put into queue
> >> [4009901.438684] svc: got len=136
> >> [4009901.442710] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.446878] svc: socket ee5fd600 sendto([e9bee000 124... ], 124) = 124 (addr 192.168.1.5, port=910)
> >> [4009901.450767] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009901.454672] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.458754] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.462690] svc: server eef31000 waiting for data (to = 900000)
> >> [4009901.499151] svc: transport ee5fd600 dequeued, inuse=1
> >> [4009901.502687] svc: server eefc4000, pool 0, transport ee5fd600, inuse=2
> >> [4009901.506679] svc: tcp_recv ee5fd600 data 1 conn 0 close 0
> >> [4009901.510698] svc: socket ee5fd600 recvfrom(ee5fd764, 0) = 4
> >> [4009901.514675] svc: TCP record, 140 bytes
> >> [4009901.518694] svc: socket ee5fd600 recvfrom(e996b08c, 3956) = 140
> >> [4009901.522676] svc: TCP complete record (140 bytes)
> >> [4009901.526683] svc: transport ee5fd600 put into queue
> >> [4009901.530676] svc: got len=140
> >> [4009901.534716] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.542824] svc: socket ee5fd600 sendto([dfbb0000 232... ], 232) = 232 (addr 192.168.1.5, port=910)
> >> [4009901.546806] svc: socket c1888960 TCP data ready (svsk ee5fd600)
> >> [4009901.550678] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.554700] svc: transport ee5fd600 busy, not enqueued
> >> [4009901.558682] svc: server eefc4000 waiting for data (to = 900000)
> >>
> >>
> >> --
> >> Ian Campbell
> >>
> >> GIVE: Support the helpless victims of computer error.
> >>
> >>
>
>
--
Ian Campbell
No rock so hard but that a little wave
May beat admission in a thousand years.
-- Tennyson
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-13 8:57 ` Ian Campbell
@ 2008-09-16 5:48 ` Ian Campbell
2008-09-16 11:38 ` Tom Tucker
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-09-16 5:48 UTC (permalink / raw)
To: Tom Tucker
Cc: J. Bruce Fields, Trond Myklebust, Grant Coady, linux-kernel,
neilb, linux-nfs, e1000-devel
[-- Attachment #1: Type: text/plain, Size: 1964 bytes --]
(dropping the e1000 guys, it seems unnecessary to keep spamming them
with this issue when it's unlikely to be anything to do with them. I've
left their list on for now so they know. I'd suggest dropping it from
any replies.)
On Sat, 2008-09-13 at 09:57 +0100, Ian Campbell wrote:
> On Fri, 2008-09-12 at 18:15 -0500, Tom Tucker wrote:
> > Iain sadly yes. There's a thread stuck holding the BUSY bit or a thread
> > failed to clear the bit properly
> > (maybe an error path). Data continues to arrives, but the transport
> > never gets put back on the queue
> > because it's BUSY bit is set. In other words, this is a different error
> > than the one we've been chasing.
> >
> > If I sent you a patch, could you rebuild the kernel and give it a whirl?
> > Also, can you give me a
> > kernel.org relative commit-id or tag for the kernel that you're using?
>
> I sure could. I'm using the Debian kernel which is currently at 2.6.26.3
> (pkg version 2.6.26-4) although I have an update to 2.6.26.4 (via pkg
> 2.6.26-5) pending.
>
> If I'm going to build my own I'll start with current git
> (a551b98d5f6fce5897d497abd8bfb262efb33d2a) and repro there before trying
> your patch.
FYI I've repro'd with
commit a551b98d5f6fce5897d497abd8bfb262efb33d2a
Merge: d1c6d2e... 50bed2e...
Author: Linus Torvalds <torvalds@linux-foundation.org>
Date: Thu Sep 11 11:50:15 2008 -0700
Merge branch 'for-linus' of git://git.kernel.dk/linux-2.6-block
* 'for-linus' of git://git.kernel.dk/linux-2.6-block:
sg: disable interrupts inside sg_copy_buffer
which was the latest git a few days back.
I'm going to start bisecting between v2.6.25 and v2.6.26. There's 173
commits in fs/nfs* net/sunrpc in that interval so with a day per test I
should have something next week...
Ian.
--
Ian Campbell
Vini, vidi, Linux!
-- Unknown source
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-16 5:48 ` Ian Campbell
@ 2008-09-16 11:38 ` Tom Tucker
2008-09-16 15:03 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: Tom Tucker @ 2008-09-16 11:38 UTC (permalink / raw)
To: Ian Campbell
Cc: J. Bruce Fields, Trond Myklebust, Grant Coady, linux-kernel,
neilb, linux-nfs
Ian Campbell wrote:
> (dropping the e1000 guys, it seems unnecessary to keep spamming them
> with this issue when it's unlikely to be anything to do with them. I've
> left their list on for now so they know. I'd suggest dropping it from
> any replies.)
>
> On Sat, 2008-09-13 at 09:57 +0100, Ian Campbell wrote:
>
>> On Fri, 2008-09-12 at 18:15 -0500, Tom Tucker wrote:
>>
>>> Iain sadly yes. There's a thread stuck holding the BUSY bit or a thread
>>> failed to clear the bit properly
>>> (maybe an error path). Data continues to arrives, but the transport
>>> never gets put back on the queue
>>> because it's BUSY bit is set. In other words, this is a different error
>>> than the one we've been chasing.
>>>
>>> If I sent you a patch, could you rebuild the kernel and give it a whirl?
>>> Also, can you give me a
>>> kernel.org relative commit-id or tag for the kernel that you're using?
>>>
>> I sure could. I'm using the Debian kernel which is currently at 2.6.26.3
>> (pkg version 2.6.26-4) although I have an update to 2.6.26.4 (via pkg
>> 2.6.26-5) pending.
>>
>> If I'm going to build my own I'll start with current git
>> (a551b98d5f6fce5897d497abd8bfb262efb33d2a) and repro there before trying
>> your patch.
>>
>
> FYI I've repro'd with
> commit a551b98d5f6fce5897d497abd8bfb262efb33d2a
> Merge: d1c6d2e... 50bed2e...
> Author: Linus Torvalds <torvalds@linux-foundation.org>
> Date: Thu Sep 11 11:50:15 2008 -0700
>
> Merge branch 'for-linus' of git://git.kernel.dk/linux-2.6-block
>
> * 'for-linus' of git://git.kernel.dk/linux-2.6-block:
> sg: disable interrupts inside sg_copy_buffer
> which was the latest git a few days back.
>
> I'm going to start bisecting between v2.6.25 and v2.6.26. There's 173
> commits in fs/nfs* net/sunrpc in that interval so with a day per test I
> should have something next week...
>
>
Iain:
I'm assuming you'll do this in advance of any patch from me? I was
simply going to add printk to the various shutdown paths and see if we
could get some finer grained debug output since the generic transport
debug output was too verbose.
Let me know and I'll help if I can,
Tom
> Ian.
>
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-16 11:38 ` Tom Tucker
@ 2008-09-16 15:03 ` Ian Campbell
2008-09-16 15:58 ` Tom Tucker
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-09-16 15:03 UTC (permalink / raw)
To: Tom Tucker
Cc: J. Bruce Fields, Trond Myklebust, Grant Coady, linux-kernel,
neilb, linux-nfs
On Tue, 2008-09-16 at 06:38 -0500, Tom Tucker wrote:
> Iain:
I'm actually Ian. (I don't actually mind that much, the other way round
is probably much more annoying for people called Iain).
> I'm assuming you'll do this in advance of any patch from me?
I was going to grind through it unless/until you sent a patch.
> I was simply going to add printk to the various shutdown paths and see if we
> could get some finer grained debug output since the generic transport
> debug output was too verbose.
>
> Let me know and I'll help if I can,
You mean logging on the client or server side? I'm currently bisecting
client. I'm happy to add some extra prints there if you point me in the
direction of the code paths of interest.
Ian.
--
Ian Campbell
Current Noise: Anathema - Leave No Trace
I haven't lost my mind; I know exactly where I left it.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-16 15:03 ` Ian Campbell
@ 2008-09-16 15:58 ` Tom Tucker
2008-09-16 16:24 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: Tom Tucker @ 2008-09-16 15:58 UTC (permalink / raw)
To: Ian Campbell
Cc: J. Bruce Fields, Trond Myklebust, Grant Coady, linux-kernel,
neilb, linux-nfs
Ian Campbell wrote:
> On Tue, 2008-09-16 at 06:38 -0500, Tom Tucker wrote:
>> Iain:
>
> I'm actually Ian. (I don't actually mind that much, the other way round
> is probably much more annoying for people called Iain).
>
Oops, sorry Ian.
>> I'm assuming you'll do this in advance of any patch from me?
>
> I was going to grind through it unless/until you sent a patch.
>
>> I was simply going to add printk to the various shutdown paths and see if we
>> could get some finer grained debug output since the generic transport
>> debug output was too verbose.
>>
>> Let me know and I'll help if I can,
>
> You mean logging on the client or server side? I'm currently bisecting
> client. I'm happy to add some extra prints there if you point me in the
> direction of the code paths of interest.
>
Ah. Ok. I'll send you a patch for the server.
> Ian.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-16 15:58 ` Tom Tucker
@ 2008-09-16 16:24 ` Ian Campbell
2008-09-23 7:59 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-09-16 16:24 UTC (permalink / raw)
To: Tom Tucker
Cc: J. Bruce Fields, Trond Myklebust, Grant Coady, linux-kernel,
neilb, linux-nfs
On Tue, 2008-09-16 at 10:58 -0500, Tom Tucker wrote:
> I'll send you a patch for the server.
Thanks. It's running 2.6.25.7 although I can massage a more recent patch
if needed. (I'm a bit less keen to run a variety of random kernels on
that machine).
Ian.
--
Ian Campbell
Current Noise: The Ocean - Protoerozoic - Rhyacian - Untimely Meditations
We may not return the affection of those who like us, but we always respect
their good judgement.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-16 16:24 ` Ian Campbell
@ 2008-09-23 7:59 ` Ian Campbell
2008-09-23 11:33 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-09-23 7:59 UTC (permalink / raw)
To: Tom Tucker
Cc: J. Bruce Fields, Trond Myklebust, Grant Coady, linux-kernel,
neilb, linux-nfs
[-- Attachment #1: Type: text/plain, Size: 578 bytes --]
I've found that the problem was backported into the stable stream since
I cannot reproduce the issue with 2.6.26 but I can with 2.6.26.5. This
is quite useful since there are only 3 relevant looking changesets in
that range. I will bisect between these before confirming the culprit on
mainline.
I've had a private reply from someone who is seeing the same issue with
2.6.26.5 against a server running 2.6.18 (Debian Etch) which I think
largely discounts the server from being at fault.
Ian.
--
Ian Campbell
Everything ends badly. Otherwise it wouldn't end.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-23 7:59 ` Ian Campbell
@ 2008-09-23 11:33 ` Ian Campbell
2008-09-23 17:03 ` J. Bruce Fields
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-09-23 11:33 UTC (permalink / raw)
To: Tom Tucker
Cc: J. Bruce Fields, Trond Myklebust, Grant Coady, linux-kernel,
neilb, linux-nfs
[-- Attachment #1: Type: text/plain, Size: 1331 bytes --]
On Tue, 2008-09-23 at 08:59 +0100, Ian Campbell wrote:
> I've found that the problem was backported into the stable stream since
> I cannot reproduce the issue with 2.6.26 but I can with 2.6.26.5. This
> is quite useful since there are only 3 relevant looking changesets in
> that range. I will bisect between these before confirming the culprit on
> mainline.
It reports:
daedfbe2a67628a40076a6c75fb945c60f608a2e is first bad commit
commit daedfbe2a67628a40076a6c75fb945c60f608a2e
Author: Trond Myklebust <Trond.Myklebust@netapp.com>
Date: Wed Jun 11 17:39:04 2008 -0400
NFS: Ensure we zap only the access and acl caches when setting new acls
commit f41f741838480aeaa3a189cff6e210503cf9c42d upstream
...and ensure that we obey the NFS_INO_INVALID_ACL flag when retrieving the
acls.
Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
I'm just about to build f41f741838480aeaa3a189cff6e210503cf9c42d and the
one before and try those.
I'm not using ACLs as far as I am aware.
Ian.
--
Ian Campbell
If we were meant to get up early, God would have created us with alarm clocks.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-23 11:33 ` Ian Campbell
@ 2008-09-23 17:03 ` J. Bruce Fields
2008-09-26 15:37 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: J. Bruce Fields @ 2008-09-23 17:03 UTC (permalink / raw)
To: Ian Campbell
Cc: Tom Tucker, Trond Myklebust, Grant Coady, linux-kernel, neilb,
linux-nfs
On Tue, Sep 23, 2008 at 12:33:09PM +0100, Ian Campbell wrote:
> On Tue, 2008-09-23 at 08:59 +0100, Ian Campbell wrote:
> > I've found that the problem was backported into the stable stream since
> > I cannot reproduce the issue with 2.6.26 but I can with 2.6.26.5. This
> > is quite useful since there are only 3 relevant looking changesets in
> > that range. I will bisect between these before confirming the culprit on
> > mainline.
Could you double-check that this is reproduceable with this commit
applied, and not reproduceable when it's not?
I suppose it's not impossible that this could be triggering the problem
in some very roundabout way, but it seems a bit out of left field--so I
wonder whether one of the bisection points could have gotten marked good
when it should have been bad, or vice-versa.
> It reports:
>
> daedfbe2a67628a40076a6c75fb945c60f608a2e is first bad commit
> commit daedfbe2a67628a40076a6c75fb945c60f608a2e
> Author: Trond Myklebust <Trond.Myklebust@netapp.com>
> Date: Wed Jun 11 17:39:04 2008 -0400
>
> NFS: Ensure we zap only the access and acl caches when setting new acls
>
> commit f41f741838480aeaa3a189cff6e210503cf9c42d upstream
>
> ...and ensure that we obey the NFS_INO_INVALID_ACL flag when retrieving the
> acls.
>
> Signed-off-by: Trond Myklebust <Trond.Myklebust@netapp.com>
> Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
>
> I'm just about to build f41f741838480aeaa3a189cff6e210503cf9c42d and the
> one before and try those.
>
> I'm not using ACLs as far as I am aware.
I think commands like "ls" try to get posix acls these days, so it's
possible that the nfs3_proc_getacl code at least might be getting
called. Why that would matter I can't see.
--b.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-23 17:03 ` J. Bruce Fields
@ 2008-09-26 15:37 ` Ian Campbell
2008-09-26 18:17 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-09-26 15:37 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Tom Tucker, Trond Myklebust, Grant Coady, linux-kernel, neilb,
linux-nfs
[-- Attachment #1: Type: text/plain, Size: 1544 bytes --]
On Tue, 2008-09-23 at 13:03 -0400, J. Bruce Fields wrote:
> On Tue, Sep 23, 2008 at 12:33:09PM +0100, Ian Campbell wrote:
> > On Tue, 2008-09-23 at 08:59 +0100, Ian Campbell wrote:
> > > I've found that the problem was backported into the stable stream since
> > > I cannot reproduce the issue with 2.6.26 but I can with 2.6.26.5. This
> > > is quite useful since there are only 3 relevant looking changesets in
> > > that range. I will bisect between these before confirming the culprit on
> > > mainline.
>
> Could you double-check that this is reproduceable with this commit
> applied, and not reproduceable when it's not?
I've reproduced with exactly commit
f41f741838480aeaa3a189cff6e210503cf9c42d on trunk and am now running
2e96d2867245668dbdb973729288cf69b9fafa66 which is the changeset
immediately before.
> I suppose it's not impossible that this could be triggering the problem
> in some very roundabout way, but it seems a bit out of left field--so I
> wonder whether one of the bisection points could have gotten marked good
> when it should have been bad, or vice-versa.
It's possible, the good case is naturally quite hard to establish with
100% certainty. I declared v2.6.26 OK after an uptime of 4 days and 19
hours, compared with failure normally within 1-2 days. It's possible I
was premature in doing so. I'll run 2e96d2867 for at least a full week
before reporting back.
Ian.
--
Ian Campbell
Unix is mature OS, windows is still in diapers and they smell badly.
-- Rafael Skodlar
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-26 15:37 ` Ian Campbell
@ 2008-09-26 18:17 ` Ian Campbell
2008-09-27 3:54 ` J. Bruce Fields
0 siblings, 1 reply; 64+ messages in thread
From: Ian Campbell @ 2008-09-26 18:17 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Tom Tucker, Trond Myklebust, Grant Coady, linux-kernel, neilb,
linux-nfs
[-- Attachment #1: Type: text/plain, Size: 569 bytes --]
On Fri, 2008-09-26 at 16:37 +0100, Ian Campbell wrote:
> It's possible, the good case is naturally quite hard to establish with
> 100% certainty. I declared v2.6.26 OK after an uptime of 4 days and 19
> hours, compared with failure normally within 1-2 days. It's possible I
> was premature in doing so. I'll run 2e96d2867 for at least a full week
> before reporting back.
Or it'll reproduce after just a few hours :-/
Looks like a false negative on v2.6.26 -- I'll retest from there.
Ian.
--
Ian Campbell
That secret you've been guarding, isn't.
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-26 18:17 ` Ian Campbell
@ 2008-09-27 3:54 ` J. Bruce Fields
2008-09-27 10:16 ` Ian Campbell
0 siblings, 1 reply; 64+ messages in thread
From: J. Bruce Fields @ 2008-09-27 3:54 UTC (permalink / raw)
To: Ian Campbell
Cc: Tom Tucker, Trond Myklebust, Grant Coady, linux-kernel, neilb,
linux-nfs
On Fri, Sep 26, 2008 at 07:17:33PM +0100, Ian Campbell wrote:
> On Fri, 2008-09-26 at 16:37 +0100, Ian Campbell wrote:
> > It's possible, the good case is naturally quite hard to establish with
> > 100% certainty. I declared v2.6.26 OK after an uptime of 4 days and 19
> > hours, compared with failure normally within 1-2 days. It's possible I
> > was premature in doing so. I'll run 2e96d2867 for at least a full week
> > before reporting back.
>
> Or it'll reproduce after just a few hours :-/
>
> Looks like a false negative on v2.6.26 -- I'll retest from there.
OK, so apologies, but this has been a long thread, and maybe we could
use a summary of the symptoms and the results so far. I think you said
2.6.24 or .25 was the last you're *positive* was good?
--b.
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-09-27 3:54 ` J. Bruce Fields
@ 2008-09-27 10:16 ` Ian Campbell
0 siblings, 0 replies; 64+ messages in thread
From: Ian Campbell @ 2008-09-27 10:16 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Tom Tucker, Trond Myklebust, Grant Coady, linux-kernel, neilb,
linux-nfs
[-- Attachment #1: Type: text/plain, Size: 6946 bytes --]
On Fri, 2008-09-26 at 23:54 -0400, J. Bruce Fields wrote:
> OK, so apologies, but this has been a long thread, and maybe we could
> use a summary of the symptoms and the results so far. I think you said
> 2.6.24 or .25 was the last you're *positive* was good?
2.6.24.x was good 2.6.25.y was not. The Debian packages don't include
the stable release numbers in their version so I'm unsure of x and y,
Judging from the changelog entries I think 2.6.24.3 and 2.6.25.10.
Since I have been building my own kernels I have seen repros with:
a551b98d5f6fce5897d497abd8bfb262efb33d2a Merge branch 'for-linus' of
git://git.kernel.dk/linux-2.6-block (was tip of git at the time I tested
it a while ago)
c02a79dedc7f3c3d4fdbb5eb2000cacea5df4cde v2.6.26.5
daedfbe2a67628a40076a6c75fb945c60f608a2e NFS: Ensure we zap only the
access and acl caches when setting new acls (backport of
f41f741838480aeaa3a189cff6e210503cf9c42d to 2.6.26.y stream, identified
by bisect, possibly/probably incorrectly)
f41f741838480aeaa3a189cff6e210503cf9c42d NFS: Ensure we zap only the
access and acl caches when setting new acls (original trunk commit of
above)
2e96d2867245668dbdb973729288cf69b9fafa66 NFS: Fix a warning in
nfs4_async_handle_error (commit before
f41f741838480aeaa3a189cff6e210503cf9c42d)
I did not see the problem with bce7f793daec3e65ec5c5705d2457b81fe7b5725
(v2.6.26) in 4 days 19 hours. Due to the apparent mis-bisection leading
to daedfbe2a67628a40076a6c75fb945c60f608a2e this result is in doubt, I'm
currently retesting this kernel.
The bisect was restricted to paths under fs/nfs* and net/sunrpc. I've
gone over the range v2.6.26..daedfbe2a67628a40076a6c75fb945c60f608a2e
and nothing jumps out at me. If my retest of 2.6.26 shows it to be OK
again I'll bisect over the non-NFS changesets in that range.
The symptoms are a hang of any access to an NFS mounted path, although
the rest of the system continues to work.
Other potentially interesting info cribbed from earlier postings:
While the hang is occurring rpc_debug on the server gives nothing, on
the client gives:
[144741.637997] -pid- proc flgs status -client- -prog- --rqstp- -timeout -rpcwait -action- ---ops--
[144741.637997] 3439 0004 0080 -11 f3f48200 100003 f7770000 0 xprt_sending fa0ae88e fa0bddf4
[144741.637997] 3438 0001 00a0 0 f77f2a00 100003 f77700d0 5000 xprt_pending fa0ae88e fa0bddf4
The mount points are
hopkins:/storage/music /storage/music nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
hopkins:/storage/mythtv/recordings /var/lib/mythtv/recordings nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
hopkins:/var/lib/mythvideo /var/lib/mythvideo nfs rw,nosuid,vers=3,rsize=32768,wsize=32768,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
hopkins:/storage/home/ijc /home/ijc nfs rw,vers=3,rsize=131072,wsize=131072,namlen=255,hard,nointr,proto=tcp,timeo=600,retrans=2,sec=sys,mountproto=udp,addr=192.168.1.6 0 0
all of them seem to be effected simultaneously (but I'll check to make double sure next time it occurs).
I've seen hung task backtraces from effected processes, e.g.
Aug 4 06:27:28 iranon kernel: [137969.382277] INFO: task mythbackend:3161 blocked for more than 120 seconds.
Aug 4 06:27:28 iranon kernel: [137969.382287] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 4 06:27:28 iranon kernel: [137969.382291] mythbackend D 00005dfc 0 3161 1
Aug 4 06:27:28 iranon kernel: [137969.382295] f2006c70 00000082 4b05cc7e 00005dfc f2006df0 c1c0e920 00000000 f7273390
Aug 4 06:27:28 iranon kernel: [137969.382301] 0000c8b4 00000000 00000001 f7273398 00000282 00000202 f71b0ab0 f200ddf0
Aug 4 06:27:28 iranon kernel: [137969.382306] c1c012bc fa18f0a6 fa18f0bf c02b45a7 f71b0ab0 00000000 f200de0c fa18f0a6
Aug 4 06:27:28 iranon kernel: [137969.382311] Call Trace:
Aug 4 06:27:28 iranon kernel: [137969.382347] [<fa18f0a6>] nfs_wait_schedule+0x0/0x1e [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382384] [<fa18f0bf>] nfs_wait_schedule+0x19/0x1e [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382399] [<c02b45a7>] __wait_on_bit_lock+0x2a/0x52
Aug 4 06:27:28 iranon kernel: [137969.382407] [<fa18f0a6>] nfs_wait_schedule+0x0/0x1e [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382421] [<c02b462e>] out_of_line_wait_on_bit_lock+0x5f/0x67
Aug 4 06:27:28 iranon kernel: [137969.382429] [<c0133a1d>] wake_bit_function+0x0/0x3c
Aug 4 06:27:28 iranon kernel: [137969.382441] [<fa18f16e>] __nfs_revalidate_inode+0xaa/0x211 [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382458] [<c017bac0>] do_lookup+0x53/0x145
Aug 4 06:27:28 iranon kernel: [137969.382466] [<c0187609>] mntput_no_expire+0x11/0x64
Aug 4 06:27:28 iranon kernel: [137969.382472] [<c017d971>] __link_path_walk+0xa71/0xb65
Aug 4 06:27:28 iranon kernel: [137969.382477] [<c017bac0>] do_lookup+0x53/0x145
Aug 4 06:27:28 iranon kernel: [137969.382483] [<c0187609>] mntput_no_expire+0x11/0x64
Aug 4 06:27:28 iranon kernel: [137969.382492] [<c0187609>] mntput_no_expire+0x11/0x64
Aug 4 06:27:28 iranon kernel: [137969.382496] [<c017daf5>] path_walk+0x90/0x98
Aug 4 06:27:28 iranon kernel: [137969.382502] [<fa18f935>] nfs_getattr+0x8f/0xbe [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382520] [<fa18f8a6>] nfs_getattr+0x0/0xbe [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382536] [<c01785ac>] vfs_getattr+0x36/0x4d
Aug 4 06:27:28 iranon kernel: [137969.382545] [<c0178684>] vfs_lstat_fd+0x27/0x39
Aug 4 06:27:28 iranon kernel: [137969.382550] [<fa18bca6>] nfs_permission+0x0/0x129 [nfs]
Aug 4 06:27:28 iranon kernel: [137969.382567] [<c0187609>] mntput_no_expire+0x11/0x64
Aug 4 06:27:28 iranon kernel: [137969.382572] [<c0174914>] sys_faccessat+0x11e/0x15d
Aug 4 06:27:28 iranon kernel: [137969.382582] [<c0178703>] sys_lstat64+0xf/0x23
Aug 4 06:27:28 iranon kernel: [137969.382588] [<c017603c>] vfs_read+0xe3/0x11e
Aug 4 06:27:28 iranon kernel: [137969.382598] [<c0174962>] sys_access+0xf/0x13
Aug 4 06:27:28 iranon kernel: [137969.382603] [<c01077e8>] sysenter_past_esp+0x6d/0xa5
Aug 4 06:27:28 iranon kernel: [137969.382617] =======================
Ian.
--
Ian Campbell
Contemptuous lights flashed across the computer's console.
-- Hitchhiker's Guide to the Galaxy
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2008-08-18 19:37 ` J. K. Cliburn
2008-08-18 23:13 ` Athanasius
@ 2009-05-12 20:27 ` Frank Filz
2009-05-13 0:05 ` Jay Cliburn
1 sibling, 1 reply; 64+ messages in thread
From: Frank Filz @ 2009-05-12 20:27 UTC (permalink / raw)
To: jacliburn; +Cc: Grant Coady, linux-kernel, neilb, bfields, linux-nfs
On Mon, 2008-08-18 at 14:37 -0500, J. K. Cliburn wrote:
> Athanasius wrote:
> > On Mon, Aug 18, 2008 at 12:02:20PM +1000, Grant Coady wrote:
> >> I've been using NFS here for years, lately there's something odd going on
> >> since about a month or so. Previously reported last month:
> >> http://www.gossamer-threads.com/lists/linux/kernel/951419?page=last
> >>
> >> Now with 2.6.27-rc3 on one of the client boxes I get a complete stall
> >> at odd times when accessing the server's exported directory, cannot
> >> see a pattern to it. Eventually recovers after a Ctrl-C. Nothing in
> >> the server or client log files. Not easy to reproduce either.
> >
> > I wonder if this is what I've been seeing. I've been otherwise too
> > busy to properly report it, thinking that *someone* else must also be
> > seeing it and it's being worked on, else it's a subtle configuration
> > problem my end.
> >
> > I first started seeing this with 2.6.26 on the client end,
>
> I see you're running the atl1 driver on your client. I'm chasing an
> intermittent bug in that driver that seems to be affected by certain
> offload parameters. Try disabling TSO and see if things improve.
Was this ever resolved? We're seeing a similar hangup in a 2.6.27
kernel.
Thanks
Frank Filz
^ permalink raw reply [flat|nested] 64+ messages in thread
* Re: NFS regression? Odd delays and lockups accessing an NFS export.
2009-05-12 20:27 ` Frank Filz
@ 2009-05-13 0:05 ` Jay Cliburn
0 siblings, 0 replies; 64+ messages in thread
From: Jay Cliburn @ 2009-05-13 0:05 UTC (permalink / raw)
To: Frank Filz; +Cc: Grant Coady, linux-kernel, neilb, bfields, linux-nfs
On Tue, 12 May 2009 13:27:58 -0700
Frank Filz <ffilzlnx@us.ibm.com> wrote:
> On Mon, 2008-08-18 at 14:37 -0500, J. K. Cliburn wrote:
> > Athanasius wrote:
> > > On Mon, Aug 18, 2008 at 12:02:20PM +1000, Grant Coady wrote:
> > >> I've been using NFS here for years, lately there's something odd going on
> > >> since about a month or so. Previously reported last month:
> > >> http://www.gossamer-threads.com/lists/linux/kernel/951419?page=last
[...]
> > I see you're running the atl1 driver on your client. I'm chasing an
> > intermittent bug in that driver that seems to be affected by certain
> > offload parameters. Try disabling TSO and see if things improve.
>
> Was this ever resolved? We're seeing a similar hangup in a 2.6.27
> kernel.
The TSO problem in the atl1 driver was never resolved, and TSO was turned off by default with commit 82c26a9d117f0178b8c1b33429014b6d99c470f6 in the 2.6.27 cycle. It was also added to the -stable tree at 2.6.26.4.
http://lkml.org/lkml/2008/9/3/251
Jay
^ permalink raw reply [flat|nested] 64+ messages in thread
end of thread, other threads:[~2009-05-13 0:05 UTC | newest]
Thread overview: 64+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-08-18 2:02 NFS regression? Odd delays and lockups accessing an NFS export Grant Coady
2008-08-18 18:50 ` Athanasius
2008-08-18 19:19 ` Trond Myklebust
2008-08-18 19:37 ` J. K. Cliburn
2008-08-18 23:13 ` Athanasius
2009-05-12 20:27 ` Frank Filz
2009-05-13 0:05 ` Jay Cliburn
2008-08-18 19:20 ` Trond Myklebust
2008-08-20 1:10 ` Grant Coady
2008-08-20 23:17 ` Grant Coady
2008-08-22 10:23 ` Ian Campbell
2008-08-22 18:08 ` Trond Myklebust
2008-08-22 18:13 ` Ian Campbell
2008-08-22 19:33 ` John Ronciak
2008-08-22 20:00 ` Ian Campbell
2008-08-22 21:15 ` John Ronciak
2008-08-22 21:23 ` Trond Myklebust
2008-08-22 21:37 ` Ian Campbell
2008-08-22 21:56 ` Trond Myklebust
2008-08-22 22:41 ` Ian Campbell
2008-08-24 18:53 ` Ian Campbell
2008-08-24 19:17 ` Trond Myklebust
2008-08-24 19:19 ` Trond Myklebust
2008-08-24 22:09 ` Ian Campbell
2008-08-24 22:15 ` Trond Myklebust
2008-08-25 9:59 ` Ian Campbell
2008-08-25 16:04 ` Tom Tucker
2008-08-25 16:54 ` Trond Myklebust
2008-08-25 20:15 ` Tom Tucker
2008-08-26 19:27 ` J. Bruce Fields
2008-08-27 14:43 ` Tom Tucker
2008-08-30 15:47 ` Ian Campbell
2008-08-31 19:30 ` J. Bruce Fields
2008-08-31 19:44 ` Ian Campbell
2008-08-31 19:46 ` J. Bruce Fields
2008-08-31 19:49 ` Ian Campbell
2008-08-31 19:51 ` Tom Tucker
2008-08-31 19:51 ` Tom Tucker
2008-08-31 21:18 ` Ian Campbell
2008-09-01 17:20 ` Tom Tucker
2008-09-01 17:46 ` Ian Campbell
2008-09-10 8:40 ` Ian Campbell
2008-09-12 22:43 ` J. Bruce Fields
2008-09-12 23:15 ` Tom Tucker
2008-09-13 8:57 ` Ian Campbell
2008-09-16 5:48 ` Ian Campbell
2008-09-16 11:38 ` Tom Tucker
2008-09-16 15:03 ` Ian Campbell
2008-09-16 15:58 ` Tom Tucker
2008-09-16 16:24 ` Ian Campbell
2008-09-23 7:59 ` Ian Campbell
2008-09-23 11:33 ` Ian Campbell
2008-09-23 17:03 ` J. Bruce Fields
2008-09-26 15:37 ` Ian Campbell
2008-09-26 18:17 ` Ian Campbell
2008-09-27 3:54 ` J. Bruce Fields
2008-09-27 10:16 ` Ian Campbell
2008-08-25 21:39 ` Roger Heflin
2008-08-25 20:23 ` Grant Coady
2008-08-25 22:11 ` Trond Myklebust
2008-08-26 0:29 ` Grant Coady
2008-08-26 0:59 ` Muntz, Daniel
2008-08-26 1:06 ` Grant Coady
-- strict thread matches above, loose matches on Subject: below --
2008-09-10 2:51 Benoit Plessis
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox