* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() [not found] ` <4B4F5A00.4060604-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org> @ 2010-01-22 23:03 ` Andrew Morton 2010-01-27 19:05 ` J. Bruce Fields 0 siblings, 1 reply; 12+ messages in thread From: Andrew Morton @ 2010-01-22 23:03 UTC (permalink / raw) To: Harry Edmon; +Cc: linux-kernel, linux-nfs (cc linux-nfs) On Thu, 14 Jan 2010 09:53:04 -0800 Harry Edmon <harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org> wrote: > I have a single quad-core Xeon machine running 2.6.32.2 that keeps > giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This > machine is mainaly used as a NFS server. I have attached the kern.log > and my config file. Any ideas? > -- > > Dr. Harry Edmon E-MAIL: harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org > 206-543-0547 FAX: 206-543-0308 harry-B93hV6UPU7Z2icitjWtXSw@public.gmane.org, harry@uw.edu > Director of IT, College of the Environment and > Director of Computing, Dept of Atmospheric Sciences > University of Washington, Box 351640, Seattle, WA 98195-1640 > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() 2010-01-22 23:03 ` 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() Andrew Morton @ 2010-01-27 19:05 ` J. Bruce Fields 2010-01-27 23:01 ` Harry Edmon 0 siblings, 1 reply; 12+ messages in thread From: J. Bruce Fields @ 2010-01-27 19:05 UTC (permalink / raw) To: Andrew Morton; +Cc: Harry Edmon, linux-kernel, linux-nfs On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote: > (cc linux-nfs) > > On Thu, 14 Jan 2010 09:53:04 -0800 > Harry Edmon <harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org> wrote: > > > I have a single quad-core Xeon machine running 2.6.32.2 that keeps > > giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This > > machine is mainaly used as a NFS server. I have attached the kern.log > > and my config file. Any ideas? Are you using NFSv4, or only v2/v3? Looking at the trace: Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------ Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() OK, so that's the warning that we're trying to get something with reference count 0. Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8 Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1 What does the very first oops, bug, or warning you get look like? (If I'm reading the tain information right, this isn't the first one.) Jan 11 06:01:53 t kernel: [1311647.216100] Call Trace: Jan 11 06:01:53 t kernel: [1311647.231808] [<ffffffff81142230>] ? kref_get+0x23/0x2b Jan 11 06:01:53 t kernel: [1311647.263633] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 Jan 11 06:01:53 t kernel: [1311647.300658] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 Jan 11 06:01:53 t kernel: [1311647.336122] [<ffffffff81142230>] kref_get+0x23/0x2b Jan 11 06:01:53 t kernel: [1311647.366914] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] Jan 11 06:01:53 t kernel: [1311647.400301] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] And the only likely candidate I see for a kref is the svc_xprt_get(xprt). --b. Jan 11 06:01:53 t kernel: [1311647.436803] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] Jan 11 06:01:53 t kernel: [1311647.475903] [<ffffffff81035177>] ? default_wake_function+0x0/0xf Jan 11 06:01:53 t kernel: [1311647.513452] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] Jan 11 06:01:53 t kernel: [1311647.546838] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] Jan 11 06:01:53 t kernel: [1311647.579701] [<ffffffff8104af36>] kthread+0x7d/0x85 Jan 11 06:01:53 t kernel: [1311647.609967] [<ffffffff8100c97a>] child_rip+0xa/0x20 Jan 11 06:01:53 t kernel: [1311647.640749] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 Jan 11 06:01:53 t kernel: [1311647.671535] [<ffffffff8100c970>] ? child_rip+0x0/0x20 Jan 11 06:01:53 t kernel: [1311647.703359] ---[ end trace 1a77408724500fc0 ]--- Jan 11 06:01:53 t kernel: [1311647.732134] ------------[ cut here ]------------ Jan 11 06:01:53 t kernel: [1311647.736063] kernel BUG at fs/inode.c:1343! Jan 11 06:01:53 t kernel: [1311647.736063] invalid opcode: 0000 [#10] SMP Jan 11 06:01:53 t kernel: [1311647.736063] last sysfs file: /sys/block/md2/md/raid_disks Jan 11 06:01:53 t kernel: [1311647.736063] CPU 3 Jan 11 06:01:53 t kernel: [1311647.736063] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] Jan 11 06:01:53 t kernel: [1311647.736063] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8 Jan 11 06:01:53 t kernel: [1311647.736063] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65 Jan 11 06:01:53 t kernel: [1311647.736063] RSP: 0018:ffff88012730bd20 EFLAGS: 00010246 Jan 11 06:01:53 t kernel: [1311647.736063] RAX: 0000000000000000 RBX: ffff88010c9c7408 RCX: ffffffff8139f927 Jan 11 06:01:53 t kernel: [1311647.736063] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88010c9c7408 Jan 11 06:01:53 t kernel: [1311647.736063] RBP: ffff88012730bd30 R08: 0000000000000020 R09: ffff88002830ca70 Jan 11 06:01:53 t kernel: [1311647.736063] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff88010c9c73c0 Jan 11 06:01:53 t kernel: [1311647.736063] R13: ffffffffa028ef50 R14: ffff880018c58000 R15: ffff88012dd423c0 Jan 11 06:01:53 t kernel: [1311647.736063] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000 Jan 11 06:01:53 t kernel: [1311647.736063] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Jan 11 06:01:53 t kernel: [1311647.736063] CR2: 00007f0dc9cc5098 CR3: 0000000005bd8000 CR4: 00000000000006e0 Jan 11 06:01:53 t kernel: [1311647.736063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 11 06:01:53 t kernel: [1311647.736063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 11 06:01:53 t kernel: [1311647.736063] Process nfsd (pid: 3068, threadinfo ffff88012730a000, task ffff8801270e8150) Jan 11 06:01:53 t kernel: [1311647.736063] Stack: Jan 11 06:01:53 t kernel: [1311647.736063] ffff88012730bd90 ffff880018c58000 ffff88012730bd50 ffffffff811f4be2 Jan 11 06:01:53 t kernel: [1311647.736063] <0> ffff880018c58000 ffff880018c58010 ffff88012730bd70 ffffffffa027c1d2 Jan 11 06:01:53 t kernel: [1311647.736063] <0> ffff88012730bd70 ffff880018c58000 ffff88012730bda0 ffffffffa0286b03 Jan 11 06:01:53 t kernel: [1311647.736063] Call Trace: Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff811f4be2>] sock_release+0x5c/0x6c Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc] Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc] Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff81142230>] ? kref_get+0x23/0x2b Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc] Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff81142201>] kref_put+0x43/0x4f Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc] Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc] Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff81035177>] ? default_wake_function+0x0/0xf Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff8104af36>] kthread+0x7d/0x85 Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff8100c97a>] child_rip+0xa/0x20 Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff8100c970>] ? child_rip+0x0/0x20 Jan 11 06:01:53 t kernel: [1311647.736063] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85 Jan 11 06:01:53 t kernel: [1311647.736063] RIP [<ffffffff810bccc2>] iput+0x1b/0x65 Jan 11 06:01:53 t kernel: [1311647.736063] RSP <ffff88012730bd20> Jan 11 06:01:53 t kernel: [1311649.618340] ---[ end trace 1a77408724500fc1 ]--- ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() 2010-01-27 19:05 ` J. Bruce Fields @ 2010-01-27 23:01 ` Harry Edmon 2010-02-02 16:22 ` J. Bruce Fields 0 siblings, 1 reply; 12+ messages in thread From: Harry Edmon @ 2010-01-27 23:01 UTC (permalink / raw) To: J. Bruce Fields; +Cc: Andrew Morton, linux-kernel, linux-nfs [-- Attachment #1: Type: text/plain, Size: 2024 bytes --] J. Bruce Fields wrote: > On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote: > >> (cc linux-nfs) >> >> On Thu, 14 Jan 2010 09:53:04 -0800 >> Harry Edmon <harry@atmos.washington.edu> wrote: >> >> >>> I have a single quad-core Xeon machine running 2.6.32.2 that keeps >>> giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This >>> machine is mainaly used as a NFS server. I have attached the kern.log >>> and my config file. Any ideas? >>> > > Are you using NFSv4, or only v2/v3? Looking at the trace: > > Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------ > Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > NFSv4 and NFSv3. > OK, so that's the warning that we're trying to get something with > reference count 0. > > Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8 > Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1 > > What does the very first oops, bug, or warning you get look like? (If > I'm reading the tain information right, this isn't the first one.) > Attached is the kern.log that contains the first warning. -- Dr. Harry Edmon E-MAIL: harry@atmos.washington.edu 206-543-0547 FAX: 206-543-0308 harry@washington.edu, harry@uw.edu Director of IT, College of the Environment and Director of Computing, Dept of Atmospheric Sciences University of Washington, Box 351640, Seattle, WA 98195-1640 [-- Attachment #2: kern.log.3 --] [-- Type: text/plain, Size: 25606 bytes --] Dec 27 11:57:44 t kernel: [37000.838904] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0055): Battery charging started:. Dec 27 11:57:46 t kernel: [37002.028946] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0056): Battery charging completed:. Dec 28 00:00:51 t kernel: [80387.429120] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0. Dec 28 03:32:07 t kernel: [93063.950486] 3w-9xxx: scsi6: AEN: INFO (0x04:0x002B): Verify completed:unit=0. Dec 29 17:04:43 t kernel: [228218.883443] ------------[ cut here ]------------ Dec 29 17:04:45 t kernel: [228218.911691] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() Dec 29 17:04:45 t kernel: [228218.945083] Hardware name: X7DB8 Dec 29 17:04:45 t kernel: [228218.964948] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] Dec 29 17:04:45 t kernel: [228219.227296] Pid: 3108, comm: nfsd Not tainted 2.6.32.2 #1 Dec 29 17:04:45 t kernel: [228219.260164] Call Trace: Dec 29 17:04:45 t kernel: [228219.275350] [<ffffffff81142230>] ? kref_get+0x23/0x2b Dec 29 17:04:45 t kernel: [228219.306656] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 Dec 29 17:04:45 t kernel: [228219.343160] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 Dec 29 17:04:45 t kernel: [228219.378105] [<ffffffff81142230>] kref_get+0x23/0x2b Dec 29 17:04:45 t kernel: [228219.408379] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] Dec 29 17:04:45 t kernel: [228219.441242] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] Dec 29 17:04:45 t kernel: [228219.477225] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] Dec 29 17:04:45 t kernel: [228219.515806] [<ffffffff81035177>] ? default_wake_function+0x0/0xf Dec 29 17:04:45 t kernel: [228219.552834] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] Dec 29 17:04:45 t kernel: [228219.585698] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] Dec 29 17:04:45 t kernel: [228219.618043] [<ffffffff8104af36>] kthread+0x7d/0x85 Dec 29 17:04:45 t kernel: [228219.647789] [<ffffffff8100c97a>] child_rip+0xa/0x20 Dec 29 17:04:45 t kernel: [228219.678052] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 Dec 29 17:04:45 t kernel: [228219.708317] [<ffffffff8100c970>] ? child_rip+0x0/0x20 Dec 29 17:04:45 t kernel: [228219.739621] ---[ end trace 1a77408724500fae ]--- Dec 29 17:04:45 t kernel: [228219.767925] ------------[ cut here ]------------ Dec 29 17:04:45 t kernel: [228219.792730] kernel BUG at fs/inode.c:1343! Dec 29 17:04:45 t kernel: [228219.792730] invalid opcode: 0000 [#1] SMP Dec 29 17:04:45 t kernel: [228219.792730] last sysfs file: /sys/class/scsi_host/host6/stats Dec 29 17:04:45 t kernel: [228219.792730] CPU 0 Dec 29 17:04:45 t kernel: [228219.792730] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] Dec 29 17:04:45 t kernel: [228219.792730] Pid: 3108, comm: nfsd Tainted: G W 2.6.32.2 #1 X7DB8 Dec 29 17:04:45 t kernel: [228219.792730] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65 Dec 29 17:04:45 t kernel: [228219.792730] RSP: 0018:ffff88012588bd20 EFLAGS: 00010246 Dec 29 17:04:45 t kernel: [228219.792730] RAX: 0000000000000000 RBX: ffff88003c3c2cc8 RCX: ffffffff8139f927 Dec 29 17:04:45 t kernel: [228219.792730] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88003c3c2cc8 Dec 29 17:04:45 t kernel: [228219.792730] RBP: ffff88012588bd30 R08: 0000000000000002 R09: 0000000000000000 Dec 29 17:04:45 t kernel: [228219.792730] R10: ffffffff81142230 R11: 0000000100000001 R12: ffff88003c3c2c80 Dec 29 17:04:45 t kernel: [228219.792730] R13: ffffffffa028ef50 R14: ffff88010072d400 R15: ffff88012dd423c0 Dec 29 17:04:45 t kernel: [228219.792730] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 Dec 29 17:04:45 t kernel: [228219.792730] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Dec 29 17:04:45 t kernel: [228219.792730] CR2: 00007fad86c2e000 CR3: 00000000305e5000 CR4: 00000000000006f0 Dec 29 17:04:45 t kernel: [228219.792730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Dec 29 17:04:45 t kernel: [228219.792730] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Dec 29 17:04:45 t kernel: [228219.792730] Process nfsd (pid: 3108, threadinfo ffff88012588a000, task ffff880125e66150) Dec 29 17:04:45 t kernel: [228219.792730] Stack: Dec 29 17:04:45 t kernel: [228219.792730] ffff88012588bd90 ffff88010072d400 ffff88012588bd50 ffffffff811f4be2 Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88010072d400 ffff88010072d410 ffff88012588bd70 ffffffffa027c1d2 Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88012588bd70 ffff88010072d400 ffff88012588bda0 ffffffffa0286b03 Dec 29 17:04:45 t kernel: [228219.792730] Call Trace: Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff811f4be2>] sock_release+0x5c/0x6c Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc] Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc] Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142230>] ? kref_get+0x23/0x2b Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc] Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142201>] kref_put+0x43/0x4f Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc] Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc] Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81035177>] ? default_wake_function+0x0/0xf Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104af36>] kthread+0x7d/0x85 Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c97a>] child_rip+0xa/0x20 Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c970>] ? child_rip+0x0/0x20 Dec 29 17:04:45 t kernel: [228219.792730] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85 Dec 29 17:04:45 t kernel: [228219.792730] RIP [<ffffffff810bccc2>] iput+0x1b/0x65 Dec 29 17:04:45 t kernel: [228219.792730] RSP <ffff88012588bd20> Dec 29 17:04:45 t kernel: [228221.632694] ---[ end trace 1a77408724500faf ]--- Dec 29 20:53:01 t kernel: [241917.056374] ------------[ cut here ]------------ Dec 29 20:53:01 t kernel: [241917.084581] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() Dec 29 20:53:01 t kernel: [241917.117941] Hardware name: X7DB8 Dec 29 20:53:01 t kernel: [241917.137804] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] Dec 29 20:53:01 t kernel: [241917.400101] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 Dec 29 20:53:01 t kernel: [241917.437653] Call Trace: Dec 29 20:53:01 t kernel: [241917.452843] [<ffffffffa0286b0b>] ? svc_xprt_free+0x46/0x4e [sunrpc] Dec 29 20:53:01 t kernel: [241917.491420] [<ffffffff81142230>] ? kref_get+0x23/0x2b Dec 29 20:53:01 t kernel: [241917.522728] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 Dec 29 20:53:01 t kernel: [241917.559230] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 Dec 29 20:53:01 t kernel: [241917.594177] [<ffffffff81142230>] kref_get+0x23/0x2b Dec 29 20:53:01 t kernel: [241917.624446] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] Dec 29 20:53:01 t kernel: [241917.660432] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] Dec 29 20:53:01 t kernel: [241917.699013] [<ffffffff81035177>] ? default_wake_function+0x0/0xf Dec 29 20:53:01 t kernel: [241917.736041] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] Dec 29 20:53:01 t kernel: [241917.768905] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] Dec 29 20:53:01 t kernel: [241917.801249] [<ffffffff8104af36>] kthread+0x7d/0x85 Dec 29 20:53:01 t kernel: [241917.830993] [<ffffffff8100c97a>] child_rip+0xa/0x20 Dec 29 20:53:01 t kernel: [241917.861258] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 Dec 29 20:53:01 t kernel: [241917.891522] [<ffffffff8100c970>] ? child_rip+0x0/0x20 Dec 29 20:53:01 t kernel: [241917.922827] ---[ end trace 1a77408724500fb0 ]--- Dec 29 20:53:01 t kernel: [241917.951084] ------------[ cut here ]------------ Dec 29 20:53:03 t kernel: [241917.974874] kernel BUG at fs/inode.c:1343! Dec 29 20:53:03 t kernel: [241917.988010] invalid opcode: 0000 [#2] SMP Dec 29 20:53:03 t kernel: [241917.988010] last sysfs file: /sys/class/scsi_host/host6/stats Dec 29 20:53:03 t kernel: [241917.988010] CPU 1 Dec 29 20:53:03 t kernel: [241917.988010] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] Dec 29 20:53:03 t kernel: [241918.102960] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8 Dec 29 20:53:03 t kernel: [241918.102960] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65 Dec 29 20:53:03 t kernel: [241918.102960] RSP: 0000:ffff8801225b9d20 EFLAGS: 00010246 Dec 29 20:53:03 t kernel: [241918.102960] RAX: 0000000000000000 RBX: ffff880120435148 RCX: ffffffff8139f927 Dec 29 20:53:03 t kernel: [241918.102960] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff880120435148 Dec 29 20:53:03 t kernel: [241918.102960] RBP: ffff8801225b9d30 R08: 0000000000000020 R09: ffff88002820c9d0 Dec 29 20:53:03 t kernel: [241918.102960] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff880120435100 Dec 29 20:53:03 t kernel: [241918.102960] R13: ffffffffa028ef50 R14: ffff8800c1b3b800 R15: ffff88012dd423c0 Dec 29 20:53:03 t kernel: [241918.102960] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000 Dec 29 20:53:03 t kernel: [241918.102960] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Dec 29 20:53:03 t kernel: [241918.102960] CR2: 00007fad86c2e000 CR3: 0000000129ae6000 CR4: 00000000000006e0 Dec 29 20:53:03 t kernel: [241918.102960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Dec 29 20:53:03 t kernel: [241918.102960] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Dec 29 20:53:03 t kernel: [241918.102960] Process nfsd (pid: 3207, threadinfo ffff8801225b8000, task ffff880122416640) Dec 29 20:53:03 t kernel: [241918.102960] Stack: Dec 29 20:53:03 t kernel: [241918.102960] ffff8801225b9d90 ffff8800c1b3b800 ffff8801225b9d50 ffffffff811f4be2 Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8800c1b3b800 ffff8800c1b3b810 ffff8801225b9d70 ffffffffa027c1d2 Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8801225b9d70 ffff8800c1b3b800 ffff8801225b9da0 ffffffffa0286b03 Dec 29 20:53:03 t kernel: [241918.102960] Call Trace: Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff811f4be2>] sock_release+0x5c/0x6c Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc] Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc] Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff81142230>] ? kref_get+0x23/0x2b Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc] Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81142201>] kref_put+0x43/0x4f Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc] Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc] Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81035177>] ? default_wake_function+0x0/0xf Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104af36>] kthread+0x7d/0x85 Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c97a>] child_rip+0xa/0x20 Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c970>] ? child_rip+0x0/0x20 Dec 29 20:53:04 t kernel: [241918.102960] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85 Dec 29 20:53:04 t kernel: [241918.102960] RIP [<ffffffff810bccc2>] iput+0x1b/0x65 Dec 29 20:53:04 t kernel: [241918.102960] RSP <ffff8801225b9d20> Dec 29 20:53:04 t kernel: [241919.822190] ---[ end trace 1a77408724500fb1 ]--- Dec 30 11:22:26 t kernel: [294082.152979] 3w-9xxx: scsi6: AEN: ERROR (0x04:0x0009): Drive timeout detected:port=0. Dec 31 11:24:44 t kernel: [380620.428029] br0: port 2(eth2) entering disabled state Dec 31 11:24:44 t kernel: [380620.458826] br0: port 1(eth1) entering disabled state Dec 31 11:24:44 t kernel: [380620.666738] device eth1 left promiscuous mode Dec 31 11:24:44 t kernel: [380620.693361] br0: port 1(eth1) entering disabled state Dec 31 11:24:44 t kernel: [380620.909921] device eth2 left promiscuous mode Dec 31 11:24:44 t kernel: [380620.936603] br0: port 2(eth2) entering disabled state Dec 31 11:24:48 t kernel: [380624.449637] device eth1 entered promiscuous mode Dec 31 11:24:48 t kernel: [380624.533887] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X Dec 31 11:24:48 t kernel: [380624.620065] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X Dec 31 11:24:48 t kernel: [380624.651581] ADDRCONF(NETDEV_UP): eth1: link is not ready Dec 31 11:24:48 t kernel: [380624.687675] device eth2 entered promiscuous mode Dec 31 11:24:48 t kernel: [380624.860445] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X Dec 31 11:24:48 t kernel: [380624.944084] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X Dec 31 11:24:48 t kernel: [380624.975824] ADDRCONF(NETDEV_UP): eth2: link is not ready Dec 31 11:24:50 t kernel: [380626.984998] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Dec 31 11:24:51 t kernel: [380627.031517] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready Dec 31 11:24:51 t kernel: [380627.066844] br0: port 2(eth2) entering forwarding state Dec 31 11:24:51 t kernel: [380627.251584] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Dec 31 11:24:51 t kernel: [380627.300433] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready Dec 31 11:24:51 t kernel: [380627.335625] br0: port 1(eth1) entering forwarding state Dec 31 11:24:59 t kernel: [380635.148016] br0: no IPv6 routers present Dec 31 11:25:01 t kernel: [380637.504511] eth1: no IPv6 routers present Dec 31 11:25:01 t kernel: [380637.720515] eth2: no IPv6 routers present Dec 31 11:28:30 t kernel: [380846.205549] e1000e: eth1 NIC Link is Down Dec 31 11:28:30 t kernel: [380846.248659] br0: port 1(eth1) entering disabled state Dec 31 11:28:32 t kernel: [380848.693501] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX Dec 31 11:28:32 t kernel: [380848.739167] 0000:06:00.1: eth1: 10/100 speed: disabling TSO Dec 31 11:28:32 t kernel: [380848.776396] br0: port 1(eth1) entering forwarding state Dec 31 11:28:49 t kernel: [380865.803038] e1000e: eth1 NIC Link is Down Dec 31 11:28:49 t kernel: [380865.874086] br0: port 1(eth1) entering disabled state Dec 31 11:28:52 t kernel: [380868.267943] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Dec 31 11:28:52 t kernel: [380868.316772] br0: port 1(eth1) entering forwarding state Dec 31 11:29:18 t kernel: [380894.253543] e1000e: eth1 NIC Link is Down Dec 31 11:29:18 t kernel: [380894.293597] br0: port 1(eth1) entering disabled state Dec 31 11:29:20 t kernel: [380896.688442] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Dec 31 11:29:20 t kernel: [380896.737255] br0: port 1(eth1) entering forwarding state Dec 31 11:29:25 t kernel: [380901.493543] e1000e: eth1 NIC Link is Down Dec 31 11:29:25 t kernel: [380901.534074] br0: port 1(eth1) entering disabled state Dec 31 11:29:28 t kernel: [380904.161008] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX Dec 31 11:29:28 t kernel: [380904.209842] br0: port 1(eth1) entering forwarding state Jan 1 22:40:06 t kernel: [507542.268700] ------------[ cut here ]------------ Jan 1 22:40:08 t kernel: [507542.296925] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() Jan 1 22:40:08 t kernel: [507542.330301] Hardware name: X7DB8 Jan 1 22:40:08 t kernel: [507542.350163] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] Jan 1 22:40:08 t kernel: [507542.614635] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 Jan 1 22:40:08 t kernel: [507542.652191] Call Trace: Jan 1 22:40:08 t kernel: [507542.667375] [<ffffffff81142230>] ? kref_get+0x23/0x2b Jan 1 22:40:08 t kernel: [507542.698681] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 Jan 1 22:40:08 t kernel: [507542.735186] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 Jan 1 22:40:08 t kernel: [507542.770130] [<ffffffff81142230>] kref_get+0x23/0xBj Jan 1 22:40:08 t kernel: [507542.800400] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] Jan 1 22:40:08 t kernel: [507542.833269] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] Jan 1 22:40:08 t kernel: [507542.869251] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] Jan 1 22:40:08 t kernel: [507542.907825] [<ffffffff81035177>] ? default_wake_function+0x0/0xf Jan 1 22:40:08 t kernel: [507542.944853] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] Jan 1 22:40:08 t kernel: [507542.977720] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] Jan 1 22:40:08 t kernel: [507543.010063] [<ffffffff8104af36>] kthread+0x7d/0x85 Jan 1 22:40:08 t kernel: [507543.039808] [<ffffffff8100c97a>] child_rip+0xa/0x20 Jan 1 22:40:08 t kernel: [507543.070071] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 Jan 1 22:40:08 t kernel: [507543.100337] [<ffffffff8100c970>] ? child_rip+0x0/0x20 Jan 1 22:40:08 t kernel: [507543.131640] ---[ end trace 1a77408724500fb2 ]--- Jan 1 22:40:08 t kernel: [507543.160026] general protection fault: 0000 [#3] SMP Jan 1 22:40:08 t kernel: [507543.165025] last sysfs file: /sys/class/net/eth2/brport/hold_timer Jan 1 22:40:08 t kernel: [507543.165025] CPU 3 Jan 1 22:40:08 t kernel: [507543.165025] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] Jan 1 22:40:08 t kernel: [507543.165025] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8 Jan 1 22:40:08 t kernel: [507543.165025] RIP: 0010:[<ffffffffa0286797>] [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc] Jan 1 22:40:08 t kernel: [507543.165025] RSP: 0018:ffff880124ea1e00 EFLAGS: 00010246 Jan 1 22:40:08 t kernel: [507543.165025] RAX: 3a30343a32322031 RBX: 0000000000000000 RCX: 0000000000000000 Jan 1 22:40:08 t kernel: [507543.165025] RDX: 0000000000004c86 RSI: ffff880124ea1df0 RDI: ffff880124e00000 Jan 1 22:40:08 t kernel: [507543.165025] RBP: ffff880124ea1eb0 R08: 0000000000000020 R09: ffff88002830ca70 Jan 1 22:40:08 t kernel: [507543.165025] R10: ffff88002838cf40 R11: 0000000000000000 R12: 0000000000000082 Jan 1 22:40:08 t kernel: [507543.165025] R13: ffff880124e00000 R14: ffff8800c7444000 R15: ffff88012dd423c0 Jan 1 22:40:08 t kernel: [507543.165025] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000 Jan 1 22:40:08 t kernel: [507543.165025] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Jan 1 22:40:08 t kernel: [507543.165025] CR2: 00007f0dc9cc5098 CR3: 000000012c266000 CR4: 00000000000006e0 Jan 1 22:40:08 t kernel: [507543.165025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 1 22:40:08 t kernel: [507543.165025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 1 22:40:08 t kernel: [507543.165025] Process nfsd (pid: 3134, threadinfo ffff880124ea0000, task ffff880124e9f890) Jan 1 22:40:08 t kernel: [507543.165025] Stack: Jan 1 22:40:08 t kernel: [507543.165025] ffff880124e00000 ffff880124e9f890 ffff880124ea1e40 00000000000dbba0 Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff88012dd423c4 ffff88012c0bea40 ffff880000000082 0000000000000001 Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff880124ea1eb0 ffffffffa027abe4 0000000000000000 ffff880124e9f890 Jan 1 22:40:08 t kernel: [507543.165025] Call Trace: Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff81035177>] ? default_wake_function+0x0/0xf Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104af36>] kthread+0x7d/0x85 Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c97a>] child_rip+0xa/0x20 Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c970>] ? child_rip+0x0/0x20 Jan 1 22:40:08 t kernel: [507543.165025] Code: 00 49 8d 85 88 01 00 00 49 89 85 a0 09 00 00 8b 82 e8 00 00 00 8d 1c 85 00 00 00 00 2b 9a e0 00 00 00 eb 0c 49 8b 46 08 4c 89 ef <ff> 50 18 89 c3 f6 05 56 ce 00 00 01 74 10 89 de 48 c7 c7 cf b6 Jan 1 22:40:08 t kernel: [507543.165025] RIP [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc] Jan 1 22:40:08 t kernel: [507543.165025] RSP <ffff880124ea1e00> Jan 1 22:40:08 t kernel: [507544.713111] ---[ end trace 1a77408724500fb3 ]--- Jan 2 12:06:02 t kernel: [555898.365986] RPC: multiple fragments per record not supported Jan 2 12:06:02 t kernel: [555898.527050] RPC: multiple fragments per record not supported Jan 2 12:06:05 t kernel: [555901.700485] RPC: multiple fragments per record not supported Jan 2 12:06:08 t kernel: [555904.880825] RPC: multiple fragments per record not supported Jan 3 00:01:08 t kernel: [598804.533706] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0. Jan 3 00:57:01 t kernel: [602157.347383] md: data-check of RAID array md0 Jan 3 00:57:01 t kernel: [602157.350802] md: delaying data-check of md2 until md0 has finished (they share one or more physical units) Jan 3 00:57:01 t kernel: [602157.431453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. Jan 3 00:57:01 t kernel: [602157.467054] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. Jan 3 00:57:01 t kernel: [602157.526113] md: using 128k window, over a total of 19542976 blocks. Jan 3 01:03:47 t kernel: [602563.525129] md: md0: data-check done. Jan 3 01:03:47 t kernel: [602563.554886] md: data-check of RAID array md2 Jan 3 01:03:47 t kernel: [602563.581194] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. Jan 3 01:03:47 t kernel: [602563.616879] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. Jan 3 01:03:47 t kernel: [602563.675944] md: using 128k window, over a total of 460928832 blocks. Jan 3 04:47:09 t kernel: [615965.563359] md: md2: data-check done. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() 2010-01-27 23:01 ` Harry Edmon @ 2010-02-02 16:22 ` J. Bruce Fields 2010-02-16 18:22 ` Nikola Ciprich 0 siblings, 1 reply; 12+ messages in thread From: J. Bruce Fields @ 2010-02-02 16:22 UTC (permalink / raw) To: Harry Edmon; +Cc: Andrew Morton, linux-kernel, linux-nfs On Wed, Jan 27, 2010 at 03:01:40PM -0800, Harry Edmon wrote: > J. Bruce Fields wrote: >> On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote: >> >>> (cc linux-nfs) >>> >>> On Thu, 14 Jan 2010 09:53:04 -0800 >>> Harry Edmon <harry@atmos.washington.edu> wrote: >>> >>> >>>> I have a single quad-core Xeon machine running 2.6.32.2 that keeps >>>> giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This >>>> machine is mainaly used as a NFS server. I have attached the >>>> kern.log and my config file. Any ideas? >>>> >> >> Are you using NFSv4, or only v2/v3? Looking at the trace: >> >> Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------ >> Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() >> > NFSv4 and NFSv3. Thanks. And you aren't using rdma are you? >> OK, so that's the warning that we're trying to get something with >> reference count 0. >> >> Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8 >> Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] >> Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1 >> >> What does the very first oops, bug, or warning you get look like? (If >> I'm reading the tain information right, this isn't the first one.) >> > Attached is the kern.log that contains the first warning. Thanks. So yes, the first warning is a stray kref_put() in svc_recv(), and the such put I see on a quick skim is a svc_xprt_put(). The following BUG() is confirmation, since it's an attempt inside scv-xprt_release() to clear an inode with I_CLEAR already set; so we dropped the reference count to zero, cleared it, then tried to do another get/put pair, resulting in an attempt to free the same socket twice. --b. > > -- > Dr. Harry Edmon E-MAIL: harry@atmos.washington.edu > 206-543-0547 FAX: 206-543-0308 harry@washington.edu, harry@uw.edu > Director of IT, College of the Environment and > Director of Computing, Dept of Atmospheric Sciences > University of Washington, Box 351640, Seattle, WA 98195-1640 > > Dec 27 11:57:44 t kernel: [37000.838904] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0055): Battery charging started:. > Dec 27 11:57:46 t kernel: [37002.028946] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0056): Battery charging completed:. > Dec 28 00:00:51 t kernel: [80387.429120] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0. > Dec 28 03:32:07 t kernel: [93063.950486] 3w-9xxx: scsi6: AEN: INFO (0x04:0x002B): Verify completed:unit=0. > Dec 29 17:04:43 t kernel: [228218.883443] ------------[ cut here ]------------ > Dec 29 17:04:45 t kernel: [228218.911691] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > Dec 29 17:04:45 t kernel: [228218.945083] Hardware name: X7DB8 > Dec 29 17:04:45 t kernel: [228218.964948] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > Dec 29 17:04:45 t kernel: [228219.227296] Pid: 3108, comm: nfsd Not tainted 2.6.32.2 #1 > Dec 29 17:04:45 t kernel: [228219.260164] Call Trace: > Dec 29 17:04:45 t kernel: [228219.275350] [<ffffffff81142230>] ? kref_get+0x23/0x2b > Dec 29 17:04:45 t kernel: [228219.306656] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > Dec 29 17:04:45 t kernel: [228219.343160] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > Dec 29 17:04:45 t kernel: [228219.378105] [<ffffffff81142230>] kref_get+0x23/0x2b > Dec 29 17:04:45 t kernel: [228219.408379] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > Dec 29 17:04:45 t kernel: [228219.441242] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > Dec 29 17:04:45 t kernel: [228219.477225] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > Dec 29 17:04:45 t kernel: [228219.515806] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > Dec 29 17:04:45 t kernel: [228219.552834] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > Dec 29 17:04:45 t kernel: [228219.585698] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > Dec 29 17:04:45 t kernel: [228219.618043] [<ffffffff8104af36>] kthread+0x7d/0x85 > Dec 29 17:04:45 t kernel: [228219.647789] [<ffffffff8100c97a>] child_rip+0xa/0x20 > Dec 29 17:04:45 t kernel: [228219.678052] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > Dec 29 17:04:45 t kernel: [228219.708317] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > Dec 29 17:04:45 t kernel: [228219.739621] ---[ end trace 1a77408724500fae ]--- > Dec 29 17:04:45 t kernel: [228219.767925] ------------[ cut here ]------------ > Dec 29 17:04:45 t kernel: [228219.792730] kernel BUG at fs/inode.c:1343! > Dec 29 17:04:45 t kernel: [228219.792730] invalid opcode: 0000 [#1] SMP > Dec 29 17:04:45 t kernel: [228219.792730] last sysfs file: /sys/class/scsi_host/host6/stats > Dec 29 17:04:45 t kernel: [228219.792730] CPU 0 > Dec 29 17:04:45 t kernel: [228219.792730] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > Dec 29 17:04:45 t kernel: [228219.792730] Pid: 3108, comm: nfsd Tainted: G W 2.6.32.2 #1 X7DB8 > Dec 29 17:04:45 t kernel: [228219.792730] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65 > Dec 29 17:04:45 t kernel: [228219.792730] RSP: 0018:ffff88012588bd20 EFLAGS: 00010246 > Dec 29 17:04:45 t kernel: [228219.792730] RAX: 0000000000000000 RBX: ffff88003c3c2cc8 RCX: ffffffff8139f927 > Dec 29 17:04:45 t kernel: [228219.792730] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88003c3c2cc8 > Dec 29 17:04:45 t kernel: [228219.792730] RBP: ffff88012588bd30 R08: 0000000000000002 R09: 0000000000000000 > Dec 29 17:04:45 t kernel: [228219.792730] R10: ffffffff81142230 R11: 0000000100000001 R12: ffff88003c3c2c80 > Dec 29 17:04:45 t kernel: [228219.792730] R13: ffffffffa028ef50 R14: ffff88010072d400 R15: ffff88012dd423c0 > Dec 29 17:04:45 t kernel: [228219.792730] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 > Dec 29 17:04:45 t kernel: [228219.792730] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > Dec 29 17:04:45 t kernel: [228219.792730] CR2: 00007fad86c2e000 CR3: 00000000305e5000 CR4: 00000000000006f0 > Dec 29 17:04:45 t kernel: [228219.792730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > Dec 29 17:04:45 t kernel: [228219.792730] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Dec 29 17:04:45 t kernel: [228219.792730] Process nfsd (pid: 3108, threadinfo ffff88012588a000, task ffff880125e66150) > Dec 29 17:04:45 t kernel: [228219.792730] Stack: > Dec 29 17:04:45 t kernel: [228219.792730] ffff88012588bd90 ffff88010072d400 ffff88012588bd50 ffffffff811f4be2 > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88010072d400 ffff88010072d410 ffff88012588bd70 ffffffffa027c1d2 > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88012588bd70 ffff88010072d400 ffff88012588bda0 ffffffffa0286b03 > Dec 29 17:04:45 t kernel: [228219.792730] Call Trace: > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff811f4be2>] sock_release+0x5c/0x6c > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc] > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc] > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142230>] ? kref_get+0x23/0x2b > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc] > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142201>] kref_put+0x43/0x4f > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc] > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc] > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104af36>] kthread+0x7d/0x85 > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c97a>] child_rip+0xa/0x20 > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > Dec 29 17:04:45 t kernel: [228219.792730] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85 > Dec 29 17:04:45 t kernel: [228219.792730] RIP [<ffffffff810bccc2>] iput+0x1b/0x65 > Dec 29 17:04:45 t kernel: [228219.792730] RSP <ffff88012588bd20> > Dec 29 17:04:45 t kernel: [228221.632694] ---[ end trace 1a77408724500faf ]--- > Dec 29 20:53:01 t kernel: [241917.056374] ------------[ cut here ]------------ > Dec 29 20:53:01 t kernel: [241917.084581] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > Dec 29 20:53:01 t kernel: [241917.117941] Hardware name: X7DB8 > Dec 29 20:53:01 t kernel: [241917.137804] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > Dec 29 20:53:01 t kernel: [241917.400101] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 > Dec 29 20:53:01 t kernel: [241917.437653] Call Trace: > Dec 29 20:53:01 t kernel: [241917.452843] [<ffffffffa0286b0b>] ? svc_xprt_free+0x46/0x4e [sunrpc] > Dec 29 20:53:01 t kernel: [241917.491420] [<ffffffff81142230>] ? kref_get+0x23/0x2b > Dec 29 20:53:01 t kernel: [241917.522728] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > Dec 29 20:53:01 t kernel: [241917.559230] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > Dec 29 20:53:01 t kernel: [241917.594177] [<ffffffff81142230>] kref_get+0x23/0x2b > Dec 29 20:53:01 t kernel: [241917.624446] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > Dec 29 20:53:01 t kernel: [241917.660432] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > Dec 29 20:53:01 t kernel: [241917.699013] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > Dec 29 20:53:01 t kernel: [241917.736041] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > Dec 29 20:53:01 t kernel: [241917.768905] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > Dec 29 20:53:01 t kernel: [241917.801249] [<ffffffff8104af36>] kthread+0x7d/0x85 > Dec 29 20:53:01 t kernel: [241917.830993] [<ffffffff8100c97a>] child_rip+0xa/0x20 > Dec 29 20:53:01 t kernel: [241917.861258] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > Dec 29 20:53:01 t kernel: [241917.891522] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > Dec 29 20:53:01 t kernel: [241917.922827] ---[ end trace 1a77408724500fb0 ]--- > Dec 29 20:53:01 t kernel: [241917.951084] ------------[ cut here ]------------ > Dec 29 20:53:03 t kernel: [241917.974874] kernel BUG at fs/inode.c:1343! > Dec 29 20:53:03 t kernel: [241917.988010] invalid opcode: 0000 [#2] SMP > Dec 29 20:53:03 t kernel: [241917.988010] last sysfs file: /sys/class/scsi_host/host6/stats > Dec 29 20:53:03 t kernel: [241917.988010] CPU 1 > Dec 29 20:53:03 t kernel: [241917.988010] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > Dec 29 20:53:03 t kernel: [241918.102960] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8 > Dec 29 20:53:03 t kernel: [241918.102960] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65 > Dec 29 20:53:03 t kernel: [241918.102960] RSP: 0000:ffff8801225b9d20 EFLAGS: 00010246 > Dec 29 20:53:03 t kernel: [241918.102960] RAX: 0000000000000000 RBX: ffff880120435148 RCX: ffffffff8139f927 > Dec 29 20:53:03 t kernel: [241918.102960] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff880120435148 > Dec 29 20:53:03 t kernel: [241918.102960] RBP: ffff8801225b9d30 R08: 0000000000000020 R09: ffff88002820c9d0 > Dec 29 20:53:03 t kernel: [241918.102960] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff880120435100 > Dec 29 20:53:03 t kernel: [241918.102960] R13: ffffffffa028ef50 R14: ffff8800c1b3b800 R15: ffff88012dd423c0 > Dec 29 20:53:03 t kernel: [241918.102960] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000 > Dec 29 20:53:03 t kernel: [241918.102960] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > Dec 29 20:53:03 t kernel: [241918.102960] CR2: 00007fad86c2e000 CR3: 0000000129ae6000 CR4: 00000000000006e0 > Dec 29 20:53:03 t kernel: [241918.102960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > Dec 29 20:53:03 t kernel: [241918.102960] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Dec 29 20:53:03 t kernel: [241918.102960] Process nfsd (pid: 3207, threadinfo ffff8801225b8000, task ffff880122416640) > Dec 29 20:53:03 t kernel: [241918.102960] Stack: > Dec 29 20:53:03 t kernel: [241918.102960] ffff8801225b9d90 ffff8800c1b3b800 ffff8801225b9d50 ffffffff811f4be2 > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8800c1b3b800 ffff8800c1b3b810 ffff8801225b9d70 ffffffffa027c1d2 > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8801225b9d70 ffff8800c1b3b800 ffff8801225b9da0 ffffffffa0286b03 > Dec 29 20:53:03 t kernel: [241918.102960] Call Trace: > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff811f4be2>] sock_release+0x5c/0x6c > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc] > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc] > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff81142230>] ? kref_get+0x23/0x2b > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc] > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81142201>] kref_put+0x43/0x4f > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc] > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc] > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104af36>] kthread+0x7d/0x85 > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c97a>] child_rip+0xa/0x20 > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > Dec 29 20:53:04 t kernel: [241918.102960] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85 > Dec 29 20:53:04 t kernel: [241918.102960] RIP [<ffffffff810bccc2>] iput+0x1b/0x65 > Dec 29 20:53:04 t kernel: [241918.102960] RSP <ffff8801225b9d20> > Dec 29 20:53:04 t kernel: [241919.822190] ---[ end trace 1a77408724500fb1 ]--- > Dec 30 11:22:26 t kernel: [294082.152979] 3w-9xxx: scsi6: AEN: ERROR (0x04:0x0009): Drive timeout detected:port=0. > Dec 31 11:24:44 t kernel: [380620.428029] br0: port 2(eth2) entering disabled state > Dec 31 11:24:44 t kernel: [380620.458826] br0: port 1(eth1) entering disabled state > Dec 31 11:24:44 t kernel: [380620.666738] device eth1 left promiscuous mode > Dec 31 11:24:44 t kernel: [380620.693361] br0: port 1(eth1) entering disabled state > Dec 31 11:24:44 t kernel: [380620.909921] device eth2 left promiscuous mode > Dec 31 11:24:44 t kernel: [380620.936603] br0: port 2(eth2) entering disabled state > Dec 31 11:24:48 t kernel: [380624.449637] device eth1 entered promiscuous mode > Dec 31 11:24:48 t kernel: [380624.533887] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X > Dec 31 11:24:48 t kernel: [380624.620065] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X > Dec 31 11:24:48 t kernel: [380624.651581] ADDRCONF(NETDEV_UP): eth1: link is not ready > Dec 31 11:24:48 t kernel: [380624.687675] device eth2 entered promiscuous mode > Dec 31 11:24:48 t kernel: [380624.860445] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X > Dec 31 11:24:48 t kernel: [380624.944084] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X > Dec 31 11:24:48 t kernel: [380624.975824] ADDRCONF(NETDEV_UP): eth2: link is not ready > Dec 31 11:24:50 t kernel: [380626.984998] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > Dec 31 11:24:51 t kernel: [380627.031517] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready > Dec 31 11:24:51 t kernel: [380627.066844] br0: port 2(eth2) entering forwarding state > Dec 31 11:24:51 t kernel: [380627.251584] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > Dec 31 11:24:51 t kernel: [380627.300433] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready > Dec 31 11:24:51 t kernel: [380627.335625] br0: port 1(eth1) entering forwarding state > Dec 31 11:24:59 t kernel: [380635.148016] br0: no IPv6 routers present > Dec 31 11:25:01 t kernel: [380637.504511] eth1: no IPv6 routers present > Dec 31 11:25:01 t kernel: [380637.720515] eth2: no IPv6 routers present > Dec 31 11:28:30 t kernel: [380846.205549] e1000e: eth1 NIC Link is Down > Dec 31 11:28:30 t kernel: [380846.248659] br0: port 1(eth1) entering disabled state > Dec 31 11:28:32 t kernel: [380848.693501] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX > Dec 31 11:28:32 t kernel: [380848.739167] 0000:06:00.1: eth1: 10/100 speed: disabling TSO > Dec 31 11:28:32 t kernel: [380848.776396] br0: port 1(eth1) entering forwarding state > Dec 31 11:28:49 t kernel: [380865.803038] e1000e: eth1 NIC Link is Down > Dec 31 11:28:49 t kernel: [380865.874086] br0: port 1(eth1) entering disabled state > Dec 31 11:28:52 t kernel: [380868.267943] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > Dec 31 11:28:52 t kernel: [380868.316772] br0: port 1(eth1) entering forwarding state > Dec 31 11:29:18 t kernel: [380894.253543] e1000e: eth1 NIC Link is Down > Dec 31 11:29:18 t kernel: [380894.293597] br0: port 1(eth1) entering disabled state > Dec 31 11:29:20 t kernel: [380896.688442] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > Dec 31 11:29:20 t kernel: [380896.737255] br0: port 1(eth1) entering forwarding state > Dec 31 11:29:25 t kernel: [380901.493543] e1000e: eth1 NIC Link is Down > Dec 31 11:29:25 t kernel: [380901.534074] br0: port 1(eth1) entering disabled state > Dec 31 11:29:28 t kernel: [380904.161008] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > Dec 31 11:29:28 t kernel: [380904.209842] br0: port 1(eth1) entering forwarding state > Jan 1 22:40:06 t kernel: [507542.268700] ------------[ cut here ]------------ > Jan 1 22:40:08 t kernel: [507542.296925] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > Jan 1 22:40:08 t kernel: [507542.330301] Hardware name: X7DB8 > Jan 1 22:40:08 t kernel: [507542.350163] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > Jan 1 22:40:08 t kernel: [507542.614635] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 > Jan 1 22:40:08 t kernel: [507542.652191] Call Trace: > Jan 1 22:40:08 t kernel: [507542.667375] [<ffffffff81142230>] ? kref_get+0x23/0x2b > Jan 1 22:40:08 t kernel: [507542.698681] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > Jan 1 22:40:08 t kernel: [507542.735186] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > Jan 1 22:40:08 t kernel: [507542.770130] [<ffffffff81142230>] kref_get+0x23/0xBj > Jan 1 22:40:08 t kernel: [507542.800400] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > Jan 1 22:40:08 t kernel: [507542.833269] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > Jan 1 22:40:08 t kernel: [507542.869251] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > Jan 1 22:40:08 t kernel: [507542.907825] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > Jan 1 22:40:08 t kernel: [507542.944853] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > Jan 1 22:40:08 t kernel: [507542.977720] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > Jan 1 22:40:08 t kernel: [507543.010063] [<ffffffff8104af36>] kthread+0x7d/0x85 > Jan 1 22:40:08 t kernel: [507543.039808] [<ffffffff8100c97a>] child_rip+0xa/0x20 > Jan 1 22:40:08 t kernel: [507543.070071] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > Jan 1 22:40:08 t kernel: [507543.100337] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > Jan 1 22:40:08 t kernel: [507543.131640] ---[ end trace 1a77408724500fb2 ]--- > Jan 1 22:40:08 t kernel: [507543.160026] general protection fault: 0000 [#3] SMP > Jan 1 22:40:08 t kernel: [507543.165025] last sysfs file: /sys/class/net/eth2/brport/hold_timer > Jan 1 22:40:08 t kernel: [507543.165025] CPU 3 > Jan 1 22:40:08 t kernel: [507543.165025] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > Jan 1 22:40:08 t kernel: [507543.165025] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8 > Jan 1 22:40:08 t kernel: [507543.165025] RIP: 0010:[<ffffffffa0286797>] [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc] > Jan 1 22:40:08 t kernel: [507543.165025] RSP: 0018:ffff880124ea1e00 EFLAGS: 00010246 > Jan 1 22:40:08 t kernel: [507543.165025] RAX: 3a30343a32322031 RBX: 0000000000000000 RCX: 0000000000000000 > Jan 1 22:40:08 t kernel: [507543.165025] RDX: 0000000000004c86 RSI: ffff880124ea1df0 RDI: ffff880124e00000 > Jan 1 22:40:08 t kernel: [507543.165025] RBP: ffff880124ea1eb0 R08: 0000000000000020 R09: ffff88002830ca70 > Jan 1 22:40:08 t kernel: [507543.165025] R10: ffff88002838cf40 R11: 0000000000000000 R12: 0000000000000082 > Jan 1 22:40:08 t kernel: [507543.165025] R13: ffff880124e00000 R14: ffff8800c7444000 R15: ffff88012dd423c0 > Jan 1 22:40:08 t kernel: [507543.165025] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000 > Jan 1 22:40:08 t kernel: [507543.165025] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > Jan 1 22:40:08 t kernel: [507543.165025] CR2: 00007f0dc9cc5098 CR3: 000000012c266000 CR4: 00000000000006e0 > Jan 1 22:40:08 t kernel: [507543.165025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > Jan 1 22:40:08 t kernel: [507543.165025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Jan 1 22:40:08 t kernel: [507543.165025] Process nfsd (pid: 3134, threadinfo ffff880124ea0000, task ffff880124e9f890) > Jan 1 22:40:08 t kernel: [507543.165025] Stack: > Jan 1 22:40:08 t kernel: [507543.165025] ffff880124e00000 ffff880124e9f890 ffff880124ea1e40 00000000000dbba0 > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff88012dd423c4 ffff88012c0bea40 ffff880000000082 0000000000000001 > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff880124ea1eb0 ffffffffa027abe4 0000000000000000 ffff880124e9f890 > Jan 1 22:40:08 t kernel: [507543.165025] Call Trace: > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104af36>] kthread+0x7d/0x85 > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c97a>] child_rip+0xa/0x20 > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > Jan 1 22:40:08 t kernel: [507543.165025] Code: 00 49 8d 85 88 01 00 00 49 89 85 a0 09 00 00 8b 82 e8 00 00 00 8d 1c 85 00 00 00 00 2b 9a e0 00 00 00 eb 0c 49 8b 46 08 4c 89 ef <ff> 50 18 89 c3 f6 05 56 ce 00 00 01 74 10 89 de 48 c7 c7 cf b6 > Jan 1 22:40:08 t kernel: [507543.165025] RIP [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc] > Jan 1 22:40:08 t kernel: [507543.165025] RSP <ffff880124ea1e00> > Jan 1 22:40:08 t kernel: [507544.713111] ---[ end trace 1a77408724500fb3 ]--- > Jan 2 12:06:02 t kernel: [555898.365986] RPC: multiple fragments per record not supported > Jan 2 12:06:02 t kernel: [555898.527050] RPC: multiple fragments per record not supported > Jan 2 12:06:05 t kernel: [555901.700485] RPC: multiple fragments per record not supported > Jan 2 12:06:08 t kernel: [555904.880825] RPC: multiple fragments per record not supported > Jan 3 00:01:08 t kernel: [598804.533706] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0. > Jan 3 00:57:01 t kernel: [602157.347383] md: data-check of RAID array md0 > Jan 3 00:57:01 t kernel: [602157.350802] md: delaying data-check of md2 until md0 has finished (they share one or more physical units) > Jan 3 00:57:01 t kernel: [602157.431453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. > Jan 3 00:57:01 t kernel: [602157.467054] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. > Jan 3 00:57:01 t kernel: [602157.526113] md: using 128k window, over a total of 19542976 blocks. > Jan 3 01:03:47 t kernel: [602563.525129] md: md0: data-check done. > Jan 3 01:03:47 t kernel: [602563.554886] md: data-check of RAID array md2 > Jan 3 01:03:47 t kernel: [602563.581194] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. > Jan 3 01:03:47 t kernel: [602563.616879] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. > Jan 3 01:03:47 t kernel: [602563.675944] md: using 128k window, over a total of 460928832 blocks. > Jan 3 04:47:09 t kernel: [615965.563359] md: md2: data-check done. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() 2010-02-02 16:22 ` J. Bruce Fields @ 2010-02-16 18:22 ` Nikola Ciprich [not found] ` <20100216182248.GA4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org> 0 siblings, 1 reply; 12+ messages in thread From: Nikola Ciprich @ 2010-02-16 18:22 UTC (permalink / raw) To: J. Bruce Fields Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs, nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g Hello, is there any conclusion regarding this problem? I'm seeing it too on one of my machines, running 2.6.32.8 thanks a lot in advance for any informamation.. regards nik On Tue, Feb 02, 2010 at 11:22:28AM -0500, J. Bruce Fields wrote: > On Wed, Jan 27, 2010 at 03:01:40PM -0800, Harry Edmon wrote: > > J. Bruce Fields wrote: > >> On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote: > >> > >>> (cc linux-nfs) > >>> > >>> On Thu, 14 Jan 2010 09:53:04 -0800 > >>> Harry Edmon <harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org> wrote: > >>> > >>> > >>>> I have a single quad-core Xeon machine running 2.6.32.2 that keeps > >>>> giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This > >>>> machine is mainaly used as a NFS server. I have attached the > >>>> kern.log and my config file. Any ideas? > >>>> > >> > >> Are you using NFSv4, or only v2/v3? Looking at the trace: > >> > >> Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------ > >> Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > >> > > NFSv4 and NFSv3. > > Thanks. And you aren't using rdma are you? > > >> OK, so that's the warning that we're trying to get something with > >> reference count 0. > >> > >> Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8 > >> Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > >> Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1 > >> > >> What does the very first oops, bug, or warning you get look like? (If > >> I'm reading the tain information right, this isn't the first one.) > >> > > Attached is the kern.log that contains the first warning. > > Thanks. So yes, the first warning is a stray kref_put() in svc_recv(), > and the such put I see on a quick skim is a svc_xprt_put(). The > following BUG() is confirmation, since it's an attempt inside > scv-xprt_release() to clear an inode with I_CLEAR already set; so we > dropped the reference count to zero, cleared it, then tried to do > another get/put pair, resulting in an attempt to free the same socket > twice. > > --b. > > > > > -- > > Dr. Harry Edmon E-MAIL: harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org > > 206-543-0547 FAX: 206-543-0308 harry-B93hV6UPU7Z2icitjWtXSw@public.gmane.org, harry@uw.edu > > Director of IT, College of the Environment and > > Director of Computing, Dept of Atmospheric Sciences > > University of Washington, Box 351640, Seattle, WA 98195-1640 > > > > > Dec 27 11:57:44 t kernel: [37000.838904] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0055): Battery charging started:. > > Dec 27 11:57:46 t kernel: [37002.028946] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0056): Battery charging completed:. > > Dec 28 00:00:51 t kernel: [80387.429120] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0. > > Dec 28 03:32:07 t kernel: [93063.950486] 3w-9xxx: scsi6: AEN: INFO (0x04:0x002B): Verify completed:unit=0. > > Dec 29 17:04:43 t kernel: [228218.883443] ------------[ cut here ]------------ > > Dec 29 17:04:45 t kernel: [228218.911691] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > > Dec 29 17:04:45 t kernel: [228218.945083] Hardware name: X7DB8 > > Dec 29 17:04:45 t kernel: [228218.964948] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > Dec 29 17:04:45 t kernel: [228219.227296] Pid: 3108, comm: nfsd Not tainted 2.6.32.2 #1 > > Dec 29 17:04:45 t kernel: [228219.260164] Call Trace: > > Dec 29 17:04:45 t kernel: [228219.275350] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > Dec 29 17:04:45 t kernel: [228219.306656] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > > Dec 29 17:04:45 t kernel: [228219.343160] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > > Dec 29 17:04:45 t kernel: [228219.378105] [<ffffffff81142230>] kref_get+0x23/0x2b > > Dec 29 17:04:45 t kernel: [228219.408379] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > Dec 29 17:04:45 t kernel: [228219.441242] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > > Dec 29 17:04:45 t kernel: [228219.477225] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > Dec 29 17:04:45 t kernel: [228219.515806] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > Dec 29 17:04:45 t kernel: [228219.552834] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > Dec 29 17:04:45 t kernel: [228219.585698] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > Dec 29 17:04:45 t kernel: [228219.618043] [<ffffffff8104af36>] kthread+0x7d/0x85 > > Dec 29 17:04:45 t kernel: [228219.647789] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > Dec 29 17:04:45 t kernel: [228219.678052] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > Dec 29 17:04:45 t kernel: [228219.708317] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > Dec 29 17:04:45 t kernel: [228219.739621] ---[ end trace 1a77408724500fae ]--- > > Dec 29 17:04:45 t kernel: [228219.767925] ------------[ cut here ]------------ > > Dec 29 17:04:45 t kernel: [228219.792730] kernel BUG at fs/inode.c:1343! > > Dec 29 17:04:45 t kernel: [228219.792730] invalid opcode: 0000 [#1] SMP > > Dec 29 17:04:45 t kernel: [228219.792730] last sysfs file: /sys/class/scsi_host/host6/stats > > Dec 29 17:04:45 t kernel: [228219.792730] CPU 0 > > Dec 29 17:04:45 t kernel: [228219.792730] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > Dec 29 17:04:45 t kernel: [228219.792730] Pid: 3108, comm: nfsd Tainted: G W 2.6.32.2 #1 X7DB8 > > Dec 29 17:04:45 t kernel: [228219.792730] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65 > > Dec 29 17:04:45 t kernel: [228219.792730] RSP: 0018:ffff88012588bd20 EFLAGS: 00010246 > > Dec 29 17:04:45 t kernel: [228219.792730] RAX: 0000000000000000 RBX: ffff88003c3c2cc8 RCX: ffffffff8139f927 > > Dec 29 17:04:45 t kernel: [228219.792730] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88003c3c2cc8 > > Dec 29 17:04:45 t kernel: [228219.792730] RBP: ffff88012588bd30 R08: 0000000000000002 R09: 0000000000000000 > > Dec 29 17:04:45 t kernel: [228219.792730] R10: ffffffff81142230 R11: 0000000100000001 R12: ffff88003c3c2c80 > > Dec 29 17:04:45 t kernel: [228219.792730] R13: ffffffffa028ef50 R14: ffff88010072d400 R15: ffff88012dd423c0 > > Dec 29 17:04:45 t kernel: [228219.792730] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 > > Dec 29 17:04:45 t kernel: [228219.792730] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Dec 29 17:04:45 t kernel: [228219.792730] CR2: 00007fad86c2e000 CR3: 00000000305e5000 CR4: 00000000000006f0 > > Dec 29 17:04:45 t kernel: [228219.792730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > Dec 29 17:04:45 t kernel: [228219.792730] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > Dec 29 17:04:45 t kernel: [228219.792730] Process nfsd (pid: 3108, threadinfo ffff88012588a000, task ffff880125e66150) > > Dec 29 17:04:45 t kernel: [228219.792730] Stack: > > Dec 29 17:04:45 t kernel: [228219.792730] ffff88012588bd90 ffff88010072d400 ffff88012588bd50 ffffffff811f4be2 > > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88010072d400 ffff88010072d410 ffff88012588bd70 ffffffffa027c1d2 > > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88012588bd70 ffff88010072d400 ffff88012588bda0 ffffffffa0286b03 > > Dec 29 17:04:45 t kernel: [228219.792730] Call Trace: > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff811f4be2>] sock_release+0x5c/0x6c > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc] > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc] > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc] > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142201>] kref_put+0x43/0x4f > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc] > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc] > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104af36>] kthread+0x7d/0x85 > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > Dec 29 17:04:45 t kernel: [228219.792730] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85 > > Dec 29 17:04:45 t kernel: [228219.792730] RIP [<ffffffff810bccc2>] iput+0x1b/0x65 > > Dec 29 17:04:45 t kernel: [228219.792730] RSP <ffff88012588bd20> > > Dec 29 17:04:45 t kernel: [228221.632694] ---[ end trace 1a77408724500faf ]--- > > Dec 29 20:53:01 t kernel: [241917.056374] ------------[ cut here ]------------ > > Dec 29 20:53:01 t kernel: [241917.084581] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > > Dec 29 20:53:01 t kernel: [241917.117941] Hardware name: X7DB8 > > Dec 29 20:53:01 t kernel: [241917.137804] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > Dec 29 20:53:01 t kernel: [241917.400101] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 > > Dec 29 20:53:01 t kernel: [241917.437653] Call Trace: > > Dec 29 20:53:01 t kernel: [241917.452843] [<ffffffffa0286b0b>] ? svc_xprt_free+0x46/0x4e [sunrpc] > > Dec 29 20:53:01 t kernel: [241917.491420] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > Dec 29 20:53:01 t kernel: [241917.522728] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > > Dec 29 20:53:01 t kernel: [241917.559230] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > > Dec 29 20:53:01 t kernel: [241917.594177] [<ffffffff81142230>] kref_get+0x23/0x2b > > Dec 29 20:53:01 t kernel: [241917.624446] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > > Dec 29 20:53:01 t kernel: [241917.660432] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > Dec 29 20:53:01 t kernel: [241917.699013] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > Dec 29 20:53:01 t kernel: [241917.736041] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > Dec 29 20:53:01 t kernel: [241917.768905] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > Dec 29 20:53:01 t kernel: [241917.801249] [<ffffffff8104af36>] kthread+0x7d/0x85 > > Dec 29 20:53:01 t kernel: [241917.830993] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > Dec 29 20:53:01 t kernel: [241917.861258] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > Dec 29 20:53:01 t kernel: [241917.891522] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > Dec 29 20:53:01 t kernel: [241917.922827] ---[ end trace 1a77408724500fb0 ]--- > > Dec 29 20:53:01 t kernel: [241917.951084] ------------[ cut here ]------------ > > Dec 29 20:53:03 t kernel: [241917.974874] kernel BUG at fs/inode.c:1343! > > Dec 29 20:53:03 t kernel: [241917.988010] invalid opcode: 0000 [#2] SMP > > Dec 29 20:53:03 t kernel: [241917.988010] last sysfs file: /sys/class/scsi_host/host6/stats > > Dec 29 20:53:03 t kernel: [241917.988010] CPU 1 > > Dec 29 20:53:03 t kernel: [241917.988010] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > Dec 29 20:53:03 t kernel: [241918.102960] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8 > > Dec 29 20:53:03 t kernel: [241918.102960] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65 > > Dec 29 20:53:03 t kernel: [241918.102960] RSP: 0000:ffff8801225b9d20 EFLAGS: 00010246 > > Dec 29 20:53:03 t kernel: [241918.102960] RAX: 0000000000000000 RBX: ffff880120435148 RCX: ffffffff8139f927 > > Dec 29 20:53:03 t kernel: [241918.102960] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff880120435148 > > Dec 29 20:53:03 t kernel: [241918.102960] RBP: ffff8801225b9d30 R08: 0000000000000020 R09: ffff88002820c9d0 > > Dec 29 20:53:03 t kernel: [241918.102960] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff880120435100 > > Dec 29 20:53:03 t kernel: [241918.102960] R13: ffffffffa028ef50 R14: ffff8800c1b3b800 R15: ffff88012dd423c0 > > Dec 29 20:53:03 t kernel: [241918.102960] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000 > > Dec 29 20:53:03 t kernel: [241918.102960] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Dec 29 20:53:03 t kernel: [241918.102960] CR2: 00007fad86c2e000 CR3: 0000000129ae6000 CR4: 00000000000006e0 > > Dec 29 20:53:03 t kernel: [241918.102960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > Dec 29 20:53:03 t kernel: [241918.102960] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > Dec 29 20:53:03 t kernel: [241918.102960] Process nfsd (pid: 3207, threadinfo ffff8801225b8000, task ffff880122416640) > > Dec 29 20:53:03 t kernel: [241918.102960] Stack: > > Dec 29 20:53:03 t kernel: [241918.102960] ffff8801225b9d90 ffff8800c1b3b800 ffff8801225b9d50 ffffffff811f4be2 > > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8800c1b3b800 ffff8800c1b3b810 ffff8801225b9d70 ffffffffa027c1d2 > > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8801225b9d70 ffff8800c1b3b800 ffff8801225b9da0 ffffffffa0286b03 > > Dec 29 20:53:03 t kernel: [241918.102960] Call Trace: > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff811f4be2>] sock_release+0x5c/0x6c > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc] > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc] > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc] > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81142201>] kref_put+0x43/0x4f > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc] > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc] > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104af36>] kthread+0x7d/0x85 > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > Dec 29 20:53:04 t kernel: [241918.102960] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85 > > Dec 29 20:53:04 t kernel: [241918.102960] RIP [<ffffffff810bccc2>] iput+0x1b/0x65 > > Dec 29 20:53:04 t kernel: [241918.102960] RSP <ffff8801225b9d20> > > Dec 29 20:53:04 t kernel: [241919.822190] ---[ end trace 1a77408724500fb1 ]--- > > Dec 30 11:22:26 t kernel: [294082.152979] 3w-9xxx: scsi6: AEN: ERROR (0x04:0x0009): Drive timeout detected:port=0. > > Dec 31 11:24:44 t kernel: [380620.428029] br0: port 2(eth2) entering disabled state > > Dec 31 11:24:44 t kernel: [380620.458826] br0: port 1(eth1) entering disabled state > > Dec 31 11:24:44 t kernel: [380620.666738] device eth1 left promiscuous mode > > Dec 31 11:24:44 t kernel: [380620.693361] br0: port 1(eth1) entering disabled state > > Dec 31 11:24:44 t kernel: [380620.909921] device eth2 left promiscuous mode > > Dec 31 11:24:44 t kernel: [380620.936603] br0: port 2(eth2) entering disabled state > > Dec 31 11:24:48 t kernel: [380624.449637] device eth1 entered promiscuous mode > > Dec 31 11:24:48 t kernel: [380624.533887] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X > > Dec 31 11:24:48 t kernel: [380624.620065] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X > > Dec 31 11:24:48 t kernel: [380624.651581] ADDRCONF(NETDEV_UP): eth1: link is not ready > > Dec 31 11:24:48 t kernel: [380624.687675] device eth2 entered promiscuous mode > > Dec 31 11:24:48 t kernel: [380624.860445] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X > > Dec 31 11:24:48 t kernel: [380624.944084] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X > > Dec 31 11:24:48 t kernel: [380624.975824] ADDRCONF(NETDEV_UP): eth2: link is not ready > > Dec 31 11:24:50 t kernel: [380626.984998] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > Dec 31 11:24:51 t kernel: [380627.031517] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready > > Dec 31 11:24:51 t kernel: [380627.066844] br0: port 2(eth2) entering forwarding state > > Dec 31 11:24:51 t kernel: [380627.251584] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > Dec 31 11:24:51 t kernel: [380627.300433] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready > > Dec 31 11:24:51 t kernel: [380627.335625] br0: port 1(eth1) entering forwarding state > > Dec 31 11:24:59 t kernel: [380635.148016] br0: no IPv6 routers present > > Dec 31 11:25:01 t kernel: [380637.504511] eth1: no IPv6 routers present > > Dec 31 11:25:01 t kernel: [380637.720515] eth2: no IPv6 routers present > > Dec 31 11:28:30 t kernel: [380846.205549] e1000e: eth1 NIC Link is Down > > Dec 31 11:28:30 t kernel: [380846.248659] br0: port 1(eth1) entering disabled state > > Dec 31 11:28:32 t kernel: [380848.693501] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX > > Dec 31 11:28:32 t kernel: [380848.739167] 0000:06:00.1: eth1: 10/100 speed: disabling TSO > > Dec 31 11:28:32 t kernel: [380848.776396] br0: port 1(eth1) entering forwarding state > > Dec 31 11:28:49 t kernel: [380865.803038] e1000e: eth1 NIC Link is Down > > Dec 31 11:28:49 t kernel: [380865.874086] br0: port 1(eth1) entering disabled state > > Dec 31 11:28:52 t kernel: [380868.267943] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > Dec 31 11:28:52 t kernel: [380868.316772] br0: port 1(eth1) entering forwarding state > > Dec 31 11:29:18 t kernel: [380894.253543] e1000e: eth1 NIC Link is Down > > Dec 31 11:29:18 t kernel: [380894.293597] br0: port 1(eth1) entering disabled state > > Dec 31 11:29:20 t kernel: [380896.688442] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > Dec 31 11:29:20 t kernel: [380896.737255] br0: port 1(eth1) entering forwarding state > > Dec 31 11:29:25 t kernel: [380901.493543] e1000e: eth1 NIC Link is Down > > Dec 31 11:29:25 t kernel: [380901.534074] br0: port 1(eth1) entering disabled state > > Dec 31 11:29:28 t kernel: [380904.161008] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > Dec 31 11:29:28 t kernel: [380904.209842] br0: port 1(eth1) entering forwarding state > > Jan 1 22:40:06 t kernel: [507542.268700] ------------[ cut here ]------------ > > Jan 1 22:40:08 t kernel: [507542.296925] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > > Jan 1 22:40:08 t kernel: [507542.330301] Hardware name: X7DB8 > > Jan 1 22:40:08 t kernel: [507542.350163] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > Jan 1 22:40:08 t kernel: [507542.614635] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 > > Jan 1 22:40:08 t kernel: [507542.652191] Call Trace: > > Jan 1 22:40:08 t kernel: [507542.667375] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > Jan 1 22:40:08 t kernel: [507542.698681] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > > Jan 1 22:40:08 t kernel: [507542.735186] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > > Jan 1 22:40:08 t kernel: [507542.770130] [<ffffffff81142230>] kref_get+0x23/0xBj > > Jan 1 22:40:08 t kernel: [507542.800400] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > Jan 1 22:40:08 t kernel: [507542.833269] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > > Jan 1 22:40:08 t kernel: [507542.869251] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > Jan 1 22:40:08 t kernel: [507542.907825] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > Jan 1 22:40:08 t kernel: [507542.944853] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > Jan 1 22:40:08 t kernel: [507542.977720] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > Jan 1 22:40:08 t kernel: [507543.010063] [<ffffffff8104af36>] kthread+0x7d/0x85 > > Jan 1 22:40:08 t kernel: [507543.039808] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > Jan 1 22:40:08 t kernel: [507543.070071] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > Jan 1 22:40:08 t kernel: [507543.100337] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > Jan 1 22:40:08 t kernel: [507543.131640] ---[ end trace 1a77408724500fb2 ]--- > > Jan 1 22:40:08 t kernel: [507543.160026] general protection fault: 0000 [#3] SMP > > Jan 1 22:40:08 t kernel: [507543.165025] last sysfs file: /sys/class/net/eth2/brport/hold_timer > > Jan 1 22:40:08 t kernel: [507543.165025] CPU 3 > > Jan 1 22:40:08 t kernel: [507543.165025] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > Jan 1 22:40:08 t kernel: [507543.165025] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8 > > Jan 1 22:40:08 t kernel: [507543.165025] RIP: 0010:[<ffffffffa0286797>] [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc] > > Jan 1 22:40:08 t kernel: [507543.165025] RSP: 0018:ffff880124ea1e00 EFLAGS: 00010246 > > Jan 1 22:40:08 t kernel: [507543.165025] RAX: 3a30343a32322031 RBX: 0000000000000000 RCX: 0000000000000000 > > Jan 1 22:40:08 t kernel: [507543.165025] RDX: 0000000000004c86 RSI: ffff880124ea1df0 RDI: ffff880124e00000 > > Jan 1 22:40:08 t kernel: [507543.165025] RBP: ffff880124ea1eb0 R08: 0000000000000020 R09: ffff88002830ca70 > > Jan 1 22:40:08 t kernel: [507543.165025] R10: ffff88002838cf40 R11: 0000000000000000 R12: 0000000000000082 > > Jan 1 22:40:08 t kernel: [507543.165025] R13: ffff880124e00000 R14: ffff8800c7444000 R15: ffff88012dd423c0 > > Jan 1 22:40:08 t kernel: [507543.165025] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000 > > Jan 1 22:40:08 t kernel: [507543.165025] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Jan 1 22:40:08 t kernel: [507543.165025] CR2: 00007f0dc9cc5098 CR3: 000000012c266000 CR4: 00000000000006e0 > > Jan 1 22:40:08 t kernel: [507543.165025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > Jan 1 22:40:08 t kernel: [507543.165025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > Jan 1 22:40:08 t kernel: [507543.165025] Process nfsd (pid: 3134, threadinfo ffff880124ea0000, task ffff880124e9f890) > > Jan 1 22:40:08 t kernel: [507543.165025] Stack: > > Jan 1 22:40:08 t kernel: [507543.165025] ffff880124e00000 ffff880124e9f890 ffff880124ea1e40 00000000000dbba0 > > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff88012dd423c4 ffff88012c0bea40 ffff880000000082 0000000000000001 > > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff880124ea1eb0 ffffffffa027abe4 0000000000000000 ffff880124e9f890 > > Jan 1 22:40:08 t kernel: [507543.165025] Call Trace: > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104af36>] kthread+0x7d/0x85 > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > Jan 1 22:40:08 t kernel: [507543.165025] Code: 00 49 8d 85 88 01 00 00 49 89 85 a0 09 00 00 8b 82 e8 00 00 00 8d 1c 85 00 00 00 00 2b 9a e0 00 00 00 eb 0c 49 8b 46 08 4c 89 ef <ff> 50 18 89 c3 f6 05 56 ce 00 00 01 74 10 89 de 48 c7 c7 cf b6 > > Jan 1 22:40:08 t kernel: [507543.165025] RIP [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc] > > Jan 1 22:40:08 t kernel: [507543.165025] RSP <ffff880124ea1e00> > > Jan 1 22:40:08 t kernel: [507544.713111] ---[ end trace 1a77408724500fb3 ]--- > > Jan 2 12:06:02 t kernel: [555898.365986] RPC: multiple fragments per record not supported > > Jan 2 12:06:02 t kernel: [555898.527050] RPC: multiple fragments per record not supported > > Jan 2 12:06:05 t kernel: [555901.700485] RPC: multiple fragments per record not supported > > Jan 2 12:06:08 t kernel: [555904.880825] RPC: multiple fragments per record not supported > > Jan 3 00:01:08 t kernel: [598804.533706] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0. > > Jan 3 00:57:01 t kernel: [602157.347383] md: data-check of RAID array md0 > > Jan 3 00:57:01 t kernel: [602157.350802] md: delaying data-check of md2 until md0 has finished (they share one or more physical units) > > Jan 3 00:57:01 t kernel: [602157.431453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. > > Jan 3 00:57:01 t kernel: [602157.467054] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. > > Jan 3 00:57:01 t kernel: [602157.526113] md: using 128k window, over a total of 19542976 blocks. > > Jan 3 01:03:47 t kernel: [602563.525129] md: md0: data-check done. > > Jan 3 01:03:47 t kernel: [602563.554886] md: data-check of RAID array md2 > > Jan 3 01:03:47 t kernel: [602563.581194] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. > > Jan 3 01:03:47 t kernel: [602563.616879] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. > > Jan 3 01:03:47 t kernel: [602563.675944] md: using 128k window, over a total of 460928832 blocks. > > Jan 3 04:47:09 t kernel: [615965.563359] md: md2: data-check done. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- ------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis-Jp3n8lUXroTtwjQa/ONI9g@public.gmane.org ------------------------------------- ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <20100216182248.GA4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org>]
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() [not found] ` <20100216182248.GA4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org> @ 2010-02-16 20:21 ` J. Bruce Fields 2010-02-16 20:27 ` Nikola Ciprich 0 siblings, 1 reply; 12+ messages in thread From: J. Bruce Fields @ 2010-02-16 20:21 UTC (permalink / raw) To: Nikola Ciprich Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs, nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g On Tue, Feb 16, 2010 at 07:22:48PM +0100, Nikola Ciprich wrote: > Hello, > is there any conclusion regarding this problem? > I'm seeing it too on one of my machines, running 2.6.32.8 > thanks a lot in advance for any informamation.. > regards Nothing yet. I'm taking another look. Do you know what the most recent version was where you did *not* see these warnings? --b. > nik > > > On Tue, Feb 02, 2010 at 11:22:28AM -0500, J. Bruce Fields wrote: > > On Wed, Jan 27, 2010 at 03:01:40PM -0800, Harry Edmon wrote: > > > J. Bruce Fields wrote: > > >> On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote: > > >> > > >>> (cc linux-nfs) > > >>> > > >>> On Thu, 14 Jan 2010 09:53:04 -0800 > > >>> Harry Edmon <harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org> wrote: > > >>> > > >>> > > >>>> I have a single quad-core Xeon machine running 2.6.32.2 that keeps > > >>>> giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This > > >>>> machine is mainaly used as a NFS server. I have attached the > > >>>> kern.log and my config file. Any ideas? > > >>>> > > >> > > >> Are you using NFSv4, or only v2/v3? Looking at the trace: > > >> > > >> Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------ > > >> Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > > >> > > > NFSv4 and NFSv3. > > > > Thanks. And you aren't using rdma are you? > > > > >> OK, so that's the warning that we're trying to get something with > > >> reference count 0. > > >> > > >> Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8 > > >> Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > >> Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1 > > >> > > >> What does the very first oops, bug, or warning you get look like? (If > > >> I'm reading the tain information right, this isn't the first one.) > > >> > > > Attached is the kern.log that contains the first warning. > > > > Thanks. So yes, the first warning is a stray kref_put() in svc_recv(), > > and the such put I see on a quick skim is a svc_xprt_put(). The > > following BUG() is confirmation, since it's an attempt inside > > scv-xprt_release() to clear an inode with I_CLEAR already set; so we > > dropped the reference count to zero, cleared it, then tried to do > > another get/put pair, resulting in an attempt to free the same socket > > twice. > > > > --b. > > > > > > > > -- > > > Dr. Harry Edmon E-MAIL: harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org > > > 206-543-0547 FAX: 206-543-0308 harry-B93hV6UPU7Z2icitjWtXSw@public.gmane.org, harry@uw.edu > > > Director of IT, College of the Environment and > > > Director of Computing, Dept of Atmospheric Sciences > > > University of Washington, Box 351640, Seattle, WA 98195-1640 > > > > > > > > Dec 27 11:57:44 t kernel: [37000.838904] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0055): Battery charging started:. > > > Dec 27 11:57:46 t kernel: [37002.028946] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0056): Battery charging completed:. > > > Dec 28 00:00:51 t kernel: [80387.429120] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0. > > > Dec 28 03:32:07 t kernel: [93063.950486] 3w-9xxx: scsi6: AEN: INFO (0x04:0x002B): Verify completed:unit=0. > > > Dec 29 17:04:43 t kernel: [228218.883443] ------------[ cut here ]------------ > > > Dec 29 17:04:45 t kernel: [228218.911691] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > > > Dec 29 17:04:45 t kernel: [228218.945083] Hardware name: X7DB8 > > > Dec 29 17:04:45 t kernel: [228218.964948] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > Dec 29 17:04:45 t kernel: [228219.227296] Pid: 3108, comm: nfsd Not tainted 2.6.32.2 #1 > > > Dec 29 17:04:45 t kernel: [228219.260164] Call Trace: > > > Dec 29 17:04:45 t kernel: [228219.275350] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > > Dec 29 17:04:45 t kernel: [228219.306656] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > > > Dec 29 17:04:45 t kernel: [228219.343160] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > > > Dec 29 17:04:45 t kernel: [228219.378105] [<ffffffff81142230>] kref_get+0x23/0x2b > > > Dec 29 17:04:45 t kernel: [228219.408379] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > Dec 29 17:04:45 t kernel: [228219.441242] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > > > Dec 29 17:04:45 t kernel: [228219.477225] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > Dec 29 17:04:45 t kernel: [228219.515806] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > Dec 29 17:04:45 t kernel: [228219.552834] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > Dec 29 17:04:45 t kernel: [228219.585698] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > Dec 29 17:04:45 t kernel: [228219.618043] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > Dec 29 17:04:45 t kernel: [228219.647789] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > Dec 29 17:04:45 t kernel: [228219.678052] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > Dec 29 17:04:45 t kernel: [228219.708317] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > Dec 29 17:04:45 t kernel: [228219.739621] ---[ end trace 1a77408724500fae ]--- > > > Dec 29 17:04:45 t kernel: [228219.767925] ------------[ cut here ]------------ > > > Dec 29 17:04:45 t kernel: [228219.792730] kernel BUG at fs/inode.c:1343! > > > Dec 29 17:04:45 t kernel: [228219.792730] invalid opcode: 0000 [#1] SMP > > > Dec 29 17:04:45 t kernel: [228219.792730] last sysfs file: /sys/class/scsi_host/host6/stats > > > Dec 29 17:04:45 t kernel: [228219.792730] CPU 0 > > > Dec 29 17:04:45 t kernel: [228219.792730] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > Dec 29 17:04:45 t kernel: [228219.792730] Pid: 3108, comm: nfsd Tainted: G W 2.6.32.2 #1 X7DB8 > > > Dec 29 17:04:45 t kernel: [228219.792730] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65 > > > Dec 29 17:04:45 t kernel: [228219.792730] RSP: 0018:ffff88012588bd20 EFLAGS: 00010246 > > > Dec 29 17:04:45 t kernel: [228219.792730] RAX: 0000000000000000 RBX: ffff88003c3c2cc8 RCX: ffffffff8139f927 > > > Dec 29 17:04:45 t kernel: [228219.792730] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88003c3c2cc8 > > > Dec 29 17:04:45 t kernel: [228219.792730] RBP: ffff88012588bd30 R08: 0000000000000002 R09: 0000000000000000 > > > Dec 29 17:04:45 t kernel: [228219.792730] R10: ffffffff81142230 R11: 0000000100000001 R12: ffff88003c3c2c80 > > > Dec 29 17:04:45 t kernel: [228219.792730] R13: ffffffffa028ef50 R14: ffff88010072d400 R15: ffff88012dd423c0 > > > Dec 29 17:04:45 t kernel: [228219.792730] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 > > > Dec 29 17:04:45 t kernel: [228219.792730] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > > Dec 29 17:04:45 t kernel: [228219.792730] CR2: 00007fad86c2e000 CR3: 00000000305e5000 CR4: 00000000000006f0 > > > Dec 29 17:04:45 t kernel: [228219.792730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > Dec 29 17:04:45 t kernel: [228219.792730] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > > Dec 29 17:04:45 t kernel: [228219.792730] Process nfsd (pid: 3108, threadinfo ffff88012588a000, task ffff880125e66150) > > > Dec 29 17:04:45 t kernel: [228219.792730] Stack: > > > Dec 29 17:04:45 t kernel: [228219.792730] ffff88012588bd90 ffff88010072d400 ffff88012588bd50 ffffffff811f4be2 > > > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88010072d400 ffff88010072d410 ffff88012588bd70 ffffffffa027c1d2 > > > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88012588bd70 ffff88010072d400 ffff88012588bda0 ffffffffa0286b03 > > > Dec 29 17:04:45 t kernel: [228219.792730] Call Trace: > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff811f4be2>] sock_release+0x5c/0x6c > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc] > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc] > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc] > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142201>] kref_put+0x43/0x4f > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc] > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc] > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > Dec 29 17:04:45 t kernel: [228219.792730] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85 > > > Dec 29 17:04:45 t kernel: [228219.792730] RIP [<ffffffff810bccc2>] iput+0x1b/0x65 > > > Dec 29 17:04:45 t kernel: [228219.792730] RSP <ffff88012588bd20> > > > Dec 29 17:04:45 t kernel: [228221.632694] ---[ end trace 1a77408724500faf ]--- > > > Dec 29 20:53:01 t kernel: [241917.056374] ------------[ cut here ]------------ > > > Dec 29 20:53:01 t kernel: [241917.084581] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > > > Dec 29 20:53:01 t kernel: [241917.117941] Hardware name: X7DB8 > > > Dec 29 20:53:01 t kernel: [241917.137804] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > Dec 29 20:53:01 t kernel: [241917.400101] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 > > > Dec 29 20:53:01 t kernel: [241917.437653] Call Trace: > > > Dec 29 20:53:01 t kernel: [241917.452843] [<ffffffffa0286b0b>] ? svc_xprt_free+0x46/0x4e [sunrpc] > > > Dec 29 20:53:01 t kernel: [241917.491420] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > > Dec 29 20:53:01 t kernel: [241917.522728] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > > > Dec 29 20:53:01 t kernel: [241917.559230] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > > > Dec 29 20:53:01 t kernel: [241917.594177] [<ffffffff81142230>] kref_get+0x23/0x2b > > > Dec 29 20:53:01 t kernel: [241917.624446] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > > > Dec 29 20:53:01 t kernel: [241917.660432] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > Dec 29 20:53:01 t kernel: [241917.699013] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > Dec 29 20:53:01 t kernel: [241917.736041] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > Dec 29 20:53:01 t kernel: [241917.768905] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > Dec 29 20:53:01 t kernel: [241917.801249] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > Dec 29 20:53:01 t kernel: [241917.830993] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > Dec 29 20:53:01 t kernel: [241917.861258] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > Dec 29 20:53:01 t kernel: [241917.891522] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > Dec 29 20:53:01 t kernel: [241917.922827] ---[ end trace 1a77408724500fb0 ]--- > > > Dec 29 20:53:01 t kernel: [241917.951084] ------------[ cut here ]------------ > > > Dec 29 20:53:03 t kernel: [241917.974874] kernel BUG at fs/inode.c:1343! > > > Dec 29 20:53:03 t kernel: [241917.988010] invalid opcode: 0000 [#2] SMP > > > Dec 29 20:53:03 t kernel: [241917.988010] last sysfs file: /sys/class/scsi_host/host6/stats > > > Dec 29 20:53:03 t kernel: [241917.988010] CPU 1 > > > Dec 29 20:53:03 t kernel: [241917.988010] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > Dec 29 20:53:03 t kernel: [241918.102960] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8 > > > Dec 29 20:53:03 t kernel: [241918.102960] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65 > > > Dec 29 20:53:03 t kernel: [241918.102960] RSP: 0000:ffff8801225b9d20 EFLAGS: 00010246 > > > Dec 29 20:53:03 t kernel: [241918.102960] RAX: 0000000000000000 RBX: ffff880120435148 RCX: ffffffff8139f927 > > > Dec 29 20:53:03 t kernel: [241918.102960] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff880120435148 > > > Dec 29 20:53:03 t kernel: [241918.102960] RBP: ffff8801225b9d30 R08: 0000000000000020 R09: ffff88002820c9d0 > > > Dec 29 20:53:03 t kernel: [241918.102960] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff880120435100 > > > Dec 29 20:53:03 t kernel: [241918.102960] R13: ffffffffa028ef50 R14: ffff8800c1b3b800 R15: ffff88012dd423c0 > > > Dec 29 20:53:03 t kernel: [241918.102960] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000 > > > Dec 29 20:53:03 t kernel: [241918.102960] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > > Dec 29 20:53:03 t kernel: [241918.102960] CR2: 00007fad86c2e000 CR3: 0000000129ae6000 CR4: 00000000000006e0 > > > Dec 29 20:53:03 t kernel: [241918.102960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > Dec 29 20:53:03 t kernel: [241918.102960] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > > Dec 29 20:53:03 t kernel: [241918.102960] Process nfsd (pid: 3207, threadinfo ffff8801225b8000, task ffff880122416640) > > > Dec 29 20:53:03 t kernel: [241918.102960] Stack: > > > Dec 29 20:53:03 t kernel: [241918.102960] ffff8801225b9d90 ffff8800c1b3b800 ffff8801225b9d50 ffffffff811f4be2 > > > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8800c1b3b800 ffff8800c1b3b810 ffff8801225b9d70 ffffffffa027c1d2 > > > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8801225b9d70 ffff8800c1b3b800 ffff8801225b9da0 ffffffffa0286b03 > > > Dec 29 20:53:03 t kernel: [241918.102960] Call Trace: > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff811f4be2>] sock_release+0x5c/0x6c > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc] > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc] > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc] > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81142201>] kref_put+0x43/0x4f > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc] > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc] > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > Dec 29 20:53:04 t kernel: [241918.102960] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85 > > > Dec 29 20:53:04 t kernel: [241918.102960] RIP [<ffffffff810bccc2>] iput+0x1b/0x65 > > > Dec 29 20:53:04 t kernel: [241918.102960] RSP <ffff8801225b9d20> > > > Dec 29 20:53:04 t kernel: [241919.822190] ---[ end trace 1a77408724500fb1 ]--- > > > Dec 30 11:22:26 t kernel: [294082.152979] 3w-9xxx: scsi6: AEN: ERROR (0x04:0x0009): Drive timeout detected:port=0. > > > Dec 31 11:24:44 t kernel: [380620.428029] br0: port 2(eth2) entering disabled state > > > Dec 31 11:24:44 t kernel: [380620.458826] br0: port 1(eth1) entering disabled state > > > Dec 31 11:24:44 t kernel: [380620.666738] device eth1 left promiscuous mode > > > Dec 31 11:24:44 t kernel: [380620.693361] br0: port 1(eth1) entering disabled state > > > Dec 31 11:24:44 t kernel: [380620.909921] device eth2 left promiscuous mode > > > Dec 31 11:24:44 t kernel: [380620.936603] br0: port 2(eth2) entering disabled state > > > Dec 31 11:24:48 t kernel: [380624.449637] device eth1 entered promiscuous mode > > > Dec 31 11:24:48 t kernel: [380624.533887] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X > > > Dec 31 11:24:48 t kernel: [380624.620065] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X > > > Dec 31 11:24:48 t kernel: [380624.651581] ADDRCONF(NETDEV_UP): eth1: link is not ready > > > Dec 31 11:24:48 t kernel: [380624.687675] device eth2 entered promiscuous mode > > > Dec 31 11:24:48 t kernel: [380624.860445] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X > > > Dec 31 11:24:48 t kernel: [380624.944084] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X > > > Dec 31 11:24:48 t kernel: [380624.975824] ADDRCONF(NETDEV_UP): eth2: link is not ready > > > Dec 31 11:24:50 t kernel: [380626.984998] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > > Dec 31 11:24:51 t kernel: [380627.031517] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready > > > Dec 31 11:24:51 t kernel: [380627.066844] br0: port 2(eth2) entering forwarding state > > > Dec 31 11:24:51 t kernel: [380627.251584] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > > Dec 31 11:24:51 t kernel: [380627.300433] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready > > > Dec 31 11:24:51 t kernel: [380627.335625] br0: port 1(eth1) entering forwarding state > > > Dec 31 11:24:59 t kernel: [380635.148016] br0: no IPv6 routers present > > > Dec 31 11:25:01 t kernel: [380637.504511] eth1: no IPv6 routers present > > > Dec 31 11:25:01 t kernel: [380637.720515] eth2: no IPv6 routers present > > > Dec 31 11:28:30 t kernel: [380846.205549] e1000e: eth1 NIC Link is Down > > > Dec 31 11:28:30 t kernel: [380846.248659] br0: port 1(eth1) entering disabled state > > > Dec 31 11:28:32 t kernel: [380848.693501] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX > > > Dec 31 11:28:32 t kernel: [380848.739167] 0000:06:00.1: eth1: 10/100 speed: disabling TSO > > > Dec 31 11:28:32 t kernel: [380848.776396] br0: port 1(eth1) entering forwarding state > > > Dec 31 11:28:49 t kernel: [380865.803038] e1000e: eth1 NIC Link is Down > > > Dec 31 11:28:49 t kernel: [380865.874086] br0: port 1(eth1) entering disabled state > > > Dec 31 11:28:52 t kernel: [380868.267943] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > > Dec 31 11:28:52 t kernel: [380868.316772] br0: port 1(eth1) entering forwarding state > > > Dec 31 11:29:18 t kernel: [380894.253543] e1000e: eth1 NIC Link is Down > > > Dec 31 11:29:18 t kernel: [380894.293597] br0: port 1(eth1) entering disabled state > > > Dec 31 11:29:20 t kernel: [380896.688442] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > > Dec 31 11:29:20 t kernel: [380896.737255] br0: port 1(eth1) entering forwarding state > > > Dec 31 11:29:25 t kernel: [380901.493543] e1000e: eth1 NIC Link is Down > > > Dec 31 11:29:25 t kernel: [380901.534074] br0: port 1(eth1) entering disabled state > > > Dec 31 11:29:28 t kernel: [380904.161008] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > > Dec 31 11:29:28 t kernel: [380904.209842] br0: port 1(eth1) entering forwarding state > > > Jan 1 22:40:06 t kernel: [507542.268700] ------------[ cut here ]------------ > > > Jan 1 22:40:08 t kernel: [507542.296925] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > > > Jan 1 22:40:08 t kernel: [507542.330301] Hardware name: X7DB8 > > > Jan 1 22:40:08 t kernel: [507542.350163] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > Jan 1 22:40:08 t kernel: [507542.614635] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 > > > Jan 1 22:40:08 t kernel: [507542.652191] Call Trace: > > > Jan 1 22:40:08 t kernel: [507542.667375] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > > Jan 1 22:40:08 t kernel: [507542.698681] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > > > Jan 1 22:40:08 t kernel: [507542.735186] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > > > Jan 1 22:40:08 t kernel: [507542.770130] [<ffffffff81142230>] kref_get+0x23/0xBj > > > Jan 1 22:40:08 t kernel: [507542.800400] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > Jan 1 22:40:08 t kernel: [507542.833269] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > > > Jan 1 22:40:08 t kernel: [507542.869251] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > Jan 1 22:40:08 t kernel: [507542.907825] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > Jan 1 22:40:08 t kernel: [507542.944853] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > Jan 1 22:40:08 t kernel: [507542.977720] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > Jan 1 22:40:08 t kernel: [507543.010063] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > Jan 1 22:40:08 t kernel: [507543.039808] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > Jan 1 22:40:08 t kernel: [507543.070071] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > Jan 1 22:40:08 t kernel: [507543.100337] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > Jan 1 22:40:08 t kernel: [507543.131640] ---[ end trace 1a77408724500fb2 ]--- > > > Jan 1 22:40:08 t kernel: [507543.160026] general protection fault: 0000 [#3] SMP > > > Jan 1 22:40:08 t kernel: [507543.165025] last sysfs file: /sys/class/net/eth2/brport/hold_timer > > > Jan 1 22:40:08 t kernel: [507543.165025] CPU 3 > > > Jan 1 22:40:08 t kernel: [507543.165025] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > Jan 1 22:40:08 t kernel: [507543.165025] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8 > > > Jan 1 22:40:08 t kernel: [507543.165025] RIP: 0010:[<ffffffffa0286797>] [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc] > > > Jan 1 22:40:08 t kernel: [507543.165025] RSP: 0018:ffff880124ea1e00 EFLAGS: 00010246 > > > Jan 1 22:40:08 t kernel: [507543.165025] RAX: 3a30343a32322031 RBX: 0000000000000000 RCX: 0000000000000000 > > > Jan 1 22:40:08 t kernel: [507543.165025] RDX: 0000000000004c86 RSI: ffff880124ea1df0 RDI: ffff880124e00000 > > > Jan 1 22:40:08 t kernel: [507543.165025] RBP: ffff880124ea1eb0 R08: 0000000000000020 R09: ffff88002830ca70 > > > Jan 1 22:40:08 t kernel: [507543.165025] R10: ffff88002838cf40 R11: 0000000000000000 R12: 0000000000000082 > > > Jan 1 22:40:08 t kernel: [507543.165025] R13: ffff880124e00000 R14: ffff8800c7444000 R15: ffff88012dd423c0 > > > Jan 1 22:40:08 t kernel: [507543.165025] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000 > > > Jan 1 22:40:08 t kernel: [507543.165025] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > > Jan 1 22:40:08 t kernel: [507543.165025] CR2: 00007f0dc9cc5098 CR3: 000000012c266000 CR4: 00000000000006e0 > > > Jan 1 22:40:08 t kernel: [507543.165025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > Jan 1 22:40:08 t kernel: [507543.165025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > > Jan 1 22:40:08 t kernel: [507543.165025] Process nfsd (pid: 3134, threadinfo ffff880124ea0000, task ffff880124e9f890) > > > Jan 1 22:40:08 t kernel: [507543.165025] Stack: > > > Jan 1 22:40:08 t kernel: [507543.165025] ffff880124e00000 ffff880124e9f890 ffff880124ea1e40 00000000000dbba0 > > > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff88012dd423c4 ffff88012c0bea40 ffff880000000082 0000000000000001 > > > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff880124ea1eb0 ffffffffa027abe4 0000000000000000 ffff880124e9f890 > > > Jan 1 22:40:08 t kernel: [507543.165025] Call Trace: > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > Jan 1 22:40:08 t kernel: [507543.165025] Code: 00 49 8d 85 88 01 00 00 49 89 85 a0 09 00 00 8b 82 e8 00 00 00 8d 1c 85 00 00 00 00 2b 9a e0 00 00 00 eb 0c 49 8b 46 08 4c 89 ef <ff> 50 18 89 c3 f6 05 56 ce 00 00 01 74 10 89 de 48 c7 c7 cf b6 > > > Jan 1 22:40:08 t kernel: [507543.165025] RIP [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc] > > > Jan 1 22:40:08 t kernel: [507543.165025] RSP <ffff880124ea1e00> > > > Jan 1 22:40:08 t kernel: [507544.713111] ---[ end trace 1a77408724500fb3 ]--- > > > Jan 2 12:06:02 t kernel: [555898.365986] RPC: multiple fragments per record not supported > > > Jan 2 12:06:02 t kernel: [555898.527050] RPC: multiple fragments per record not supported > > > Jan 2 12:06:05 t kernel: [555901.700485] RPC: multiple fragments per record not supported > > > Jan 2 12:06:08 t kernel: [555904.880825] RPC: multiple fragments per record not supported > > > Jan 3 00:01:08 t kernel: [598804.533706] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0. > > > Jan 3 00:57:01 t kernel: [602157.347383] md: data-check of RAID array md0 > > > Jan 3 00:57:01 t kernel: [602157.350802] md: delaying data-check of md2 until md0 has finished (they share one or more physical units) > > > Jan 3 00:57:01 t kernel: [602157.431453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. > > > Jan 3 00:57:01 t kernel: [602157.467054] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. > > > Jan 3 00:57:01 t kernel: [602157.526113] md: using 128k window, over a total of 19542976 blocks. > > > Jan 3 01:03:47 t kernel: [602563.525129] md: md0: data-check done. > > > Jan 3 01:03:47 t kernel: [602563.554886] md: data-check of RAID array md2 > > > Jan 3 01:03:47 t kernel: [602563.581194] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. > > > Jan 3 01:03:47 t kernel: [602563.616879] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. > > > Jan 3 01:03:47 t kernel: [602563.675944] md: using 128k window, over a total of 460928832 blocks. > > > Jan 3 04:47:09 t kernel: [615965.563359] md: md2: data-check done. > > > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- > ------------------------------------- > Nikola CIPRICH > LinuxBox.cz, s.r.o. > 28. rijna 168, 709 01 Ostrava > > tel.: +420 596 603 142 > fax: +420 596 621 273 > mobil: +420 777 093 799 > > www.linuxbox.cz > > mobil servis: +420 737 238 656 > email servis: servis-Jp3n8lUXroTtwjQa/ONI9g@public.gmane.org > ------------------------------------- ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() 2010-02-16 20:21 ` J. Bruce Fields @ 2010-02-16 20:27 ` Nikola Ciprich [not found] ` <20100216202720.GB4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org> 0 siblings, 1 reply; 12+ messages in thread From: Nikola Ciprich @ 2010-02-16 20:27 UTC (permalink / raw) To: J. Bruce Fields Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs, nikola.ciprich > Nothing yet. I'm taking another look. > > Do you know what the most recent version was where you did *not* see > these warnings? well, actually I decided I'll bisect this one no matter what, so hopefully we'll see soon :) I'll report n. > > --b. > > > nik > > > > > > On Tue, Feb 02, 2010 at 11:22:28AM -0500, J. Bruce Fields wrote: > > > On Wed, Jan 27, 2010 at 03:01:40PM -0800, Harry Edmon wrote: > > > > J. Bruce Fields wrote: > > > >> On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote: > > > >> > > > >>> (cc linux-nfs) > > > >>> > > > >>> On Thu, 14 Jan 2010 09:53:04 -0800 > > > >>> Harry Edmon <harry@atmos.washington.edu> wrote: > > > >>> > > > >>> > > > >>>> I have a single quad-core Xeon machine running 2.6.32.2 that keeps > > > >>>> giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This > > > >>>> machine is mainaly used as a NFS server. I have attached the > > > >>>> kern.log and my config file. Any ideas? > > > >>>> > > > >> > > > >> Are you using NFSv4, or only v2/v3? Looking at the trace: > > > >> > > > >> Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------ > > > >> Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > > > >> > > > > NFSv4 and NFSv3. > > > > > > Thanks. And you aren't using rdma are you? > > > > > > >> OK, so that's the warning that we're trying to get something with > > > >> reference count 0. > > > >> > > > >> Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8 > > > >> Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > >> Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1 > > > >> > > > >> What does the very first oops, bug, or warning you get look like? (If > > > >> I'm reading the tain information right, this isn't the first one.) > > > >> > > > > Attached is the kern.log that contains the first warning. > > > > > > Thanks. So yes, the first warning is a stray kref_put() in svc_recv(), > > > and the such put I see on a quick skim is a svc_xprt_put(). The > > > following BUG() is confirmation, since it's an attempt inside > > > scv-xprt_release() to clear an inode with I_CLEAR already set; so we > > > dropped the reference count to zero, cleared it, then tried to do > > > another get/put pair, resulting in an attempt to free the same socket > > > twice. > > > > > > --b. > > > > > > > > > > > -- > > > > Dr. Harry Edmon E-MAIL: harry@atmos.washington.edu > > > > 206-543-0547 FAX: 206-543-0308 harry@washington.edu, harry@uw.edu > > > > Director of IT, College of the Environment and > > > > Director of Computing, Dept of Atmospheric Sciences > > > > University of Washington, Box 351640, Seattle, WA 98195-1640 > > > > > > > > > > > Dec 27 11:57:44 t kernel: [37000.838904] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0055): Battery charging started:. > > > > Dec 27 11:57:46 t kernel: [37002.028946] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0056): Battery charging completed:. > > > > Dec 28 00:00:51 t kernel: [80387.429120] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0. > > > > Dec 28 03:32:07 t kernel: [93063.950486] 3w-9xxx: scsi6: AEN: INFO (0x04:0x002B): Verify completed:unit=0. > > > > Dec 29 17:04:43 t kernel: [228218.883443] ------------[ cut here ]------------ > > > > Dec 29 17:04:45 t kernel: [228218.911691] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > > > > Dec 29 17:04:45 t kernel: [228218.945083] Hardware name: X7DB8 > > > > Dec 29 17:04:45 t kernel: [228218.964948] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > > Dec 29 17:04:45 t kernel: [228219.227296] Pid: 3108, comm: nfsd Not tainted 2.6.32.2 #1 > > > > Dec 29 17:04:45 t kernel: [228219.260164] Call Trace: > > > > Dec 29 17:04:45 t kernel: [228219.275350] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > > > Dec 29 17:04:45 t kernel: [228219.306656] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > > > > Dec 29 17:04:45 t kernel: [228219.343160] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > > > > Dec 29 17:04:45 t kernel: [228219.378105] [<ffffffff81142230>] kref_get+0x23/0x2b > > > > Dec 29 17:04:45 t kernel: [228219.408379] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > > Dec 29 17:04:45 t kernel: [228219.441242] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > > > > Dec 29 17:04:45 t kernel: [228219.477225] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > > Dec 29 17:04:45 t kernel: [228219.515806] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > > Dec 29 17:04:45 t kernel: [228219.552834] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > > Dec 29 17:04:45 t kernel: [228219.585698] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > > Dec 29 17:04:45 t kernel: [228219.618043] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > > Dec 29 17:04:45 t kernel: [228219.647789] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > > Dec 29 17:04:45 t kernel: [228219.678052] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > > Dec 29 17:04:45 t kernel: [228219.708317] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > > Dec 29 17:04:45 t kernel: [228219.739621] ---[ end trace 1a77408724500fae ]--- > > > > Dec 29 17:04:45 t kernel: [228219.767925] ------------[ cut here ]------------ > > > > Dec 29 17:04:45 t kernel: [228219.792730] kernel BUG at fs/inode.c:1343! > > > > Dec 29 17:04:45 t kernel: [228219.792730] invalid opcode: 0000 [#1] SMP > > > > Dec 29 17:04:45 t kernel: [228219.792730] last sysfs file: /sys/class/scsi_host/host6/stats > > > > Dec 29 17:04:45 t kernel: [228219.792730] CPU 0 > > > > Dec 29 17:04:45 t kernel: [228219.792730] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > > Dec 29 17:04:45 t kernel: [228219.792730] Pid: 3108, comm: nfsd Tainted: G W 2.6.32.2 #1 X7DB8 > > > > Dec 29 17:04:45 t kernel: [228219.792730] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65 > > > > Dec 29 17:04:45 t kernel: [228219.792730] RSP: 0018:ffff88012588bd20 EFLAGS: 00010246 > > > > Dec 29 17:04:45 t kernel: [228219.792730] RAX: 0000000000000000 RBX: ffff88003c3c2cc8 RCX: ffffffff8139f927 > > > > Dec 29 17:04:45 t kernel: [228219.792730] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88003c3c2cc8 > > > > Dec 29 17:04:45 t kernel: [228219.792730] RBP: ffff88012588bd30 R08: 0000000000000002 R09: 0000000000000000 > > > > Dec 29 17:04:45 t kernel: [228219.792730] R10: ffffffff81142230 R11: 0000000100000001 R12: ffff88003c3c2c80 > > > > Dec 29 17:04:45 t kernel: [228219.792730] R13: ffffffffa028ef50 R14: ffff88010072d400 R15: ffff88012dd423c0 > > > > Dec 29 17:04:45 t kernel: [228219.792730] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 > > > > Dec 29 17:04:45 t kernel: [228219.792730] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > > > Dec 29 17:04:45 t kernel: [228219.792730] CR2: 00007fad86c2e000 CR3: 00000000305e5000 CR4: 00000000000006f0 > > > > Dec 29 17:04:45 t kernel: [228219.792730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > > Dec 29 17:04:45 t kernel: [228219.792730] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > > > Dec 29 17:04:45 t kernel: [228219.792730] Process nfsd (pid: 3108, threadinfo ffff88012588a000, task ffff880125e66150) > > > > Dec 29 17:04:45 t kernel: [228219.792730] Stack: > > > > Dec 29 17:04:45 t kernel: [228219.792730] ffff88012588bd90 ffff88010072d400 ffff88012588bd50 ffffffff811f4be2 > > > > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88010072d400 ffff88010072d410 ffff88012588bd70 ffffffffa027c1d2 > > > > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88012588bd70 ffff88010072d400 ffff88012588bda0 ffffffffa0286b03 > > > > Dec 29 17:04:45 t kernel: [228219.792730] Call Trace: > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff811f4be2>] sock_release+0x5c/0x6c > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc] > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc] > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc] > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142201>] kref_put+0x43/0x4f > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc] > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc] > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > > Dec 29 17:04:45 t kernel: [228219.792730] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85 > > > > Dec 29 17:04:45 t kernel: [228219.792730] RIP [<ffffffff810bccc2>] iput+0x1b/0x65 > > > > Dec 29 17:04:45 t kernel: [228219.792730] RSP <ffff88012588bd20> > > > > Dec 29 17:04:45 t kernel: [228221.632694] ---[ end trace 1a77408724500faf ]--- > > > > Dec 29 20:53:01 t kernel: [241917.056374] ------------[ cut here ]------------ > > > > Dec 29 20:53:01 t kernel: [241917.084581] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > > > > Dec 29 20:53:01 t kernel: [241917.117941] Hardware name: X7DB8 > > > > Dec 29 20:53:01 t kernel: [241917.137804] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > > Dec 29 20:53:01 t kernel: [241917.400101] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 > > > > Dec 29 20:53:01 t kernel: [241917.437653] Call Trace: > > > > Dec 29 20:53:01 t kernel: [241917.452843] [<ffffffffa0286b0b>] ? svc_xprt_free+0x46/0x4e [sunrpc] > > > > Dec 29 20:53:01 t kernel: [241917.491420] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > > > Dec 29 20:53:01 t kernel: [241917.522728] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > > > > Dec 29 20:53:01 t kernel: [241917.559230] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > > > > Dec 29 20:53:01 t kernel: [241917.594177] [<ffffffff81142230>] kref_get+0x23/0x2b > > > > Dec 29 20:53:01 t kernel: [241917.624446] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > > > > Dec 29 20:53:01 t kernel: [241917.660432] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > > Dec 29 20:53:01 t kernel: [241917.699013] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > > Dec 29 20:53:01 t kernel: [241917.736041] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > > Dec 29 20:53:01 t kernel: [241917.768905] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > > Dec 29 20:53:01 t kernel: [241917.801249] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > > Dec 29 20:53:01 t kernel: [241917.830993] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > > Dec 29 20:53:01 t kernel: [241917.861258] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > > Dec 29 20:53:01 t kernel: [241917.891522] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > > Dec 29 20:53:01 t kernel: [241917.922827] ---[ end trace 1a77408724500fb0 ]--- > > > > Dec 29 20:53:01 t kernel: [241917.951084] ------------[ cut here ]------------ > > > > Dec 29 20:53:03 t kernel: [241917.974874] kernel BUG at fs/inode.c:1343! > > > > Dec 29 20:53:03 t kernel: [241917.988010] invalid opcode: 0000 [#2] SMP > > > > Dec 29 20:53:03 t kernel: [241917.988010] last sysfs file: /sys/class/scsi_host/host6/stats > > > > Dec 29 20:53:03 t kernel: [241917.988010] CPU 1 > > > > Dec 29 20:53:03 t kernel: [241917.988010] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > > Dec 29 20:53:03 t kernel: [241918.102960] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8 > > > > Dec 29 20:53:03 t kernel: [241918.102960] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65 > > > > Dec 29 20:53:03 t kernel: [241918.102960] RSP: 0000:ffff8801225b9d20 EFLAGS: 00010246 > > > > Dec 29 20:53:03 t kernel: [241918.102960] RAX: 0000000000000000 RBX: ffff880120435148 RCX: ffffffff8139f927 > > > > Dec 29 20:53:03 t kernel: [241918.102960] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff880120435148 > > > > Dec 29 20:53:03 t kernel: [241918.102960] RBP: ffff8801225b9d30 R08: 0000000000000020 R09: ffff88002820c9d0 > > > > Dec 29 20:53:03 t kernel: [241918.102960] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff880120435100 > > > > Dec 29 20:53:03 t kernel: [241918.102960] R13: ffffffffa028ef50 R14: ffff8800c1b3b800 R15: ffff88012dd423c0 > > > > Dec 29 20:53:03 t kernel: [241918.102960] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000 > > > > Dec 29 20:53:03 t kernel: [241918.102960] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > > > Dec 29 20:53:03 t kernel: [241918.102960] CR2: 00007fad86c2e000 CR3: 0000000129ae6000 CR4: 00000000000006e0 > > > > Dec 29 20:53:03 t kernel: [241918.102960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > > Dec 29 20:53:03 t kernel: [241918.102960] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > > > Dec 29 20:53:03 t kernel: [241918.102960] Process nfsd (pid: 3207, threadinfo ffff8801225b8000, task ffff880122416640) > > > > Dec 29 20:53:03 t kernel: [241918.102960] Stack: > > > > Dec 29 20:53:03 t kernel: [241918.102960] ffff8801225b9d90 ffff8800c1b3b800 ffff8801225b9d50 ffffffff811f4be2 > > > > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8800c1b3b800 ffff8800c1b3b810 ffff8801225b9d70 ffffffffa027c1d2 > > > > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8801225b9d70 ffff8800c1b3b800 ffff8801225b9da0 ffffffffa0286b03 > > > > Dec 29 20:53:03 t kernel: [241918.102960] Call Trace: > > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff811f4be2>] sock_release+0x5c/0x6c > > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc] > > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc] > > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc] > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81142201>] kref_put+0x43/0x4f > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc] > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc] > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > > Dec 29 20:53:04 t kernel: [241918.102960] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85 > > > > Dec 29 20:53:04 t kernel: [241918.102960] RIP [<ffffffff810bccc2>] iput+0x1b/0x65 > > > > Dec 29 20:53:04 t kernel: [241918.102960] RSP <ffff8801225b9d20> > > > > Dec 29 20:53:04 t kernel: [241919.822190] ---[ end trace 1a77408724500fb1 ]--- > > > > Dec 30 11:22:26 t kernel: [294082.152979] 3w-9xxx: scsi6: AEN: ERROR (0x04:0x0009): Drive timeout detected:port=0. > > > > Dec 31 11:24:44 t kernel: [380620.428029] br0: port 2(eth2) entering disabled state > > > > Dec 31 11:24:44 t kernel: [380620.458826] br0: port 1(eth1) entering disabled state > > > > Dec 31 11:24:44 t kernel: [380620.666738] device eth1 left promiscuous mode > > > > Dec 31 11:24:44 t kernel: [380620.693361] br0: port 1(eth1) entering disabled state > > > > Dec 31 11:24:44 t kernel: [380620.909921] device eth2 left promiscuous mode > > > > Dec 31 11:24:44 t kernel: [380620.936603] br0: port 2(eth2) entering disabled state > > > > Dec 31 11:24:48 t kernel: [380624.449637] device eth1 entered promiscuous mode > > > > Dec 31 11:24:48 t kernel: [380624.533887] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X > > > > Dec 31 11:24:48 t kernel: [380624.620065] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X > > > > Dec 31 11:24:48 t kernel: [380624.651581] ADDRCONF(NETDEV_UP): eth1: link is not ready > > > > Dec 31 11:24:48 t kernel: [380624.687675] device eth2 entered promiscuous mode > > > > Dec 31 11:24:48 t kernel: [380624.860445] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X > > > > Dec 31 11:24:48 t kernel: [380624.944084] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X > > > > Dec 31 11:24:48 t kernel: [380624.975824] ADDRCONF(NETDEV_UP): eth2: link is not ready > > > > Dec 31 11:24:50 t kernel: [380626.984998] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > > > Dec 31 11:24:51 t kernel: [380627.031517] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready > > > > Dec 31 11:24:51 t kernel: [380627.066844] br0: port 2(eth2) entering forwarding state > > > > Dec 31 11:24:51 t kernel: [380627.251584] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > > > Dec 31 11:24:51 t kernel: [380627.300433] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready > > > > Dec 31 11:24:51 t kernel: [380627.335625] br0: port 1(eth1) entering forwarding state > > > > Dec 31 11:24:59 t kernel: [380635.148016] br0: no IPv6 routers present > > > > Dec 31 11:25:01 t kernel: [380637.504511] eth1: no IPv6 routers present > > > > Dec 31 11:25:01 t kernel: [380637.720515] eth2: no IPv6 routers present > > > > Dec 31 11:28:30 t kernel: [380846.205549] e1000e: eth1 NIC Link is Down > > > > Dec 31 11:28:30 t kernel: [380846.248659] br0: port 1(eth1) entering disabled state > > > > Dec 31 11:28:32 t kernel: [380848.693501] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX > > > > Dec 31 11:28:32 t kernel: [380848.739167] 0000:06:00.1: eth1: 10/100 speed: disabling TSO > > > > Dec 31 11:28:32 t kernel: [380848.776396] br0: port 1(eth1) entering forwarding state > > > > Dec 31 11:28:49 t kernel: [380865.803038] e1000e: eth1 NIC Link is Down > > > > Dec 31 11:28:49 t kernel: [380865.874086] br0: port 1(eth1) entering disabled state > > > > Dec 31 11:28:52 t kernel: [380868.267943] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > > > Dec 31 11:28:52 t kernel: [380868.316772] br0: port 1(eth1) entering forwarding state > > > > Dec 31 11:29:18 t kernel: [380894.253543] e1000e: eth1 NIC Link is Down > > > > Dec 31 11:29:18 t kernel: [380894.293597] br0: port 1(eth1) entering disabled state > > > > Dec 31 11:29:20 t kernel: [380896.688442] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > > > Dec 31 11:29:20 t kernel: [380896.737255] br0: port 1(eth1) entering forwarding state > > > > Dec 31 11:29:25 t kernel: [380901.493543] e1000e: eth1 NIC Link is Down > > > > Dec 31 11:29:25 t kernel: [380901.534074] br0: port 1(eth1) entering disabled state > > > > Dec 31 11:29:28 t kernel: [380904.161008] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX > > > > Dec 31 11:29:28 t kernel: [380904.209842] br0: port 1(eth1) entering forwarding state > > > > Jan 1 22:40:06 t kernel: [507542.268700] ------------[ cut here ]------------ > > > > Jan 1 22:40:08 t kernel: [507542.296925] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b() > > > > Jan 1 22:40:08 t kernel: [507542.330301] Hardware name: X7DB8 > > > > Jan 1 22:40:08 t kernel: [507542.350163] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > > Jan 1 22:40:08 t kernel: [507542.614635] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 > > > > Jan 1 22:40:08 t kernel: [507542.652191] Call Trace: > > > > Jan 1 22:40:08 t kernel: [507542.667375] [<ffffffff81142230>] ? kref_get+0x23/0x2b > > > > Jan 1 22:40:08 t kernel: [507542.698681] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4 > > > > Jan 1 22:40:08 t kernel: [507542.735186] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11 > > > > Jan 1 22:40:08 t kernel: [507542.770130] [<ffffffff81142230>] kref_get+0x23/0xBj > > > > Jan 1 22:40:08 t kernel: [507542.800400] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > > Jan 1 22:40:08 t kernel: [507542.833269] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc] > > > > Jan 1 22:40:08 t kernel: [507542.869251] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > > Jan 1 22:40:08 t kernel: [507542.907825] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > > Jan 1 22:40:08 t kernel: [507542.944853] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > > Jan 1 22:40:08 t kernel: [507542.977720] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > > Jan 1 22:40:08 t kernel: [507543.010063] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > > Jan 1 22:40:08 t kernel: [507543.039808] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > > Jan 1 22:40:08 t kernel: [507543.070071] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > > Jan 1 22:40:08 t kernel: [507543.100337] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > > Jan 1 22:40:08 t kernel: [507543.131640] ---[ end trace 1a77408724500fb2 ]--- > > > > Jan 1 22:40:08 t kernel: [507543.160026] general protection fault: 0000 [#3] SMP > > > > Jan 1 22:40:08 t kernel: [507543.165025] last sysfs file: /sys/class/net/eth2/brport/hold_timer > > > > Jan 1 22:40:08 t kernel: [507543.165025] CPU 3 > > > > Jan 1 22:40:08 t kernel: [507543.165025] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan] > > > > Jan 1 22:40:08 t kernel: [507543.165025] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8 > > > > Jan 1 22:40:08 t kernel: [507543.165025] RIP: 0010:[<ffffffffa0286797>] [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc] > > > > Jan 1 22:40:08 t kernel: [507543.165025] RSP: 0018:ffff880124ea1e00 EFLAGS: 00010246 > > > > Jan 1 22:40:08 t kernel: [507543.165025] RAX: 3a30343a32322031 RBX: 0000000000000000 RCX: 0000000000000000 > > > > Jan 1 22:40:08 t kernel: [507543.165025] RDX: 0000000000004c86 RSI: ffff880124ea1df0 RDI: ffff880124e00000 > > > > Jan 1 22:40:08 t kernel: [507543.165025] RBP: ffff880124ea1eb0 R08: 0000000000000020 R09: ffff88002830ca70 > > > > Jan 1 22:40:08 t kernel: [507543.165025] R10: ffff88002838cf40 R11: 0000000000000000 R12: 0000000000000082 > > > > Jan 1 22:40:08 t kernel: [507543.165025] R13: ffff880124e00000 R14: ffff8800c7444000 R15: ffff88012dd423c0 > > > > Jan 1 22:40:08 t kernel: [507543.165025] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000 > > > > Jan 1 22:40:08 t kernel: [507543.165025] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > > > Jan 1 22:40:08 t kernel: [507543.165025] CR2: 00007f0dc9cc5098 CR3: 000000012c266000 CR4: 00000000000006e0 > > > > Jan 1 22:40:08 t kernel: [507543.165025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > > > Jan 1 22:40:08 t kernel: [507543.165025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > > > Jan 1 22:40:08 t kernel: [507543.165025] Process nfsd (pid: 3134, threadinfo ffff880124ea0000, task ffff880124e9f890) > > > > Jan 1 22:40:08 t kernel: [507543.165025] Stack: > > > > Jan 1 22:40:08 t kernel: [507543.165025] ffff880124e00000 ffff880124e9f890 ffff880124ea1e40 00000000000dbba0 > > > > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff88012dd423c4 ffff88012c0bea40 ffff880000000082 0000000000000001 > > > > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff880124ea1eb0 ffffffffa027abe4 0000000000000000 ffff880124e9f890 > > > > Jan 1 22:40:08 t kernel: [507543.165025] Call Trace: > > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc] > > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff81035177>] ? default_wake_function+0x0/0xf > > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd] > > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd] > > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104af36>] kthread+0x7d/0x85 > > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c97a>] child_rip+0xa/0x20 > > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104aeb9>] ? kthread+0x0/0x85 > > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c970>] ? child_rip+0x0/0x20 > > > > Jan 1 22:40:08 t kernel: [507543.165025] Code: 00 49 8d 85 88 01 00 00 49 89 85 a0 09 00 00 8b 82 e8 00 00 00 8d 1c 85 00 00 00 00 2b 9a e0 00 00 00 eb 0c 49 8b 46 08 4c 89 ef <ff> 50 18 89 c3 f6 05 56 ce 00 00 01 74 10 89 de 48 c7 c7 cf b6 > > > > Jan 1 22:40:08 t kernel: [507543.165025] RIP [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc] > > > > Jan 1 22:40:08 t kernel: [507543.165025] RSP <ffff880124ea1e00> > > > > Jan 1 22:40:08 t kernel: [507544.713111] ---[ end trace 1a77408724500fb3 ]--- > > > > Jan 2 12:06:02 t kernel: [555898.365986] RPC: multiple fragments per record not supported > > > > Jan 2 12:06:02 t kernel: [555898.527050] RPC: multiple fragments per record not supported > > > > Jan 2 12:06:05 t kernel: [555901.700485] RPC: multiple fragments per record not supported > > > > Jan 2 12:06:08 t kernel: [555904.880825] RPC: multiple fragments per record not supported > > > > Jan 3 00:01:08 t kernel: [598804.533706] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0. > > > > Jan 3 00:57:01 t kernel: [602157.347383] md: data-check of RAID array md0 > > > > Jan 3 00:57:01 t kernel: [602157.350802] md: delaying data-check of md2 until md0 has finished (they share one or more physical units) > > > > Jan 3 00:57:01 t kernel: [602157.431453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. > > > > Jan 3 00:57:01 t kernel: [602157.467054] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. > > > > Jan 3 00:57:01 t kernel: [602157.526113] md: using 128k window, over a total of 19542976 blocks. > > > > Jan 3 01:03:47 t kernel: [602563.525129] md: md0: data-check done. > > > > Jan 3 01:03:47 t kernel: [602563.554886] md: data-check of RAID array md2 > > > > Jan 3 01:03:47 t kernel: [602563.581194] md: minimum _guaranteed_ speed: 1000 KB/sec/disk. > > > > Jan 3 01:03:47 t kernel: [602563.616879] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check. > > > > Jan 3 01:03:47 t kernel: [602563.675944] md: using 128k window, over a total of 460928832 blocks. > > > > Jan 3 04:47:09 t kernel: [615965.563359] md: md2: data-check done. > > > > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > -- > > ------------------------------------- > > Nikola CIPRICH > > LinuxBox.cz, s.r.o. > > 28. rijna 168, 709 01 Ostrava > > > > tel.: +420 596 603 142 > > fax: +420 596 621 273 > > mobil: +420 777 093 799 > > > > www.linuxbox.cz > > > > mobil servis: +420 737 238 656 > > email servis: servis@linuxbox.cz > > ------------------------------------- > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- ------------------------------------- Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis@linuxbox.cz ------------------------------------- ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <20100216202720.GB4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org>]
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() [not found] ` <20100216202720.GB4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org> @ 2010-02-16 20:29 ` J. Bruce Fields 2010-02-17 15:14 ` Nikola Ciprich 0 siblings, 1 reply; 12+ messages in thread From: J. Bruce Fields @ 2010-02-16 20:29 UTC (permalink / raw) To: Nikola Ciprich Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs, nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g On Tue, Feb 16, 2010 at 09:27:20PM +0100, Nikola Ciprich wrote: > > Nothing yet. I'm taking another look. > > > > Do you know what the most recent version was where you did *not* see > > these warnings? > well, actually I decided I'll bisect this one no matter what, so hopefully > we'll see soon :) > I'll report That would be fantastic, thanks. What test-case are you using? Any partial information ("xxxx good, xxxx bad") would also be useful. --b. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() 2010-02-16 20:29 ` J. Bruce Fields @ 2010-02-17 15:14 ` Nikola Ciprich [not found] ` <20100217151435.GC30897-K8br+RTdgN68z6Q06MQEFCF40CQGThNF@public.gmane.org> 0 siblings, 1 reply; 12+ messages in thread From: Nikola Ciprich @ 2010-02-17 15:14 UTC (permalink / raw) To: J. Bruce Fields Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs, nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g Well, I did a lot of stress tests, and it seems to be pretty random, and what's worse, I think it is more then one problem, oh great... Seems that at least one of my NFS problems was in fact caused by ext4 regression (what FS are you using?) applying this one: http://bugzilla.kernel.org/attachment.cgi?id=24975 improved things for me a lot (see http://bugzilla.kernel.org/show_bug.cgi?id=15231 for full discussion). I'll watch the machine for next few days and report... regards nik On Tue, Feb 16, 2010 at 03:29:47PM -0500, J. Bruce Fields wrote: > On Tue, Feb 16, 2010 at 09:27:20PM +0100, Nikola Ciprich wrote: > > > Nothing yet. I'm taking another look. > > > > > > Do you know what the most recent version was where you did *not* see > > > these warnings? > > well, actually I decided I'll bisect this one no matter what, so hopefully > > we'll see soon :) > > I'll report > > That would be fantastic, thanks. What test-case are you using? > > Any partial information ("xxxx good, xxxx bad") would also be useful. > > --b. > -- > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis-Jp3n8lUXroTtwjQa/ONI9g@public.gmane.org ------------------------------------- ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <20100217151435.GC30897-K8br+RTdgN68z6Q06MQEFCF40CQGThNF@public.gmane.org>]
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() [not found] ` <20100217151435.GC30897-K8br+RTdgN68z6Q06MQEFCF40CQGThNF@public.gmane.org> @ 2010-02-20 23:51 ` J. Bruce Fields 2010-02-22 5:37 ` Nikola Ciprich 2010-02-24 7:19 ` Jesper Krogh 0 siblings, 2 replies; 12+ messages in thread From: J. Bruce Fields @ 2010-02-20 23:51 UTC (permalink / raw) To: Nikola Ciprich Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs, nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g On Wed, Feb 17, 2010 at 04:14:35PM +0100, Nikola Ciprich wrote: > Well, > I did a lot of stress tests, and it seems to be pretty random, and what's > worse, I think it is more then one problem, oh great... > Seems that at least one of my NFS problems was in fact caused by ext4 regression > (what FS are you using?) > applying this one: http://bugzilla.kernel.org/attachment.cgi?id=24975 > improved things for me a lot (see http://bugzilla.kernel.org/show_bug.cgi?id=15231 for > full discussion). > I'll watch the machine for next few days and report... Is there *any* kernel (however old) on which you are confident you can *not* reproduce the kref bug? --b. > regards > nik > > On Tue, Feb 16, 2010 at 03:29:47PM -0500, J. Bruce Fields wrote: > > On Tue, Feb 16, 2010 at 09:27:20PM +0100, Nikola Ciprich wrote: > > > > Nothing yet. I'm taking another look. > > > > > > > > Do you know what the most recent version was where you did *not* see > > > > these warnings? > > > well, actually I decided I'll bisect this one no matter what, so hopefully > > > we'll see soon :) > > > I'll report > > > > That would be fantastic, thanks. What test-case are you using? > > > > Any partial information ("xxxx good, xxxx bad") would also be useful. > > > > --b. > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > -- > ------------------------------------- > Ing. Nikola CIPRICH > LinuxBox.cz, s.r.o. > 28. rijna 168, 709 01 Ostrava > > tel.: +420 596 603 142 > fax: +420 596 621 273 > mobil: +420 777 093 799 > www.linuxbox.cz > > mobil servis: +420 737 238 656 > email servis: servis-Jp3n8lUXroTtwjQa/ONI9g@public.gmane.org > ------------------------------------- ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() 2010-02-20 23:51 ` J. Bruce Fields @ 2010-02-22 5:37 ` Nikola Ciprich 2010-02-24 7:19 ` Jesper Krogh 1 sibling, 0 replies; 12+ messages in thread From: Nikola Ciprich @ 2010-02-22 5:37 UTC (permalink / raw) To: J. Bruce Fields Cc: Nikola Ciprich, Harry Edmon, Andrew Morton, linux-kernel, linux-nfs [-- Attachment #1: Type: text/plain, Size: 2758 bytes --] It was working pretty well when we were using 2.6.27.x, but after upgrade, I also changed FS from XFS to ext4, (so I can't easily test with 2.6.27 now) and since then NFS is almost unusable on this machine. But 2.6.27 is pretty old, so there's been a lot of changes. But greg released 2.6.32.9-rc1 which contains bunch of NFS-related fixes, so I'll give it a try and report. nik On Sat, Feb 20, 2010 at 06:51:05PM -0500, J. Bruce Fields wrote: > On Wed, Feb 17, 2010 at 04:14:35PM +0100, Nikola Ciprich wrote: > > Well, > > I did a lot of stress tests, and it seems to be pretty random, and what's > > worse, I think it is more then one problem, oh great... > > Seems that at least one of my NFS problems was in fact caused by ext4 regression > > (what FS are you using?) > > applying this one: http://bugzilla.kernel.org/attachment.cgi?id=24975 > > improved things for me a lot (see http://bugzilla.kernel.org/show_bug.cgi?id=15231 for > > full discussion). > > I'll watch the machine for next few days and report... > > Is there *any* kernel (however old) on which you are confident you can > *not* reproduce the kref bug? > > --b. > > > regards > > nik > > > > On Tue, Feb 16, 2010 at 03:29:47PM -0500, J. Bruce Fields wrote: > > > On Tue, Feb 16, 2010 at 09:27:20PM +0100, Nikola Ciprich wrote: > > > > > Nothing yet. I'm taking another look. > > > > > > > > > > Do you know what the most recent version was where you did *not* see > > > > > these warnings? > > > > well, actually I decided I'll bisect this one no matter what, so hopefully > > > > we'll see soon :) > > > > I'll report > > > > > > That would be fantastic, thanks. What test-case are you using? > > > > > > Any partial information ("xxxx good, xxxx bad") would also be useful. > > > > > > --b. > > > -- > > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in > > > the body of a message to majordomo@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > -- > > ------------------------------------- > > Ing. Nikola CIPRICH > > LinuxBox.cz, s.r.o. > > 28. rijna 168, 709 01 Ostrava > > > > tel.: +420 596 603 142 > > fax: +420 596 621 273 > > mobil: +420 777 093 799 > > www.linuxbox.cz > > > > mobil servis: +420 737 238 656 > > email servis: servis@linuxbox.cz > > ------------------------------------- > -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28. rijna 168, 709 01 Ostrava tel.: +420 596 603 142 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis@linuxbox.cz ------------------------------------- [-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() 2010-02-20 23:51 ` J. Bruce Fields 2010-02-22 5:37 ` Nikola Ciprich @ 2010-02-24 7:19 ` Jesper Krogh 1 sibling, 0 replies; 12+ messages in thread From: Jesper Krogh @ 2010-02-24 7:19 UTC (permalink / raw) To: J. Bruce Fields Cc: Nikola Ciprich, Harry Edmon, Andrew Morton, public-linux-kernel-u79uwXL29TY76Z2rM5mHXA-wOFGN7rlS/M9smdsby/KFg, public-linux-nfs-u79uwXL29TY76Z2rM5mHXA-wOFGN7rlS/M9smdsby/KFg, public-nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g-wOFGN7rlS/M9smdsby/KFg J. Bruce Fields wrote: > On Wed, Feb 17, 2010 at 04:14:35PM +0100, Nikola Ciprich wrote: >> Well, >> I did a lot of stress tests, and it seems to be pretty random, and what's >> worse, I think it is more then one problem, oh great... >> Seems that at least one of my NFS problems was in fact caused by ext4 regression >> (what FS are you using?) >> applying this one: http://bugzilla.kernel.org/attachment.cgi?id=24975 >> improved things for me a lot (see http://bugzilla.kernel.org/show_bug.cgi?id=15231 for >> full discussion). >> I'll watch the machine for next few days and report... > > Is there *any* kernel (however old) on which you are confident you can > *not* reproduce the kref bug? I have got a bug that looks a lot like this on 2.6.32.8 yesterday less than 24 hours after booting the first 2.6.32 kernel but we have been running on 2.6.31 for 3 months before, so I'm fairly confident that it is a .31 -> .32 regression. http://lkml.org/lkml/2010/2/23/70 Jesper -- Jesper ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2010-02-24 8:21 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <4B4F5A00.4060604@atmos.washington.edu>
[not found] ` <4B4F5A00.4060604-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org>
2010-01-22 23:03 ` 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() Andrew Morton
2010-01-27 19:05 ` J. Bruce Fields
2010-01-27 23:01 ` Harry Edmon
2010-02-02 16:22 ` J. Bruce Fields
2010-02-16 18:22 ` Nikola Ciprich
[not found] ` <20100216182248.GA4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org>
2010-02-16 20:21 ` J. Bruce Fields
2010-02-16 20:27 ` Nikola Ciprich
[not found] ` <20100216202720.GB4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org>
2010-02-16 20:29 ` J. Bruce Fields
2010-02-17 15:14 ` Nikola Ciprich
[not found] ` <20100217151435.GC30897-K8br+RTdgN68z6Q06MQEFCF40CQGThNF@public.gmane.org>
2010-02-20 23:51 ` J. Bruce Fields
2010-02-22 5:37 ` Nikola Ciprich
2010-02-24 7:19 ` Jesper Krogh
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox