* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
[not found] ` <4B4F5A00.4060604-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org>
@ 2010-01-22 23:03 ` Andrew Morton
2010-01-27 19:05 ` J. Bruce Fields
0 siblings, 1 reply; 12+ messages in thread
From: Andrew Morton @ 2010-01-22 23:03 UTC (permalink / raw)
To: Harry Edmon; +Cc: linux-kernel, linux-nfs
(cc linux-nfs)
On Thu, 14 Jan 2010 09:53:04 -0800
Harry Edmon <harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org> wrote:
> I have a single quad-core Xeon machine running 2.6.32.2 that keeps
> giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This
> machine is mainaly used as a NFS server. I have attached the kern.log
> and my config file. Any ideas?
> --
>
> Dr. Harry Edmon E-MAIL: harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org
> 206-543-0547 FAX: 206-543-0308 harry-B93hV6UPU7Z2icitjWtXSw@public.gmane.org, harry@uw.edu
> Director of IT, College of the Environment and
> Director of Computing, Dept of Atmospheric Sciences
> University of Washington, Box 351640, Seattle, WA 98195-1640
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
2010-01-22 23:03 ` 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() Andrew Morton
@ 2010-01-27 19:05 ` J. Bruce Fields
2010-01-27 23:01 ` Harry Edmon
0 siblings, 1 reply; 12+ messages in thread
From: J. Bruce Fields @ 2010-01-27 19:05 UTC (permalink / raw)
To: Andrew Morton; +Cc: Harry Edmon, linux-kernel, linux-nfs
On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote:
> (cc linux-nfs)
>
> On Thu, 14 Jan 2010 09:53:04 -0800
> Harry Edmon <harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org> wrote:
>
> > I have a single quad-core Xeon machine running 2.6.32.2 that keeps
> > giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This
> > machine is mainaly used as a NFS server. I have attached the kern.log
> > and my config file. Any ideas?
Are you using NFSv4, or only v2/v3? Looking at the trace:
Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------
Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
OK, so that's the warning that we're trying to get something with
reference count 0.
Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8
Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1
What does the very first oops, bug, or warning you get look like? (If
I'm reading the tain information right, this isn't the first one.)
Jan 11 06:01:53 t kernel: [1311647.216100] Call Trace:
Jan 11 06:01:53 t kernel: [1311647.231808] [<ffffffff81142230>] ? kref_get+0x23/0x2b
Jan 11 06:01:53 t kernel: [1311647.263633] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
Jan 11 06:01:53 t kernel: [1311647.300658] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
Jan 11 06:01:53 t kernel: [1311647.336122] [<ffffffff81142230>] kref_get+0x23/0x2b
Jan 11 06:01:53 t kernel: [1311647.366914] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
Jan 11 06:01:53 t kernel: [1311647.400301] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
And the only likely candidate I see for a kref is the
svc_xprt_get(xprt).
--b.
Jan 11 06:01:53 t kernel: [1311647.436803] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
Jan 11 06:01:53 t kernel: [1311647.475903] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
Jan 11 06:01:53 t kernel: [1311647.513452] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
Jan 11 06:01:53 t kernel: [1311647.546838] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
Jan 11 06:01:53 t kernel: [1311647.579701] [<ffffffff8104af36>] kthread+0x7d/0x85
Jan 11 06:01:53 t kernel: [1311647.609967] [<ffffffff8100c97a>] child_rip+0xa/0x20
Jan 11 06:01:53 t kernel: [1311647.640749] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
Jan 11 06:01:53 t kernel: [1311647.671535] [<ffffffff8100c970>] ? child_rip+0x0/0x20
Jan 11 06:01:53 t kernel: [1311647.703359] ---[ end trace 1a77408724500fc0 ]---
Jan 11 06:01:53 t kernel: [1311647.732134] ------------[ cut here ]------------
Jan 11 06:01:53 t kernel: [1311647.736063] kernel BUG at fs/inode.c:1343!
Jan 11 06:01:53 t kernel: [1311647.736063] invalid opcode: 0000 [#10] SMP
Jan 11 06:01:53 t kernel: [1311647.736063] last sysfs file: /sys/block/md2/md/raid_disks
Jan 11 06:01:53 t kernel: [1311647.736063] CPU 3
Jan 11 06:01:53 t kernel: [1311647.736063] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
Jan 11 06:01:53 t kernel: [1311647.736063] Pid: 3068, comm: nfsd
Tainted: G D W 2.6.32.2 #1 X7DB8
Jan 11 06:01:53 t kernel: [1311647.736063] RIP:
0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65
Jan 11 06:01:53 t kernel: [1311647.736063] RSP: 0018:ffff88012730bd20 EFLAGS: 00010246
Jan 11 06:01:53 t kernel: [1311647.736063] RAX: 0000000000000000 RBX: ffff88010c9c7408 RCX: ffffffff8139f927
Jan 11 06:01:53 t kernel: [1311647.736063] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88010c9c7408
Jan 11 06:01:53 t kernel: [1311647.736063] RBP: ffff88012730bd30 R08: 0000000000000020 R09: ffff88002830ca70
Jan 11 06:01:53 t kernel: [1311647.736063] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff88010c9c73c0
Jan 11 06:01:53 t kernel: [1311647.736063] R13: ffffffffa028ef50 R14: ffff880018c58000 R15: ffff88012dd423c0
Jan 11 06:01:53 t kernel: [1311647.736063] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000
Jan 11 06:01:53 t kernel: [1311647.736063] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Jan 11 06:01:53 t kernel: [1311647.736063] CR2: 00007f0dc9cc5098 CR3: 0000000005bd8000 CR4: 00000000000006e0
Jan 11 06:01:53 t kernel: [1311647.736063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 11 06:01:53 t kernel: [1311647.736063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 11 06:01:53 t kernel: [1311647.736063] Process nfsd (pid: 3068, threadinfo ffff88012730a000, task ffff8801270e8150)
Jan 11 06:01:53 t kernel: [1311647.736063] Stack:
Jan 11 06:01:53 t kernel: [1311647.736063] ffff88012730bd90 ffff880018c58000 ffff88012730bd50 ffffffff811f4be2
Jan 11 06:01:53 t kernel: [1311647.736063] <0> ffff880018c58000 ffff880018c58010 ffff88012730bd70 ffffffffa027c1d2
Jan 11 06:01:53 t kernel: [1311647.736063] <0> ffff88012730bd70 ffff880018c58000 ffff88012730bda0 ffffffffa0286b03
Jan 11 06:01:53 t kernel: [1311647.736063] Call Trace:
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff811f4be2>] sock_release+0x5c/0x6c
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc]
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc]
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff81142230>] ? kref_get+0x23/0x2b
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc]
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff81142201>] kref_put+0x43/0x4f
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc]
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc]
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff8104af36>] kthread+0x7d/0x85
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff8100c97a>] child_rip+0xa/0x20
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
Jan 11 06:01:53 t kernel: [1311647.736063] [<ffffffff8100c970>] ? child_rip+0x0/0x20
Jan 11 06:01:53 t kernel: [1311647.736063] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85
Jan 11 06:01:53 t kernel: [1311647.736063] RIP [<ffffffff810bccc2>] iput+0x1b/0x65
Jan 11 06:01:53 t kernel: [1311647.736063] RSP <ffff88012730bd20>
Jan 11 06:01:53 t kernel: [1311649.618340] ---[ end trace 1a77408724500fc1 ]---
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
2010-01-27 19:05 ` J. Bruce Fields
@ 2010-01-27 23:01 ` Harry Edmon
2010-02-02 16:22 ` J. Bruce Fields
0 siblings, 1 reply; 12+ messages in thread
From: Harry Edmon @ 2010-01-27 23:01 UTC (permalink / raw)
To: J. Bruce Fields; +Cc: Andrew Morton, linux-kernel, linux-nfs
[-- Attachment #1: Type: text/plain, Size: 2024 bytes --]
J. Bruce Fields wrote:
> On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote:
>
>> (cc linux-nfs)
>>
>> On Thu, 14 Jan 2010 09:53:04 -0800
>> Harry Edmon <harry@atmos.washington.edu> wrote:
>>
>>
>>> I have a single quad-core Xeon machine running 2.6.32.2 that keeps
>>> giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This
>>> machine is mainaly used as a NFS server. I have attached the kern.log
>>> and my config file. Any ideas?
>>>
>
> Are you using NFSv4, or only v2/v3? Looking at the trace:
>
> Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------
> Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
>
NFSv4 and NFSv3.
> OK, so that's the warning that we're trying to get something with
> reference count 0.
>
> Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8
> Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1
>
> What does the very first oops, bug, or warning you get look like? (If
> I'm reading the tain information right, this isn't the first one.)
>
Attached is the kern.log that contains the first warning.
--
Dr. Harry Edmon E-MAIL: harry@atmos.washington.edu
206-543-0547 FAX: 206-543-0308 harry@washington.edu, harry@uw.edu
Director of IT, College of the Environment and
Director of Computing, Dept of Atmospheric Sciences
University of Washington, Box 351640, Seattle, WA 98195-1640
[-- Attachment #2: kern.log.3 --]
[-- Type: text/plain, Size: 25606 bytes --]
Dec 27 11:57:44 t kernel: [37000.838904] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0055): Battery charging started:.
Dec 27 11:57:46 t kernel: [37002.028946] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0056): Battery charging completed:.
Dec 28 00:00:51 t kernel: [80387.429120] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0.
Dec 28 03:32:07 t kernel: [93063.950486] 3w-9xxx: scsi6: AEN: INFO (0x04:0x002B): Verify completed:unit=0.
Dec 29 17:04:43 t kernel: [228218.883443] ------------[ cut here ]------------
Dec 29 17:04:45 t kernel: [228218.911691] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
Dec 29 17:04:45 t kernel: [228218.945083] Hardware name: X7DB8
Dec 29 17:04:45 t kernel: [228218.964948] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
Dec 29 17:04:45 t kernel: [228219.227296] Pid: 3108, comm: nfsd Not tainted 2.6.32.2 #1
Dec 29 17:04:45 t kernel: [228219.260164] Call Trace:
Dec 29 17:04:45 t kernel: [228219.275350] [<ffffffff81142230>] ? kref_get+0x23/0x2b
Dec 29 17:04:45 t kernel: [228219.306656] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
Dec 29 17:04:45 t kernel: [228219.343160] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
Dec 29 17:04:45 t kernel: [228219.378105] [<ffffffff81142230>] kref_get+0x23/0x2b
Dec 29 17:04:45 t kernel: [228219.408379] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
Dec 29 17:04:45 t kernel: [228219.441242] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
Dec 29 17:04:45 t kernel: [228219.477225] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
Dec 29 17:04:45 t kernel: [228219.515806] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
Dec 29 17:04:45 t kernel: [228219.552834] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
Dec 29 17:04:45 t kernel: [228219.585698] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
Dec 29 17:04:45 t kernel: [228219.618043] [<ffffffff8104af36>] kthread+0x7d/0x85
Dec 29 17:04:45 t kernel: [228219.647789] [<ffffffff8100c97a>] child_rip+0xa/0x20
Dec 29 17:04:45 t kernel: [228219.678052] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
Dec 29 17:04:45 t kernel: [228219.708317] [<ffffffff8100c970>] ? child_rip+0x0/0x20
Dec 29 17:04:45 t kernel: [228219.739621] ---[ end trace 1a77408724500fae ]---
Dec 29 17:04:45 t kernel: [228219.767925] ------------[ cut here ]------------
Dec 29 17:04:45 t kernel: [228219.792730] kernel BUG at fs/inode.c:1343!
Dec 29 17:04:45 t kernel: [228219.792730] invalid opcode: 0000 [#1] SMP
Dec 29 17:04:45 t kernel: [228219.792730] last sysfs file: /sys/class/scsi_host/host6/stats
Dec 29 17:04:45 t kernel: [228219.792730] CPU 0
Dec 29 17:04:45 t kernel: [228219.792730] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
Dec 29 17:04:45 t kernel: [228219.792730] Pid: 3108, comm: nfsd Tainted: G W 2.6.32.2 #1 X7DB8
Dec 29 17:04:45 t kernel: [228219.792730] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65
Dec 29 17:04:45 t kernel: [228219.792730] RSP: 0018:ffff88012588bd20 EFLAGS: 00010246
Dec 29 17:04:45 t kernel: [228219.792730] RAX: 0000000000000000 RBX: ffff88003c3c2cc8 RCX: ffffffff8139f927
Dec 29 17:04:45 t kernel: [228219.792730] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88003c3c2cc8
Dec 29 17:04:45 t kernel: [228219.792730] RBP: ffff88012588bd30 R08: 0000000000000002 R09: 0000000000000000
Dec 29 17:04:45 t kernel: [228219.792730] R10: ffffffff81142230 R11: 0000000100000001 R12: ffff88003c3c2c80
Dec 29 17:04:45 t kernel: [228219.792730] R13: ffffffffa028ef50 R14: ffff88010072d400 R15: ffff88012dd423c0
Dec 29 17:04:45 t kernel: [228219.792730] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
Dec 29 17:04:45 t kernel: [228219.792730] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Dec 29 17:04:45 t kernel: [228219.792730] CR2: 00007fad86c2e000 CR3: 00000000305e5000 CR4: 00000000000006f0
Dec 29 17:04:45 t kernel: [228219.792730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 29 17:04:45 t kernel: [228219.792730] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec 29 17:04:45 t kernel: [228219.792730] Process nfsd (pid: 3108, threadinfo ffff88012588a000, task ffff880125e66150)
Dec 29 17:04:45 t kernel: [228219.792730] Stack:
Dec 29 17:04:45 t kernel: [228219.792730] ffff88012588bd90 ffff88010072d400 ffff88012588bd50 ffffffff811f4be2
Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88010072d400 ffff88010072d410 ffff88012588bd70 ffffffffa027c1d2
Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88012588bd70 ffff88010072d400 ffff88012588bda0 ffffffffa0286b03
Dec 29 17:04:45 t kernel: [228219.792730] Call Trace:
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff811f4be2>] sock_release+0x5c/0x6c
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc]
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc]
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142230>] ? kref_get+0x23/0x2b
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc]
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142201>] kref_put+0x43/0x4f
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc]
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc]
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104af36>] kthread+0x7d/0x85
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c97a>] child_rip+0xa/0x20
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c970>] ? child_rip+0x0/0x20
Dec 29 17:04:45 t kernel: [228219.792730] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85
Dec 29 17:04:45 t kernel: [228219.792730] RIP [<ffffffff810bccc2>] iput+0x1b/0x65
Dec 29 17:04:45 t kernel: [228219.792730] RSP <ffff88012588bd20>
Dec 29 17:04:45 t kernel: [228221.632694] ---[ end trace 1a77408724500faf ]---
Dec 29 20:53:01 t kernel: [241917.056374] ------------[ cut here ]------------
Dec 29 20:53:01 t kernel: [241917.084581] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
Dec 29 20:53:01 t kernel: [241917.117941] Hardware name: X7DB8
Dec 29 20:53:01 t kernel: [241917.137804] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
Dec 29 20:53:01 t kernel: [241917.400101] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1
Dec 29 20:53:01 t kernel: [241917.437653] Call Trace:
Dec 29 20:53:01 t kernel: [241917.452843] [<ffffffffa0286b0b>] ? svc_xprt_free+0x46/0x4e [sunrpc]
Dec 29 20:53:01 t kernel: [241917.491420] [<ffffffff81142230>] ? kref_get+0x23/0x2b
Dec 29 20:53:01 t kernel: [241917.522728] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
Dec 29 20:53:01 t kernel: [241917.559230] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
Dec 29 20:53:01 t kernel: [241917.594177] [<ffffffff81142230>] kref_get+0x23/0x2b
Dec 29 20:53:01 t kernel: [241917.624446] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
Dec 29 20:53:01 t kernel: [241917.660432] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
Dec 29 20:53:01 t kernel: [241917.699013] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
Dec 29 20:53:01 t kernel: [241917.736041] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
Dec 29 20:53:01 t kernel: [241917.768905] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
Dec 29 20:53:01 t kernel: [241917.801249] [<ffffffff8104af36>] kthread+0x7d/0x85
Dec 29 20:53:01 t kernel: [241917.830993] [<ffffffff8100c97a>] child_rip+0xa/0x20
Dec 29 20:53:01 t kernel: [241917.861258] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
Dec 29 20:53:01 t kernel: [241917.891522] [<ffffffff8100c970>] ? child_rip+0x0/0x20
Dec 29 20:53:01 t kernel: [241917.922827] ---[ end trace 1a77408724500fb0 ]---
Dec 29 20:53:01 t kernel: [241917.951084] ------------[ cut here ]------------
Dec 29 20:53:03 t kernel: [241917.974874] kernel BUG at fs/inode.c:1343!
Dec 29 20:53:03 t kernel: [241917.988010] invalid opcode: 0000 [#2] SMP
Dec 29 20:53:03 t kernel: [241917.988010] last sysfs file: /sys/class/scsi_host/host6/stats
Dec 29 20:53:03 t kernel: [241917.988010] CPU 1
Dec 29 20:53:03 t kernel: [241917.988010] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
Dec 29 20:53:03 t kernel: [241918.102960] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8
Dec 29 20:53:03 t kernel: [241918.102960] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65
Dec 29 20:53:03 t kernel: [241918.102960] RSP: 0000:ffff8801225b9d20 EFLAGS: 00010246
Dec 29 20:53:03 t kernel: [241918.102960] RAX: 0000000000000000 RBX: ffff880120435148 RCX: ffffffff8139f927
Dec 29 20:53:03 t kernel: [241918.102960] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff880120435148
Dec 29 20:53:03 t kernel: [241918.102960] RBP: ffff8801225b9d30 R08: 0000000000000020 R09: ffff88002820c9d0
Dec 29 20:53:03 t kernel: [241918.102960] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff880120435100
Dec 29 20:53:03 t kernel: [241918.102960] R13: ffffffffa028ef50 R14: ffff8800c1b3b800 R15: ffff88012dd423c0
Dec 29 20:53:03 t kernel: [241918.102960] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
Dec 29 20:53:03 t kernel: [241918.102960] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Dec 29 20:53:03 t kernel: [241918.102960] CR2: 00007fad86c2e000 CR3: 0000000129ae6000 CR4: 00000000000006e0
Dec 29 20:53:03 t kernel: [241918.102960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 29 20:53:03 t kernel: [241918.102960] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec 29 20:53:03 t kernel: [241918.102960] Process nfsd (pid: 3207, threadinfo ffff8801225b8000, task ffff880122416640)
Dec 29 20:53:03 t kernel: [241918.102960] Stack:
Dec 29 20:53:03 t kernel: [241918.102960] ffff8801225b9d90 ffff8800c1b3b800 ffff8801225b9d50 ffffffff811f4be2
Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8800c1b3b800 ffff8800c1b3b810 ffff8801225b9d70 ffffffffa027c1d2
Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8801225b9d70 ffff8800c1b3b800 ffff8801225b9da0 ffffffffa0286b03
Dec 29 20:53:03 t kernel: [241918.102960] Call Trace:
Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff811f4be2>] sock_release+0x5c/0x6c
Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc]
Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc]
Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff81142230>] ? kref_get+0x23/0x2b
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc]
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81142201>] kref_put+0x43/0x4f
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc]
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc]
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104af36>] kthread+0x7d/0x85
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c97a>] child_rip+0xa/0x20
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c970>] ? child_rip+0x0/0x20
Dec 29 20:53:04 t kernel: [241918.102960] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85
Dec 29 20:53:04 t kernel: [241918.102960] RIP [<ffffffff810bccc2>] iput+0x1b/0x65
Dec 29 20:53:04 t kernel: [241918.102960] RSP <ffff8801225b9d20>
Dec 29 20:53:04 t kernel: [241919.822190] ---[ end trace 1a77408724500fb1 ]---
Dec 30 11:22:26 t kernel: [294082.152979] 3w-9xxx: scsi6: AEN: ERROR (0x04:0x0009): Drive timeout detected:port=0.
Dec 31 11:24:44 t kernel: [380620.428029] br0: port 2(eth2) entering disabled state
Dec 31 11:24:44 t kernel: [380620.458826] br0: port 1(eth1) entering disabled state
Dec 31 11:24:44 t kernel: [380620.666738] device eth1 left promiscuous mode
Dec 31 11:24:44 t kernel: [380620.693361] br0: port 1(eth1) entering disabled state
Dec 31 11:24:44 t kernel: [380620.909921] device eth2 left promiscuous mode
Dec 31 11:24:44 t kernel: [380620.936603] br0: port 2(eth2) entering disabled state
Dec 31 11:24:48 t kernel: [380624.449637] device eth1 entered promiscuous mode
Dec 31 11:24:48 t kernel: [380624.533887] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X
Dec 31 11:24:48 t kernel: [380624.620065] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X
Dec 31 11:24:48 t kernel: [380624.651581] ADDRCONF(NETDEV_UP): eth1: link is not ready
Dec 31 11:24:48 t kernel: [380624.687675] device eth2 entered promiscuous mode
Dec 31 11:24:48 t kernel: [380624.860445] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X
Dec 31 11:24:48 t kernel: [380624.944084] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X
Dec 31 11:24:48 t kernel: [380624.975824] ADDRCONF(NETDEV_UP): eth2: link is not ready
Dec 31 11:24:50 t kernel: [380626.984998] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Dec 31 11:24:51 t kernel: [380627.031517] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
Dec 31 11:24:51 t kernel: [380627.066844] br0: port 2(eth2) entering forwarding state
Dec 31 11:24:51 t kernel: [380627.251584] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Dec 31 11:24:51 t kernel: [380627.300433] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Dec 31 11:24:51 t kernel: [380627.335625] br0: port 1(eth1) entering forwarding state
Dec 31 11:24:59 t kernel: [380635.148016] br0: no IPv6 routers present
Dec 31 11:25:01 t kernel: [380637.504511] eth1: no IPv6 routers present
Dec 31 11:25:01 t kernel: [380637.720515] eth2: no IPv6 routers present
Dec 31 11:28:30 t kernel: [380846.205549] e1000e: eth1 NIC Link is Down
Dec 31 11:28:30 t kernel: [380846.248659] br0: port 1(eth1) entering disabled state
Dec 31 11:28:32 t kernel: [380848.693501] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX
Dec 31 11:28:32 t kernel: [380848.739167] 0000:06:00.1: eth1: 10/100 speed: disabling TSO
Dec 31 11:28:32 t kernel: [380848.776396] br0: port 1(eth1) entering forwarding state
Dec 31 11:28:49 t kernel: [380865.803038] e1000e: eth1 NIC Link is Down
Dec 31 11:28:49 t kernel: [380865.874086] br0: port 1(eth1) entering disabled state
Dec 31 11:28:52 t kernel: [380868.267943] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Dec 31 11:28:52 t kernel: [380868.316772] br0: port 1(eth1) entering forwarding state
Dec 31 11:29:18 t kernel: [380894.253543] e1000e: eth1 NIC Link is Down
Dec 31 11:29:18 t kernel: [380894.293597] br0: port 1(eth1) entering disabled state
Dec 31 11:29:20 t kernel: [380896.688442] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Dec 31 11:29:20 t kernel: [380896.737255] br0: port 1(eth1) entering forwarding state
Dec 31 11:29:25 t kernel: [380901.493543] e1000e: eth1 NIC Link is Down
Dec 31 11:29:25 t kernel: [380901.534074] br0: port 1(eth1) entering disabled state
Dec 31 11:29:28 t kernel: [380904.161008] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Dec 31 11:29:28 t kernel: [380904.209842] br0: port 1(eth1) entering forwarding state
Jan 1 22:40:06 t kernel: [507542.268700] ------------[ cut here ]------------
Jan 1 22:40:08 t kernel: [507542.296925] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
Jan 1 22:40:08 t kernel: [507542.330301] Hardware name: X7DB8
Jan 1 22:40:08 t kernel: [507542.350163] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
Jan 1 22:40:08 t kernel: [507542.614635] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1
Jan 1 22:40:08 t kernel: [507542.652191] Call Trace:
Jan 1 22:40:08 t kernel: [507542.667375] [<ffffffff81142230>] ? kref_get+0x23/0x2b
Jan 1 22:40:08 t kernel: [507542.698681] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
Jan 1 22:40:08 t kernel: [507542.735186] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
Jan 1 22:40:08 t kernel: [507542.770130] [<ffffffff81142230>] kref_get+0x23/0xBj
Jan 1 22:40:08 t kernel: [507542.800400] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
Jan 1 22:40:08 t kernel: [507542.833269] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
Jan 1 22:40:08 t kernel: [507542.869251] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
Jan 1 22:40:08 t kernel: [507542.907825] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
Jan 1 22:40:08 t kernel: [507542.944853] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
Jan 1 22:40:08 t kernel: [507542.977720] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
Jan 1 22:40:08 t kernel: [507543.010063] [<ffffffff8104af36>] kthread+0x7d/0x85
Jan 1 22:40:08 t kernel: [507543.039808] [<ffffffff8100c97a>] child_rip+0xa/0x20
Jan 1 22:40:08 t kernel: [507543.070071] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
Jan 1 22:40:08 t kernel: [507543.100337] [<ffffffff8100c970>] ? child_rip+0x0/0x20
Jan 1 22:40:08 t kernel: [507543.131640] ---[ end trace 1a77408724500fb2 ]---
Jan 1 22:40:08 t kernel: [507543.160026] general protection fault: 0000 [#3] SMP
Jan 1 22:40:08 t kernel: [507543.165025] last sysfs file: /sys/class/net/eth2/brport/hold_timer
Jan 1 22:40:08 t kernel: [507543.165025] CPU 3
Jan 1 22:40:08 t kernel: [507543.165025] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
Jan 1 22:40:08 t kernel: [507543.165025] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8
Jan 1 22:40:08 t kernel: [507543.165025] RIP: 0010:[<ffffffffa0286797>] [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc]
Jan 1 22:40:08 t kernel: [507543.165025] RSP: 0018:ffff880124ea1e00 EFLAGS: 00010246
Jan 1 22:40:08 t kernel: [507543.165025] RAX: 3a30343a32322031 RBX: 0000000000000000 RCX: 0000000000000000
Jan 1 22:40:08 t kernel: [507543.165025] RDX: 0000000000004c86 RSI: ffff880124ea1df0 RDI: ffff880124e00000
Jan 1 22:40:08 t kernel: [507543.165025] RBP: ffff880124ea1eb0 R08: 0000000000000020 R09: ffff88002830ca70
Jan 1 22:40:08 t kernel: [507543.165025] R10: ffff88002838cf40 R11: 0000000000000000 R12: 0000000000000082
Jan 1 22:40:08 t kernel: [507543.165025] R13: ffff880124e00000 R14: ffff8800c7444000 R15: ffff88012dd423c0
Jan 1 22:40:08 t kernel: [507543.165025] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000
Jan 1 22:40:08 t kernel: [507543.165025] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Jan 1 22:40:08 t kernel: [507543.165025] CR2: 00007f0dc9cc5098 CR3: 000000012c266000 CR4: 00000000000006e0
Jan 1 22:40:08 t kernel: [507543.165025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 1 22:40:08 t kernel: [507543.165025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan 1 22:40:08 t kernel: [507543.165025] Process nfsd (pid: 3134, threadinfo ffff880124ea0000, task ffff880124e9f890)
Jan 1 22:40:08 t kernel: [507543.165025] Stack:
Jan 1 22:40:08 t kernel: [507543.165025] ffff880124e00000 ffff880124e9f890 ffff880124ea1e40 00000000000dbba0
Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff88012dd423c4 ffff88012c0bea40 ffff880000000082 0000000000000001
Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff880124ea1eb0 ffffffffa027abe4 0000000000000000 ffff880124e9f890
Jan 1 22:40:08 t kernel: [507543.165025] Call Trace:
Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104af36>] kthread+0x7d/0x85
Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c97a>] child_rip+0xa/0x20
Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c970>] ? child_rip+0x0/0x20
Jan 1 22:40:08 t kernel: [507543.165025] Code: 00 49 8d 85 88 01 00 00 49 89 85 a0 09 00 00 8b 82 e8 00 00 00 8d 1c 85 00 00 00 00 2b 9a e0 00 00 00 eb 0c 49 8b 46 08 4c 89 ef <ff> 50 18 89 c3 f6 05 56 ce 00 00 01 74 10 89 de 48 c7 c7 cf b6
Jan 1 22:40:08 t kernel: [507543.165025] RIP [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc]
Jan 1 22:40:08 t kernel: [507543.165025] RSP <ffff880124ea1e00>
Jan 1 22:40:08 t kernel: [507544.713111] ---[ end trace 1a77408724500fb3 ]---
Jan 2 12:06:02 t kernel: [555898.365986] RPC: multiple fragments per record not supported
Jan 2 12:06:02 t kernel: [555898.527050] RPC: multiple fragments per record not supported
Jan 2 12:06:05 t kernel: [555901.700485] RPC: multiple fragments per record not supported
Jan 2 12:06:08 t kernel: [555904.880825] RPC: multiple fragments per record not supported
Jan 3 00:01:08 t kernel: [598804.533706] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0.
Jan 3 00:57:01 t kernel: [602157.347383] md: data-check of RAID array md0
Jan 3 00:57:01 t kernel: [602157.350802] md: delaying data-check of md2 until md0 has finished (they share one or more physical units)
Jan 3 00:57:01 t kernel: [602157.431453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Jan 3 00:57:01 t kernel: [602157.467054] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Jan 3 00:57:01 t kernel: [602157.526113] md: using 128k window, over a total of 19542976 blocks.
Jan 3 01:03:47 t kernel: [602563.525129] md: md0: data-check done.
Jan 3 01:03:47 t kernel: [602563.554886] md: data-check of RAID array md2
Jan 3 01:03:47 t kernel: [602563.581194] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
Jan 3 01:03:47 t kernel: [602563.616879] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
Jan 3 01:03:47 t kernel: [602563.675944] md: using 128k window, over a total of 460928832 blocks.
Jan 3 04:47:09 t kernel: [615965.563359] md: md2: data-check done.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
2010-01-27 23:01 ` Harry Edmon
@ 2010-02-02 16:22 ` J. Bruce Fields
2010-02-16 18:22 ` Nikola Ciprich
0 siblings, 1 reply; 12+ messages in thread
From: J. Bruce Fields @ 2010-02-02 16:22 UTC (permalink / raw)
To: Harry Edmon; +Cc: Andrew Morton, linux-kernel, linux-nfs
On Wed, Jan 27, 2010 at 03:01:40PM -0800, Harry Edmon wrote:
> J. Bruce Fields wrote:
>> On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote:
>>
>>> (cc linux-nfs)
>>>
>>> On Thu, 14 Jan 2010 09:53:04 -0800
>>> Harry Edmon <harry@atmos.washington.edu> wrote:
>>>
>>>
>>>> I have a single quad-core Xeon machine running 2.6.32.2 that keeps
>>>> giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This
>>>> machine is mainaly used as a NFS server. I have attached the
>>>> kern.log and my config file. Any ideas?
>>>>
>>
>> Are you using NFSv4, or only v2/v3? Looking at the trace:
>>
>> Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------
>> Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
>>
> NFSv4 and NFSv3.
Thanks. And you aren't using rdma are you?
>> OK, so that's the warning that we're trying to get something with
>> reference count 0.
>>
>> Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8
>> Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
>> Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1
>>
>> What does the very first oops, bug, or warning you get look like? (If
>> I'm reading the tain information right, this isn't the first one.)
>>
> Attached is the kern.log that contains the first warning.
Thanks. So yes, the first warning is a stray kref_put() in svc_recv(),
and the such put I see on a quick skim is a svc_xprt_put(). The
following BUG() is confirmation, since it's an attempt inside
scv-xprt_release() to clear an inode with I_CLEAR already set; so we
dropped the reference count to zero, cleared it, then tried to do
another get/put pair, resulting in an attempt to free the same socket
twice.
--b.
>
> --
> Dr. Harry Edmon E-MAIL: harry@atmos.washington.edu
> 206-543-0547 FAX: 206-543-0308 harry@washington.edu, harry@uw.edu
> Director of IT, College of the Environment and
> Director of Computing, Dept of Atmospheric Sciences
> University of Washington, Box 351640, Seattle, WA 98195-1640
>
> Dec 27 11:57:44 t kernel: [37000.838904] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0055): Battery charging started:.
> Dec 27 11:57:46 t kernel: [37002.028946] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0056): Battery charging completed:.
> Dec 28 00:00:51 t kernel: [80387.429120] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0.
> Dec 28 03:32:07 t kernel: [93063.950486] 3w-9xxx: scsi6: AEN: INFO (0x04:0x002B): Verify completed:unit=0.
> Dec 29 17:04:43 t kernel: [228218.883443] ------------[ cut here ]------------
> Dec 29 17:04:45 t kernel: [228218.911691] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> Dec 29 17:04:45 t kernel: [228218.945083] Hardware name: X7DB8
> Dec 29 17:04:45 t kernel: [228218.964948] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> Dec 29 17:04:45 t kernel: [228219.227296] Pid: 3108, comm: nfsd Not tainted 2.6.32.2 #1
> Dec 29 17:04:45 t kernel: [228219.260164] Call Trace:
> Dec 29 17:04:45 t kernel: [228219.275350] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> Dec 29 17:04:45 t kernel: [228219.306656] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> Dec 29 17:04:45 t kernel: [228219.343160] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> Dec 29 17:04:45 t kernel: [228219.378105] [<ffffffff81142230>] kref_get+0x23/0x2b
> Dec 29 17:04:45 t kernel: [228219.408379] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> Dec 29 17:04:45 t kernel: [228219.441242] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> Dec 29 17:04:45 t kernel: [228219.477225] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> Dec 29 17:04:45 t kernel: [228219.515806] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> Dec 29 17:04:45 t kernel: [228219.552834] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> Dec 29 17:04:45 t kernel: [228219.585698] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> Dec 29 17:04:45 t kernel: [228219.618043] [<ffffffff8104af36>] kthread+0x7d/0x85
> Dec 29 17:04:45 t kernel: [228219.647789] [<ffffffff8100c97a>] child_rip+0xa/0x20
> Dec 29 17:04:45 t kernel: [228219.678052] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> Dec 29 17:04:45 t kernel: [228219.708317] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> Dec 29 17:04:45 t kernel: [228219.739621] ---[ end trace 1a77408724500fae ]---
> Dec 29 17:04:45 t kernel: [228219.767925] ------------[ cut here ]------------
> Dec 29 17:04:45 t kernel: [228219.792730] kernel BUG at fs/inode.c:1343!
> Dec 29 17:04:45 t kernel: [228219.792730] invalid opcode: 0000 [#1] SMP
> Dec 29 17:04:45 t kernel: [228219.792730] last sysfs file: /sys/class/scsi_host/host6/stats
> Dec 29 17:04:45 t kernel: [228219.792730] CPU 0
> Dec 29 17:04:45 t kernel: [228219.792730] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> Dec 29 17:04:45 t kernel: [228219.792730] Pid: 3108, comm: nfsd Tainted: G W 2.6.32.2 #1 X7DB8
> Dec 29 17:04:45 t kernel: [228219.792730] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65
> Dec 29 17:04:45 t kernel: [228219.792730] RSP: 0018:ffff88012588bd20 EFLAGS: 00010246
> Dec 29 17:04:45 t kernel: [228219.792730] RAX: 0000000000000000 RBX: ffff88003c3c2cc8 RCX: ffffffff8139f927
> Dec 29 17:04:45 t kernel: [228219.792730] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88003c3c2cc8
> Dec 29 17:04:45 t kernel: [228219.792730] RBP: ffff88012588bd30 R08: 0000000000000002 R09: 0000000000000000
> Dec 29 17:04:45 t kernel: [228219.792730] R10: ffffffff81142230 R11: 0000000100000001 R12: ffff88003c3c2c80
> Dec 29 17:04:45 t kernel: [228219.792730] R13: ffffffffa028ef50 R14: ffff88010072d400 R15: ffff88012dd423c0
> Dec 29 17:04:45 t kernel: [228219.792730] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
> Dec 29 17:04:45 t kernel: [228219.792730] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> Dec 29 17:04:45 t kernel: [228219.792730] CR2: 00007fad86c2e000 CR3: 00000000305e5000 CR4: 00000000000006f0
> Dec 29 17:04:45 t kernel: [228219.792730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Dec 29 17:04:45 t kernel: [228219.792730] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Dec 29 17:04:45 t kernel: [228219.792730] Process nfsd (pid: 3108, threadinfo ffff88012588a000, task ffff880125e66150)
> Dec 29 17:04:45 t kernel: [228219.792730] Stack:
> Dec 29 17:04:45 t kernel: [228219.792730] ffff88012588bd90 ffff88010072d400 ffff88012588bd50 ffffffff811f4be2
> Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88010072d400 ffff88010072d410 ffff88012588bd70 ffffffffa027c1d2
> Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88012588bd70 ffff88010072d400 ffff88012588bda0 ffffffffa0286b03
> Dec 29 17:04:45 t kernel: [228219.792730] Call Trace:
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff811f4be2>] sock_release+0x5c/0x6c
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc]
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc]
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc]
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142201>] kref_put+0x43/0x4f
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc]
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc]
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104af36>] kthread+0x7d/0x85
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c97a>] child_rip+0xa/0x20
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> Dec 29 17:04:45 t kernel: [228219.792730] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85
> Dec 29 17:04:45 t kernel: [228219.792730] RIP [<ffffffff810bccc2>] iput+0x1b/0x65
> Dec 29 17:04:45 t kernel: [228219.792730] RSP <ffff88012588bd20>
> Dec 29 17:04:45 t kernel: [228221.632694] ---[ end trace 1a77408724500faf ]---
> Dec 29 20:53:01 t kernel: [241917.056374] ------------[ cut here ]------------
> Dec 29 20:53:01 t kernel: [241917.084581] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> Dec 29 20:53:01 t kernel: [241917.117941] Hardware name: X7DB8
> Dec 29 20:53:01 t kernel: [241917.137804] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> Dec 29 20:53:01 t kernel: [241917.400101] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1
> Dec 29 20:53:01 t kernel: [241917.437653] Call Trace:
> Dec 29 20:53:01 t kernel: [241917.452843] [<ffffffffa0286b0b>] ? svc_xprt_free+0x46/0x4e [sunrpc]
> Dec 29 20:53:01 t kernel: [241917.491420] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> Dec 29 20:53:01 t kernel: [241917.522728] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> Dec 29 20:53:01 t kernel: [241917.559230] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> Dec 29 20:53:01 t kernel: [241917.594177] [<ffffffff81142230>] kref_get+0x23/0x2b
> Dec 29 20:53:01 t kernel: [241917.624446] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> Dec 29 20:53:01 t kernel: [241917.660432] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> Dec 29 20:53:01 t kernel: [241917.699013] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> Dec 29 20:53:01 t kernel: [241917.736041] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> Dec 29 20:53:01 t kernel: [241917.768905] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> Dec 29 20:53:01 t kernel: [241917.801249] [<ffffffff8104af36>] kthread+0x7d/0x85
> Dec 29 20:53:01 t kernel: [241917.830993] [<ffffffff8100c97a>] child_rip+0xa/0x20
> Dec 29 20:53:01 t kernel: [241917.861258] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> Dec 29 20:53:01 t kernel: [241917.891522] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> Dec 29 20:53:01 t kernel: [241917.922827] ---[ end trace 1a77408724500fb0 ]---
> Dec 29 20:53:01 t kernel: [241917.951084] ------------[ cut here ]------------
> Dec 29 20:53:03 t kernel: [241917.974874] kernel BUG at fs/inode.c:1343!
> Dec 29 20:53:03 t kernel: [241917.988010] invalid opcode: 0000 [#2] SMP
> Dec 29 20:53:03 t kernel: [241917.988010] last sysfs file: /sys/class/scsi_host/host6/stats
> Dec 29 20:53:03 t kernel: [241917.988010] CPU 1
> Dec 29 20:53:03 t kernel: [241917.988010] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> Dec 29 20:53:03 t kernel: [241918.102960] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8
> Dec 29 20:53:03 t kernel: [241918.102960] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65
> Dec 29 20:53:03 t kernel: [241918.102960] RSP: 0000:ffff8801225b9d20 EFLAGS: 00010246
> Dec 29 20:53:03 t kernel: [241918.102960] RAX: 0000000000000000 RBX: ffff880120435148 RCX: ffffffff8139f927
> Dec 29 20:53:03 t kernel: [241918.102960] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff880120435148
> Dec 29 20:53:03 t kernel: [241918.102960] RBP: ffff8801225b9d30 R08: 0000000000000020 R09: ffff88002820c9d0
> Dec 29 20:53:03 t kernel: [241918.102960] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff880120435100
> Dec 29 20:53:03 t kernel: [241918.102960] R13: ffffffffa028ef50 R14: ffff8800c1b3b800 R15: ffff88012dd423c0
> Dec 29 20:53:03 t kernel: [241918.102960] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
> Dec 29 20:53:03 t kernel: [241918.102960] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> Dec 29 20:53:03 t kernel: [241918.102960] CR2: 00007fad86c2e000 CR3: 0000000129ae6000 CR4: 00000000000006e0
> Dec 29 20:53:03 t kernel: [241918.102960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Dec 29 20:53:03 t kernel: [241918.102960] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Dec 29 20:53:03 t kernel: [241918.102960] Process nfsd (pid: 3207, threadinfo ffff8801225b8000, task ffff880122416640)
> Dec 29 20:53:03 t kernel: [241918.102960] Stack:
> Dec 29 20:53:03 t kernel: [241918.102960] ffff8801225b9d90 ffff8800c1b3b800 ffff8801225b9d50 ffffffff811f4be2
> Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8800c1b3b800 ffff8800c1b3b810 ffff8801225b9d70 ffffffffa027c1d2
> Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8801225b9d70 ffff8800c1b3b800 ffff8801225b9da0 ffffffffa0286b03
> Dec 29 20:53:03 t kernel: [241918.102960] Call Trace:
> Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff811f4be2>] sock_release+0x5c/0x6c
> Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc]
> Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc]
> Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc]
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81142201>] kref_put+0x43/0x4f
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc]
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc]
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104af36>] kthread+0x7d/0x85
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c97a>] child_rip+0xa/0x20
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> Dec 29 20:53:04 t kernel: [241918.102960] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85
> Dec 29 20:53:04 t kernel: [241918.102960] RIP [<ffffffff810bccc2>] iput+0x1b/0x65
> Dec 29 20:53:04 t kernel: [241918.102960] RSP <ffff8801225b9d20>
> Dec 29 20:53:04 t kernel: [241919.822190] ---[ end trace 1a77408724500fb1 ]---
> Dec 30 11:22:26 t kernel: [294082.152979] 3w-9xxx: scsi6: AEN: ERROR (0x04:0x0009): Drive timeout detected:port=0.
> Dec 31 11:24:44 t kernel: [380620.428029] br0: port 2(eth2) entering disabled state
> Dec 31 11:24:44 t kernel: [380620.458826] br0: port 1(eth1) entering disabled state
> Dec 31 11:24:44 t kernel: [380620.666738] device eth1 left promiscuous mode
> Dec 31 11:24:44 t kernel: [380620.693361] br0: port 1(eth1) entering disabled state
> Dec 31 11:24:44 t kernel: [380620.909921] device eth2 left promiscuous mode
> Dec 31 11:24:44 t kernel: [380620.936603] br0: port 2(eth2) entering disabled state
> Dec 31 11:24:48 t kernel: [380624.449637] device eth1 entered promiscuous mode
> Dec 31 11:24:48 t kernel: [380624.533887] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X
> Dec 31 11:24:48 t kernel: [380624.620065] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X
> Dec 31 11:24:48 t kernel: [380624.651581] ADDRCONF(NETDEV_UP): eth1: link is not ready
> Dec 31 11:24:48 t kernel: [380624.687675] device eth2 entered promiscuous mode
> Dec 31 11:24:48 t kernel: [380624.860445] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X
> Dec 31 11:24:48 t kernel: [380624.944084] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X
> Dec 31 11:24:48 t kernel: [380624.975824] ADDRCONF(NETDEV_UP): eth2: link is not ready
> Dec 31 11:24:50 t kernel: [380626.984998] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> Dec 31 11:24:51 t kernel: [380627.031517] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
> Dec 31 11:24:51 t kernel: [380627.066844] br0: port 2(eth2) entering forwarding state
> Dec 31 11:24:51 t kernel: [380627.251584] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> Dec 31 11:24:51 t kernel: [380627.300433] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
> Dec 31 11:24:51 t kernel: [380627.335625] br0: port 1(eth1) entering forwarding state
> Dec 31 11:24:59 t kernel: [380635.148016] br0: no IPv6 routers present
> Dec 31 11:25:01 t kernel: [380637.504511] eth1: no IPv6 routers present
> Dec 31 11:25:01 t kernel: [380637.720515] eth2: no IPv6 routers present
> Dec 31 11:28:30 t kernel: [380846.205549] e1000e: eth1 NIC Link is Down
> Dec 31 11:28:30 t kernel: [380846.248659] br0: port 1(eth1) entering disabled state
> Dec 31 11:28:32 t kernel: [380848.693501] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX
> Dec 31 11:28:32 t kernel: [380848.739167] 0000:06:00.1: eth1: 10/100 speed: disabling TSO
> Dec 31 11:28:32 t kernel: [380848.776396] br0: port 1(eth1) entering forwarding state
> Dec 31 11:28:49 t kernel: [380865.803038] e1000e: eth1 NIC Link is Down
> Dec 31 11:28:49 t kernel: [380865.874086] br0: port 1(eth1) entering disabled state
> Dec 31 11:28:52 t kernel: [380868.267943] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> Dec 31 11:28:52 t kernel: [380868.316772] br0: port 1(eth1) entering forwarding state
> Dec 31 11:29:18 t kernel: [380894.253543] e1000e: eth1 NIC Link is Down
> Dec 31 11:29:18 t kernel: [380894.293597] br0: port 1(eth1) entering disabled state
> Dec 31 11:29:20 t kernel: [380896.688442] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> Dec 31 11:29:20 t kernel: [380896.737255] br0: port 1(eth1) entering forwarding state
> Dec 31 11:29:25 t kernel: [380901.493543] e1000e: eth1 NIC Link is Down
> Dec 31 11:29:25 t kernel: [380901.534074] br0: port 1(eth1) entering disabled state
> Dec 31 11:29:28 t kernel: [380904.161008] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> Dec 31 11:29:28 t kernel: [380904.209842] br0: port 1(eth1) entering forwarding state
> Jan 1 22:40:06 t kernel: [507542.268700] ------------[ cut here ]------------
> Jan 1 22:40:08 t kernel: [507542.296925] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> Jan 1 22:40:08 t kernel: [507542.330301] Hardware name: X7DB8
> Jan 1 22:40:08 t kernel: [507542.350163] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> Jan 1 22:40:08 t kernel: [507542.614635] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1
> Jan 1 22:40:08 t kernel: [507542.652191] Call Trace:
> Jan 1 22:40:08 t kernel: [507542.667375] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> Jan 1 22:40:08 t kernel: [507542.698681] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> Jan 1 22:40:08 t kernel: [507542.735186] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> Jan 1 22:40:08 t kernel: [507542.770130] [<ffffffff81142230>] kref_get+0x23/0xBj
> Jan 1 22:40:08 t kernel: [507542.800400] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> Jan 1 22:40:08 t kernel: [507542.833269] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> Jan 1 22:40:08 t kernel: [507542.869251] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> Jan 1 22:40:08 t kernel: [507542.907825] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> Jan 1 22:40:08 t kernel: [507542.944853] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> Jan 1 22:40:08 t kernel: [507542.977720] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> Jan 1 22:40:08 t kernel: [507543.010063] [<ffffffff8104af36>] kthread+0x7d/0x85
> Jan 1 22:40:08 t kernel: [507543.039808] [<ffffffff8100c97a>] child_rip+0xa/0x20
> Jan 1 22:40:08 t kernel: [507543.070071] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> Jan 1 22:40:08 t kernel: [507543.100337] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> Jan 1 22:40:08 t kernel: [507543.131640] ---[ end trace 1a77408724500fb2 ]---
> Jan 1 22:40:08 t kernel: [507543.160026] general protection fault: 0000 [#3] SMP
> Jan 1 22:40:08 t kernel: [507543.165025] last sysfs file: /sys/class/net/eth2/brport/hold_timer
> Jan 1 22:40:08 t kernel: [507543.165025] CPU 3
> Jan 1 22:40:08 t kernel: [507543.165025] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> Jan 1 22:40:08 t kernel: [507543.165025] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8
> Jan 1 22:40:08 t kernel: [507543.165025] RIP: 0010:[<ffffffffa0286797>] [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc]
> Jan 1 22:40:08 t kernel: [507543.165025] RSP: 0018:ffff880124ea1e00 EFLAGS: 00010246
> Jan 1 22:40:08 t kernel: [507543.165025] RAX: 3a30343a32322031 RBX: 0000000000000000 RCX: 0000000000000000
> Jan 1 22:40:08 t kernel: [507543.165025] RDX: 0000000000004c86 RSI: ffff880124ea1df0 RDI: ffff880124e00000
> Jan 1 22:40:08 t kernel: [507543.165025] RBP: ffff880124ea1eb0 R08: 0000000000000020 R09: ffff88002830ca70
> Jan 1 22:40:08 t kernel: [507543.165025] R10: ffff88002838cf40 R11: 0000000000000000 R12: 0000000000000082
> Jan 1 22:40:08 t kernel: [507543.165025] R13: ffff880124e00000 R14: ffff8800c7444000 R15: ffff88012dd423c0
> Jan 1 22:40:08 t kernel: [507543.165025] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000
> Jan 1 22:40:08 t kernel: [507543.165025] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> Jan 1 22:40:08 t kernel: [507543.165025] CR2: 00007f0dc9cc5098 CR3: 000000012c266000 CR4: 00000000000006e0
> Jan 1 22:40:08 t kernel: [507543.165025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Jan 1 22:40:08 t kernel: [507543.165025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Jan 1 22:40:08 t kernel: [507543.165025] Process nfsd (pid: 3134, threadinfo ffff880124ea0000, task ffff880124e9f890)
> Jan 1 22:40:08 t kernel: [507543.165025] Stack:
> Jan 1 22:40:08 t kernel: [507543.165025] ffff880124e00000 ffff880124e9f890 ffff880124ea1e40 00000000000dbba0
> Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff88012dd423c4 ffff88012c0bea40 ffff880000000082 0000000000000001
> Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff880124ea1eb0 ffffffffa027abe4 0000000000000000 ffff880124e9f890
> Jan 1 22:40:08 t kernel: [507543.165025] Call Trace:
> Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104af36>] kthread+0x7d/0x85
> Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c97a>] child_rip+0xa/0x20
> Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> Jan 1 22:40:08 t kernel: [507543.165025] Code: 00 49 8d 85 88 01 00 00 49 89 85 a0 09 00 00 8b 82 e8 00 00 00 8d 1c 85 00 00 00 00 2b 9a e0 00 00 00 eb 0c 49 8b 46 08 4c 89 ef <ff> 50 18 89 c3 f6 05 56 ce 00 00 01 74 10 89 de 48 c7 c7 cf b6
> Jan 1 22:40:08 t kernel: [507543.165025] RIP [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc]
> Jan 1 22:40:08 t kernel: [507543.165025] RSP <ffff880124ea1e00>
> Jan 1 22:40:08 t kernel: [507544.713111] ---[ end trace 1a77408724500fb3 ]---
> Jan 2 12:06:02 t kernel: [555898.365986] RPC: multiple fragments per record not supported
> Jan 2 12:06:02 t kernel: [555898.527050] RPC: multiple fragments per record not supported
> Jan 2 12:06:05 t kernel: [555901.700485] RPC: multiple fragments per record not supported
> Jan 2 12:06:08 t kernel: [555904.880825] RPC: multiple fragments per record not supported
> Jan 3 00:01:08 t kernel: [598804.533706] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0.
> Jan 3 00:57:01 t kernel: [602157.347383] md: data-check of RAID array md0
> Jan 3 00:57:01 t kernel: [602157.350802] md: delaying data-check of md2 until md0 has finished (they share one or more physical units)
> Jan 3 00:57:01 t kernel: [602157.431453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> Jan 3 00:57:01 t kernel: [602157.467054] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> Jan 3 00:57:01 t kernel: [602157.526113] md: using 128k window, over a total of 19542976 blocks.
> Jan 3 01:03:47 t kernel: [602563.525129] md: md0: data-check done.
> Jan 3 01:03:47 t kernel: [602563.554886] md: data-check of RAID array md2
> Jan 3 01:03:47 t kernel: [602563.581194] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> Jan 3 01:03:47 t kernel: [602563.616879] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> Jan 3 01:03:47 t kernel: [602563.675944] md: using 128k window, over a total of 460928832 blocks.
> Jan 3 04:47:09 t kernel: [615965.563359] md: md2: data-check done.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
2010-02-02 16:22 ` J. Bruce Fields
@ 2010-02-16 18:22 ` Nikola Ciprich
[not found] ` <20100216182248.GA4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org>
0 siblings, 1 reply; 12+ messages in thread
From: Nikola Ciprich @ 2010-02-16 18:22 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs,
nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g
Hello,
is there any conclusion regarding this problem?
I'm seeing it too on one of my machines, running 2.6.32.8
thanks a lot in advance for any informamation..
regards
nik
On Tue, Feb 02, 2010 at 11:22:28AM -0500, J. Bruce Fields wrote:
> On Wed, Jan 27, 2010 at 03:01:40PM -0800, Harry Edmon wrote:
> > J. Bruce Fields wrote:
> >> On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote:
> >>
> >>> (cc linux-nfs)
> >>>
> >>> On Thu, 14 Jan 2010 09:53:04 -0800
> >>> Harry Edmon <harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org> wrote:
> >>>
> >>>
> >>>> I have a single quad-core Xeon machine running 2.6.32.2 that keeps
> >>>> giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This
> >>>> machine is mainaly used as a NFS server. I have attached the
> >>>> kern.log and my config file. Any ideas?
> >>>>
> >>
> >> Are you using NFSv4, or only v2/v3? Looking at the trace:
> >>
> >> Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------
> >> Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> >>
> > NFSv4 and NFSv3.
>
> Thanks. And you aren't using rdma are you?
>
> >> OK, so that's the warning that we're trying to get something with
> >> reference count 0.
> >>
> >> Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8
> >> Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> >> Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1
> >>
> >> What does the very first oops, bug, or warning you get look like? (If
> >> I'm reading the tain information right, this isn't the first one.)
> >>
> > Attached is the kern.log that contains the first warning.
>
> Thanks. So yes, the first warning is a stray kref_put() in svc_recv(),
> and the such put I see on a quick skim is a svc_xprt_put(). The
> following BUG() is confirmation, since it's an attempt inside
> scv-xprt_release() to clear an inode with I_CLEAR already set; so we
> dropped the reference count to zero, cleared it, then tried to do
> another get/put pair, resulting in an attempt to free the same socket
> twice.
>
> --b.
>
> >
> > --
> > Dr. Harry Edmon E-MAIL: harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org
> > 206-543-0547 FAX: 206-543-0308 harry-B93hV6UPU7Z2icitjWtXSw@public.gmane.org, harry@uw.edu
> > Director of IT, College of the Environment and
> > Director of Computing, Dept of Atmospheric Sciences
> > University of Washington, Box 351640, Seattle, WA 98195-1640
> >
>
> > Dec 27 11:57:44 t kernel: [37000.838904] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0055): Battery charging started:.
> > Dec 27 11:57:46 t kernel: [37002.028946] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0056): Battery charging completed:.
> > Dec 28 00:00:51 t kernel: [80387.429120] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0.
> > Dec 28 03:32:07 t kernel: [93063.950486] 3w-9xxx: scsi6: AEN: INFO (0x04:0x002B): Verify completed:unit=0.
> > Dec 29 17:04:43 t kernel: [228218.883443] ------------[ cut here ]------------
> > Dec 29 17:04:45 t kernel: [228218.911691] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> > Dec 29 17:04:45 t kernel: [228218.945083] Hardware name: X7DB8
> > Dec 29 17:04:45 t kernel: [228218.964948] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > Dec 29 17:04:45 t kernel: [228219.227296] Pid: 3108, comm: nfsd Not tainted 2.6.32.2 #1
> > Dec 29 17:04:45 t kernel: [228219.260164] Call Trace:
> > Dec 29 17:04:45 t kernel: [228219.275350] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > Dec 29 17:04:45 t kernel: [228219.306656] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> > Dec 29 17:04:45 t kernel: [228219.343160] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> > Dec 29 17:04:45 t kernel: [228219.378105] [<ffffffff81142230>] kref_get+0x23/0x2b
> > Dec 29 17:04:45 t kernel: [228219.408379] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > Dec 29 17:04:45 t kernel: [228219.441242] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> > Dec 29 17:04:45 t kernel: [228219.477225] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > Dec 29 17:04:45 t kernel: [228219.515806] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > Dec 29 17:04:45 t kernel: [228219.552834] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > Dec 29 17:04:45 t kernel: [228219.585698] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > Dec 29 17:04:45 t kernel: [228219.618043] [<ffffffff8104af36>] kthread+0x7d/0x85
> > Dec 29 17:04:45 t kernel: [228219.647789] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > Dec 29 17:04:45 t kernel: [228219.678052] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > Dec 29 17:04:45 t kernel: [228219.708317] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > Dec 29 17:04:45 t kernel: [228219.739621] ---[ end trace 1a77408724500fae ]---
> > Dec 29 17:04:45 t kernel: [228219.767925] ------------[ cut here ]------------
> > Dec 29 17:04:45 t kernel: [228219.792730] kernel BUG at fs/inode.c:1343!
> > Dec 29 17:04:45 t kernel: [228219.792730] invalid opcode: 0000 [#1] SMP
> > Dec 29 17:04:45 t kernel: [228219.792730] last sysfs file: /sys/class/scsi_host/host6/stats
> > Dec 29 17:04:45 t kernel: [228219.792730] CPU 0
> > Dec 29 17:04:45 t kernel: [228219.792730] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > Dec 29 17:04:45 t kernel: [228219.792730] Pid: 3108, comm: nfsd Tainted: G W 2.6.32.2 #1 X7DB8
> > Dec 29 17:04:45 t kernel: [228219.792730] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65
> > Dec 29 17:04:45 t kernel: [228219.792730] RSP: 0018:ffff88012588bd20 EFLAGS: 00010246
> > Dec 29 17:04:45 t kernel: [228219.792730] RAX: 0000000000000000 RBX: ffff88003c3c2cc8 RCX: ffffffff8139f927
> > Dec 29 17:04:45 t kernel: [228219.792730] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88003c3c2cc8
> > Dec 29 17:04:45 t kernel: [228219.792730] RBP: ffff88012588bd30 R08: 0000000000000002 R09: 0000000000000000
> > Dec 29 17:04:45 t kernel: [228219.792730] R10: ffffffff81142230 R11: 0000000100000001 R12: ffff88003c3c2c80
> > Dec 29 17:04:45 t kernel: [228219.792730] R13: ffffffffa028ef50 R14: ffff88010072d400 R15: ffff88012dd423c0
> > Dec 29 17:04:45 t kernel: [228219.792730] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
> > Dec 29 17:04:45 t kernel: [228219.792730] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> > Dec 29 17:04:45 t kernel: [228219.792730] CR2: 00007fad86c2e000 CR3: 00000000305e5000 CR4: 00000000000006f0
> > Dec 29 17:04:45 t kernel: [228219.792730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > Dec 29 17:04:45 t kernel: [228219.792730] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Dec 29 17:04:45 t kernel: [228219.792730] Process nfsd (pid: 3108, threadinfo ffff88012588a000, task ffff880125e66150)
> > Dec 29 17:04:45 t kernel: [228219.792730] Stack:
> > Dec 29 17:04:45 t kernel: [228219.792730] ffff88012588bd90 ffff88010072d400 ffff88012588bd50 ffffffff811f4be2
> > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88010072d400 ffff88010072d410 ffff88012588bd70 ffffffffa027c1d2
> > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88012588bd70 ffff88010072d400 ffff88012588bda0 ffffffffa0286b03
> > Dec 29 17:04:45 t kernel: [228219.792730] Call Trace:
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff811f4be2>] sock_release+0x5c/0x6c
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc]
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc]
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc]
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142201>] kref_put+0x43/0x4f
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc]
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc]
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104af36>] kthread+0x7d/0x85
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > Dec 29 17:04:45 t kernel: [228219.792730] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85
> > Dec 29 17:04:45 t kernel: [228219.792730] RIP [<ffffffff810bccc2>] iput+0x1b/0x65
> > Dec 29 17:04:45 t kernel: [228219.792730] RSP <ffff88012588bd20>
> > Dec 29 17:04:45 t kernel: [228221.632694] ---[ end trace 1a77408724500faf ]---
> > Dec 29 20:53:01 t kernel: [241917.056374] ------------[ cut here ]------------
> > Dec 29 20:53:01 t kernel: [241917.084581] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> > Dec 29 20:53:01 t kernel: [241917.117941] Hardware name: X7DB8
> > Dec 29 20:53:01 t kernel: [241917.137804] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > Dec 29 20:53:01 t kernel: [241917.400101] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1
> > Dec 29 20:53:01 t kernel: [241917.437653] Call Trace:
> > Dec 29 20:53:01 t kernel: [241917.452843] [<ffffffffa0286b0b>] ? svc_xprt_free+0x46/0x4e [sunrpc]
> > Dec 29 20:53:01 t kernel: [241917.491420] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > Dec 29 20:53:01 t kernel: [241917.522728] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> > Dec 29 20:53:01 t kernel: [241917.559230] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> > Dec 29 20:53:01 t kernel: [241917.594177] [<ffffffff81142230>] kref_get+0x23/0x2b
> > Dec 29 20:53:01 t kernel: [241917.624446] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> > Dec 29 20:53:01 t kernel: [241917.660432] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > Dec 29 20:53:01 t kernel: [241917.699013] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > Dec 29 20:53:01 t kernel: [241917.736041] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > Dec 29 20:53:01 t kernel: [241917.768905] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > Dec 29 20:53:01 t kernel: [241917.801249] [<ffffffff8104af36>] kthread+0x7d/0x85
> > Dec 29 20:53:01 t kernel: [241917.830993] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > Dec 29 20:53:01 t kernel: [241917.861258] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > Dec 29 20:53:01 t kernel: [241917.891522] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > Dec 29 20:53:01 t kernel: [241917.922827] ---[ end trace 1a77408724500fb0 ]---
> > Dec 29 20:53:01 t kernel: [241917.951084] ------------[ cut here ]------------
> > Dec 29 20:53:03 t kernel: [241917.974874] kernel BUG at fs/inode.c:1343!
> > Dec 29 20:53:03 t kernel: [241917.988010] invalid opcode: 0000 [#2] SMP
> > Dec 29 20:53:03 t kernel: [241917.988010] last sysfs file: /sys/class/scsi_host/host6/stats
> > Dec 29 20:53:03 t kernel: [241917.988010] CPU 1
> > Dec 29 20:53:03 t kernel: [241917.988010] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > Dec 29 20:53:03 t kernel: [241918.102960] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8
> > Dec 29 20:53:03 t kernel: [241918.102960] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65
> > Dec 29 20:53:03 t kernel: [241918.102960] RSP: 0000:ffff8801225b9d20 EFLAGS: 00010246
> > Dec 29 20:53:03 t kernel: [241918.102960] RAX: 0000000000000000 RBX: ffff880120435148 RCX: ffffffff8139f927
> > Dec 29 20:53:03 t kernel: [241918.102960] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff880120435148
> > Dec 29 20:53:03 t kernel: [241918.102960] RBP: ffff8801225b9d30 R08: 0000000000000020 R09: ffff88002820c9d0
> > Dec 29 20:53:03 t kernel: [241918.102960] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff880120435100
> > Dec 29 20:53:03 t kernel: [241918.102960] R13: ffffffffa028ef50 R14: ffff8800c1b3b800 R15: ffff88012dd423c0
> > Dec 29 20:53:03 t kernel: [241918.102960] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
> > Dec 29 20:53:03 t kernel: [241918.102960] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> > Dec 29 20:53:03 t kernel: [241918.102960] CR2: 00007fad86c2e000 CR3: 0000000129ae6000 CR4: 00000000000006e0
> > Dec 29 20:53:03 t kernel: [241918.102960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > Dec 29 20:53:03 t kernel: [241918.102960] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Dec 29 20:53:03 t kernel: [241918.102960] Process nfsd (pid: 3207, threadinfo ffff8801225b8000, task ffff880122416640)
> > Dec 29 20:53:03 t kernel: [241918.102960] Stack:
> > Dec 29 20:53:03 t kernel: [241918.102960] ffff8801225b9d90 ffff8800c1b3b800 ffff8801225b9d50 ffffffff811f4be2
> > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8800c1b3b800 ffff8800c1b3b810 ffff8801225b9d70 ffffffffa027c1d2
> > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8801225b9d70 ffff8800c1b3b800 ffff8801225b9da0 ffffffffa0286b03
> > Dec 29 20:53:03 t kernel: [241918.102960] Call Trace:
> > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff811f4be2>] sock_release+0x5c/0x6c
> > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc]
> > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc]
> > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc]
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81142201>] kref_put+0x43/0x4f
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc]
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc]
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104af36>] kthread+0x7d/0x85
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > Dec 29 20:53:04 t kernel: [241918.102960] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85
> > Dec 29 20:53:04 t kernel: [241918.102960] RIP [<ffffffff810bccc2>] iput+0x1b/0x65
> > Dec 29 20:53:04 t kernel: [241918.102960] RSP <ffff8801225b9d20>
> > Dec 29 20:53:04 t kernel: [241919.822190] ---[ end trace 1a77408724500fb1 ]---
> > Dec 30 11:22:26 t kernel: [294082.152979] 3w-9xxx: scsi6: AEN: ERROR (0x04:0x0009): Drive timeout detected:port=0.
> > Dec 31 11:24:44 t kernel: [380620.428029] br0: port 2(eth2) entering disabled state
> > Dec 31 11:24:44 t kernel: [380620.458826] br0: port 1(eth1) entering disabled state
> > Dec 31 11:24:44 t kernel: [380620.666738] device eth1 left promiscuous mode
> > Dec 31 11:24:44 t kernel: [380620.693361] br0: port 1(eth1) entering disabled state
> > Dec 31 11:24:44 t kernel: [380620.909921] device eth2 left promiscuous mode
> > Dec 31 11:24:44 t kernel: [380620.936603] br0: port 2(eth2) entering disabled state
> > Dec 31 11:24:48 t kernel: [380624.449637] device eth1 entered promiscuous mode
> > Dec 31 11:24:48 t kernel: [380624.533887] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X
> > Dec 31 11:24:48 t kernel: [380624.620065] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X
> > Dec 31 11:24:48 t kernel: [380624.651581] ADDRCONF(NETDEV_UP): eth1: link is not ready
> > Dec 31 11:24:48 t kernel: [380624.687675] device eth2 entered promiscuous mode
> > Dec 31 11:24:48 t kernel: [380624.860445] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X
> > Dec 31 11:24:48 t kernel: [380624.944084] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X
> > Dec 31 11:24:48 t kernel: [380624.975824] ADDRCONF(NETDEV_UP): eth2: link is not ready
> > Dec 31 11:24:50 t kernel: [380626.984998] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > Dec 31 11:24:51 t kernel: [380627.031517] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
> > Dec 31 11:24:51 t kernel: [380627.066844] br0: port 2(eth2) entering forwarding state
> > Dec 31 11:24:51 t kernel: [380627.251584] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > Dec 31 11:24:51 t kernel: [380627.300433] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
> > Dec 31 11:24:51 t kernel: [380627.335625] br0: port 1(eth1) entering forwarding state
> > Dec 31 11:24:59 t kernel: [380635.148016] br0: no IPv6 routers present
> > Dec 31 11:25:01 t kernel: [380637.504511] eth1: no IPv6 routers present
> > Dec 31 11:25:01 t kernel: [380637.720515] eth2: no IPv6 routers present
> > Dec 31 11:28:30 t kernel: [380846.205549] e1000e: eth1 NIC Link is Down
> > Dec 31 11:28:30 t kernel: [380846.248659] br0: port 1(eth1) entering disabled state
> > Dec 31 11:28:32 t kernel: [380848.693501] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX
> > Dec 31 11:28:32 t kernel: [380848.739167] 0000:06:00.1: eth1: 10/100 speed: disabling TSO
> > Dec 31 11:28:32 t kernel: [380848.776396] br0: port 1(eth1) entering forwarding state
> > Dec 31 11:28:49 t kernel: [380865.803038] e1000e: eth1 NIC Link is Down
> > Dec 31 11:28:49 t kernel: [380865.874086] br0: port 1(eth1) entering disabled state
> > Dec 31 11:28:52 t kernel: [380868.267943] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > Dec 31 11:28:52 t kernel: [380868.316772] br0: port 1(eth1) entering forwarding state
> > Dec 31 11:29:18 t kernel: [380894.253543] e1000e: eth1 NIC Link is Down
> > Dec 31 11:29:18 t kernel: [380894.293597] br0: port 1(eth1) entering disabled state
> > Dec 31 11:29:20 t kernel: [380896.688442] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > Dec 31 11:29:20 t kernel: [380896.737255] br0: port 1(eth1) entering forwarding state
> > Dec 31 11:29:25 t kernel: [380901.493543] e1000e: eth1 NIC Link is Down
> > Dec 31 11:29:25 t kernel: [380901.534074] br0: port 1(eth1) entering disabled state
> > Dec 31 11:29:28 t kernel: [380904.161008] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > Dec 31 11:29:28 t kernel: [380904.209842] br0: port 1(eth1) entering forwarding state
> > Jan 1 22:40:06 t kernel: [507542.268700] ------------[ cut here ]------------
> > Jan 1 22:40:08 t kernel: [507542.296925] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> > Jan 1 22:40:08 t kernel: [507542.330301] Hardware name: X7DB8
> > Jan 1 22:40:08 t kernel: [507542.350163] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > Jan 1 22:40:08 t kernel: [507542.614635] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1
> > Jan 1 22:40:08 t kernel: [507542.652191] Call Trace:
> > Jan 1 22:40:08 t kernel: [507542.667375] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > Jan 1 22:40:08 t kernel: [507542.698681] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> > Jan 1 22:40:08 t kernel: [507542.735186] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> > Jan 1 22:40:08 t kernel: [507542.770130] [<ffffffff81142230>] kref_get+0x23/0xBj
> > Jan 1 22:40:08 t kernel: [507542.800400] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > Jan 1 22:40:08 t kernel: [507542.833269] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> > Jan 1 22:40:08 t kernel: [507542.869251] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > Jan 1 22:40:08 t kernel: [507542.907825] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > Jan 1 22:40:08 t kernel: [507542.944853] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > Jan 1 22:40:08 t kernel: [507542.977720] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > Jan 1 22:40:08 t kernel: [507543.010063] [<ffffffff8104af36>] kthread+0x7d/0x85
> > Jan 1 22:40:08 t kernel: [507543.039808] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > Jan 1 22:40:08 t kernel: [507543.070071] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > Jan 1 22:40:08 t kernel: [507543.100337] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > Jan 1 22:40:08 t kernel: [507543.131640] ---[ end trace 1a77408724500fb2 ]---
> > Jan 1 22:40:08 t kernel: [507543.160026] general protection fault: 0000 [#3] SMP
> > Jan 1 22:40:08 t kernel: [507543.165025] last sysfs file: /sys/class/net/eth2/brport/hold_timer
> > Jan 1 22:40:08 t kernel: [507543.165025] CPU 3
> > Jan 1 22:40:08 t kernel: [507543.165025] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > Jan 1 22:40:08 t kernel: [507543.165025] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8
> > Jan 1 22:40:08 t kernel: [507543.165025] RIP: 0010:[<ffffffffa0286797>] [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc]
> > Jan 1 22:40:08 t kernel: [507543.165025] RSP: 0018:ffff880124ea1e00 EFLAGS: 00010246
> > Jan 1 22:40:08 t kernel: [507543.165025] RAX: 3a30343a32322031 RBX: 0000000000000000 RCX: 0000000000000000
> > Jan 1 22:40:08 t kernel: [507543.165025] RDX: 0000000000004c86 RSI: ffff880124ea1df0 RDI: ffff880124e00000
> > Jan 1 22:40:08 t kernel: [507543.165025] RBP: ffff880124ea1eb0 R08: 0000000000000020 R09: ffff88002830ca70
> > Jan 1 22:40:08 t kernel: [507543.165025] R10: ffff88002838cf40 R11: 0000000000000000 R12: 0000000000000082
> > Jan 1 22:40:08 t kernel: [507543.165025] R13: ffff880124e00000 R14: ffff8800c7444000 R15: ffff88012dd423c0
> > Jan 1 22:40:08 t kernel: [507543.165025] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000
> > Jan 1 22:40:08 t kernel: [507543.165025] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> > Jan 1 22:40:08 t kernel: [507543.165025] CR2: 00007f0dc9cc5098 CR3: 000000012c266000 CR4: 00000000000006e0
> > Jan 1 22:40:08 t kernel: [507543.165025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > Jan 1 22:40:08 t kernel: [507543.165025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Jan 1 22:40:08 t kernel: [507543.165025] Process nfsd (pid: 3134, threadinfo ffff880124ea0000, task ffff880124e9f890)
> > Jan 1 22:40:08 t kernel: [507543.165025] Stack:
> > Jan 1 22:40:08 t kernel: [507543.165025] ffff880124e00000 ffff880124e9f890 ffff880124ea1e40 00000000000dbba0
> > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff88012dd423c4 ffff88012c0bea40 ffff880000000082 0000000000000001
> > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff880124ea1eb0 ffffffffa027abe4 0000000000000000 ffff880124e9f890
> > Jan 1 22:40:08 t kernel: [507543.165025] Call Trace:
> > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104af36>] kthread+0x7d/0x85
> > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > Jan 1 22:40:08 t kernel: [507543.165025] Code: 00 49 8d 85 88 01 00 00 49 89 85 a0 09 00 00 8b 82 e8 00 00 00 8d 1c 85 00 00 00 00 2b 9a e0 00 00 00 eb 0c 49 8b 46 08 4c 89 ef <ff> 50 18 89 c3 f6 05 56 ce 00 00 01 74 10 89 de 48 c7 c7 cf b6
> > Jan 1 22:40:08 t kernel: [507543.165025] RIP [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc]
> > Jan 1 22:40:08 t kernel: [507543.165025] RSP <ffff880124ea1e00>
> > Jan 1 22:40:08 t kernel: [507544.713111] ---[ end trace 1a77408724500fb3 ]---
> > Jan 2 12:06:02 t kernel: [555898.365986] RPC: multiple fragments per record not supported
> > Jan 2 12:06:02 t kernel: [555898.527050] RPC: multiple fragments per record not supported
> > Jan 2 12:06:05 t kernel: [555901.700485] RPC: multiple fragments per record not supported
> > Jan 2 12:06:08 t kernel: [555904.880825] RPC: multiple fragments per record not supported
> > Jan 3 00:01:08 t kernel: [598804.533706] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0.
> > Jan 3 00:57:01 t kernel: [602157.347383] md: data-check of RAID array md0
> > Jan 3 00:57:01 t kernel: [602157.350802] md: delaying data-check of md2 until md0 has finished (they share one or more physical units)
> > Jan 3 00:57:01 t kernel: [602157.431453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> > Jan 3 00:57:01 t kernel: [602157.467054] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> > Jan 3 00:57:01 t kernel: [602157.526113] md: using 128k window, over a total of 19542976 blocks.
> > Jan 3 01:03:47 t kernel: [602563.525129] md: md0: data-check done.
> > Jan 3 01:03:47 t kernel: [602563.554886] md: data-check of RAID array md2
> > Jan 3 01:03:47 t kernel: [602563.581194] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> > Jan 3 01:03:47 t kernel: [602563.616879] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> > Jan 3 01:03:47 t kernel: [602563.675944] md: using 128k window, over a total of 460928832 blocks.
> > Jan 3 04:47:09 t kernel: [615965.563359] md: md2: data-check done.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava
tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: servis-Jp3n8lUXroTtwjQa/ONI9g@public.gmane.org
-------------------------------------
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
[not found] ` <20100216182248.GA4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org>
@ 2010-02-16 20:21 ` J. Bruce Fields
2010-02-16 20:27 ` Nikola Ciprich
0 siblings, 1 reply; 12+ messages in thread
From: J. Bruce Fields @ 2010-02-16 20:21 UTC (permalink / raw)
To: Nikola Ciprich
Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs,
nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g
On Tue, Feb 16, 2010 at 07:22:48PM +0100, Nikola Ciprich wrote:
> Hello,
> is there any conclusion regarding this problem?
> I'm seeing it too on one of my machines, running 2.6.32.8
> thanks a lot in advance for any informamation..
> regards
Nothing yet. I'm taking another look.
Do you know what the most recent version was where you did *not* see
these warnings?
--b.
> nik
>
>
> On Tue, Feb 02, 2010 at 11:22:28AM -0500, J. Bruce Fields wrote:
> > On Wed, Jan 27, 2010 at 03:01:40PM -0800, Harry Edmon wrote:
> > > J. Bruce Fields wrote:
> > >> On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote:
> > >>
> > >>> (cc linux-nfs)
> > >>>
> > >>> On Thu, 14 Jan 2010 09:53:04 -0800
> > >>> Harry Edmon <harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org> wrote:
> > >>>
> > >>>
> > >>>> I have a single quad-core Xeon machine running 2.6.32.2 that keeps
> > >>>> giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This
> > >>>> machine is mainaly used as a NFS server. I have attached the
> > >>>> kern.log and my config file. Any ideas?
> > >>>>
> > >>
> > >> Are you using NFSv4, or only v2/v3? Looking at the trace:
> > >>
> > >> Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------
> > >> Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> > >>
> > > NFSv4 and NFSv3.
> >
> > Thanks. And you aren't using rdma are you?
> >
> > >> OK, so that's the warning that we're trying to get something with
> > >> reference count 0.
> > >>
> > >> Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8
> > >> Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > >> Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1
> > >>
> > >> What does the very first oops, bug, or warning you get look like? (If
> > >> I'm reading the tain information right, this isn't the first one.)
> > >>
> > > Attached is the kern.log that contains the first warning.
> >
> > Thanks. So yes, the first warning is a stray kref_put() in svc_recv(),
> > and the such put I see on a quick skim is a svc_xprt_put(). The
> > following BUG() is confirmation, since it's an attempt inside
> > scv-xprt_release() to clear an inode with I_CLEAR already set; so we
> > dropped the reference count to zero, cleared it, then tried to do
> > another get/put pair, resulting in an attempt to free the same socket
> > twice.
> >
> > --b.
> >
> > >
> > > --
> > > Dr. Harry Edmon E-MAIL: harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org
> > > 206-543-0547 FAX: 206-543-0308 harry-B93hV6UPU7Z2icitjWtXSw@public.gmane.org, harry@uw.edu
> > > Director of IT, College of the Environment and
> > > Director of Computing, Dept of Atmospheric Sciences
> > > University of Washington, Box 351640, Seattle, WA 98195-1640
> > >
> >
> > > Dec 27 11:57:44 t kernel: [37000.838904] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0055): Battery charging started:.
> > > Dec 27 11:57:46 t kernel: [37002.028946] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0056): Battery charging completed:.
> > > Dec 28 00:00:51 t kernel: [80387.429120] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0.
> > > Dec 28 03:32:07 t kernel: [93063.950486] 3w-9xxx: scsi6: AEN: INFO (0x04:0x002B): Verify completed:unit=0.
> > > Dec 29 17:04:43 t kernel: [228218.883443] ------------[ cut here ]------------
> > > Dec 29 17:04:45 t kernel: [228218.911691] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> > > Dec 29 17:04:45 t kernel: [228218.945083] Hardware name: X7DB8
> > > Dec 29 17:04:45 t kernel: [228218.964948] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > Dec 29 17:04:45 t kernel: [228219.227296] Pid: 3108, comm: nfsd Not tainted 2.6.32.2 #1
> > > Dec 29 17:04:45 t kernel: [228219.260164] Call Trace:
> > > Dec 29 17:04:45 t kernel: [228219.275350] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > > Dec 29 17:04:45 t kernel: [228219.306656] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> > > Dec 29 17:04:45 t kernel: [228219.343160] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> > > Dec 29 17:04:45 t kernel: [228219.378105] [<ffffffff81142230>] kref_get+0x23/0x2b
> > > Dec 29 17:04:45 t kernel: [228219.408379] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > Dec 29 17:04:45 t kernel: [228219.441242] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> > > Dec 29 17:04:45 t kernel: [228219.477225] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > Dec 29 17:04:45 t kernel: [228219.515806] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > Dec 29 17:04:45 t kernel: [228219.552834] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > Dec 29 17:04:45 t kernel: [228219.585698] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > Dec 29 17:04:45 t kernel: [228219.618043] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > Dec 29 17:04:45 t kernel: [228219.647789] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > Dec 29 17:04:45 t kernel: [228219.678052] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > Dec 29 17:04:45 t kernel: [228219.708317] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > Dec 29 17:04:45 t kernel: [228219.739621] ---[ end trace 1a77408724500fae ]---
> > > Dec 29 17:04:45 t kernel: [228219.767925] ------------[ cut here ]------------
> > > Dec 29 17:04:45 t kernel: [228219.792730] kernel BUG at fs/inode.c:1343!
> > > Dec 29 17:04:45 t kernel: [228219.792730] invalid opcode: 0000 [#1] SMP
> > > Dec 29 17:04:45 t kernel: [228219.792730] last sysfs file: /sys/class/scsi_host/host6/stats
> > > Dec 29 17:04:45 t kernel: [228219.792730] CPU 0
> > > Dec 29 17:04:45 t kernel: [228219.792730] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > Dec 29 17:04:45 t kernel: [228219.792730] Pid: 3108, comm: nfsd Tainted: G W 2.6.32.2 #1 X7DB8
> > > Dec 29 17:04:45 t kernel: [228219.792730] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65
> > > Dec 29 17:04:45 t kernel: [228219.792730] RSP: 0018:ffff88012588bd20 EFLAGS: 00010246
> > > Dec 29 17:04:45 t kernel: [228219.792730] RAX: 0000000000000000 RBX: ffff88003c3c2cc8 RCX: ffffffff8139f927
> > > Dec 29 17:04:45 t kernel: [228219.792730] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88003c3c2cc8
> > > Dec 29 17:04:45 t kernel: [228219.792730] RBP: ffff88012588bd30 R08: 0000000000000002 R09: 0000000000000000
> > > Dec 29 17:04:45 t kernel: [228219.792730] R10: ffffffff81142230 R11: 0000000100000001 R12: ffff88003c3c2c80
> > > Dec 29 17:04:45 t kernel: [228219.792730] R13: ffffffffa028ef50 R14: ffff88010072d400 R15: ffff88012dd423c0
> > > Dec 29 17:04:45 t kernel: [228219.792730] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
> > > Dec 29 17:04:45 t kernel: [228219.792730] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> > > Dec 29 17:04:45 t kernel: [228219.792730] CR2: 00007fad86c2e000 CR3: 00000000305e5000 CR4: 00000000000006f0
> > > Dec 29 17:04:45 t kernel: [228219.792730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > Dec 29 17:04:45 t kernel: [228219.792730] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > Dec 29 17:04:45 t kernel: [228219.792730] Process nfsd (pid: 3108, threadinfo ffff88012588a000, task ffff880125e66150)
> > > Dec 29 17:04:45 t kernel: [228219.792730] Stack:
> > > Dec 29 17:04:45 t kernel: [228219.792730] ffff88012588bd90 ffff88010072d400 ffff88012588bd50 ffffffff811f4be2
> > > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88010072d400 ffff88010072d410 ffff88012588bd70 ffffffffa027c1d2
> > > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88012588bd70 ffff88010072d400 ffff88012588bda0 ffffffffa0286b03
> > > Dec 29 17:04:45 t kernel: [228219.792730] Call Trace:
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff811f4be2>] sock_release+0x5c/0x6c
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc]
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc]
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc]
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142201>] kref_put+0x43/0x4f
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc]
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc]
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > Dec 29 17:04:45 t kernel: [228219.792730] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85
> > > Dec 29 17:04:45 t kernel: [228219.792730] RIP [<ffffffff810bccc2>] iput+0x1b/0x65
> > > Dec 29 17:04:45 t kernel: [228219.792730] RSP <ffff88012588bd20>
> > > Dec 29 17:04:45 t kernel: [228221.632694] ---[ end trace 1a77408724500faf ]---
> > > Dec 29 20:53:01 t kernel: [241917.056374] ------------[ cut here ]------------
> > > Dec 29 20:53:01 t kernel: [241917.084581] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> > > Dec 29 20:53:01 t kernel: [241917.117941] Hardware name: X7DB8
> > > Dec 29 20:53:01 t kernel: [241917.137804] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > Dec 29 20:53:01 t kernel: [241917.400101] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1
> > > Dec 29 20:53:01 t kernel: [241917.437653] Call Trace:
> > > Dec 29 20:53:01 t kernel: [241917.452843] [<ffffffffa0286b0b>] ? svc_xprt_free+0x46/0x4e [sunrpc]
> > > Dec 29 20:53:01 t kernel: [241917.491420] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > > Dec 29 20:53:01 t kernel: [241917.522728] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> > > Dec 29 20:53:01 t kernel: [241917.559230] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> > > Dec 29 20:53:01 t kernel: [241917.594177] [<ffffffff81142230>] kref_get+0x23/0x2b
> > > Dec 29 20:53:01 t kernel: [241917.624446] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> > > Dec 29 20:53:01 t kernel: [241917.660432] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > Dec 29 20:53:01 t kernel: [241917.699013] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > Dec 29 20:53:01 t kernel: [241917.736041] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > Dec 29 20:53:01 t kernel: [241917.768905] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > Dec 29 20:53:01 t kernel: [241917.801249] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > Dec 29 20:53:01 t kernel: [241917.830993] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > Dec 29 20:53:01 t kernel: [241917.861258] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > Dec 29 20:53:01 t kernel: [241917.891522] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > Dec 29 20:53:01 t kernel: [241917.922827] ---[ end trace 1a77408724500fb0 ]---
> > > Dec 29 20:53:01 t kernel: [241917.951084] ------------[ cut here ]------------
> > > Dec 29 20:53:03 t kernel: [241917.974874] kernel BUG at fs/inode.c:1343!
> > > Dec 29 20:53:03 t kernel: [241917.988010] invalid opcode: 0000 [#2] SMP
> > > Dec 29 20:53:03 t kernel: [241917.988010] last sysfs file: /sys/class/scsi_host/host6/stats
> > > Dec 29 20:53:03 t kernel: [241917.988010] CPU 1
> > > Dec 29 20:53:03 t kernel: [241917.988010] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > Dec 29 20:53:03 t kernel: [241918.102960] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8
> > > Dec 29 20:53:03 t kernel: [241918.102960] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65
> > > Dec 29 20:53:03 t kernel: [241918.102960] RSP: 0000:ffff8801225b9d20 EFLAGS: 00010246
> > > Dec 29 20:53:03 t kernel: [241918.102960] RAX: 0000000000000000 RBX: ffff880120435148 RCX: ffffffff8139f927
> > > Dec 29 20:53:03 t kernel: [241918.102960] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff880120435148
> > > Dec 29 20:53:03 t kernel: [241918.102960] RBP: ffff8801225b9d30 R08: 0000000000000020 R09: ffff88002820c9d0
> > > Dec 29 20:53:03 t kernel: [241918.102960] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff880120435100
> > > Dec 29 20:53:03 t kernel: [241918.102960] R13: ffffffffa028ef50 R14: ffff8800c1b3b800 R15: ffff88012dd423c0
> > > Dec 29 20:53:03 t kernel: [241918.102960] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
> > > Dec 29 20:53:03 t kernel: [241918.102960] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> > > Dec 29 20:53:03 t kernel: [241918.102960] CR2: 00007fad86c2e000 CR3: 0000000129ae6000 CR4: 00000000000006e0
> > > Dec 29 20:53:03 t kernel: [241918.102960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > Dec 29 20:53:03 t kernel: [241918.102960] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > Dec 29 20:53:03 t kernel: [241918.102960] Process nfsd (pid: 3207, threadinfo ffff8801225b8000, task ffff880122416640)
> > > Dec 29 20:53:03 t kernel: [241918.102960] Stack:
> > > Dec 29 20:53:03 t kernel: [241918.102960] ffff8801225b9d90 ffff8800c1b3b800 ffff8801225b9d50 ffffffff811f4be2
> > > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8800c1b3b800 ffff8800c1b3b810 ffff8801225b9d70 ffffffffa027c1d2
> > > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8801225b9d70 ffff8800c1b3b800 ffff8801225b9da0 ffffffffa0286b03
> > > Dec 29 20:53:03 t kernel: [241918.102960] Call Trace:
> > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff811f4be2>] sock_release+0x5c/0x6c
> > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc]
> > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc]
> > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc]
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81142201>] kref_put+0x43/0x4f
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc]
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc]
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > Dec 29 20:53:04 t kernel: [241918.102960] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85
> > > Dec 29 20:53:04 t kernel: [241918.102960] RIP [<ffffffff810bccc2>] iput+0x1b/0x65
> > > Dec 29 20:53:04 t kernel: [241918.102960] RSP <ffff8801225b9d20>
> > > Dec 29 20:53:04 t kernel: [241919.822190] ---[ end trace 1a77408724500fb1 ]---
> > > Dec 30 11:22:26 t kernel: [294082.152979] 3w-9xxx: scsi6: AEN: ERROR (0x04:0x0009): Drive timeout detected:port=0.
> > > Dec 31 11:24:44 t kernel: [380620.428029] br0: port 2(eth2) entering disabled state
> > > Dec 31 11:24:44 t kernel: [380620.458826] br0: port 1(eth1) entering disabled state
> > > Dec 31 11:24:44 t kernel: [380620.666738] device eth1 left promiscuous mode
> > > Dec 31 11:24:44 t kernel: [380620.693361] br0: port 1(eth1) entering disabled state
> > > Dec 31 11:24:44 t kernel: [380620.909921] device eth2 left promiscuous mode
> > > Dec 31 11:24:44 t kernel: [380620.936603] br0: port 2(eth2) entering disabled state
> > > Dec 31 11:24:48 t kernel: [380624.449637] device eth1 entered promiscuous mode
> > > Dec 31 11:24:48 t kernel: [380624.533887] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X
> > > Dec 31 11:24:48 t kernel: [380624.620065] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X
> > > Dec 31 11:24:48 t kernel: [380624.651581] ADDRCONF(NETDEV_UP): eth1: link is not ready
> > > Dec 31 11:24:48 t kernel: [380624.687675] device eth2 entered promiscuous mode
> > > Dec 31 11:24:48 t kernel: [380624.860445] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X
> > > Dec 31 11:24:48 t kernel: [380624.944084] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X
> > > Dec 31 11:24:48 t kernel: [380624.975824] ADDRCONF(NETDEV_UP): eth2: link is not ready
> > > Dec 31 11:24:50 t kernel: [380626.984998] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > > Dec 31 11:24:51 t kernel: [380627.031517] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
> > > Dec 31 11:24:51 t kernel: [380627.066844] br0: port 2(eth2) entering forwarding state
> > > Dec 31 11:24:51 t kernel: [380627.251584] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > > Dec 31 11:24:51 t kernel: [380627.300433] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
> > > Dec 31 11:24:51 t kernel: [380627.335625] br0: port 1(eth1) entering forwarding state
> > > Dec 31 11:24:59 t kernel: [380635.148016] br0: no IPv6 routers present
> > > Dec 31 11:25:01 t kernel: [380637.504511] eth1: no IPv6 routers present
> > > Dec 31 11:25:01 t kernel: [380637.720515] eth2: no IPv6 routers present
> > > Dec 31 11:28:30 t kernel: [380846.205549] e1000e: eth1 NIC Link is Down
> > > Dec 31 11:28:30 t kernel: [380846.248659] br0: port 1(eth1) entering disabled state
> > > Dec 31 11:28:32 t kernel: [380848.693501] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX
> > > Dec 31 11:28:32 t kernel: [380848.739167] 0000:06:00.1: eth1: 10/100 speed: disabling TSO
> > > Dec 31 11:28:32 t kernel: [380848.776396] br0: port 1(eth1) entering forwarding state
> > > Dec 31 11:28:49 t kernel: [380865.803038] e1000e: eth1 NIC Link is Down
> > > Dec 31 11:28:49 t kernel: [380865.874086] br0: port 1(eth1) entering disabled state
> > > Dec 31 11:28:52 t kernel: [380868.267943] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > > Dec 31 11:28:52 t kernel: [380868.316772] br0: port 1(eth1) entering forwarding state
> > > Dec 31 11:29:18 t kernel: [380894.253543] e1000e: eth1 NIC Link is Down
> > > Dec 31 11:29:18 t kernel: [380894.293597] br0: port 1(eth1) entering disabled state
> > > Dec 31 11:29:20 t kernel: [380896.688442] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > > Dec 31 11:29:20 t kernel: [380896.737255] br0: port 1(eth1) entering forwarding state
> > > Dec 31 11:29:25 t kernel: [380901.493543] e1000e: eth1 NIC Link is Down
> > > Dec 31 11:29:25 t kernel: [380901.534074] br0: port 1(eth1) entering disabled state
> > > Dec 31 11:29:28 t kernel: [380904.161008] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > > Dec 31 11:29:28 t kernel: [380904.209842] br0: port 1(eth1) entering forwarding state
> > > Jan 1 22:40:06 t kernel: [507542.268700] ------------[ cut here ]------------
> > > Jan 1 22:40:08 t kernel: [507542.296925] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> > > Jan 1 22:40:08 t kernel: [507542.330301] Hardware name: X7DB8
> > > Jan 1 22:40:08 t kernel: [507542.350163] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > Jan 1 22:40:08 t kernel: [507542.614635] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1
> > > Jan 1 22:40:08 t kernel: [507542.652191] Call Trace:
> > > Jan 1 22:40:08 t kernel: [507542.667375] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > > Jan 1 22:40:08 t kernel: [507542.698681] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> > > Jan 1 22:40:08 t kernel: [507542.735186] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> > > Jan 1 22:40:08 t kernel: [507542.770130] [<ffffffff81142230>] kref_get+0x23/0xBj
> > > Jan 1 22:40:08 t kernel: [507542.800400] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > Jan 1 22:40:08 t kernel: [507542.833269] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> > > Jan 1 22:40:08 t kernel: [507542.869251] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > Jan 1 22:40:08 t kernel: [507542.907825] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > Jan 1 22:40:08 t kernel: [507542.944853] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > Jan 1 22:40:08 t kernel: [507542.977720] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > Jan 1 22:40:08 t kernel: [507543.010063] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > Jan 1 22:40:08 t kernel: [507543.039808] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > Jan 1 22:40:08 t kernel: [507543.070071] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > Jan 1 22:40:08 t kernel: [507543.100337] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > Jan 1 22:40:08 t kernel: [507543.131640] ---[ end trace 1a77408724500fb2 ]---
> > > Jan 1 22:40:08 t kernel: [507543.160026] general protection fault: 0000 [#3] SMP
> > > Jan 1 22:40:08 t kernel: [507543.165025] last sysfs file: /sys/class/net/eth2/brport/hold_timer
> > > Jan 1 22:40:08 t kernel: [507543.165025] CPU 3
> > > Jan 1 22:40:08 t kernel: [507543.165025] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > Jan 1 22:40:08 t kernel: [507543.165025] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8
> > > Jan 1 22:40:08 t kernel: [507543.165025] RIP: 0010:[<ffffffffa0286797>] [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc]
> > > Jan 1 22:40:08 t kernel: [507543.165025] RSP: 0018:ffff880124ea1e00 EFLAGS: 00010246
> > > Jan 1 22:40:08 t kernel: [507543.165025] RAX: 3a30343a32322031 RBX: 0000000000000000 RCX: 0000000000000000
> > > Jan 1 22:40:08 t kernel: [507543.165025] RDX: 0000000000004c86 RSI: ffff880124ea1df0 RDI: ffff880124e00000
> > > Jan 1 22:40:08 t kernel: [507543.165025] RBP: ffff880124ea1eb0 R08: 0000000000000020 R09: ffff88002830ca70
> > > Jan 1 22:40:08 t kernel: [507543.165025] R10: ffff88002838cf40 R11: 0000000000000000 R12: 0000000000000082
> > > Jan 1 22:40:08 t kernel: [507543.165025] R13: ffff880124e00000 R14: ffff8800c7444000 R15: ffff88012dd423c0
> > > Jan 1 22:40:08 t kernel: [507543.165025] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000
> > > Jan 1 22:40:08 t kernel: [507543.165025] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> > > Jan 1 22:40:08 t kernel: [507543.165025] CR2: 00007f0dc9cc5098 CR3: 000000012c266000 CR4: 00000000000006e0
> > > Jan 1 22:40:08 t kernel: [507543.165025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > Jan 1 22:40:08 t kernel: [507543.165025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > Jan 1 22:40:08 t kernel: [507543.165025] Process nfsd (pid: 3134, threadinfo ffff880124ea0000, task ffff880124e9f890)
> > > Jan 1 22:40:08 t kernel: [507543.165025] Stack:
> > > Jan 1 22:40:08 t kernel: [507543.165025] ffff880124e00000 ffff880124e9f890 ffff880124ea1e40 00000000000dbba0
> > > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff88012dd423c4 ffff88012c0bea40 ffff880000000082 0000000000000001
> > > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff880124ea1eb0 ffffffffa027abe4 0000000000000000 ffff880124e9f890
> > > Jan 1 22:40:08 t kernel: [507543.165025] Call Trace:
> > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > Jan 1 22:40:08 t kernel: [507543.165025] Code: 00 49 8d 85 88 01 00 00 49 89 85 a0 09 00 00 8b 82 e8 00 00 00 8d 1c 85 00 00 00 00 2b 9a e0 00 00 00 eb 0c 49 8b 46 08 4c 89 ef <ff> 50 18 89 c3 f6 05 56 ce 00 00 01 74 10 89 de 48 c7 c7 cf b6
> > > Jan 1 22:40:08 t kernel: [507543.165025] RIP [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc]
> > > Jan 1 22:40:08 t kernel: [507543.165025] RSP <ffff880124ea1e00>
> > > Jan 1 22:40:08 t kernel: [507544.713111] ---[ end trace 1a77408724500fb3 ]---
> > > Jan 2 12:06:02 t kernel: [555898.365986] RPC: multiple fragments per record not supported
> > > Jan 2 12:06:02 t kernel: [555898.527050] RPC: multiple fragments per record not supported
> > > Jan 2 12:06:05 t kernel: [555901.700485] RPC: multiple fragments per record not supported
> > > Jan 2 12:06:08 t kernel: [555904.880825] RPC: multiple fragments per record not supported
> > > Jan 3 00:01:08 t kernel: [598804.533706] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0.
> > > Jan 3 00:57:01 t kernel: [602157.347383] md: data-check of RAID array md0
> > > Jan 3 00:57:01 t kernel: [602157.350802] md: delaying data-check of md2 until md0 has finished (they share one or more physical units)
> > > Jan 3 00:57:01 t kernel: [602157.431453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> > > Jan 3 00:57:01 t kernel: [602157.467054] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> > > Jan 3 00:57:01 t kernel: [602157.526113] md: using 128k window, over a total of 19542976 blocks.
> > > Jan 3 01:03:47 t kernel: [602563.525129] md: md0: data-check done.
> > > Jan 3 01:03:47 t kernel: [602563.554886] md: data-check of RAID array md2
> > > Jan 3 01:03:47 t kernel: [602563.581194] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> > > Jan 3 01:03:47 t kernel: [602563.616879] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> > > Jan 3 01:03:47 t kernel: [602563.675944] md: using 128k window, over a total of 460928832 blocks.
> > > Jan 3 04:47:09 t kernel: [615965.563359] md: md2: data-check done.
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
> --
> -------------------------------------
> Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28. rijna 168, 709 01 Ostrava
>
> tel.: +420 596 603 142
> fax: +420 596 621 273
> mobil: +420 777 093 799
>
> www.linuxbox.cz
>
> mobil servis: +420 737 238 656
> email servis: servis-Jp3n8lUXroTtwjQa/ONI9g@public.gmane.org
> -------------------------------------
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
2010-02-16 20:21 ` J. Bruce Fields
@ 2010-02-16 20:27 ` Nikola Ciprich
[not found] ` <20100216202720.GB4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org>
0 siblings, 1 reply; 12+ messages in thread
From: Nikola Ciprich @ 2010-02-16 20:27 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs,
nikola.ciprich
> Nothing yet. I'm taking another look.
>
> Do you know what the most recent version was where you did *not* see
> these warnings?
well, actually I decided I'll bisect this one no matter what, so hopefully
we'll see soon :)
I'll report
n.
>
> --b.
>
> > nik
> >
> >
> > On Tue, Feb 02, 2010 at 11:22:28AM -0500, J. Bruce Fields wrote:
> > > On Wed, Jan 27, 2010 at 03:01:40PM -0800, Harry Edmon wrote:
> > > > J. Bruce Fields wrote:
> > > >> On Fri, Jan 22, 2010 at 03:03:25PM -0800, Andrew Morton wrote:
> > > >>
> > > >>> (cc linux-nfs)
> > > >>>
> > > >>> On Thu, 14 Jan 2010 09:53:04 -0800
> > > >>> Harry Edmon <harry@atmos.washington.edu> wrote:
> > > >>>
> > > >>>
> > > >>>> I have a single quad-core Xeon machine running 2.6.32.2 that keeps
> > > >>>> giving me "WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()". This
> > > >>>> machine is mainaly used as a NFS server. I have attached the
> > > >>>> kern.log and my config file. Any ideas?
> > > >>>>
> > > >>
> > > >> Are you using NFSv4, or only v2/v3? Looking at the trace:
> > > >>
> > > >> Jan 11 06:01:51 t kernel: [1311646.832173] ------------[ cut here ]------------
> > > >> Jan 11 06:01:53 t kernel: [1311646.860891] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> > > >>
> > > > NFSv4 and NFSv3.
> > >
> > > Thanks. And you aren't using rdma are you?
> > >
> > > >> OK, so that's the warning that we're trying to get something with
> > > >> reference count 0.
> > > >>
> > > >> Jan 11 06:01:53 t kernel: [1311646.894782] Hardware name: X7DB8
> > > >> Jan 11 06:01:53 t kernel: [1311646.915165] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > >> Jan 11 06:01:53 t kernel: [1311647.178038] Pid: 3068, comm: nfsd Tainted: G D W 2.6.32.2 #1
> > > >>
> > > >> What does the very first oops, bug, or warning you get look like? (If
> > > >> I'm reading the tain information right, this isn't the first one.)
> > > >>
> > > > Attached is the kern.log that contains the first warning.
> > >
> > > Thanks. So yes, the first warning is a stray kref_put() in svc_recv(),
> > > and the such put I see on a quick skim is a svc_xprt_put(). The
> > > following BUG() is confirmation, since it's an attempt inside
> > > scv-xprt_release() to clear an inode with I_CLEAR already set; so we
> > > dropped the reference count to zero, cleared it, then tried to do
> > > another get/put pair, resulting in an attempt to free the same socket
> > > twice.
> > >
> > > --b.
> > >
> > > >
> > > > --
> > > > Dr. Harry Edmon E-MAIL: harry@atmos.washington.edu
> > > > 206-543-0547 FAX: 206-543-0308 harry@washington.edu, harry@uw.edu
> > > > Director of IT, College of the Environment and
> > > > Director of Computing, Dept of Atmospheric Sciences
> > > > University of Washington, Box 351640, Seattle, WA 98195-1640
> > > >
> > >
> > > > Dec 27 11:57:44 t kernel: [37000.838904] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0055): Battery charging started:.
> > > > Dec 27 11:57:46 t kernel: [37002.028946] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0056): Battery charging completed:.
> > > > Dec 28 00:00:51 t kernel: [80387.429120] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0.
> > > > Dec 28 03:32:07 t kernel: [93063.950486] 3w-9xxx: scsi6: AEN: INFO (0x04:0x002B): Verify completed:unit=0.
> > > > Dec 29 17:04:43 t kernel: [228218.883443] ------------[ cut here ]------------
> > > > Dec 29 17:04:45 t kernel: [228218.911691] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> > > > Dec 29 17:04:45 t kernel: [228218.945083] Hardware name: X7DB8
> > > > Dec 29 17:04:45 t kernel: [228218.964948] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > > Dec 29 17:04:45 t kernel: [228219.227296] Pid: 3108, comm: nfsd Not tainted 2.6.32.2 #1
> > > > Dec 29 17:04:45 t kernel: [228219.260164] Call Trace:
> > > > Dec 29 17:04:45 t kernel: [228219.275350] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > > > Dec 29 17:04:45 t kernel: [228219.306656] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> > > > Dec 29 17:04:45 t kernel: [228219.343160] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> > > > Dec 29 17:04:45 t kernel: [228219.378105] [<ffffffff81142230>] kref_get+0x23/0x2b
> > > > Dec 29 17:04:45 t kernel: [228219.408379] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > > Dec 29 17:04:45 t kernel: [228219.441242] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> > > > Dec 29 17:04:45 t kernel: [228219.477225] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > > Dec 29 17:04:45 t kernel: [228219.515806] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > > Dec 29 17:04:45 t kernel: [228219.552834] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > > Dec 29 17:04:45 t kernel: [228219.585698] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > > Dec 29 17:04:45 t kernel: [228219.618043] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > > Dec 29 17:04:45 t kernel: [228219.647789] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > > Dec 29 17:04:45 t kernel: [228219.678052] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > > Dec 29 17:04:45 t kernel: [228219.708317] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > > Dec 29 17:04:45 t kernel: [228219.739621] ---[ end trace 1a77408724500fae ]---
> > > > Dec 29 17:04:45 t kernel: [228219.767925] ------------[ cut here ]------------
> > > > Dec 29 17:04:45 t kernel: [228219.792730] kernel BUG at fs/inode.c:1343!
> > > > Dec 29 17:04:45 t kernel: [228219.792730] invalid opcode: 0000 [#1] SMP
> > > > Dec 29 17:04:45 t kernel: [228219.792730] last sysfs file: /sys/class/scsi_host/host6/stats
> > > > Dec 29 17:04:45 t kernel: [228219.792730] CPU 0
> > > > Dec 29 17:04:45 t kernel: [228219.792730] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > > Dec 29 17:04:45 t kernel: [228219.792730] Pid: 3108, comm: nfsd Tainted: G W 2.6.32.2 #1 X7DB8
> > > > Dec 29 17:04:45 t kernel: [228219.792730] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65
> > > > Dec 29 17:04:45 t kernel: [228219.792730] RSP: 0018:ffff88012588bd20 EFLAGS: 00010246
> > > > Dec 29 17:04:45 t kernel: [228219.792730] RAX: 0000000000000000 RBX: ffff88003c3c2cc8 RCX: ffffffff8139f927
> > > > Dec 29 17:04:45 t kernel: [228219.792730] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff88003c3c2cc8
> > > > Dec 29 17:04:45 t kernel: [228219.792730] RBP: ffff88012588bd30 R08: 0000000000000002 R09: 0000000000000000
> > > > Dec 29 17:04:45 t kernel: [228219.792730] R10: ffffffff81142230 R11: 0000000100000001 R12: ffff88003c3c2c80
> > > > Dec 29 17:04:45 t kernel: [228219.792730] R13: ffffffffa028ef50 R14: ffff88010072d400 R15: ffff88012dd423c0
> > > > Dec 29 17:04:45 t kernel: [228219.792730] FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
> > > > Dec 29 17:04:45 t kernel: [228219.792730] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> > > > Dec 29 17:04:45 t kernel: [228219.792730] CR2: 00007fad86c2e000 CR3: 00000000305e5000 CR4: 00000000000006f0
> > > > Dec 29 17:04:45 t kernel: [228219.792730] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > Dec 29 17:04:45 t kernel: [228219.792730] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > Dec 29 17:04:45 t kernel: [228219.792730] Process nfsd (pid: 3108, threadinfo ffff88012588a000, task ffff880125e66150)
> > > > Dec 29 17:04:45 t kernel: [228219.792730] Stack:
> > > > Dec 29 17:04:45 t kernel: [228219.792730] ffff88012588bd90 ffff88010072d400 ffff88012588bd50 ffffffff811f4be2
> > > > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88010072d400 ffff88010072d410 ffff88012588bd70 ffffffffa027c1d2
> > > > Dec 29 17:04:45 t kernel: [228219.792730] <0> ffff88012588bd70 ffff88010072d400 ffff88012588bda0 ffffffffa0286b03
> > > > Dec 29 17:04:45 t kernel: [228219.792730] Call Trace:
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff811f4be2>] sock_release+0x5c/0x6c
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc]
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc]
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc]
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81142201>] kref_put+0x43/0x4f
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc]
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc]
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > > Dec 29 17:04:45 t kernel: [228219.792730] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > > Dec 29 17:04:45 t kernel: [228219.792730] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85
> > > > Dec 29 17:04:45 t kernel: [228219.792730] RIP [<ffffffff810bccc2>] iput+0x1b/0x65
> > > > Dec 29 17:04:45 t kernel: [228219.792730] RSP <ffff88012588bd20>
> > > > Dec 29 17:04:45 t kernel: [228221.632694] ---[ end trace 1a77408724500faf ]---
> > > > Dec 29 20:53:01 t kernel: [241917.056374] ------------[ cut here ]------------
> > > > Dec 29 20:53:01 t kernel: [241917.084581] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> > > > Dec 29 20:53:01 t kernel: [241917.117941] Hardware name: X7DB8
> > > > Dec 29 20:53:01 t kernel: [241917.137804] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > > Dec 29 20:53:01 t kernel: [241917.400101] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1
> > > > Dec 29 20:53:01 t kernel: [241917.437653] Call Trace:
> > > > Dec 29 20:53:01 t kernel: [241917.452843] [<ffffffffa0286b0b>] ? svc_xprt_free+0x46/0x4e [sunrpc]
> > > > Dec 29 20:53:01 t kernel: [241917.491420] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > > > Dec 29 20:53:01 t kernel: [241917.522728] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> > > > Dec 29 20:53:01 t kernel: [241917.559230] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> > > > Dec 29 20:53:01 t kernel: [241917.594177] [<ffffffff81142230>] kref_get+0x23/0x2b
> > > > Dec 29 20:53:01 t kernel: [241917.624446] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> > > > Dec 29 20:53:01 t kernel: [241917.660432] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > > Dec 29 20:53:01 t kernel: [241917.699013] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > > Dec 29 20:53:01 t kernel: [241917.736041] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > > Dec 29 20:53:01 t kernel: [241917.768905] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > > Dec 29 20:53:01 t kernel: [241917.801249] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > > Dec 29 20:53:01 t kernel: [241917.830993] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > > Dec 29 20:53:01 t kernel: [241917.861258] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > > Dec 29 20:53:01 t kernel: [241917.891522] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > > Dec 29 20:53:01 t kernel: [241917.922827] ---[ end trace 1a77408724500fb0 ]---
> > > > Dec 29 20:53:01 t kernel: [241917.951084] ------------[ cut here ]------------
> > > > Dec 29 20:53:03 t kernel: [241917.974874] kernel BUG at fs/inode.c:1343!
> > > > Dec 29 20:53:03 t kernel: [241917.988010] invalid opcode: 0000 [#2] SMP
> > > > Dec 29 20:53:03 t kernel: [241917.988010] last sysfs file: /sys/class/scsi_host/host6/stats
> > > > Dec 29 20:53:03 t kernel: [241917.988010] CPU 1
> > > > Dec 29 20:53:03 t kernel: [241917.988010] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > > Dec 29 20:53:03 t kernel: [241918.102960] Pid: 3207, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8
> > > > Dec 29 20:53:03 t kernel: [241918.102960] RIP: 0010:[<ffffffff810bccc2>] [<ffffffff810bccc2>] iput+0x1b/0x65
> > > > Dec 29 20:53:03 t kernel: [241918.102960] RSP: 0000:ffff8801225b9d20 EFLAGS: 00010246
> > > > Dec 29 20:53:03 t kernel: [241918.102960] RAX: 0000000000000000 RBX: ffff880120435148 RCX: ffffffff8139f927
> > > > Dec 29 20:53:03 t kernel: [241918.102960] RDX: 0000000000000000 RSI: 0000000000000040 RDI: ffff880120435148
> > > > Dec 29 20:53:03 t kernel: [241918.102960] RBP: ffff8801225b9d30 R08: 0000000000000020 R09: ffff88002820c9d0
> > > > Dec 29 20:53:03 t kernel: [241918.102960] R10: ffffffff81142230 R11: 0000000000000000 R12: ffff880120435100
> > > > Dec 29 20:53:03 t kernel: [241918.102960] R13: ffffffffa028ef50 R14: ffff8800c1b3b800 R15: ffff88012dd423c0
> > > > Dec 29 20:53:03 t kernel: [241918.102960] FS: 0000000000000000(0000) GS:ffff880028280000(0000) knlGS:0000000000000000
> > > > Dec 29 20:53:03 t kernel: [241918.102960] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> > > > Dec 29 20:53:03 t kernel: [241918.102960] CR2: 00007fad86c2e000 CR3: 0000000129ae6000 CR4: 00000000000006e0
> > > > Dec 29 20:53:03 t kernel: [241918.102960] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > Dec 29 20:53:03 t kernel: [241918.102960] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > Dec 29 20:53:03 t kernel: [241918.102960] Process nfsd (pid: 3207, threadinfo ffff8801225b8000, task ffff880122416640)
> > > > Dec 29 20:53:03 t kernel: [241918.102960] Stack:
> > > > Dec 29 20:53:03 t kernel: [241918.102960] ffff8801225b9d90 ffff8800c1b3b800 ffff8801225b9d50 ffffffff811f4be2
> > > > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8800c1b3b800 ffff8800c1b3b810 ffff8801225b9d70 ffffffffa027c1d2
> > > > Dec 29 20:53:03 t kernel: [241918.102960] <0> ffff8801225b9d70 ffff8800c1b3b800 ffff8801225b9da0 ffffffffa0286b03
> > > > Dec 29 20:53:03 t kernel: [241918.102960] Call Trace:
> > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff811f4be2>] sock_release+0x5c/0x6c
> > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa027c1d2>] svc_sock_free+0x45/0x51 [sunrpc]
> > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffffa0286b03>] svc_xprt_free+0x3e/0x4e [sunrpc]
> > > > Dec 29 20:53:03 t kernel: [241918.102960] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0286ac5>] ? svc_xprt_free+0x0/0x4e [sunrpc]
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81142201>] kref_put+0x43/0x4f
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa0285f9f>] svc_xprt_release+0xc6/0xcf [sunrpc]
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa02867fc>] svc_recv+0x782/0x7e6 [sunrpc]
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > > Dec 29 20:53:04 t kernel: [241918.102960] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > > Dec 29 20:53:04 t kernel: [241918.102960] Code: c0 74 d5 48 83 c4 20 5b 41 5c 41 5d 41 5e c9 c3 55 48 89 e5 53 48 89 fb 48 83 ec 08 48 85 ff 74 50 48 83 bf 10 02 00 00 40 75 04 <0f> 0b eb fe 48 8d 7f 48 48 c7 c6 a0 7d 4a 81 e8 e2 31 08 00 85
> > > > Dec 29 20:53:04 t kernel: [241918.102960] RIP [<ffffffff810bccc2>] iput+0x1b/0x65
> > > > Dec 29 20:53:04 t kernel: [241918.102960] RSP <ffff8801225b9d20>
> > > > Dec 29 20:53:04 t kernel: [241919.822190] ---[ end trace 1a77408724500fb1 ]---
> > > > Dec 30 11:22:26 t kernel: [294082.152979] 3w-9xxx: scsi6: AEN: ERROR (0x04:0x0009): Drive timeout detected:port=0.
> > > > Dec 31 11:24:44 t kernel: [380620.428029] br0: port 2(eth2) entering disabled state
> > > > Dec 31 11:24:44 t kernel: [380620.458826] br0: port 1(eth1) entering disabled state
> > > > Dec 31 11:24:44 t kernel: [380620.666738] device eth1 left promiscuous mode
> > > > Dec 31 11:24:44 t kernel: [380620.693361] br0: port 1(eth1) entering disabled state
> > > > Dec 31 11:24:44 t kernel: [380620.909921] device eth2 left promiscuous mode
> > > > Dec 31 11:24:44 t kernel: [380620.936603] br0: port 2(eth2) entering disabled state
> > > > Dec 31 11:24:48 t kernel: [380624.449637] device eth1 entered promiscuous mode
> > > > Dec 31 11:24:48 t kernel: [380624.533887] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X
> > > > Dec 31 11:24:48 t kernel: [380624.620065] e1000e 0000:06:00.1: irq 55 for MSI/MSI-X
> > > > Dec 31 11:24:48 t kernel: [380624.651581] ADDRCONF(NETDEV_UP): eth1: link is not ready
> > > > Dec 31 11:24:48 t kernel: [380624.687675] device eth2 entered promiscuous mode
> > > > Dec 31 11:24:48 t kernel: [380624.860445] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X
> > > > Dec 31 11:24:48 t kernel: [380624.944084] e1000e 0000:08:00.0: irq 56 for MSI/MSI-X
> > > > Dec 31 11:24:48 t kernel: [380624.975824] ADDRCONF(NETDEV_UP): eth2: link is not ready
> > > > Dec 31 11:24:50 t kernel: [380626.984998] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > > > Dec 31 11:24:51 t kernel: [380627.031517] ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
> > > > Dec 31 11:24:51 t kernel: [380627.066844] br0: port 2(eth2) entering forwarding state
> > > > Dec 31 11:24:51 t kernel: [380627.251584] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > > > Dec 31 11:24:51 t kernel: [380627.300433] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
> > > > Dec 31 11:24:51 t kernel: [380627.335625] br0: port 1(eth1) entering forwarding state
> > > > Dec 31 11:24:59 t kernel: [380635.148016] br0: no IPv6 routers present
> > > > Dec 31 11:25:01 t kernel: [380637.504511] eth1: no IPv6 routers present
> > > > Dec 31 11:25:01 t kernel: [380637.720515] eth2: no IPv6 routers present
> > > > Dec 31 11:28:30 t kernel: [380846.205549] e1000e: eth1 NIC Link is Down
> > > > Dec 31 11:28:30 t kernel: [380846.248659] br0: port 1(eth1) entering disabled state
> > > > Dec 31 11:28:32 t kernel: [380848.693501] e1000e: eth1 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX
> > > > Dec 31 11:28:32 t kernel: [380848.739167] 0000:06:00.1: eth1: 10/100 speed: disabling TSO
> > > > Dec 31 11:28:32 t kernel: [380848.776396] br0: port 1(eth1) entering forwarding state
> > > > Dec 31 11:28:49 t kernel: [380865.803038] e1000e: eth1 NIC Link is Down
> > > > Dec 31 11:28:49 t kernel: [380865.874086] br0: port 1(eth1) entering disabled state
> > > > Dec 31 11:28:52 t kernel: [380868.267943] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > > > Dec 31 11:28:52 t kernel: [380868.316772] br0: port 1(eth1) entering forwarding state
> > > > Dec 31 11:29:18 t kernel: [380894.253543] e1000e: eth1 NIC Link is Down
> > > > Dec 31 11:29:18 t kernel: [380894.293597] br0: port 1(eth1) entering disabled state
> > > > Dec 31 11:29:20 t kernel: [380896.688442] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > > > Dec 31 11:29:20 t kernel: [380896.737255] br0: port 1(eth1) entering forwarding state
> > > > Dec 31 11:29:25 t kernel: [380901.493543] e1000e: eth1 NIC Link is Down
> > > > Dec 31 11:29:25 t kernel: [380901.534074] br0: port 1(eth1) entering disabled state
> > > > Dec 31 11:29:28 t kernel: [380904.161008] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
> > > > Dec 31 11:29:28 t kernel: [380904.209842] br0: port 1(eth1) entering forwarding state
> > > > Jan 1 22:40:06 t kernel: [507542.268700] ------------[ cut here ]------------
> > > > Jan 1 22:40:08 t kernel: [507542.296925] WARNING: at lib/kref.c:43 kref_get+0x23/0x2b()
> > > > Jan 1 22:40:08 t kernel: [507542.330301] Hardware name: X7DB8
> > > > Jan 1 22:40:08 t kernel: [507542.350163] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > > Jan 1 22:40:08 t kernel: [507542.614635] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1
> > > > Jan 1 22:40:08 t kernel: [507542.652191] Call Trace:
> > > > Jan 1 22:40:08 t kernel: [507542.667375] [<ffffffff81142230>] ? kref_get+0x23/0x2b
> > > > Jan 1 22:40:08 t kernel: [507542.698681] [<ffffffff81038943>] warn_slowpath_common+0x77/0xa4
> > > > Jan 1 22:40:08 t kernel: [507542.735186] [<ffffffff8103897f>] warn_slowpath_null+0xf/0x11
> > > > Jan 1 22:40:08 t kernel: [507542.770130] [<ffffffff81142230>] kref_get+0x23/0xBj
> > > > Jan 1 22:40:08 t kernel: [507542.800400] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > > Jan 1 22:40:08 t kernel: [507542.833269] [<ffffffffa028637f>] svc_recv+0x305/0x7e6 [sunrpc]
> > > > Jan 1 22:40:08 t kernel: [507542.869251] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > > Jan 1 22:40:08 t kernel: [507542.907825] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > > Jan 1 22:40:08 t kernel: [507542.944853] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > > Jan 1 22:40:08 t kernel: [507542.977720] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > > Jan 1 22:40:08 t kernel: [507543.010063] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > > Jan 1 22:40:08 t kernel: [507543.039808] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > > Jan 1 22:40:08 t kernel: [507543.070071] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > > Jan 1 22:40:08 t kernel: [507543.100337] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > > Jan 1 22:40:08 t kernel: [507543.131640] ---[ end trace 1a77408724500fb2 ]---
> > > > Jan 1 22:40:08 t kernel: [507543.160026] general protection fault: 0000 [#3] SMP
> > > > Jan 1 22:40:08 t kernel: [507543.165025] last sysfs file: /sys/class/net/eth2/brport/hold_timer
> > > > Jan 1 22:40:08 t kernel: [507543.165025] CPU 3
> > > > Jan 1 22:40:08 t kernel: [507543.165025] Modules linked in: cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 i5000_edac edac_core i2c_i801 parport_pc parport psmouse container i2c_core ioatdma dca i5k_amb serio_raw pcspkr button processor joydev evdev xfs exportfs sg usbhid sr_mod hid cdrom ata_piix floppy uhci_hcd ehci_hcd ata_generic 3w_9xxx e1000e thermal fan raid1 md_mod sd_mod ahci libata scsi_mod [last unloaded: scsi_wait_scan]
> > > > Jan 1 22:40:08 t kernel: [507543.165025] Pid: 3134, comm: nfsd Tainted: G D W 2.6.32.2 #1 X7DB8
> > > > Jan 1 22:40:08 t kernel: [507543.165025] RIP: 0010:[<ffffffffa0286797>] [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc]
> > > > Jan 1 22:40:08 t kernel: [507543.165025] RSP: 0018:ffff880124ea1e00 EFLAGS: 00010246
> > > > Jan 1 22:40:08 t kernel: [507543.165025] RAX: 3a30343a32322031 RBX: 0000000000000000 RCX: 0000000000000000
> > > > Jan 1 22:40:08 t kernel: [507543.165025] RDX: 0000000000004c86 RSI: ffff880124ea1df0 RDI: ffff880124e00000
> > > > Jan 1 22:40:08 t kernel: [507543.165025] RBP: ffff880124ea1eb0 R08: 0000000000000020 R09: ffff88002830ca70
> > > > Jan 1 22:40:08 t kernel: [507543.165025] R10: ffff88002838cf40 R11: 0000000000000000 R12: 0000000000000082
> > > > Jan 1 22:40:08 t kernel: [507543.165025] R13: ffff880124e00000 R14: ffff8800c7444000 R15: ffff88012dd423c0
> > > > Jan 1 22:40:08 t kernel: [507543.165025] FS: 0000000000000000(0000) GS:ffff880028380000(0000) knlGS:0000000000000000
> > > > Jan 1 22:40:08 t kernel: [507543.165025] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> > > > Jan 1 22:40:08 t kernel: [507543.165025] CR2: 00007f0dc9cc5098 CR3: 000000012c266000 CR4: 00000000000006e0
> > > > Jan 1 22:40:08 t kernel: [507543.165025] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > Jan 1 22:40:08 t kernel: [507543.165025] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > Jan 1 22:40:08 t kernel: [507543.165025] Process nfsd (pid: 3134, threadinfo ffff880124ea0000, task ffff880124e9f890)
> > > > Jan 1 22:40:08 t kernel: [507543.165025] Stack:
> > > > Jan 1 22:40:08 t kernel: [507543.165025] ffff880124e00000 ffff880124e9f890 ffff880124ea1e40 00000000000dbba0
> > > > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff88012dd423c4 ffff88012c0bea40 ffff880000000082 0000000000000001
> > > > Jan 1 22:40:08 t kernel: [507543.165025] <0> ffff880124ea1eb0 ffffffffa027abe4 0000000000000000 ffff880124e9f890
> > > > Jan 1 22:40:08 t kernel: [507543.165025] Call Trace:
> > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa027abe4>] ? svc_process+0x759/0x768 [sunrpc]
> > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff81035177>] ? default_wake_function+0x0/0xf
> > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b7df>] ? nfsd+0x0/0x13a [nfsd]
> > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffffa032b886>] nfsd+0xa7/0x13a [nfsd]
> > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104af36>] kthread+0x7d/0x85
> > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c97a>] child_rip+0xa/0x20
> > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8104aeb9>] ? kthread+0x0/0x85
> > > > Jan 1 22:40:08 t kernel: [507543.165025] [<ffffffff8100c970>] ? child_rip+0x0/0x20
> > > > Jan 1 22:40:08 t kernel: [507543.165025] Code: 00 49 8d 85 88 01 00 00 49 89 85 a0 09 00 00 8b 82 e8 00 00 00 8d 1c 85 00 00 00 00 2b 9a e0 00 00 00 eb 0c 49 8b 46 08 4c 89 ef <ff> 50 18 89 c3 f6 05 56 ce 00 00 01 74 10 89 de 48 c7 c7 cf b6
> > > > Jan 1 22:40:08 t kernel: [507543.165025] RIP [<ffffffffa0286797>] svc_recv+0x71d/0x7e6 [sunrpc]
> > > > Jan 1 22:40:08 t kernel: [507543.165025] RSP <ffff880124ea1e00>
> > > > Jan 1 22:40:08 t kernel: [507544.713111] ---[ end trace 1a77408724500fb3 ]---
> > > > Jan 2 12:06:02 t kernel: [555898.365986] RPC: multiple fragments per record not supported
> > > > Jan 2 12:06:02 t kernel: [555898.527050] RPC: multiple fragments per record not supported
> > > > Jan 2 12:06:05 t kernel: [555901.700485] RPC: multiple fragments per record not supported
> > > > Jan 2 12:06:08 t kernel: [555904.880825] RPC: multiple fragments per record not supported
> > > > Jan 3 00:01:08 t kernel: [598804.533706] 3w-9xxx: scsi6: AEN: INFO (0x04:0x0029): Verify started:unit=0.
> > > > Jan 3 00:57:01 t kernel: [602157.347383] md: data-check of RAID array md0
> > > > Jan 3 00:57:01 t kernel: [602157.350802] md: delaying data-check of md2 until md0 has finished (they share one or more physical units)
> > > > Jan 3 00:57:01 t kernel: [602157.431453] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> > > > Jan 3 00:57:01 t kernel: [602157.467054] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> > > > Jan 3 00:57:01 t kernel: [602157.526113] md: using 128k window, over a total of 19542976 blocks.
> > > > Jan 3 01:03:47 t kernel: [602563.525129] md: md0: data-check done.
> > > > Jan 3 01:03:47 t kernel: [602563.554886] md: data-check of RAID array md2
> > > > Jan 3 01:03:47 t kernel: [602563.581194] md: minimum _guaranteed_ speed: 1000 KB/sec/disk.
> > > > Jan 3 01:03:47 t kernel: [602563.616879] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> > > > Jan 3 01:03:47 t kernel: [602563.675944] md: using 128k window, over a total of 460928832 blocks.
> > > > Jan 3 04:47:09 t kernel: [615965.563359] md: md2: data-check done.
> > >
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > >
> >
> > --
> > -------------------------------------
> > Nikola CIPRICH
> > LinuxBox.cz, s.r.o.
> > 28. rijna 168, 709 01 Ostrava
> >
> > tel.: +420 596 603 142
> > fax: +420 596 621 273
> > mobil: +420 777 093 799
> >
> > www.linuxbox.cz
> >
> > mobil servis: +420 737 238 656
> > email servis: servis@linuxbox.cz
> > -------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
--
-------------------------------------
Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava
tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
[not found] ` <20100216202720.GB4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org>
@ 2010-02-16 20:29 ` J. Bruce Fields
2010-02-17 15:14 ` Nikola Ciprich
0 siblings, 1 reply; 12+ messages in thread
From: J. Bruce Fields @ 2010-02-16 20:29 UTC (permalink / raw)
To: Nikola Ciprich
Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs,
nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g
On Tue, Feb 16, 2010 at 09:27:20PM +0100, Nikola Ciprich wrote:
> > Nothing yet. I'm taking another look.
> >
> > Do you know what the most recent version was where you did *not* see
> > these warnings?
> well, actually I decided I'll bisect this one no matter what, so hopefully
> we'll see soon :)
> I'll report
That would be fantastic, thanks. What test-case are you using?
Any partial information ("xxxx good, xxxx bad") would also be useful.
--b.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
2010-02-16 20:29 ` J. Bruce Fields
@ 2010-02-17 15:14 ` Nikola Ciprich
[not found] ` <20100217151435.GC30897-K8br+RTdgN68z6Q06MQEFCF40CQGThNF@public.gmane.org>
0 siblings, 1 reply; 12+ messages in thread
From: Nikola Ciprich @ 2010-02-17 15:14 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs,
nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g
Well,
I did a lot of stress tests, and it seems to be pretty random, and what's
worse, I think it is more then one problem, oh great...
Seems that at least one of my NFS problems was in fact caused by ext4 regression
(what FS are you using?)
applying this one: http://bugzilla.kernel.org/attachment.cgi?id=24975
improved things for me a lot (see http://bugzilla.kernel.org/show_bug.cgi?id=15231 for
full discussion).
I'll watch the machine for next few days and report...
regards
nik
On Tue, Feb 16, 2010 at 03:29:47PM -0500, J. Bruce Fields wrote:
> On Tue, Feb 16, 2010 at 09:27:20PM +0100, Nikola Ciprich wrote:
> > > Nothing yet. I'm taking another look.
> > >
> > > Do you know what the most recent version was where you did *not* see
> > > these warnings?
> > well, actually I decided I'll bisect this one no matter what, so hopefully
> > we'll see soon :)
> > I'll report
>
> That would be fantastic, thanks. What test-case are you using?
>
> Any partial information ("xxxx good, xxxx bad") would also be useful.
>
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava
tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: servis-Jp3n8lUXroTtwjQa/ONI9g@public.gmane.org
-------------------------------------
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
[not found] ` <20100217151435.GC30897-K8br+RTdgN68z6Q06MQEFCF40CQGThNF@public.gmane.org>
@ 2010-02-20 23:51 ` J. Bruce Fields
2010-02-22 5:37 ` Nikola Ciprich
2010-02-24 7:19 ` Jesper Krogh
0 siblings, 2 replies; 12+ messages in thread
From: J. Bruce Fields @ 2010-02-20 23:51 UTC (permalink / raw)
To: Nikola Ciprich
Cc: Harry Edmon, Andrew Morton, linux-kernel, linux-nfs,
nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g
On Wed, Feb 17, 2010 at 04:14:35PM +0100, Nikola Ciprich wrote:
> Well,
> I did a lot of stress tests, and it seems to be pretty random, and what's
> worse, I think it is more then one problem, oh great...
> Seems that at least one of my NFS problems was in fact caused by ext4 regression
> (what FS are you using?)
> applying this one: http://bugzilla.kernel.org/attachment.cgi?id=24975
> improved things for me a lot (see http://bugzilla.kernel.org/show_bug.cgi?id=15231 for
> full discussion).
> I'll watch the machine for next few days and report...
Is there *any* kernel (however old) on which you are confident you can
*not* reproduce the kref bug?
--b.
> regards
> nik
>
> On Tue, Feb 16, 2010 at 03:29:47PM -0500, J. Bruce Fields wrote:
> > On Tue, Feb 16, 2010 at 09:27:20PM +0100, Nikola Ciprich wrote:
> > > > Nothing yet. I'm taking another look.
> > > >
> > > > Do you know what the most recent version was where you did *not* see
> > > > these warnings?
> > > well, actually I decided I'll bisect this one no matter what, so hopefully
> > > we'll see soon :)
> > > I'll report
> >
> > That would be fantastic, thanks. What test-case are you using?
> >
> > Any partial information ("xxxx good, xxxx bad") would also be useful.
> >
> > --b.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
> --
> -------------------------------------
> Ing. Nikola CIPRICH
> LinuxBox.cz, s.r.o.
> 28. rijna 168, 709 01 Ostrava
>
> tel.: +420 596 603 142
> fax: +420 596 621 273
> mobil: +420 777 093 799
> www.linuxbox.cz
>
> mobil servis: +420 737 238 656
> email servis: servis-Jp3n8lUXroTtwjQa/ONI9g@public.gmane.org
> -------------------------------------
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
2010-02-20 23:51 ` J. Bruce Fields
@ 2010-02-22 5:37 ` Nikola Ciprich
2010-02-24 7:19 ` Jesper Krogh
1 sibling, 0 replies; 12+ messages in thread
From: Nikola Ciprich @ 2010-02-22 5:37 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Nikola Ciprich, Harry Edmon, Andrew Morton, linux-kernel,
linux-nfs
[-- Attachment #1: Type: text/plain, Size: 2758 bytes --]
It was working pretty well when we were using 2.6.27.x, but after upgrade, I
also changed FS from XFS to ext4, (so I can't easily test with 2.6.27 now) and
since then NFS is almost unusable on this machine.
But 2.6.27 is pretty old, so there's been a lot of changes.
But greg released 2.6.32.9-rc1 which contains bunch of NFS-related fixes,
so I'll give it a try and report.
nik
On Sat, Feb 20, 2010 at 06:51:05PM -0500, J. Bruce Fields wrote:
> On Wed, Feb 17, 2010 at 04:14:35PM +0100, Nikola Ciprich wrote:
> > Well,
> > I did a lot of stress tests, and it seems to be pretty random, and what's
> > worse, I think it is more then one problem, oh great...
> > Seems that at least one of my NFS problems was in fact caused by ext4 regression
> > (what FS are you using?)
> > applying this one: http://bugzilla.kernel.org/attachment.cgi?id=24975
> > improved things for me a lot (see http://bugzilla.kernel.org/show_bug.cgi?id=15231 for
> > full discussion).
> > I'll watch the machine for next few days and report...
>
> Is there *any* kernel (however old) on which you are confident you can
> *not* reproduce the kref bug?
>
> --b.
>
> > regards
> > nik
> >
> > On Tue, Feb 16, 2010 at 03:29:47PM -0500, J. Bruce Fields wrote:
> > > On Tue, Feb 16, 2010 at 09:27:20PM +0100, Nikola Ciprich wrote:
> > > > > Nothing yet. I'm taking another look.
> > > > >
> > > > > Do you know what the most recent version was where you did *not* see
> > > > > these warnings?
> > > > well, actually I decided I'll bisect this one no matter what, so hopefully
> > > > we'll see soon :)
> > > > I'll report
> > >
> > > That would be fantastic, thanks. What test-case are you using?
> > >
> > > Any partial information ("xxxx good, xxxx bad") would also be useful.
> > >
> > > --b.
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at http://vger.kernel.org/majordomo-info.html
> > >
> >
> > --
> > -------------------------------------
> > Ing. Nikola CIPRICH
> > LinuxBox.cz, s.r.o.
> > 28. rijna 168, 709 01 Ostrava
> >
> > tel.: +420 596 603 142
> > fax: +420 596 621 273
> > mobil: +420 777 093 799
> > www.linuxbox.cz
> >
> > mobil servis: +420 737 238 656
> > email servis: servis@linuxbox.cz
> > -------------------------------------
>
--
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava
tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b()
2010-02-20 23:51 ` J. Bruce Fields
2010-02-22 5:37 ` Nikola Ciprich
@ 2010-02-24 7:19 ` Jesper Krogh
1 sibling, 0 replies; 12+ messages in thread
From: Jesper Krogh @ 2010-02-24 7:19 UTC (permalink / raw)
To: J. Bruce Fields
Cc: Nikola Ciprich, Harry Edmon, Andrew Morton,
public-linux-kernel-u79uwXL29TY76Z2rM5mHXA-wOFGN7rlS/M9smdsby/KFg,
public-linux-nfs-u79uwXL29TY76Z2rM5mHXA-wOFGN7rlS/M9smdsby/KFg,
public-nikola.ciprich-Jp3n8lUXroTtwjQa/ONI9g-wOFGN7rlS/M9smdsby/KFg
J. Bruce Fields wrote:
> On Wed, Feb 17, 2010 at 04:14:35PM +0100, Nikola Ciprich wrote:
>> Well,
>> I did a lot of stress tests, and it seems to be pretty random, and what's
>> worse, I think it is more then one problem, oh great...
>> Seems that at least one of my NFS problems was in fact caused by ext4 regression
>> (what FS are you using?)
>> applying this one: http://bugzilla.kernel.org/attachment.cgi?id=24975
>> improved things for me a lot (see http://bugzilla.kernel.org/show_bug.cgi?id=15231 for
>> full discussion).
>> I'll watch the machine for next few days and report...
>
> Is there *any* kernel (however old) on which you are confident you can
> *not* reproduce the kref bug?
I have got a bug that looks a lot like this on 2.6.32.8 yesterday less
than 24 hours after booting the first 2.6.32 kernel but we have been
running on 2.6.31 for 3 months before, so I'm fairly confident that it
is a .31 -> .32 regression.
http://lkml.org/lkml/2010/2/23/70
Jesper
--
Jesper
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2010-02-24 8:21 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <4B4F5A00.4060604@atmos.washington.edu>
[not found] ` <4B4F5A00.4060604-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org>
2010-01-22 23:03 ` 2.6.32.2 - WARNING: at lib/kref.c:43 kref_get+0x,23/0x2b() Andrew Morton
2010-01-27 19:05 ` J. Bruce Fields
2010-01-27 23:01 ` Harry Edmon
2010-02-02 16:22 ` J. Bruce Fields
2010-02-16 18:22 ` Nikola Ciprich
[not found] ` <20100216182248.GA4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org>
2010-02-16 20:21 ` J. Bruce Fields
2010-02-16 20:27 ` Nikola Ciprich
[not found] ` <20100216202720.GB4922-YtsXe8XL0VS8z6Q06MQEFCF40CQGThNF@public.gmane.org>
2010-02-16 20:29 ` J. Bruce Fields
2010-02-17 15:14 ` Nikola Ciprich
[not found] ` <20100217151435.GC30897-K8br+RTdgN68z6Q06MQEFCF40CQGThNF@public.gmane.org>
2010-02-20 23:51 ` J. Bruce Fields
2010-02-22 5:37 ` Nikola Ciprich
2010-02-24 7:19 ` Jesper Krogh
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox