* Apparent Deadlock with nfsd/jfs on 2.6.21.1 under bonnie.
@ 2007-05-15 14:08 Roger Heflin
[not found] ` <pan.2007.05.15.22.03.07.359081@linux.vnet.ibm.com>
0 siblings, 1 reply; 9+ messages in thread
From: Roger Heflin @ 2007-05-15 14:08 UTC (permalink / raw)
To: Linux-Kernel, NFS
[-- Attachment #1: Type: text/plain, Size: 13135 bytes --]
Hello,
Running 2.6.21.1 (FC6 Dist), with a RHEL client (client
appears to not be having issues) I am getting what I believe
is a deadlock on the server end. This is with JFS and
NFSD, I have not tested yet with a non-JFS filesystem,
though our customer indicated that they have duplicated it with
the ext3 filesystem.
The basic setup is:
fiber channel array -> qlogic fiber card -> /dev/sdx ->
LVM stripe -> jfs -> nfs.
Running bonnie on a NFS share has apparently produced a
deadlock. I have ran bonnie several times without having
any issues, I don't believe this is a HW issue, we have a
couple of other machines configured with slightly different
HW and are also able to duplicate this problem on those
machines. There are no abnormal messages in dmesg or
in the messages file.
After having the apparent deadlock I started a dd of a on
the deadlocked filesystem and according to vmstat 1 that was
actually working, I then did a "mkdir junk" on the deadlocked
filesystem and that apparently put the cat into a permanent
"D" state. I will include the sysrq -t from before the cat/mkdir
and after the cat/mkdir.
I believe I can duplicate this again, and other than the processes
going into the "D" state everything else seems to work. Other
filesytems appear to be functional, I can still login to the
machine.
Right now the machine is in the deadlocked state, and I will
wait for any suggestions of more data to collect or other tests
to try.
Here is the ps -axu list:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 10304 680 ? Ss May14 0:00 init
[3]
root 2 0.0 0.0 0 0 ? S May14 0:00
[migration/0]
root 3 0.0 0.0 0 0 ? SN May14 0:00
[ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S May14 0:00
[watchdog/0]
root 5 0.0 0.0 0 0 ? S May14 0:00
[migration/1]
root 6 0.0 0.0 0 0 ? SN May14 0:00
[ksoftirqd/1]
root 7 0.0 0.0 0 0 ? S May14 0:00
[watchdog/1]
root 8 0.0 0.0 0 0 ? S May14 0:00
[migration/2]
root 9 0.0 0.0 0 0 ? SN May14 0:00
[ksoftirqd/2]
root 10 0.0 0.0 0 0 ? S May14 0:00
[watchdog/2]
root 11 0.0 0.0 0 0 ? S May14 0:00
[migration/3]
root 12 0.1 0.0 0 0 ? SN May14 1:21
[ksoftirqd/3]
root 13 0.0 0.0 0 0 ? S May14 0:00
[watchdog/3]
root 14 0.0 0.0 0 0 ? S< May14 0:00 [events/0]
root 15 0.0 0.0 0 0 ? S< May14 0:00 [events/1]
root 16 0.0 0.0 0 0 ? S< May14 0:00 [events/2]
root 17 0.0 0.0 0 0 ? S< May14 0:00 [events/3]
root 18 0.0 0.0 0 0 ? S< May14 0:00 [khelper]
root 19 0.0 0.0 0 0 ? S< May14 0:00 [kthread]
root 59 0.0 0.0 0 0 ? S< May14 0:00 [kblockd/0]
root 60 0.0 0.0 0 0 ? S< May14 0:00 [kblockd/1]
root 61 0.0 0.0 0 0 ? S< May14 0:00 [kblockd/2]
root 62 0.0 0.0 0 0 ? S< May14 0:00 [kblockd/3]
root 63 0.0 0.0 0 0 ? S< May14 0:00 [kacpid]
root 235 0.0 0.0 0 0 ? S< May14 0:00 [cqueue/0]
root 236 0.0 0.0 0 0 ? S< May14 0:00 [cqueue/1]
root 237 0.0 0.0 0 0 ? S< May14 0:00 [cqueue/2]
root 238 0.0 0.0 0 0 ? S< May14 0:00 [cqueue/3]
root 239 0.0 0.0 0 0 ? S< May14 0:00
[ksuspend_usbd]
root 242 0.0 0.0 0 0 ? S< May14 0:00 [khubd]
root 244 0.0 0.0 0 0 ? S< May14 0:00 [kseriod]
root 298 0.0 0.0 0 0 ? S May14 0:00 [pdflush]
root 299 0.0 0.0 0 0 ? D May14 0:36 [pdflush]
root 300 0.0 0.0 0 0 ? S< May14 0:25 [kswapd0]
root 301 0.0 0.0 0 0 ? S< May14 0:20 [kswapd1]
root 302 0.0 0.0 0 0 ? S< May14 0:00 [aio/0]
root 303 0.0 0.0 0 0 ? S< May14 0:00 [aio/1]
root 304 0.0 0.0 0 0 ? S< May14 0:00 [aio/2]
root 305 0.0 0.0 0 0 ? S< May14 0:00 [aio/3]
root 461 0.0 0.0 0 0 ? S< May14 0:00 [kpsmoused]
root 520 0.0 0.0 0 0 ? S< May14 0:00 [scsi_eh_0]
root 536 0.0 0.0 0 0 ? S< May14 0:00 [scsi_eh_1]
root 537 0.0 0.0 0 0 ? S< May14 0:00
[qla2xxx_1_dpc]
root 538 0.0 0.0 0 0 ? S< May14 0:00 [scsi_wq_1]
root 539 0.0 0.0 0 0 ? S< May14 0:00 [fc_wq_1]
root 540 0.0 0.0 0 0 ? S< May14 0:00 [fc_dl_1]
root 541 0.0 0.0 0 0 ? S< May14 0:00 [scsi_eh_2]
root 542 0.0 0.0 0 0 ? S< May14 0:00
[qla2xxx_2_dpc]
root 543 0.0 0.0 0 0 ? S< May14 0:00 [scsi_wq_2]
root 544 0.0 0.0 0 0 ? S< May14 0:00 [fc_wq_2]
root 545 0.0 0.0 0 0 ? S< May14 0:00 [fc_dl_2]
root 546 0.0 0.0 0 0 ? S< May14 0:00 [scsi_eh_3]
root 547 0.0 0.0 0 0 ? S< May14 0:00
[qla2xxx_3_dpc]
root 548 0.0 0.0 0 0 ? S< May14 0:00 [scsi_wq_3]
root 549 0.0 0.0 0 0 ? S< May14 0:00 [fc_wq_3]
root 550 0.0 0.0 0 0 ? S< May14 0:00 [fc_dl_3]
root 551 0.0 0.0 0 0 ? S< May14 0:00 [scsi_eh_4]
root 552 0.0 0.0 0 0 ? S< May14 0:00
[qla2xxx_4_dpc]
root 553 0.0 0.0 0 0 ? S< May14 0:00 [scsi_wq_4]
root 554 0.0 0.0 0 0 ? S< May14 0:00 [fc_wq_4]
root 555 0.0 0.0 0 0 ? S< May14 0:00 [fc_dl_4]
root 556 0.0 0.0 0 0 ? S< May14 0:00 [kjournald]
root 583 0.0 0.0 0 0 ? S< May14 0:00 [kauditd]
root 617 0.0 0.0 12564 704 ? S<s May14 0:03
/sbin/udevd -d
root 1220 0.0 0.0 0 0 ? S< May14 0:00 [khpsbpkt]
root 1271 0.0 0.0 0 0 ? S< May14 0:00
[knodemgrd_0]
root 1364 0.0 0.0 0 0 ? S< May14 0:00 [ata/0]
root 1365 0.0 0.0 0 0 ? S< May14 0:00 [ata/1]
root 1366 0.0 0.0 0 0 ? S< May14 0:00 [ata/2]
root 1367 0.0 0.0 0 0 ? S< May14 0:00 [ata/3]
root 1368 0.0 0.0 0 0 ? S< May14 0:00 [ata_aux]
root 1675 0.0 0.0 0 0 ? S< May14 0:00 [scsi_eh_5]
root 1677 0.0 0.0 0 0 ? S< May14 0:00 [scsi_eh_6]
root 2076 0.0 0.0 0 0 ? S< May14 0:00 [scsi_eh_7]
root 2077 0.0 0.0 0 0 ? S< May14 0:00 [scsi_eh_8]
root 2515 0.0 0.0 0 0 ? S< May14 0:00 [kmpathd/0]
root 2516 0.0 0.0 0 0 ? S< May14 0:00 [kmpathd/1]
root 2517 0.0 0.0 0 0 ? S< May14 0:00 [kmpathd/2]
root 2518 0.0 0.0 0 0 ? S< May14 0:00 [kmpathd/3]
root 2533 0.0 0.0 0 0 ? S< May14 0:00 [kmirrord]
root 2571 0.0 0.0 0 0 ? S< May14 0:02 [jfsIO]
root 2572 0.0 0.0 0 0 ? S< May14 0:30 [jfsCommit]
root 2573 0.0 0.0 0 0 ? S< May14 0:32 [jfsCommit]
root 2574 0.0 0.0 0 0 ? S< May14 0:30 [jfsCommit]
root 2575 0.0 0.0 0 0 ? S< May14 0:31 [jfsCommit]
root 2576 0.0 0.0 0 0 ? S< May14 0:00 [jfsSync]
root 2578 0.0 0.0 0 0 ? S< May14 0:00 [kjournald]
root 2580 0.0 0.0 0 0 ? S< May14 0:00 [kjournald]
root 2582 0.0 0.0 0 0 ? S< May14 0:01 [kjournald]
root 2712 0.0 0.0 0 0 ? S< May14 0:00
[kondemand/0]
root 2713 0.0 0.0 0 0 ? S< May14 0:00
[kondemand/1]
root 2714 0.0 0.0 0 0 ? S< May14 0:00
[kondemand/2]
root 2715 0.0 0.0 0 0 ? S< May14 0:00
[kondemand/3]
root 3260 0.0 0.0 6700 888 ? Ss May14 0:00
/sbin/dhclient -1 -q -lf /var/lib/dhclient/dhcli
ent-eth1.leases -pf /var/run/dhclient-eth1.pid eth1
root 3374 0.0 0.0 5868 604 ? Ss May14 0:01 syslogd
-m 0
root 3377 0.0 0.0 3760 448 ? Ss May14 0:00 klogd -x
root 3393 0.0 0.0 6436 332 ? Ss May14 0:22 irqbalance
rpc 3428 0.0 0.0 8008 632 ? Ss May14 0:00 portmap
root 3453 0.0 0.0 10100 784 ? Ss May14 0:00 rpc.statd
dbus 3481 0.0 0.0 25452 780 ? Ss May14 0:00
dbus-daemon --system
root 3518 0.0 0.0 8472 480 ? Ss May14 0:00
/usr/bin/hidd --server
root 3535 0.0 0.0 3760 556 ? Ss May14 0:00
/usr/sbin/acpid
root 3553 0.0 0.0 44288 1076 ? Ss May14 0:00
/usr/sbin/sshd
root 3679 0.0 0.0 42004 720 ? Ss May14 0:00 rpc.idmapd
root 3694 0.0 0.0 73876 1124 ? Ss May14 0:00 crond
root 3709 0.0 0.0 18672 460 ? Ss May14 0:00
/usr/sbin/atd
68 3724 0.0 0.0 27240 4600 ? Ss May14 0:02 hald
root 3725 0.0 0.0 17384 952 ? S May14 0:00 hald-runner
68 3790 0.0 0.0 12264 792 ? S May14 0:00
hald-addon-keyboard: listening on /dev/input/eve
nt1
68 3797 0.0 0.0 12264 800 ? S May14 0:00
hald-addon-acpi: listening on acpid socket /var/
run/acpid.socket
root 3798 0.0 0.0 19484 900 ? S May14 0:00
/usr/libexec/hald-addon-cpufreq
68 3799 0.0 0.0 12268 796 ? S May14 0:00
hald-addon-keyboard: listening on /dev/input/eve
nt4
68 3802 0.0 0.0 12268 796 ? S May14 0:00
hald-addon-keyboard: listening on /dev/input/eve
nt3
root 3812 0.0 0.0 10168 636 ? S May14 0:05
hald-addon-storage: polling /dev/hda
root 3876 0.0 0.0 4040 384 ? S May14 0:00
/usr/sbin/smartd -q never
root 3879 0.0 0.0 3748 464 tty1 Ss+ May14 0:00
/sbin/mingetty tty1
root 3880 0.0 0.0 3748 464 tty2 Ss+ May14 0:00
/sbin/mingetty tty2
root 3881 0.0 0.0 3752 468 tty3 Ss+ May14 0:00
/sbin/mingetty tty3
root 3882 0.0 0.0 3752 468 tty4 Ss+ May14 0:00
/sbin/mingetty tty4
root 3883 0.0 0.0 3752 468 tty5 Ss+ May14 0:00
/sbin/mingetty tty5
root 3884 0.0 0.0 3752 468 tty6 Ss+ May14 0:00
/sbin/mingetty tty6
root 3943 0.0 0.0 80308 3244 ? Ss May14 0:00 sshd:
root@pts/0
root 3945 0.0 0.0 65136 1588 pts/0 Ss May14 0:00 -bash
root 4049 0.0 0.0 66444 280 ? Ss May14 0:00 rpc.rquotad
root 4052 0.0 0.0 0 0 ? S May14 0:00 [lockd]
root 4053 0.0 0.0 0 0 ? S< May14 0:00 [rpciod/0]
root 4054 0.0 0.0 0 0 ? S< May14 0:00 [rpciod/1]
root 4055 0.0 0.0 0 0 ? S< May14 0:00 [rpciod/2]
root 4056 0.0 0.0 0 0 ? S< May14 0:00 [rpciod/3]
root 4057 0.0 0.0 0 0 ? S< May14 0:00 [nfsd4]
root 4058 0.6 0.0 0 0 ? D May14 5:48 [nfsd]
root 4059 0.6 0.0 0 0 ? D May14 5:39 [nfsd]
root 4060 0.6 0.0 0 0 ? D May14 5:42 [nfsd]
root 4061 0.6 0.0 0 0 ? D May14 5:48 [nfsd]
root 4062 0.5 0.0 0 0 ? D May14 5:28 [nfsd]
root 4063 0.6 0.0 0 0 ? D May14 5:46 [nfsd]
root 4064 0.5 0.0 0 0 ? D May14 5:31 [nfsd]
root 4065 0.5 0.0 0 0 ? D May14 5:24 [nfsd]
root 4068 0.0 0.0 14372 876 ? Ss May14 0:00 rpc.mountd
root 4862 3.1 0.0 57984 460 pts/0 D 05:32 0:14 cat
/test/Bonnie.25846.051 /test/Bonnie.25846.05
2 /test/Bonnie.25846.053 /test/Bonnie.25846.054
root 4864 0.0 0.0 64524 612 pts/0 D+ 05:32 0:00 mkdir junk
root 4865 0.0 0.0 80308 3240 ? Ss 05:33 0:00 sshd:
root@pts/1
root 4867 0.0 0.0 65136 1588 pts/1 Ss 05:33 0:00 -bash
root 4892 0.0 0.0 64680 956 pts/1 R+ 05:40 0:00 ps -axuww
root 4893 0.0 0.0 60104 656 pts/1 S+ 05:40 0:00 more
[-- Attachment #2: messages_nfs_hang.txt --]
[-- Type: text/plain, Size: 454838 bytes --]
May 14 14:05:39 vault1 kernel: Linux version 2.6.21.1 (root@vault1) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-51)) #1 SMP Tue May 8 17:55:41 EDT 2007
May 14 14:05:39 vault1 kernel: Command line: ro root=LABEL=/ elevator=deadline
May 14 14:05:39 vault1 kernel: BIOS-provided physical RAM map:
May 14 14:05:39 vault1 kernel: BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
May 14 14:05:39 vault1 kernel: BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
May 14 14:05:39 vault1 kernel: BIOS-e820: 00000000000e8000 - 0000000000100000 (reserved)
May 14 14:05:39 vault1 kernel: BIOS-e820: 0000000000100000 - 00000000bffd0000 (usable)
May 14 14:05:39 vault1 kernel: BIOS-e820: 00000000bffd0000 - 00000000bffde000 (ACPI data)
May 14 14:05:39 vault1 kernel: BIOS-e820: 00000000bffde000 - 00000000c0000000 (ACPI NVS)
May 14 14:05:39 vault1 kernel: BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved)
May 14 14:05:39 vault1 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
May 14 14:05:39 vault1 kernel: BIOS-e820: 00000000ff700000 - 0000000100000000 (reserved)
May 14 14:05:39 vault1 kernel: BIOS-e820: 0000000100000000 - 0000000231000000 (usable)
May 14 14:05:39 vault1 kernel: end_pfn_map = 2297856
May 14 14:05:39 vault1 kernel: DMI 2.3 present.
May 14 14:05:39 vault1 kernel: ACPI: RSDP 000FA370, 0014 (r0 ACPIAM)
May 14 14:05:39 vault1 kernel: ACPI: RSDT BFFD0000, 0034 (r1 A M I OEMRSDT 2000627 MSFT 97)
May 14 14:05:39 vault1 kernel: ACPI: FACP BFFD0200, 0084 (r2 A M I OEMFACP 2000627 MSFT 97)
May 14 14:05:39 vault1 kernel: ACPI: DSDT BFFD0420, 5BB6 (r1 1HDC8 1HDC8003 3 INTL 2002026)
May 14 14:05:39 vault1 kernel: ACPI: FACS BFFDE000, 0040
May 14 14:05:39 vault1 kernel: ACPI: APIC BFFD0390, 0090 (r1 A M I OEMAPIC 2000627 MSFT 97)
May 14 14:05:39 vault1 kernel: ACPI: OEMB BFFDE040, 0063 (r1 A M I AMI_OEM 2000627 MSFT 97)
May 14 14:05:39 vault1 kernel: ACPI: SRAT BFFD5FE0, 0110 (r1 A M I OEMSRAT 2000627 MSFT 97)
May 14 14:05:39 vault1 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
May 14 14:05:39 vault1 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0
May 14 14:05:39 vault1 kernel: SRAT: PXM 1 -> APIC 2 -> Node 1
May 14 14:05:39 vault1 kernel: SRAT: PXM 1 -> APIC 3 -> Node 1
May 14 14:05:39 vault1 kernel: SRAT: Node 0 PXM 0 100000-c0000000
May 14 14:05:39 vault1 kernel: SRAT: Node 1 PXM 1 131000000-231000000
May 14 14:05:39 vault1 kernel: SRAT: Node 0 PXM 0 100000-131000000
May 14 14:05:39 vault1 kernel: SRAT: Node 0 PXM 0 0-131000000
May 14 14:05:39 vault1 kernel: Bootmem setup node 0 0000000000000000-0000000131000000
May 14 14:05:39 vault1 kernel: Bootmem setup node 1 0000000131000000-0000000231000000
May 14 14:05:39 vault1 kernel: Zone PFN ranges:
May 14 14:05:39 vault1 kernel: DMA 0 -> 4096
May 14 14:05:39 vault1 kernel: DMA32 4096 -> 1048576
May 14 14:05:39 vault1 kernel: Normal 1048576 -> 2297856
May 14 14:05:39 vault1 kernel: early_node_map[4] active PFN ranges
May 14 14:05:39 vault1 kernel: 0: 0 -> 159
May 14 14:05:39 vault1 kernel: 0: 256 -> 786384
May 14 14:05:39 vault1 kernel: 0: 1048576 -> 1249280
May 14 14:05:39 vault1 kernel: 1: 1249280 -> 2297856
May 14 14:05:40 vault1 kernel: Nvidia board detected. Ignoring ACPI timer override.
May 14 14:05:40 vault1 kernel: If you got timer trouble try acpi_use_timer_override
May 14 14:05:40 vault1 kernel: ACPI: PM-Timer IO Port: 0x4008
May 14 14:05:40 vault1 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
May 14 14:05:40 vault1 kernel: Processor #0 (Bootup-CPU)
May 14 14:05:40 vault1 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
May 14 14:05:40 vault1 kernel: Processor #1
May 14 14:05:40 vault1 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
May 14 14:05:40 vault1 kernel: Processor #2
May 14 14:05:40 vault1 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
May 14 14:05:40 vault1 kernel: Processor #3
May 14 14:05:40 vault1 kernel: ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
May 14 14:05:40 vault1 rpc.statd[3453]: Version 1.0.10 Starting
May 14 14:05:40 vault1 kernel: IOAPIC[0]: apic_id 4, address 0xfec00000, GSI 0-23
May 14 14:05:40 vault1 kernel: ACPI: IOAPIC (id[0x05] address[0xfeafe000] gsi_base[24])
May 14 14:05:40 vault1 kernel: IOAPIC[1]: apic_id 5, address 0xfeafe000, GSI 24-47
May 14 14:05:40 vault1 kernel: ACPI: IOAPIC (id[0x09] address[0xfe6fe000] gsi_base[80])
May 14 14:05:40 vault1 kernel: IOAPIC[2]: apic_id 9, address 0xfe6fe000, GSI 80-86
May 14 14:05:40 vault1 kernel: ACPI: IOAPIC (id[0x0a] address[0xfe6ff000] gsi_base[87])
May 14 14:05:40 vault1 kernel: IOAPIC[3]: apic_id 10, address 0xfe6ff000, GSI 87-93
May 14 14:05:40 vault1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
May 14 14:05:40 vault1 kernel: ACPI: BIOS IRQ0 pin2 override ignored.
May 14 14:05:40 vault1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
May 14 14:05:40 vault1 rpc.statd[3453]: statd running as root. chown /var/lib/nfs/statd/sm to choose different user
May 14 14:05:40 vault1 kernel: Setting APIC routing to physical flat
May 14 14:05:40 vault1 kernel: Using ACPI (MADT) for SMP configuration information
May 14 14:05:40 vault1 kernel: Nosave address range: 000000000009f000 - 00000000000a0000
May 14 14:05:40 vault1 kernel: Nosave address range: 00000000000a0000 - 00000000000e8000
May 14 14:05:40 vault1 kernel: Nosave address range: 00000000000e8000 - 0000000000100000
May 14 14:05:40 vault1 kernel: Nosave address range: 00000000bffd0000 - 00000000bffde000
May 14 14:05:40 vault1 kernel: Nosave address range: 00000000bffde000 - 00000000c0000000
May 14 14:05:40 vault1 kernel: Nosave address range: 00000000c0000000 - 00000000fec00000
May 14 14:05:40 vault1 kernel: Nosave address range: 00000000fec00000 - 00000000fec01000
May 14 14:05:40 vault1 kernel: Nosave address range: 00000000fec01000 - 00000000fee00000
May 14 14:05:40 vault1 kernel: Nosave address range: 00000000fee00000 - 00000000fee01000
May 14 14:05:40 vault1 kernel: Nosave address range: 00000000fee01000 - 00000000ff700000
May 14 14:05:40 vault1 kernel: Nosave address range: 00000000ff700000 - 0000000100000000
May 14 14:05:40 vault1 kernel: Allocating PCI resources starting at c4000000 (gap: c0000000:3ec00000)
May 14 14:05:40 vault1 kernel: SMP: Allowing 4 CPUs, 0 hotplug CPUs
May 14 14:05:40 vault1 kernel: PERCPU: Allocating 44544 bytes of per cpu data
May 14 14:05:40 vault1 kernel: Built 2 zonelists. Total pages: 1983833
May 14 14:05:40 vault1 kernel: Kernel command line: ro root=LABEL=/ elevator=deadline
May 14 14:05:40 vault1 kernel: Initializing CPU#0
May 14 14:05:40 vault1 kernel: PID hash table entries: 4096 (order: 12, 32768 bytes)
May 14 14:05:40 vault1 kernel: time.c: Detected 2400.028 MHz processor.
May 14 14:05:40 vault1 kernel: Console: colour VGA+ 80x25
May 14 14:05:40 vault1 kernel: Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
May 14 14:05:40 vault1 kernel: ... MAX_LOCKDEP_SUBCLASSES: 8
May 14 14:05:40 vault1 kernel: ... MAX_LOCK_DEPTH: 30
May 14 14:05:40 vault1 kernel: ... MAX_LOCKDEP_KEYS: 2048
May 14 14:05:40 vault1 kernel: ... CLASSHASH_SIZE: 1024
May 14 14:05:40 vault1 kernel: ... MAX_LOCKDEP_ENTRIES: 8192
May 14 14:05:40 vault1 kernel: ... MAX_LOCKDEP_CHAINS: 16384
May 14 14:05:41 vault1 kernel: ... CHAINHASH_SIZE: 8192
May 14 14:05:41 vault1 kernel: memory used by lock dependency info: 1648 kB
May 14 14:05:41 vault1 kernel: per task-struct memory footprint: 1680 bytes
May 14 14:05:41 vault1 kernel: Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
May 14 14:05:41 vault1 kernel: Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
May 14 14:05:41 vault1 kernel: Checking aperture...
May 14 14:05:41 vault1 kernel: CPU 0: aperture @ 9280000000 size 32 MB
May 14 14:05:41 vault1 kernel: Aperture too small (32 MB)
May 14 14:05:41 vault1 hidd[3518]: Bluetooth HID daemon
May 14 14:05:41 vault1 kernel: No AGP bridge found
May 14 14:05:41 vault1 kernel: Your BIOS doesn't leave a aperture memory hole
May 14 14:05:41 vault1 kernel: Please enable the IOMMU option in the BIOS setup
May 14 14:05:41 vault1 kernel: This costs you 64 MB of RAM
May 14 14:05:41 vault1 kernel: Mapping aperture over 65536 KB of RAM @ 8000000
May 14 14:05:41 vault1 kernel: Memory: 7875072k/9191424k available (2463k kernel code, 267196k reserved, 1519k data, 328k init)
May 14 14:05:41 vault1 kernel: Calibrating delay using timer specific routine.. 4802.07 BogoMIPS (lpj=2401037)
May 14 14:05:41 vault1 kernel: Security Framework v1.0.0 initialized
May 14 14:05:41 vault1 kernel: SELinux: Initializing.
May 14 14:05:41 vault1 kernel: selinux_register_security: Registering secondary module capability
May 14 14:05:41 vault1 kernel: Capability LSM initialized as secondary
May 14 14:05:41 vault1 kernel: Mount-cache hash table entries: 256
May 14 14:05:41 vault1 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
May 14 14:05:41 vault1 kernel: CPU: L2 Cache: 1024K (64 bytes/line)
May 14 14:05:41 vault1 kernel: CPU 0/0 -> Node 0
May 14 14:05:41 vault1 kernel: CPU: Physical Processor ID: 0
May 14 14:05:41 vault1 kernel: CPU: Processor Core ID: 0
May 14 14:05:41 vault1 kernel: lockdep: not fixing up alternatives.
May 14 14:05:41 vault1 kernel: ACPI: Core revision 20070126
May 14 14:05:41 vault1 kernel: Using local APIC timer interrupts.
May 14 14:05:41 vault1 kernel: result 12500157
May 14 14:05:41 vault1 kernel: Detected 12.500 MHz APIC timer.
May 14 14:05:41 vault1 kernel: lockdep: not fixing up alternatives.
May 14 14:05:41 vault1 kernel: Booting processor 1/4 APIC 0x1
May 14 14:05:41 vault1 kernel: Initializing CPU#1
May 14 14:05:41 vault1 kernel: Calibrating delay using timer specific routine.. 4799.35 BogoMIPS (lpj=2399678)
May 14 14:05:41 vault1 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
May 14 14:05:41 vault1 kernel: CPU: L2 Cache: 1024K (64 bytes/line)
May 14 14:05:41 vault1 kernel: CPU 1/1 -> Node 0
May 14 14:05:41 vault1 kernel: CPU: Physical Processor ID: 0
May 14 14:05:41 vault1 kernel: CPU: Processor Core ID: 1
May 14 14:05:41 vault1 kernel: Dual Core AMD Opteron(tm) Processor 280 stepping 02
May 14 14:05:41 vault1 kernel: lockdep: not fixing up alternatives.
May 14 14:05:41 vault1 kernel: Booting processor 2/4 APIC 0x2
May 14 14:05:41 vault1 kernel: Initializing CPU#2
May 14 14:05:41 vault1 kernel: Calibrating delay using timer specific routine.. 4799.28 BogoMIPS (lpj=2399642)
May 14 14:05:41 vault1 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
May 14 14:05:41 vault1 kernel: CPU: L2 Cache: 1024K (64 bytes/line)
May 14 14:05:42 vault1 kernel: CPU 2/2 -> Node 1
May 14 14:05:42 vault1 kernel: CPU: Physical Processor ID: 1
May 14 14:05:42 vault1 kernel: CPU: Processor Core ID: 0
May 14 14:05:42 vault1 kernel: Dual Core AMD Opteron(tm) Processor 280 stepping 02
May 14 14:05:42 vault1 kernel: lockdep: not fixing up alternatives.
May 14 14:05:42 vault1 kernel: Booting processor 3/4 APIC 0x3
May 14 14:05:42 vault1 kernel: Initializing CPU#3
May 14 14:05:42 vault1 kernel: Calibrating delay using timer specific routine.. 4799.30 BogoMIPS (lpj=2399652)
May 14 14:05:42 vault1 kernel: CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
May 14 14:05:42 vault1 kernel: CPU: L2 Cache: 1024K (64 bytes/line)
May 14 14:05:42 vault1 kernel: CPU 3/3 -> Node 1
May 14 14:05:42 vault1 kernel: CPU: Physical Processor ID: 1
May 14 14:05:42 vault1 kernel: CPU: Processor Core ID: 1
May 14 14:05:42 vault1 kernel: Dual Core AMD Opteron(tm) Processor 280 stepping 02
May 14 14:05:42 vault1 kernel: Brought up 4 CPUs
May 14 14:05:42 vault1 kernel: migration_cost=361,469
May 14 14:05:42 vault1 kernel: NET: Registered protocol family 16
May 14 14:05:42 vault1 kernel: ACPI: bus type pci registered
May 14 14:05:42 vault1 kernel: PCI: Using configuration type 1
May 14 14:05:42 vault1 kernel: ACPI: Interpreter enabled
May 14 14:05:42 vault1 kernel: ACPI: (supports S0 S1 S3 S4 S5)
May 14 14:05:42 vault1 kernel: ACPI: Using IOAPIC for interrupt routing
May 14 14:05:42 vault1 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00)
May 14 14:05:42 vault1 kernel: PCI: Transparent bridge - 0000:00:09.0
May 14 14:05:42 vault1 kernel: ACPI: PCI Root Bridge [PCIC] (0000:04)
May 14 14:05:42 vault1 kernel: ACPI: PCI Root Bridge [PCIB] (0000:80)
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 16 17 18 19) *10
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 16 17 18 19) *9
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 16 17 18 19) *9
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 16 17 18 19) *9
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LNKE] (IRQs 16 17 18 19) *0, disabled.
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LUS0] (IRQs 20 21 22) *9
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LUS1] (IRQs 20 21 22) *0, disabled.
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LUS2] (IRQs 20 21 22) *9
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LKLN] (IRQs 20 21 22) *11
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LAUI] (IRQs 20 21 22) *9
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LKMO] (IRQs 20 21 22) *0, disabled.
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LKSM] (IRQs 20 21 22) *0, disabled.
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LTID] (IRQs 20 21 22) *9
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LTIE] (IRQs 20 21 22) *5
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LATA] (IRQs 20 21 22) *14
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LN2A] (IRQs 40 41 42 43) *0, disabled.
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LN2B] (IRQs 40 41 42 43) *0, disabled.
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LN2C] (IRQs 40 41 42 43) *0, disabled.
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LN2D] (IRQs 40 41 42 43) *0, disabled.
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LK2N] (IRQs 44 45 46 47) *0, disabled.
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LT3D] (IRQs 44 45 46 47) *0, disabled.
May 14 14:05:42 vault1 kernel: ACPI: PCI Interrupt Link [LT2E] (IRQs 44 45 46 47) *0, disabled.
May 14 14:05:42 vault1 kernel: Linux Plug and Play Support v0.97 (c) Adam Belay
May 14 14:05:42 vault1 kernel: pnp: PnP ACPI init
May 14 14:05:42 vault1 kernel: pnp: PnP ACPI: found 17 devices
May 14 14:05:43 vault1 kernel: usbcore: registered new interface driver usbfs
May 14 14:05:43 vault1 kernel: usbcore: registered new interface driver hub
May 14 14:05:43 vault1 kernel: usbcore: registered new device driver usb
May 14 14:05:43 vault1 kernel: PCI: Using ACPI for IRQ routing
May 14 14:05:43 vault1 kernel: PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
May 14 14:05:43 vault1 kernel: NetLabel: Initializing
May 14 14:05:43 vault1 kernel: NetLabel: domain hash size = 128
May 14 14:05:43 vault1 kernel: NetLabel: protocols = UNLABELED CIPSOv4
May 14 14:05:43 vault1 kernel: NetLabel: unlabeled traffic allowed by default
May 14 14:05:43 vault1 kernel: PCI-DMA: Disabling AGP.
May 14 14:05:43 vault1 kernel: PCI-DMA: aperture base @ 8000000 size 65536 KB
May 14 14:05:43 vault1 kernel: PCI-DMA: using GART IOMMU.
May 14 14:05:43 vault1 kernel: PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
May 14 14:05:43 vault1 kernel: pnp: 00:09: iomem range 0xffb80000-0xfffffffe could not be reserved
May 14 14:05:43 vault1 kernel: pnp: 00:09: iomem range 0xdfefc000-0xdfefcfff has been reserved
May 14 14:05:43 vault1 kernel: pnp: 00:09: iomem range 0xdfefd000-0xdfefd3ff has been reserved
May 14 14:05:43 vault1 kernel: pnp: 00:09: iomem range 0xdfefe000-0xdfefe3ff has been reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0a: iomem range 0xfec00000-0xfec00fff could not be reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0a: iomem range 0xfee00000-0xfeefffff could not be reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0c: ioport range 0x680-0x6ff has been reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0c: ioport range 0x295-0x296 has been reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0c: ioport range 0x778-0x77f has been reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0c: ioport range 0xb78-0xb7f has been reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0c: ioport range 0xf78-0xf7f has been reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0d: iomem range 0xe0000000-0xefffffff has been reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0e: iomem range 0x0-0x9ffff could not be reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0e: iomem range 0xc0000-0xdffff has been reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0e: iomem range 0xe0000-0xfffff could not be reserved
May 14 14:05:43 vault1 kernel: pnp: 00:0e: iomem range 0x100000-0xbfffffff could not be reserved
May 14 14:05:43 vault1 kernel: PCI: Bridge: 0000:00:09.0
May 14 14:05:43 vault1 kernel: IO window: 9000-9fff
May 14 14:05:43 vault1 kernel: MEM window: fa000000-fc0fffff
May 14 14:05:43 vault1 kernel: Time: acpi_pm clocksource has been installed.
May 14 14:05:43 vault1 kernel: PREFETCH window: c4000000-c40fffff
May 14 14:05:43 vault1 kernel: PCI: Bridge: 0000:00:0d.0
May 14 14:05:43 vault1 kernel: IO window: disabled.
May 14 14:05:43 vault1 kernel: MEM window: disabled.
May 14 14:05:43 vault1 kernel: PREFETCH window: disabled.
May 14 14:05:43 vault1 kernel: PCI: Bridge: 0000:00:0e.0
May 14 14:05:43 vault1 kernel: IO window: disabled.
May 14 14:05:44 vault1 kernel: MEM window: f9d00000-f9ffffff
May 14 14:05:44 vault1 kernel: PREFETCH window: fc200000-fe1fffff
May 14 14:05:44 vault1 kernel: PCI: Bridge: 0000:04:01.0
May 14 14:05:44 vault1 kernel: IO window: disabled.
May 14 14:05:44 vault1 kernel: MEM window: fe500000-fe5fffff
May 14 14:05:44 vault1 kernel: PREFETCH window: fe700000-fe7fffff
May 14 14:05:44 vault1 kernel: PCI: Bridge: 0000:05:01.0
May 14 14:05:44 vault1 kernel: IO window: e000-efff
May 14 14:05:44 vault1 kernel: MEM window: fe300000-fe4fffff
May 14 14:05:44 vault1 kernel: PREFETCH window: disabled.
May 14 14:05:44 vault1 kernel: PCI: Bridge: 0000:04:02.0
May 14 14:05:44 vault1 kernel: IO window: e000-efff
May 14 14:05:44 vault1 kernel: MEM window: fe300000-fe4fffff
May 14 14:05:44 vault1 kernel: PREFETCH window: disabled.
May 14 14:05:44 vault1 kernel: PCI: Bridge: 0000:80:0d.0
May 14 14:05:44 vault1 kernel: IO window: disabled.
May 14 14:05:44 vault1 kernel: MEM window: disabled.
May 14 14:05:44 vault1 kernel: PREFETCH window: disabled.
May 14 14:05:44 vault1 kernel: PCI: Bridge: 0000:81:00.0
May 14 14:05:44 vault1 kernel: IO window: disabled.
May 14 14:05:44 vault1 kernel: MEM window: fe900000-fe9fffff
May 14 14:05:44 vault1 kernel: PREFETCH window: disabled.
May 14 14:05:44 vault1 kernel: PCI: Bridge: 0000:80:0e.0
May 14 14:05:44 vault1 kernel: IO window: disabled.
May 14 14:05:44 vault1 kernel: MEM window: fe900000-fe9fffff
May 14 14:05:44 vault1 kernel: PREFETCH window: disabled.
May 14 14:05:44 vault1 kernel: NET: Registered protocol family 2
May 14 14:05:44 vault1 kernel: IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
May 14 14:05:44 vault1 kernel: TCP established hash table entries: 65536 (order: 10, 4194304 bytes)
May 14 14:05:44 vault1 kernel: TCP bind hash table entries: 65536 (order: 9, 3670016 bytes)
May 14 14:05:44 vault1 kernel: TCP: Hash tables configured (established 65536 bind 65536)
May 14 14:05:44 vault1 kernel: TCP reno registered
May 14 14:05:44 vault1 kernel: checking if image is initramfs... it is
May 14 14:05:44 vault1 kernel: Freeing initrd memory: 1592k freed
May 14 14:05:44 vault1 kernel: audit: initializing netlink socket (disabled)
May 14 14:05:44 vault1 kernel: audit(1179151443.002:1): initialized
May 14 14:05:44 vault1 kernel: Total HugeTLB memory allocated, 0
May 14 14:05:44 vault1 kernel: VFS: Disk quotas dquot_6.5.1
May 14 14:05:44 vault1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
May 14 14:05:44 vault1 kernel: io scheduler noop registered
May 14 14:05:44 vault1 kernel: io scheduler anticipatory registered
May 14 14:05:44 vault1 kernel: io scheduler deadline registered (default)
May 14 14:05:44 vault1 kernel: io scheduler cfq registered
May 14 14:05:44 vault1 kernel: PCI: Found disabled HT MSI Mapping on 0000:00:0d.0
May 14 14:05:44 vault1 kernel: PCI: Found enabled HT MSI Mapping on 0000:00:00.0
May 14 14:05:44 vault1 kernel: PCI: Found disabled HT MSI Mapping on 0000:00:0e.0
May 14 14:05:44 vault1 kernel: PCI: Found enabled HT MSI Mapping on 0000:00:00.0
May 14 14:05:44 vault1 kernel: PCI: Found disabled HT MSI Mapping on 0000:80:0d.0
May 14 14:05:44 vault1 kernel: PCI: Found disabled HT MSI Mapping on 0000:80:00.0
May 14 14:05:44 vault1 kernel: PCI: MSI quirk detected. MSI disabled on chipset 0000:80:0d.0.
May 14 14:05:44 vault1 kernel: PCI: Found disabled HT MSI Mapping on 0000:80:0e.0
May 14 14:05:44 vault1 kernel: PCI: Found disabled HT MSI Mapping on 0000:80:00.0
May 14 14:05:44 vault1 kernel: PCI: MSI quirk detected. MSI disabled on chipset 0000:80:0e.0.
May 14 14:05:44 vault1 kernel: assign_interrupt_mode Found MSI capability
May 14 14:05:45 vault1 last message repeated 3 times
May 14 14:05:45 vault1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
May 14 14:05:45 vault1 kernel: Real Time Clock Driver v1.12ac
May 14 14:05:45 vault1 kernel: Non-volatile memory driver v1.2
May 14 14:05:45 vault1 kernel: Linux agpgart interface v0.102 (c) Dave Jones
May 14 14:05:45 vault1 kernel: Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
May 14 14:05:45 vault1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
May 14 14:05:45 vault1 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
May 14 14:05:45 vault1 kernel: 00:05: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
May 14 14:05:45 vault1 kernel: 00:06: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
May 14 14:05:45 vault1 kernel: RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
May 14 14:05:45 vault1 kernel: input: Macintosh mouse button emulation as /class/input/input0
May 14 14:05:45 vault1 kernel: Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
May 14 14:05:45 vault1 kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
May 14 14:05:45 vault1 kernel: NFORCE-CK804: IDE controller at PCI slot 0000:00:06.0
May 14 14:05:45 vault1 kernel: NFORCE-CK804: chipset revision 242
May 14 14:05:45 vault1 kernel: NFORCE-CK804: not 100% native mode: will probe irqs later
May 14 14:05:45 vault1 kernel: NFORCE-CK804: 0000:00:06.0 (rev f2) UDMA133 controller
May 14 14:05:45 vault1 kernel: ide0: BM-DMA at 0x3000-0x3007, BIOS settings: hda:DMA, hdb:DMA
May 14 14:05:45 vault1 kernel: ide1: BM-DMA at 0x3008-0x300f, BIOS settings: hdc:DMA, hdd:DMA
May 14 14:05:45 vault1 kernel: hda: DVDR PX-750A, ATAPI CD/DVD-ROM drive
May 14 14:05:45 vault1 kernel: ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
May 14 14:05:45 vault1 kernel: ide-floppy driver 0.99.newide
May 14 14:05:45 vault1 kernel: usbcore: registered new interface driver libusual
May 14 14:05:45 vault1 kernel: usbcore: registered new interface driver hiddev
May 14 14:05:45 vault1 kernel: usbcore: registered new interface driver usbhid
May 14 14:05:45 vault1 kernel: drivers/usb/input/hid-core.c: v2.6:USB HID core driver
May 14 14:05:45 vault1 kernel: PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
May 14 14:05:45 vault1 kernel: PNP: PS/2 controller doesn't have AUX irq; using default 12
May 14 14:05:45 vault1 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
May 14 14:05:45 vault1 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12
May 14 14:05:45 vault1 kernel: mice: PS/2 mouse device common for all mice
May 14 14:05:45 vault1 kernel: input: AT Translated Set 2 keyboard as /class/input/input1
May 14 14:05:45 vault1 kernel: input: AT Translated Set 2 keyboard as /class/input/input2
May 14 14:05:45 vault1 kernel: TCP bic registered
May 14 14:05:45 vault1 kernel: Initializing XFRM netlink socket
May 14 14:05:45 vault1 kernel: NET: Registered protocol family 1
May 14 14:05:45 vault1 kernel: NET: Registered protocol family 17
May 14 14:05:45 vault1 kernel: powernow-k8: Found 4 Dual Core AMD Opteron(tm) Processor 280 processors (version 2.00.00)
May 14 14:05:45 vault1 kernel: powernow-k8: 0 : fid 0x10 (2400 MHz), vid 0x8
May 14 14:05:45 vault1 kernel: powernow-k8: 1 : fid 0xe (2200 MHz), vid 0xa
May 14 14:05:45 vault1 kernel: powernow-k8: 2 : fid 0xc (2000 MHz), vid 0xc
May 14 14:05:45 vault1 kernel: powernow-k8: 3 : fid 0xa (1800 MHz), vid 0xe
May 14 14:05:45 vault1 kernel: powernow-k8: 4 : fid 0x2 (1000 MHz), vid 0x12
May 14 14:05:45 vault1 kernel: powernow-k8: 0 : fid 0x10 (2400 MHz), vid 0x8
May 14 14:05:45 vault1 kernel: powernow-k8: 1 : fid 0xe (2200 MHz), vid 0xa
May 14 14:05:45 vault1 kernel: powernow-k8: 2 : fid 0xc (2000 MHz), vid 0xc
May 14 14:05:45 vault1 kernel: powernow-k8: 3 : fid 0xa (1800 MHz), vid 0xe
May 14 14:05:45 vault1 kernel: powernow-k8: 4 : fid 0x2 (1000 MHz), vid 0x12
May 14 14:05:45 vault1 kernel: Freeing unused kernel memory: 328k freed
May 14 14:05:45 vault1 kernel: Write protecting the kernel read-only data: 1101k
May 14 14:05:46 vault1 kernel: USB Universal Host Controller Interface driver v3.0
May 14 14:05:46 vault1 kernel: ACPI: PCI Interrupt Link [LUS0] enabled at IRQ 22
May 14 14:05:46 vault1 kernel: ACPI: PCI Interrupt 0000:00:02.0[A] -> Link [LUS0] -> GSI 22 (level, low) -> IRQ 22
May 14 14:05:46 vault1 kernel: ohci_hcd 0000:00:02.0: OHCI Host Controller
May 14 14:05:46 vault1 kernel: ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 1
May 14 14:05:46 vault1 kernel: ohci_hcd 0000:00:02.0: irq 22, io mem 0xfc1fb000
May 14 14:05:46 vault1 kernel: usb usb1: configuration #1 chosen from 1 choice
May 14 14:05:46 vault1 kernel: hub 1-0:1.0: USB hub found
May 14 14:05:46 vault1 kernel: hub 1-0:1.0: 10 ports detected
May 14 14:05:46 vault1 kernel: ACPI: PCI Interrupt Link [LUS2] enabled at IRQ 21
May 14 14:05:46 vault1 kernel: ACPI: PCI Interrupt 0000:00:02.1[B] -> Link [LUS2] -> GSI 21 (level, low) -> IRQ 21
May 14 14:05:46 vault1 kernel: ehci_hcd 0000:00:02.1: EHCI Host Controller
May 14 14:05:46 vault1 kernel: ehci_hcd 0000:00:02.1: new USB bus registered, assigned bus number 2
May 14 14:05:46 vault1 kernel: ehci_hcd 0000:00:02.1: debug port 1
May 14 14:05:46 vault1 kernel: ehci_hcd 0000:00:02.1: irq 21, io mem 0xfc1e0000
May 14 14:05:46 vault1 kernel: ehci_hcd 0000:00:02.1: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
May 14 14:05:46 vault1 kernel: usb usb2: configuration #1 chosen from 1 choice
May 14 14:05:46 vault1 kernel: hub 2-0:1.0: USB hub found
May 14 14:05:46 vault1 kernel: hub 2-0:1.0: 10 ports detected
May 14 14:05:46 vault1 kernel: megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
May 14 14:05:46 vault1 kernel: SCSI subsystem initialized
May 14 14:05:46 vault1 kernel: megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
May 14 14:05:46 vault1 kernel: megaraid: probe new device 0x1000:0x1960:0x1000:0x0520: bus 7:slot 1:func 0
May 14 14:05:46 vault1 kernel: ACPI: PCI Interrupt 0000:07:01.0[A] -> GSI 82 (level, low) -> IRQ 82
May 14 14:05:46 vault1 kernel: megaraid: fw version:[1L45] bios version:[G121]
May 14 14:05:46 vault1 kernel: scsi0 : LSI Logic MegaRAID driver
May 14 14:05:46 vault1 kernel: scsi[0]: scanning scsi channel 0 [Phy 0] for non-raid devices
May 14 14:05:46 vault1 kernel: scsi 0:0:8:0: Processor SUPER GEM359 REV001 1.09 PQ: 0 ANSI: 2
May 14 14:05:46 vault1 kernel: scsi[0]: scanning scsi channel 1 [virtual] for logical drives
May 14 14:05:46 vault1 kernel: scsi 0:1:0:0: Direct-Access MegaRAID LD 0 RAID1 69G 1L45 PQ: 0 ANSI: 2
May 14 14:05:46 vault1 kernel: SCSI device sda: 142577664 512-byte hdwr sectors (73000 MB)
May 14 14:05:46 vault1 kernel: sda: Write Protect is off
May 14 14:05:46 vault1 kernel: sda: asking for cache data failed
May 14 14:05:46 vault1 kernel: sda: assuming drive cache: write through
May 14 14:05:46 vault1 kernel: SCSI device sda: 142577664 512-byte hdwr sectors (73000 MB)
May 14 14:05:46 vault1 kernel: sda: Write Protect is off
May 14 14:05:46 vault1 kernel: sda: asking for cache data failed
May 14 14:05:46 vault1 kernel: sda: assuming drive cache: write through
May 14 14:05:46 vault1 kernel: sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 >
May 14 14:05:46 vault1 kernel: sd 0:1:0:0: Attached scsi disk sda
May 14 14:05:46 vault1 kernel: scsi 0:1:1:0: Direct-Access MegaRAID LD 1 RAID0 69G 1L45 PQ: 0 ANSI: 2
May 14 14:05:46 vault1 kernel: SCSI device sdb: 142577664 512-byte hdwr sectors (73000 MB)
May 14 14:05:46 vault1 kernel: sdb: Write Protect is off
May 14 14:05:46 vault1 kernel: sdb: asking for cache data failed
May 14 14:05:46 vault1 kernel: sdb: assuming drive cache: write through
May 14 14:05:46 vault1 kernel: SCSI device sdb: 142577664 512-byte hdwr sectors (73000 MB)
May 14 14:05:46 vault1 kernel: sdb: Write Protect is off
May 14 14:05:46 vault1 kernel: sdb: asking for cache data failed
May 14 14:05:46 vault1 kernel: sdb: assuming drive cache: write through
May 14 14:05:46 vault1 kernel: sdb: sdb1
May 14 14:05:46 vault1 kernel: sdb1: <solaris: [s0] sdb5 [s1] sdb6 [s2] sdb7 [s4] sdb8 [s7] sdb9 >
May 14 14:05:46 vault1 kernel: sd 0:1:1:0: Attached scsi disk sdb
May 14 14:05:46 vault1 kernel: shpchp: HPC vendor_id 1022 device_id 7458 ss_vid 0 ss_did 0
May 14 14:05:46 vault1 kernel: shpchp: shpc_init: cannot reserve MMIO region
May 14 14:05:46 vault1 kernel: shpchp: HPC vendor_id 1022 device_id 7458 ss_vid 0 ss_did 0
May 14 14:05:46 vault1 kernel: shpchp: shpc_init: cannot reserve MMIO region
May 14 14:05:46 vault1 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
May 14 14:05:46 vault1 kernel: QLogic Fibre Channel HBA Driver
May 14 14:05:46 vault1 kernel: ACPI: PCI Interrupt 0000:06:04.0[A] -> GSI 87 (level, low) -> IRQ 87
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.0: Found an ISP2312, irq 87, iobase 0xffffc20000020000
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.0: Configuring PCI space...
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.0: Configure NVRAM parameters...
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.0: Verifying loaded RISC code...
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.0: Firmware image unavailable.
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.0: Firmware images can be retrieved from: ftp://ftp.qlogic.com/outgoing/linux/firmware/.
May 14 14:05:46 vault1 kernel: scsi1 : qla2xxx
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.0: Parity error -- HCCR=74a0, Dumping firmware!
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.0: No buffer available for dump!!!
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.0:
May 14 14:05:46 vault1 kernel: QLogic Fibre Channel HBA Driver: 8.01.07-k5
May 14 14:05:46 vault1 kernel: QLogic QLA2344 - 133MHz PCI-X to 2Gb FC, Quad Channel
May 14 14:05:46 vault1 kernel: ISP2312: PCI-X (133 MHz) @ 0000:06:04.0 hdma+, host#=1, fw=0.00.00 (0)
May 14 14:05:46 vault1 kernel: ACPI: PCI Interrupt 0000:06:04.1[B] -> GSI 88 (level, low) -> IRQ 88
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.1: Found an ISP2312, irq 88, iobase 0xffffc20000022000
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.1: Configuring PCI space...
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.1: Configure NVRAM parameters...
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.1: Verifying loaded RISC code...
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.1: Firmware image unavailable.
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.1: Firmware images can be retrieved from: ftp://ftp.qlogic.com/outgoing/linux/firmware/.
May 14 14:05:46 vault1 kernel: scsi2 : qla2xxx
May 14 14:05:46 vault1 kernel: qla2xxx 0000:06:04.1: Parity error -- HCCR=74a0, Dumping firmware!
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:04.1: No buffer available for dump!!!
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:04.1:
May 14 14:05:47 vault1 kernel: QLogic Fibre Channel HBA Driver: 8.01.07-k5
May 14 14:05:47 vault1 kernel: QLogic QLA2344 - 133MHz PCI-X to 2Gb FC, Quad Channel
May 14 14:05:47 vault1 kernel: ISP2312: PCI-X (133 MHz) @ 0000:06:04.1 hdma+, host#=2, fw=0.00.00 (0)
May 14 14:05:47 vault1 kernel: ACPI: PCI Interrupt 0000:06:06.0[A] -> GSI 89 (level, low) -> IRQ 89
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.0: Found an ISP2312, irq 89, iobase 0xffffc20000024000
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.0: Configuring PCI space...
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.0: Configure NVRAM parameters...
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.0: Verifying loaded RISC code...
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.0: Firmware image unavailable.
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.0: Firmware images can be retrieved from: ftp://ftp.qlogic.com/outgoing/linux/firmware/.
May 14 14:05:47 vault1 kernel: scsi3 : qla2xxx
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:04.0: Mailbox command timeout occured. Issuing ISP abort.
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:04.0: Performing ISP error recovery - ha= ffff810130e5c750.
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.0: Parity error -- HCCR=74a0, Dumping firmware!
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.0: No buffer available for dump!!!
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.0:
May 14 14:05:47 vault1 kernel: QLogic Fibre Channel HBA Driver: 8.01.07-k5
May 14 14:05:47 vault1 kernel: QLogic QLA2344 - 133MHz PCI-X to 2Gb FC, Quad Channel
May 14 14:05:47 vault1 kernel: ISP2312: PCI-X (133 MHz) @ 0000:06:06.0 hdma+, host#=3, fw=0.00.00 (0)
May 14 14:05:47 vault1 kernel: ACPI: PCI Interrupt 0000:06:06.1[B] -> GSI 90 (level, low) -> IRQ 90
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.1: Found an ISP2312, irq 90, iobase 0xffffc20000026000
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.1: Configuring PCI space...
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.1: Configure NVRAM parameters...
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.1: Verifying loaded RISC code...
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:04.0: Firmware image unavailable.
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:04.0: Firmware images can be retrieved from: ftp://ftp.qlogic.com/outgoing/linux/firmware/.
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:04.0: qla2x00_abort_isp: **** FAILED ****
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:04.0: Performing ISP error recovery - ha= ffff810130e5c750.
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:04.1: Mailbox command timeout occured. Issuing ISP abort.
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:04.1: Performing ISP error recovery - ha= ffff810130fc4750.
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.1: Firmware image unavailable.
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.1: Firmware images can be retrieved from: ftp://ftp.qlogic.com/outgoing/linux/firmware/.
May 14 14:05:47 vault1 kernel: scsi4 : qla2xxx
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.1: Parity error -- HCCR=74a0, Dumping firmware!
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.1: No buffer available for dump!!!
May 14 14:05:47 vault1 kernel: qla2xxx 0000:06:06.1:
May 14 14:05:47 vault1 kernel: QLogic Fibre Channel HBA Driver: 8.01.07-k5
May 14 14:05:47 vault1 kernel: QLogic QLA2344 - 133MHz PCI-X to 2Gb FC, Quad Channel
May 14 14:05:47 vault1 kernel: ISP2312: PCI-X (133 MHz) @ 0000:06:06.1 hdma+, host#=4, fw=0.00.00 (0)
May 14 14:05:47 vault1 kernel: kjournald starting. Commit interval 5 seconds
May 14 14:05:47 vault1 kernel: EXT3-fs: mounted filesystem with ordered data mode.
May 14 14:05:47 vault1 kernel: SELinux: Disabled at runtime.
May 14 14:05:47 vault1 kernel: audit(1179151517.943:2): selinux=0 auid=4294967295
May 14 14:05:47 vault1 kernel: parport: PnPBIOS parport detected.
May 14 14:05:47 vault1 kernel: parport0: PC-style at 0x378, irq 7 [PCSPP]
May 14 14:05:47 vault1 kernel: hda: ATAPI 40X DVD-ROM DVD-R-RAM CD-R/RW drive, 2048kB Cache, UDMA(66)
May 14 14:05:47 vault1 kernel: Uniform CD-ROM driver Revision: 3.20
May 14 14:05:47 vault1 kernel: ACPI: PCI Interrupt Link [LN2A] enabled at IRQ 43
May 14 14:05:47 vault1 kernel: ACPI: PCI Interrupt 0000:82:02.0[A] -> Link [LN2A] -> GSI 43 (level, low) -> IRQ 43
May 14 14:05:47 vault1 kernel: forcedeth.c: Reverse Engineered nForce ethernet driver. Version 0.60.
May 14 14:05:47 vault1 kernel: ACPI: PCI Interrupt Link [LKLN] enabled at IRQ 20
May 14 14:05:47 vault1 kernel: ACPI: PCI Interrupt 0000:00:0a.0[A] -> Link [LKLN] -> GSI 20 (level, low) -> IRQ 20
May 14 14:05:47 vault1 kernel: forcedeth: using HIGHDMA
May 14 14:05:47 vault1 kernel: ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[43] MMIO=[fe9ff800-fe9fffff] Max Packet=[4096] IR/IT contexts=[4/8]
May 14 14:05:47 vault1 kernel: input: PC Speaker as /class/input/input3
May 14 14:05:47 vault1 kernel: ck804xrom ck804xrom_init_one(): Unable to register resource 0x00000000ffc00000-0x00000000ffffffff - kernel bug?
May 14 14:05:48 vault1 kernel: Found: SST 49LF080A
May 14 14:05:48 vault1 kernel: ck804xrom @fff00000: Found 1 x8 devices at 0x0 in 8-bit bank
May 14 14:05:48 vault1 kernel: qla2xxx 0000:06:04.0: Allocated (412 KB) for firmware dump...
May 14 14:05:48 vault1 kernel: qla2xxx 0000:06:04.1: Allocated (412 KB) for firmware dump...
May 14 14:05:48 vault1 kernel: number of JEDEC chips: 1
May 14 14:05:48 vault1 kernel: cfi_cmdset_0002: Disabling erase-suspend-program due to code brokenness.
May 14 14:05:48 vault1 kernel: scsi 0:0:8:0: Attached scsi generic sg0 type 3
May 14 14:05:48 vault1 kernel: sd 0:1:0:0: Attached scsi generic sg1 type 0
May 14 14:05:48 vault1 kernel: sd 0:1:1:0: Attached scsi generic sg2 type 0
May 14 14:05:48 vault1 kernel: Floppy drive(s): fd0 is 1.44M
May 14 14:05:48 vault1 kernel: eth0: forcedeth.c: subsystem: 015d9:1211 bound to 0000:00:0a.0
May 14 14:05:48 vault1 kernel: ACPI: PCI Interrupt Link [LK2N] enabled at IRQ 47
May 14 14:05:48 vault1 kernel: ACPI: PCI Interrupt 0000:80:0a.0[A] -> Link [LK2N] -> GSI 47 (level, low) -> IRQ 47
May 14 14:05:48 vault1 kernel: forcedeth: using HIGHDMA
May 14 14:05:48 vault1 kernel: i2c_adapter i2c-0: nForce2 SMBus adapter at 0x4c00
May 14 14:05:48 vault1 kernel: i2c_adapter i2c-1: nForce2 SMBus adapter at 0x4c40
May 14 14:05:48 vault1 kernel: ACPI: PCI Interrupt Link [LTID] enabled at IRQ 22
May 14 14:05:48 vault1 kernel: ACPI: PCI Interrupt 0000:00:07.0[A] -> Link [LTID] -> GSI 22 (level, low) -> IRQ 22
May 14 14:05:48 vault1 kernel: sata_nv 0000:00:07.0: Using ADMA mode
May 14 14:05:48 vault1 kernel: ata1: SATA max UDMA/133 cmd 0xffffc20000054480 ctl 0xffffc200000544a0 bmdma 0x000000000001c800 irq 22
May 14 14:05:48 vault1 kernel: ata2: SATA max UDMA/133 cmd 0xffffc20000054580 ctl 0xffffc200000545a0 bmdma 0x000000000001c808 irq 22
May 14 14:05:48 vault1 kernel: scsi5 : sata_nv
May 14 14:05:48 vault1 kernel: FDC 0 is a post-1991 82077
May 14 14:05:48 vault1 kernel: ata1: SATA link down (SStatus 0 SControl 300)
May 14 14:05:48 vault1 kernel: scsi6 : sata_nv
May 14 14:05:48 vault1 kernel: eth1: forcedeth.c: subsystem: 015d9:1211 bound to 0000:80:0a.0
May 14 14:05:48 vault1 kernel: qla2xxx 0000:06:06.0: Mailbox command timeout occured. Issuing ISP abort.
May 14 14:05:48 vault1 kernel: qla2xxx 0000:06:06.0: Performing ISP error recovery - ha= ffff810130fbc750.
May 14 14:05:48 vault1 kernel: ata2: SATA link down (SStatus 0 SControl 300)
May 14 14:05:48 vault1 kernel: ACPI: PCI Interrupt Link [LTIE] enabled at IRQ 21
May 14 14:05:48 vault1 kernel: ACPI: PCI Interrupt 0000:00:08.0[A] -> Link [LTIE] -> GSI 21 (level, low) -> IRQ 21
May 14 14:05:48 vault1 kernel: sata_nv 0000:00:08.0: Using ADMA mode
May 14 14:05:48 vault1 kernel: ata3: SATA max UDMA/133 cmd 0xffffc20000056480 ctl 0xffffc200000564a0 bmdma 0x000000000001b400 irq 21
May 14 14:05:48 vault1 kernel: ata4: SATA max UDMA/133 cmd 0xffffc20000056580 ctl 0xffffc200000565a0 bmdma 0x000000000001b408 irq 21
May 14 14:05:48 vault1 kernel: scsi7 : sata_nv
May 14 14:05:48 vault1 kernel: qla2xxx 0000:06:06.0: Allocated (412 KB) for firmware dump...
May 14 14:05:48 vault1 kernel: ata3: SATA link down (SStatus 0 SControl 300)
May 14 14:05:48 vault1 kernel: scsi8 : sata_nv
May 14 14:05:48 vault1 kernel: ata4: SATA link down (SStatus 0 SControl 300)
May 14 14:05:48 vault1 kernel: ACPI: PCI Interrupt Link [LAUI] enabled at IRQ 20
May 14 14:05:48 vault1 kernel: ACPI: PCI Interrupt 0000:00:04.0[A] -> Link [LAUI] -> GSI 20 (level, low) -> IRQ 20
May 14 14:05:48 vault1 kernel: qla2xxx 0000:06:06.0: LIP reset occured (f8f7).
May 14 14:05:48 vault1 kernel: qla2xxx 0000:06:06.0: LIP occured (f8f7).
May 14 14:05:48 vault1 kernel: qla2xxx 0000:06:06.0: LOOP UP detected (2 Gbps).
May 14 14:05:48 vault1 kernel: scsi 3:0:0:0: Direct-Access YI-16SAE FC 66a. PQ: 0 ANSI: 5
May 14 14:05:48 vault1 kernel: SCSI device sdc: 1283080192 512-byte hdwr sectors (656937 MB)
May 14 14:05:48 vault1 kernel: sdc: Write Protect is off
May 14 14:05:48 vault1 kernel: SCSI device sdc: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 14 14:05:48 vault1 kernel: SCSI device sdc: 1283080192 512-byte hdwr sectors (656937 MB)
May 14 14:05:48 vault1 kernel: sdc: Write Protect is off
May 14 14:05:48 vault1 kernel: SCSI device sdc: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 14 14:05:48 vault1 kernel: sdc: unknown partition table
May 14 14:05:48 vault1 kernel: sd 3:0:0:0: Attached scsi disk sdc
May 14 14:05:49 vault1 kernel: sd 3:0:0:0: Attached scsi generic sg3 type 0
May 14 14:05:49 vault1 kernel: scsi 3:0:0:1: Direct-Access YI-16SAE FC 66a. PQ: 0 ANSI: 5
May 14 14:05:49 vault1 kernel: SCSI device sdd: 620380160 512-byte hdwr sectors (317635 MB)
May 14 14:05:49 vault1 kernel: sdd: Write Protect is off
May 14 14:05:49 vault1 kernel: SCSI device sdd: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 14 14:05:49 vault1 kernel: SCSI device sdd: 620380160 512-byte hdwr sectors (317635 MB)
May 14 14:05:49 vault1 kernel: sdd: Write Protect is off
May 14 14:05:49 vault1 kernel: SCSI device sdd: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 14 14:05:49 vault1 kernel: sdd: unknown partition table
May 14 14:05:49 vault1 kernel: sd 3:0:0:1: Attached scsi disk sdd
May 14 14:05:49 vault1 kernel: sd 3:0:0:1: Attached scsi generic sg4 type 0
May 14 14:05:49 vault1 kernel: intel8x0_measure_ac97_clock: measured 51408 usecs
May 14 14:05:49 vault1 kernel: intel8x0: clocking to 46853
May 14 14:05:49 vault1 kernel: lp0: using parport0 (interrupt-driven).
May 14 14:05:49 vault1 kernel: lp0: console ready
May 14 14:05:49 vault1 kernel: input: Power Button (FF) as /class/input/input4
May 14 14:05:49 vault1 kernel: ACPI: Power Button (FF) [PWRF]
May 14 14:05:49 vault1 kernel: input: Power Button (CM) as /class/input/input5
May 14 14:05:49 vault1 kernel: ACPI: Power Button (CM) [PWRB]
May 14 14:05:49 vault1 kernel: No dock devices found.
May 14 14:05:49 vault1 kernel: ibm_acpi: ec object not found
May 14 14:05:49 vault1 kernel: md: Autodetecting RAID arrays.
May 14 14:05:49 vault1 kernel: md: autorun ...
May 14 14:05:49 vault1 kernel: md: ... autorun DONE.
May 14 14:05:49 vault1 kernel: device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel@redhat.com
May 14 14:05:49 vault1 kernel: device-mapper: multipath: version 1.0.5 loaded
May 14 14:05:49 vault1 kernel: EXT3 FS on sda3, internal journal
May 14 14:05:49 vault1 kernel: JFS: nTxBlock = 8192, nTxLock = 65536
May 14 14:05:49 vault1 kernel: kjournald starting. Commit interval 5 seconds
May 14 14:05:49 vault1 kernel: EXT3 FS on sda7, internal journal
May 14 14:05:49 vault1 kernel: EXT3-fs: mounted filesystem with ordered data mode.
May 14 14:05:49 vault1 kernel: kjournald starting. Commit interval 5 seconds
May 14 14:05:49 vault1 kernel: EXT3 FS on sda5, internal journal
May 14 14:05:49 vault1 kernel: EXT3-fs: mounted filesystem with ordered data mode.
May 14 14:05:49 vault1 kernel: kjournald starting. Commit interval 5 seconds
May 14 14:05:49 vault1 kernel: EXT3 FS on sda6, internal journal
May 14 14:05:49 vault1 kernel: EXT3-fs: mounted filesystem with ordered data mode.
May 14 14:05:49 vault1 kernel: Adding 8385920k swap on /dev/sda2. Priority:-1 extents:1 across:8385920k
May 14 14:05:49 vault1 kernel: NET: Registered protocol family 10
May 14 14:05:49 vault1 kernel: lo: Disabled Privacy Extensions
May 14 14:05:49 vault1 kernel: Mobile IPv6
May 14 14:05:49 vault1 kernel: eth0: no link during initialization.
May 14 14:05:49 vault1 kernel: ADDRCONF(NETDEV_UP): eth0: link is not ready
May 14 14:05:49 vault1 kernel: Bluetooth: Core ver 2.11
May 14 14:05:49 vault1 kernel: NET: Registered protocol family 31
May 14 14:05:49 vault1 kernel: Bluetooth: HCI device and connection manager initialized
May 14 14:05:49 vault1 kernel: Bluetooth: HCI socket layer initialized
May 14 14:05:49 vault1 kernel: Bluetooth: L2CAP ver 2.8
May 14 14:05:49 vault1 kernel: Bluetooth: L2CAP socket layer initialized
May 14 14:05:49 vault1 kernel: Bluetooth: HIDP (Human Interface Emulation) ver 1.2
May 14 14:05:49 vault1 kernel: qla2xxx 0000:06:04.1: Cable is unplugged...
May 14 14:05:49 vault1 kernel: qla2xxx 0000:06:04.0: Cable is unplugged...
May 14 14:05:49 vault1 kernel: Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
May 14 14:05:49 vault1 kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
May 14 14:05:49 vault1 kernel: NFSD: starting 90-second grace period
May 14 14:05:49 vault1 kernel: qla2xxx 0000:06:06.1: Mailbox command timeout occured. Issuing ISP abort.
May 14 14:05:49 vault1 kernel: qla2xxx 0000:06:06.1: Performing ISP error recovery - ha= ffff810130a70750.
May 14 14:05:49 vault1 kernel: qla2xxx 0000:06:06.1: Allocated (412 KB) for firmware dump...
May 14 14:05:50 vault1 kernel: qla2xxx 0000:06:06.1: LIP reset occured (f8f7).
May 14 14:05:50 vault1 kernel: qla2xxx 0000:06:06.1: LIP occured (f8f7).
May 14 14:05:50 vault1 kernel: qla2xxx 0000:06:06.1: LOOP UP detected (2 Gbps).
May 14 14:05:50 vault1 kernel: scsi 4:0:0:0: Direct-Access YI-16SAE FC 606a PQ: 1 ANSI: 5
May 14 14:05:50 vault1 kernel: scsi 4:0:0:0: Attached scsi generic sg5 type 0
May 14 14:05:50 vault1 kernel: scsi 4:0:0:1: Direct-Access YI-16SAE FC 66a. PQ: 0 ANSI: 5
May 14 14:05:50 vault1 kernel: SCSI device sde: 775471104 512-byte hdwr sectors (397041 MB)
May 14 14:05:50 vault1 kernel: sde: Write Protect is off
May 14 14:05:50 vault1 kernel: SCSI device sde: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 14 14:05:50 vault1 kernel: SCSI device sde: 775471104 512-byte hdwr sectors (397041 MB)
May 14 14:05:50 vault1 kernel: sde: Write Protect is off
May 14 14:05:50 vault1 kernel: SCSI device sde: write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 14 14:05:50 vault1 kernel: sde: unknown partition table
May 14 14:05:50 vault1 kernel: sd 4:0:0:1: Attached scsi disk sde
May 14 14:05:50 vault1 kernel: sd 4:0:0:1: Attached scsi generic sg6 type 0
May 14 14:05:51 vault1 smartd[3874]: smartd version 5.36 [x86_64-redhat-linux-gnu] Copyright (C) 2002-6 Bruce Allen
May 14 14:05:51 vault1 smartd[3874]: Home page is http://smartmontools.sourceforge.net/
May 14 14:05:51 vault1 smartd[3874]: Opened configuration file /etc/smartd.conf
May 14 14:05:51 vault1 smartd[3874]: Configuration file /etc/smartd.conf parsed.
May 14 14:05:51 vault1 smartd[3874]: Device: /dev/sde, opened
May 14 14:05:51 vault1 smartd[3874]: Device: /dev/sde, is SMART capable. Adding to "monitor" list.
May 14 14:05:51 vault1 smartd[3874]: Device: /dev/sdc, opened
May 14 14:05:51 vault1 smartd[3874]: Device: /dev/sdc, is SMART capable. Adding to "monitor" list.
May 14 14:05:51 vault1 smartd[3874]: Device: /dev/sdd, opened
May 14 14:05:51 vault1 smartd[3874]: Device: /dev/sdd, is SMART capable. Adding to "monitor" list.
May 14 14:05:51 vault1 smartd[3874]: Monitoring 0 ATA and 3 SCSI devices
May 14 14:05:51 vault1 smartd[3876]: smartd has fork()ed into background mode. New PID=3876.
May 14 14:08:11 vault1 mountd[3644]: Caught signal 15, un-registering and exiting.
May 14 14:08:15 vault1 kernel: nfsd: last server has exited
May 14 14:08:15 vault1 kernel: nfsd: unexporting all filesystems
May 14 14:08:15 vault1 kernel: RPC: failed to contact portmap (errno -5).
May 14 14:08:16 vault1 kernel: NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory
May 14 14:08:16 vault1 kernel: NFSD: starting 90-second grace period
May 14 14:08:45 vault1 mountd[4068]: authenticated mount request from 172.19.1.247:941 for /test (/test)
May 14 16:34:15 vault1 dhclient: DHCPREQUEST on eth1 to 172.19.100.1 port 67
May 14 16:34:15 vault1 dhclient: DHCPACK from 172.19.100.1
May 14 16:34:15 vault1 dhclient: bound to 172.19.1.241 -- renewal in 9293 seconds.
May 14 19:09:07 vault1 dhclient: DHCPREQUEST on eth1 to 172.19.100.1 port 67
May 14 19:09:08 vault1 dhclient: DHCPACK from 172.19.100.1
May 14 19:09:08 vault1 dhclient: bound to 172.19.1.241 -- renewal in 8182 seconds.
May 14 21:25:30 vault1 dhclient: DHCPREQUEST on eth1 to 172.19.100.1 port 67
May 14 21:25:30 vault1 dhclient: DHCPACK from 172.19.100.1
May 14 21:25:30 vault1 dhclient: bound to 172.19.1.241 -- renewal in 8851 seconds.
May 14 23:53:00 vault1 dhclient: DHCPREQUEST on eth1 to 172.19.100.1 port 67
May 14 23:53:01 vault1 dhclient: DHCPACK from 172.19.100.1
May 14 23:53:01 vault1 dhclient: bound to 172.19.1.241 -- renewal in 9286 seconds.
May 15 02:27:46 vault1 dhclient: DHCPREQUEST on eth1 to 172.19.100.1 port 67
May 15 02:27:47 vault1 dhclient: DHCPACK from 172.19.100.1
May 15 02:27:47 vault1 dhclient: bound to 172.19.1.241 -- renewal in 10425 seconds.
May 15 05:21:32 vault1 dhclient: DHCPREQUEST on eth1 to 172.19.100.1 port 67
May 15 05:21:32 vault1 dhclient: DHCPACK from 172.19.100.1
May 15 05:21:32 vault1 dhclient: bound to 172.19.1.241 -- renewal in 9545 seconds.
May 15 05:27:25 vault1 kernel: SysRq : Show State
May 15 05:27:25 vault1 kernel:
May 15 05:27:25 vault1 kernel: free sibling
May 15 05:27:25 vault1 kernel: task PC stack pid father child younger older
May 15 05:27:25 vault1 kernel: init S ffff810230f85a98 0 1 0 (NOTLB)
May 15 05:27:25 vault1 kernel: ffff810136ba3a18 0000000000000046 0000000000000000 ffff810130c70040
May 15 05:27:25 vault1 kernel: ffff810130c70040 ffffffff802638fc 000000000000000a ffff810130c70040
May 15 05:27:25 vault1 kernel: ffffffff8058a6e0 000000000000932c ffff810130c70248 0000000080733000
May 15 05:27:25 vault1 kernel: Call Trace:
May 15 05:27:25 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:25 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 15 05:27:25 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 15 05:27:25 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:27:26 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:26 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:26 vault1 kernel: [<ffffffff80209596>] __d_lookup+0x77/0x110
May 15 05:27:26 vault1 kernel: e+0x13/0x34
May 15 05:27:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:26 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:26 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:26 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:26 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:26 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:26 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:26 vault1 kernel:
May 15 05:27:26 vault1 kernel: kblockd/3 S ffff810130b704b8 0 62 19 (L-TLB)
May 15 05:27:26 vault1 kernel: ffff810230f93ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:26 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff810230f3f080
May 15 05:27:26 vault1 kernel: ffff81022fd6e0c0 000000000000660c ffff810230f3f288 0000000330f3f080
May 15 05:27:26 vault1 kernel: Call Trace:
May 15 05:27:26 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:26 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:26 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:26 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:26 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:26 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:26 vault1 kernel:
May 15 05:27:26 vault1 kernel: kacpid S ffffffff8058a280 0 63 19 (L-TLB)
May 15 05:27:26 vault1 kernel: ffff810230f99ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:26 vault1 kernel: 0000000000000000 0000000000000000 0000000000000005 ffff810230f3e0c0
May 15 05:27:26 vault1 kernel: ffff810136be9080 0000000000000cfb ffff810230f3e2c8 0000000230f3e0c0
May 15 05:27:26 vault1 kernel: Call Trace:
May 15 05:27:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:26 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:26 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:26 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:26 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:26 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:27 vault1 kernel:
May 15 05:27:27 vault1 kernel: cqueue/0 S ffffffff8058a280 0 235 19 (L-TLB)
May 15 05:27:27 vault1 kernel: ffff810230e99ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000006 ffff810230f550c0
May 15 05:27:27 vault1 kernel: ffffffff8058a6e0 0000000000001ba6 ffff810230f552c8 0000000030f550c0
May 15 05:27:27 vault1 kernel: Call Trace:
May 15 05:27:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:27 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:27 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:27 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:27 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:27 vault1 kernel:
May 15 05:27:27 vault1 kernel: cqueue/1 S ffffffff8058a280 0 236 19 (L-TLB)
May 15 05:27:27 vault1 kernel: ffff810230e87ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000005 ffff810230f4b040
May 15 05:27:27 vault1 kernel: ffff810136b92040 0000000000001a77 ffff810230f4b248 0000000130f4b040
May 15 05:27:27 vault1 kernel: Call Trace:
May 15 05:27:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:27 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:27 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:27 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:27 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:27 vault1 kernel:
May 15 05:27:27 vault1 kernel: cqueue/2 S ffffffff8058a280 0 237 19 (L-TLB)
May 15 05:27:27 vault1 kernel: ffff810230e95ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000006 ffff810230f4a080
May 15 05:27:27 vault1 kernel: ffff810130a13080 0000000000001361 ffff810230f4a288 0000000230f4a080
May 15 05:27:27 vault1 kernel: Call Trace:
May 15 05:27:27 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:27 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:27 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:27 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:27 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:27 vault1 kernel:
May 15 05:27:27 vault1 kernel: cqueue/3 S ffffffff8058a280 0 238 19 (L-TLB)
May 15 05:27:27 vault1 kernel: ffff810230e97ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000005 ffff810230f44040
May 15 05:27:27 vault1 kernel: ffff810230c4c0c0 00000000000014a6 ffff810230f44248 0000000330f44040
May 15 05:27:27 vault1 kernel: Call Trace:
May 15 05:27:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:27 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:27 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:27 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:27 vault1 kernel:
May 15 05:27:27 vault1 kernel: ksuspend_usbd S ffffffff8058a280 0 239 19 (L-TLB)
May 15 05:27:27 vault1 kernel: ffff810230eb9ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000006 ffff810230f450c0
May 15 05:27:27 vault1 kernel: ffff810136be9080 0000000000001012 ffff810230f452c8 0000000230f450c0
May 15 05:27:27 vault1 kernel: Call Trace:
May 15 05:27:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:27 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:27 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:27 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:28 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:28 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:28 vault1 kernel:
May 15 05:27:28 vault1 kernel: khubd S ffff810130b71638 0 242 19 (L-TLB)
May 15 05:27:28 vault1 kernel: ffff8102309c1e20 0000000000000046 0000000000000000 0000000000000000
May 15 05:27:28 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810230f4d080
May 15 05:27:28 vault1 kernel: ffff810136be9080 0000000000011d07 ffff810230f4d288 0000000230f4d080
May 15 05:27:28 vault1 kernel: Call Trace:
May 15 05:27:28 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:28 vault1 kernel: [<ffffffff803d6dc8>] hub_thread+0xac0/0xb3c
May 15 05:27:28 vault1 kernel: [<ffffffff80260f58>] thread_return+0x88/0x12a
May 15 05:27:28 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:28 vault1 kernel: [<ffffffff803d6308>] hub_thread+0x0/0xb3c
May 15 05:27:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:28 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:28 vault1 kernel:
May 15 05:27:28 vault1 kernel: kseriod S ffffffff8058a280 0 244 19 (L-TLB)
May 15 05:27:28 vault1 kernel: ffff8102309c7ec0 0000000000000046 0000000000000000 0000000000000000
May 15 05:27:28 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810230f610c0
May 15 05:27:28 vault1 kernel: ffffffff8058a6e0 0000000000005165 ffff810230f612c8 0000000030f610c0
May 15 05:27:28 vault1 kernel: Call Trace:
May 15 05:27:28 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:28 vault1 kernel: [<ffffffff803eaaea>] serio_thread+0x2a5/0x2f8
May 15 05:27:28 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:28 vault1 kernel: [<ffffffff803ea845>] serio_thread+0x0/0x2f8
May 15 05:27:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:28 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:28 vault1 kernel:
May 15 05:27:28 vault1 kernel: pdflush S ffff810230f84058 0 298 19 (L-TLB)
May 15 05:27:28 vault1 kernel: ffff8102308f1ed0 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:27:28 vault1 kernel: 0000000180598080 0000000000000002 000000000000000a ffff810230f20080
May 15 05:27:28 vault1 kernel: ffff810136be9080 0000000000004416 ffff810230f20288 0000000200000000
May 15 05:27:28 vault1 kernel: Call Trace:
May 15 05:27:28 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:28 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 15 05:27:28 vault1 kernel: [<ffffffff80254e21>] pdflush+0xcf/0x1e7
May 15 05:27:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:28 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:28 vault1 kernel:
May 15 05:27:28 vault1 kernel: pdflush D ffff810230f84058 0 299 19 (L-TLB)
May 15 05:27:28 vault1 kernel: ffff8102308f3a20 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:27:28 vault1 kernel: 000000018020c209 0000000000000002 000000000000000a ffff810230f1f0c0
May 15 05:27:28 vault1 kernel: ffff810136be9080 00000000000024d0 ffff810230f1f2c8 0000000200000001
May 15 05:27:28 vault1 kernel: Call Trace:
May 15 05:27:28 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:28 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:28 vault1 kernel: [<ffffffff80262fa6>] __down_write_nested+0x7e/0x96
May 15 05:27:28 vault1 kernel: [<ffffffff8832253a>] :jfs:jfs_get_block+0x45/0x237
May 15 05:27:28 vault1 kernel: [<ffffffff8021c67c>] generic_make_request+0x207/0x21e
May 15 05:27:28 vault1 kernel: [<ffffffff802e9478>] __mpage_writepage+0x1f5/0x55d
May 15 05:27:28 vault1 kernel: [<ffffffff883224f5>] :jfs:jfs_get_block+0x0/0x237
May 15 05:27:28 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 15 05:27:28 vault1 kernel: [<ffffffff802c0835>] dec_zone_page_state+0x33/0x3a
May 15 05:27:28 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:28 vault1 kernel: [<ffffffff802c0835>] dec_zone_page_state+0x33/0x3a
May 15 05:27:28 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:28 vault1 kernel: [<ffffffff8021d493>] mpage_writepages+0x228/0x375
May 15 05:27:28 vault1 kernel: [<ffffffff883220bb>] :jfs:jfs_writepage+0x0/0xf
May 15 05:27:29 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:29 vault1 kernel: [<ffffffff883224f5>] :jfs:jfs_get_block+0x0/0x237
May 15 05:27:29 vault1 kernel: [<ffffffff802593b8>] do_writepages+0x23/0x30
May 15 05:27:29 vault1 kernel: [<ffffffff80230487>] __writeback_single_inode+0x1c0/0x338
May 15 05:27:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:29 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 15 05:27:29 vault1 kernel: [<ffffffff80221c26>] sync_sb_inodes+0x1a6/0x267
May 15 05:27:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:29 vault1 kernel: [<ffffffff8024f304>] writeback_inodes+0x95/0xee
May 15 05:27:29 vault1 kernel: [<ffffffff802be735>] wb_kupdate+0x9e/0x112
May 15 05:27:29 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 15 05:27:29 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 15 05:27:29 vault1 kernel: [<ffffffff80254e8e>] pdflush+0x13c/0x1e7
May 15 05:27:29 vault1 kernel: [<ffffffff802be697>] wb_kupdate+0x0/0x112
May 15 05:27:29 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:29 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:29 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:29 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:29 vault1 kernel:
May 15 05:27:29 vault1 kernel: kswapd0 S ffff810230f85a98 0 300 19 (L-TLB)
May 15 05:27:29 vault1 kernel: ffff8102308f5e20 0000000000000046 0000000000000000 0000000000000000
May 15 05:27:29 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810230f1e040
May 15 05:27:29 vault1 kernel: ffffffff8058a6e0 00000000000a49df ffff810230f1e248 0000000030f1e040
May 15 05:27:29 vault1 kernel: Call Trace:
May 15 05:27:29 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:29 vault1 kernel: [<ffffffff802560c6>] kswapd+0xea/0x469
May 15 05:27:29 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:29 vault1 kernel: [<ffffffff80255fdc>] kswapd+0x0/0x469
May 15 05:27:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:29 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:29 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:29 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:29 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:29 vault1 kernel:
May 15 05:27:29 vault1 kernel: kswapd1 S ffff81012eb76538 0 301 19 (L-TLB)
May 15 05:27:29 vault1 kernel: ffff8102308f7e20 0000000000000046 0000000000000000 0000000000000000
May 15 05:27:29 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810230f1d080
May 15 05:27:29 vault1 kernel: ffff810136be9080 00000000000a7777 ffff810230f1d288 0000000230f1d080
May 15 05:27:29 vault1 kernel: Call Trace:
May 15 05:27:29 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:29 vault1 kernel: [<ffffffff802560c6>] kswapd+0xea/0x469
May 15 05:27:29 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:29 vault1 kernel: [<ffffffff80255fdc>] kswapd+0x0/0x469
May 15 05:27:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:29 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:29 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:29 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:29 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:29 vault1 kernel:
May 15 05:27:29 vault1 kernel: aio/0 S ffffffff8058a280 0 302 19 (L-TLB)
May 15 05:27:29 vault1 kernel: ffff8102308f9ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:29 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff810230f1c0c0
May 15 05:27:29 vault1 kernel: ffffffff8058a6e0 00000000000016fc ffff810230f1c2c8 0000000030f1c0c0
May 15 05:27:29 vault1 kernel: Call Trace:
May 15 05:27:29 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:29 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:29 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:29 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:29 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:29 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:29 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:29 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:29 vault1 kernel:
May 15 05:27:29 vault1 kernel: aio/1 S ffffffff8058a280 0 303 19 (L-TLB)
May 15 05:27:29 vault1 kernel: ffff8102308fbea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:29 vault1 kernel: 0000000000000000 0000000000000000 0000000000000006 ffff810230f1b040
May 15 05:27:29 vault1 kernel: ffff810136b92040 0000000000001bb0 ffff810230f1b248 0000000130f1b040
May 15 05:27:29 vault1 kernel: Call Trace:
May 15 05:27:29 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:29 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:29 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:30 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:30 vault1 kernel:
May 15 05:27:30 vault1 kernel: aio/2 S ffffffff8058a280 0 304 19 (L-TLB)
May 15 05:27:30 vault1 kernel: ffff8102308fdea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:30 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff810230f19080
May 15 05:27:30 vault1 kernel: ffff810130a13080 0000000000001be7 ffff810230f19288 0000000230f19080
May 15 05:27:30 vault1 kernel: Call Trace:
May 15 05:27:30 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:30 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:30 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:30 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:30 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:30 vault1 kernel:
May 15 05:27:30 vault1 kernel: aio/3 S ffffffff8058a280 0 305 19 (L-TLB)
May 15 05:27:30 vault1 kernel: ffff8102308c1ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:30 vault1 kernel: 0000000000000000 0000000000000000 0000000000000008 ffff810230f32080
May 15 05:27:30 vault1 kernel: ffff810230c4c0c0 0000000000001498 ffff810230f32288 0000000330f32080
May 15 05:27:30 vault1 kernel: Call Trace:
May 15 05:27:30 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:30 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:30 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:30 vault1 kernel:
May 15 05:27:30 vault1 kernel: kpsmoused S ffffffff8058a280 0 461 19 (L-TLB)
May 15 05:27:30 vault1 kernel: ffff810130b5dea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:30 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130f640c0
May 15 05:27:30 vault1 kernel: ffffffff8058a6e0 0000000000000fc7 ffff810130f642c8 0000000030f640c0
May 15 05:27:30 vault1 kernel: Call Trace:
May 15 05:27:30 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:30 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:30 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:30 vault1 kernel:
May 15 05:27:30 vault1 kernel: scsi_eh_0 S ffff810130b71638 0 520 19 (L-TLB)
May 15 05:27:30 vault1 kernel: ffff810130203e70 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:30 vault1 kernel: 0000000000000000 0000000000000046 0000000000000004 ffff8101309050c0
May 15 05:27:30 vault1 kernel: ffff810136be9080 00000000000009fd ffff8101309052c8 0000000280263894
May 15 05:27:30 vault1 kernel: Call Trace:
May 15 05:27:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:30 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:27:30 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:27:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:30 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:30 vault1 kernel:
May 15 05:27:30 vault1 kernel: scsi_eh_1 S ffff810130b71638 0 536 19 (L-TLB)
May 15 05:27:30 vault1 kernel: ffff81013029de70 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:30 vault1 kernel: 0000000000000000 0000000000000046 0000000000000006 ffff81013007e0c0
May 15 05:27:30 vault1 kernel: ffff810136be9080 00000000000006ce ffff81013007e2c8 0000000280263894
May 15 05:27:30 vault1 kernel: Call Trace:
May 15 05:27:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:31 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:27:31 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:27:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:31 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:31 vault1 kernel:
May 15 05:27:31 vault1 kernel: qla2xxx_1_dpc S ffff810230e444f8 0 537 19 (L-TLB)
May 15 05:27:31 vault1 kernel: ffff810130f75ee0 0000000000000046 0000000000000000 ffffffff802913d9
May 15 05:27:31 vault1 kernel: ffff810130e5c7e8 ffff810130917040 000000000000000a ffff810130917040
May 15 05:27:31 vault1 kernel: ffff810136b92040 00000000000008a0 ffff810130917248 000000018029fe09
May 15 05:27:31 vault1 kernel: Call Trace:
May 15 05:27:31 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 15 05:27:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:31 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:27:31 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 15 05:27:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:31 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:27:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:31 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:31 vault1 kernel:
May 15 05:27:31 vault1 kernel: scsi_wq_1 S ffff810130b71638 0 538 19 (L-TLB)
May 15 05:27:31 vault1 kernel: ffff810130f73ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:31 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130ff8040
May 15 05:27:31 vault1 kernel: ffff810136be9080 00000000000014b7 ffff810130ff8248 0000000230ff8040
May 15 05:27:31 vault1 kernel: Call Trace:
May 15 05:27:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:31 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:31 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:31 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:31 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:31 vault1 kernel:
May 15 05:27:31 vault1 kernel: fc_wq_1 S ffff810130b71638 0 539 19 (L-TLB)
May 15 05:27:31 vault1 kernel: ffff810130fafea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:31 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130fff080
May 15 05:27:31 vault1 kernel: ffff810136be9080 00000000000012d2 ffff810130fff288 0000000230fff080
May 15 05:27:31 vault1 kernel: Call Trace:
May 15 05:27:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:31 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:31 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:31 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:31 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:31 vault1 kernel:
May 15 05:27:31 vault1 kernel: fc_dl_1 S ffff810130b71a98 0 540 19 (L-TLB)
May 15 05:27:31 vault1 kernel: ffff810130137ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:31 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81013007f0c0
May 15 05:27:31 vault1 kernel: ffffffff8058a6e0 0000000000001268 ffff81013007f2c8 000000003007f0c0
May 15 05:27:31 vault1 kernel: Call Trace:
May 15 05:27:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:31 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:31 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:31 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:31 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:31 vault1 kernel:
May 15 05:27:31 vault1 kernel: scsi_eh_2 S ffff810130b71638 0 541 19 (L-TLB)
May 15 05:27:32 vault1 kernel: ffff81013091de70 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:32 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff810130c5a040
May 15 05:27:32 vault1 kernel: ffff810136be9080 00000000000007e9 ffff810130c5a248 0000000280263894
May 15 05:27:32 vault1 kernel: Call Trace:
May 15 05:27:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:32 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:27:32 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:27:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:32 vault1 kernel:
May 15 05:27:32 vault1 kernel: qla2xxx_2_dpc S ffff810230e444f8 0 542 19 (L-TLB)
May 15 05:27:32 vault1 kernel: ffff810130fcbee0 0000000000000046 0000000000000000 ffff810130be1080
May 15 05:27:32 vault1 kernel: ffff810130be1080 ffffffff802638fc 000000000000000a ffff810130be1080
May 15 05:27:32 vault1 kernel: ffff810136b92040 0000000000000604 ffff810130be1288 00000001880ba30f
May 15 05:27:32 vault1 kernel: Call Trace:
May 15 05:27:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:32 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:27:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:32 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:27:32 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 15 05:27:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:32 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:27:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:32 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:32 vault1 kernel:
May 15 05:27:32 vault1 kernel: scsi_wq_2 S ffff810130b71638 0 543 19 (L-TLB)
May 15 05:27:32 vault1 kernel: ffff810130911ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:32 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff8101308170c0
May 15 05:27:32 vault1 kernel: ffff810136be9080 0000000000001242 ffff8101308172c8 00000002308170c0
May 15 05:27:32 vault1 kernel: Call Trace:
May 15 05:27:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:32 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:32 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:32 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:32 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:32 vault1 kernel:
May 15 05:27:32 vault1 kernel: fc_wq_2 S ffff810130b71638 0 544 19 (L-TLB)
May 15 05:27:32 vault1 kernel: ffff810130fc1ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:32 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130852040
May 15 05:27:32 vault1 kernel: ffff810136be9080 0000000000000fdf ffff810130852248 0000000230852040
May 15 05:27:32 vault1 kernel: Call Trace:
May 15 05:27:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:32 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:32 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:32 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:32 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:32 vault1 kernel:
May 15 05:27:32 vault1 kernel: fc_dl_2 S ffff810130b71638 0 545 19 (L-TLB)
May 15 05:27:32 vault1 kernel: ffff81013009dea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:32 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130853080
May 15 05:27:32 vault1 kernel: ffff810136be9080 00000000000011d4 ffff810130853288 0000000230853080
May 15 05:27:32 vault1 kernel: Call Trace:
May 15 05:27:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:32 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:32 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:32 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:32 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:33 vault1 kernel:
May 15 05:27:33 vault1 kernel: scsi_eh_3 S ffff810130b71638 0 546 19 (L-TLB)
May 15 05:27:33 vault1 kernel: ffff8101308e5e70 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:33 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81013081c0c0
May 15 05:27:33 vault1 kernel: ffff810136be9080 000000000000074f ffff81013081c2c8 0000000280263894
May 15 05:27:33 vault1 kernel: Call Trace:
May 15 05:27:33 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:33 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:27:33 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:27:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:33 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:33 vault1 kernel:
May 15 05:27:33 vault1 kernel: qla2xxx_3_dpc S ffff81012fe02998 0 547 19 (L-TLB)
May 15 05:27:33 vault1 kernel: ffff8101309f7ee0 0000000000000046 0000000000000000 ffff810130843040
May 15 05:27:33 vault1 kernel: ffff810130843040 ffffffff802638fc 000000000000000a ffff810130843040
May 15 05:27:33 vault1 kernel: ffff810136b92040 0000000000006120 ffff810130843248 00000001880ba30f
May 15 05:27:33 vault1 kernel: Call Trace:
May 15 05:27:33 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:33 vault1 kernel: [<ffffffff880bcb27>] :qla2xxx:qla2x00_fw_ready+0x84/0x1a2
May 15 05:27:33 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:27:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:33 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:27:33 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 15 05:27:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:33 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:27:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:33 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:33 vault1 kernel:
May 15 05:27:33 vault1 kernel: scsi_wq_3 S ffff810230f851d8 0 548 19 (L-TLB)
May 15 05:27:33 vault1 kernel: ffff8101309e9ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:33 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff810130be5080
May 15 05:27:33 vault1 kernel: ffffffff8058a6e0 0000000000042b71 ffff810130be5288 0000000030be5080
May 15 05:27:33 vault1 kernel: Call Trace:
May 15 05:27:33 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:33 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:33 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:33 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:33 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:33 vault1 kernel:
May 15 05:27:33 vault1 kernel: fc_wq_3 S ffff810130b71638 0 549 19 (L-TLB)
May 15 05:27:33 vault1 kernel: ffff810130293ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:33 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff8101300810c0
May 15 05:27:33 vault1 kernel: ffff810136be9080 0000000000001055 ffff8101300812c8 00000002300810c0
May 15 05:27:33 vault1 kernel: Call Trace:
May 15 05:27:33 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:33 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:33 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:33 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:33 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:33 vault1 kernel:
May 15 05:27:33 vault1 kernel: fc_dl_3 S ffff810130b71638 0 550 19 (L-TLB)
May 15 05:27:33 vault1 kernel: ffff810130909ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:33 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81013011d040
May 15 05:27:33 vault1 kernel: ffff810136be9080 0000000000001376 ffff81013011d248 000000023011d040
May 15 05:27:33 vault1 kernel: Call Trace:
May 15 05:27:33 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:33 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:33 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:33 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:34 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:34 vault1 kernel:
May 15 05:27:34 vault1 kernel: scsi_eh_4 S ffff810130b71638 0 551 19 (L-TLB)
May 15 05:27:34 vault1 kernel: ffff810130915e70 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:34 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81013008b080
May 15 05:27:34 vault1 kernel: ffff810136be9080 000000000000086b ffff81013008b288 0000000280263894
May 15 05:27:34 vault1 kernel: Call Trace:
May 15 05:27:34 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:34 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:27:34 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:27:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:34 vault1 kernel:
May 15 05:27:34 vault1 kernel: qla2xxx_4_dpc S ffff81012f451218 0 552 19 (L-TLB)
May 15 05:27:34 vault1 kernel: ffff81013014bee0 0000000000000046 0000000000000000 ffff8101300740c0
May 15 05:27:34 vault1 kernel: ffff8101300740c0 ffffffff802638fc 000000000000000a ffff8101300740c0
May 15 05:27:34 vault1 kernel: ffff81012f4f3040 0000000000008a77 ffff8101300742c8 00000000880ba30f
May 15 05:27:34 vault1 kernel: Call Trace:
May 15 05:27:34 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:34 vault1 kernel: [<ffffffff880bcb27>] :qla2xxx:qla2x00_fw_ready+0x84/0x1a2
May 15 05:27:34 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:27:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:34 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:27:34 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 15 05:27:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:34 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:27:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:34 vault1 kernel:
May 15 05:27:34 vault1 kernel: scsi_wq_4 S ffff81012f451218 0 553 19 (L-TLB)
May 15 05:27:34 vault1 kernel: ffff81013016dea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:34 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff8101301e2040
May 15 05:27:34 vault1 kernel: ffff81012eea1080 00000000000671cb ffff8101301e2248 00000000301e2040
May 15 05:27:34 vault1 kernel: Call Trace:
May 15 05:27:34 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:34 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:34 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:34 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:34 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:34 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:34 vault1 kernel:
May 15 05:27:34 vault1 kernel: fc_wq_4 S ffff810130b71638 0 554 19 (L-TLB)
May 15 05:27:34 vault1 kernel: ffff810130099ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:34 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130185080
May 15 05:27:34 vault1 kernel: ffff810136be9080 0000000000000fd3 ffff810130185288 0000000230185080
May 15 05:27:34 vault1 kernel: Call Trace:
May 15 05:27:34 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:34 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:34 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:34 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:34 vault1 kernel:
May 15 05:27:34 vault1 kernel: fc_dl_4 S ffff810130b71638 0 555 19 (L-TLB)
May 15 05:27:34 vault1 kernel: ffff81013009bea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:34 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff8101301840c0
May 15 05:27:34 vault1 kernel: ffff810136be9080 0000000000001084 ffff8101301842c8 00000002301840c0
May 15 05:27:34 vault1 kernel: Call Trace:
May 15 05:27:34 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:34 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:34 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:34 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:34 vault1 kernel:
May 15 05:27:35 vault1 kernel: kjournald S ffff810230f84058 0 556 19 (L-TLB)
May 15 05:27:35 vault1 kernel: ffff81013017bec0 0000000000000046 0000000000000000 0000000000000000
May 15 05:27:35 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810130168040
May 15 05:27:35 vault1 kernel: ffff810136b92040 0000000000002d62 ffff810130168248 000000018029fc0f
May 15 05:27:35 vault1 kernel: Call Trace:
May 15 05:27:35 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 15 05:27:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:35 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 15 05:27:35 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:35 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 15 05:27:35 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:35 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:35 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:35 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:35 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:35 vault1 kernel:
May 15 05:27:35 vault1 kernel: kauditd S ffff81012f8576f8 0 583 19 (L-TLB)
May 15 05:27:35 vault1 kernel: ffff810130117ec0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:35 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81012fd850c0
May 15 05:27:35 vault1 kernel: ffff81012ff09040 0000000000007ed7 ffff81012fd852c8 000000002fd850c0
May 15 05:27:35 vault1 kernel: Call Trace:
May 15 05:27:35 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:35 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:35 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:35 vault1 kernel: [<ffffffff802b2b39>] kauditd_thread+0x111/0x148
May 15 05:27:35 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:35 vault1 kernel: [<ffffffff802b2a28>] kauditd_thread+0x0/0x148
May 15 05:27:35 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:35 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:35 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:35 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:35 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:35 vault1 kernel:
May 15 05:27:35 vault1 kernel: udevd S ffff81012f856e38 0 617 1 (NOTLB)
May 15 05:27:35 vault1 kernel: ffff810130acba18 0000000000000046 0000000000000000 0000000000000046
May 15 05:27:35 vault1 kernel: ffffffff802ea6a4 ffff81013025a7b8 0000000000000005 ffff81013025a080
May 15 05:27:35 vault1 kernel: ffffffff8058a6e0 0000000000024d6c ffff81013025a288 0000000080262123
May 15 05:27:35 vault1 kernel: Call Trace:
May 15 05:27:35 vault1 kernel: [<ffffffff802ea6a4>] inotify_poll+0x31/0x56
May 15 05:27:35 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:35 vault1 kernel: [<ffffffff802ea6c2>] inotify_poll+0x4f/0x56
May 15 05:27:35 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:27:35 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:35 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:35 vault1 last message repeated 3 times
May 15 05:27:35 vault1 kernel: [<ffffffff8023e153>] lookup_mnt+0x19/0x4c
May 15 05:27:35 vault1 kernel: [<ffffffff8023e17e>] lookup_mnt+0x44/0x4c
May 15 05:27:35 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 15 05:27:35 vault1 kernel: [<ffffffff80219f0b>] __follow_mount+0x26/0x7b
May 15 05:27:35 vault1 kernel: [<ffffffff8020cff9>] do_lookup+0x63/0x1ae
May 15 05:27:35 vault1 kernel: [<ffffffff802119b8>] do_wp_page+0x5af/0x60b
May 15 05:27:35 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:27:35 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:27:35 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:35 vault1 kernel: [<ffffffff8023bee1>] remove_wait_queue+0x10/0x2c
May 15 05:27:35 vault1 kernel: [<ffffffff8023bee1>] remove_wait_queue+0x10/0x2c
May 15 05:27:35 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:35 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:35 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 15 05:27:35 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:35 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:35 vault1 kernel:
May 15 05:27:35 vault1 kernel: khpsbpkt S ffff81012f856e38 0 1220 19 (L-TLB)
May 15 05:27:35 vault1 kernel: ffff81012f317f00 0000000000000046 ffffffff80246747 0000000000000002
May 15 05:27:35 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012f7f0040
May 15 05:27:35 vault1 kernel: ffff81013025a080 0000000000001ce5 ffff81012f7f0248 0000000230191d30
May 15 05:27:35 vault1 kernel: Call Trace:
May 15 05:27:35 vault1 kernel: [<ffffffff80246747>] skb_dequeue+0x13/0x50
May 15 05:27:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff8811e239>] :ieee1394:hpsbpkt_thread+0x0/0x8d
May 15 05:27:36 vault1 kernel: [<ffffffff8811e2a9>] :ieee1394:hpsbpkt_thread+0x70/0x8d
May 15 05:27:36 vault1 kernel: [<ffffffff8811e239>] :ieee1394:hpsbpkt_thread+0x0/0x8d
May 15 05:27:36 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:36 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:36 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:36 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:36 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:36 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:36 vault1 kernel:
May 15 05:27:36 vault1 kernel: knodemgrd_0 S ffff810130b71638 0 1271 19 (L-TLB)
May 15 05:27:36 vault1 kernel: ffff81012f21be70 0000000000000046 ffff810130d82080 ffffffff80261ece
May 15 05:27:36 vault1 kernel: 0000000000000246 ffff810230d3bc88 0000000000000009 ffff810130d82080
May 15 05:27:36 vault1 kernel: ffff81012f3dc040 0000000000098704 ffff810130d82288 0000000080261ece
May 15 05:27:36 vault1 kernel: Call Trace:
May 15 05:27:36 vault1 kernel: [<ffffffff80261ece>] __mutex_unlock_slowpath+0x128/0x133
May 15 05:27:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:36 vault1 kernel: [<ffffffff88122d00>] :ieee1394:nodemgr_host_thread+0x87/0xa58
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:36 vault1 kernel: [<ffffffff88122c79>] :ieee1394:nodemgr_host_thread+0x0/0xa58
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:36 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:36 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:36 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:36 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:36 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:36 vault1 kernel:
May 15 05:27:36 vault1 kernel: ata/0 S ffff810230f85a98 0 1364 19 (L-TLB)
May 15 05:27:36 vault1 kernel: ffff81012eba9ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:36 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff81012f6d2080
May 15 05:27:36 vault1 kernel: ffff810130a13080 0000000000001a45 ffff81012f6d2288 000000002f6d2080
May 15 05:27:36 vault1 kernel: Call Trace:
May 15 05:27:36 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:36 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:36 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:36 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:36 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:36 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:36 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:36 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:36 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:36 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:36 vault1 kernel:
May 15 05:27:36 vault1 kernel: ata/1 S ffff81012f856e38 0 1365 19 (L-TLB)
May 15 05:27:36 vault1 kernel: ffff81012ebabea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:36 vault1 kernel: 0000000000000000 0000000000000000 0000000000000006 ffff81012f791040
May 15 05:27:36 vault1 kernel: ffff81013025a080 000000000000208c ffff81012f791248 000000012f791040
May 15 05:27:36 vault1 kernel: Call Trace:
May 15 05:27:36 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:36 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:36 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:36 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:36 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:36 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:36 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:36 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:36 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:36 vault1 kernel:
May 15 05:27:36 vault1 kernel: ata/2 S ffff810230f84058 0 1366 19 (L-TLB)
May 15 05:27:36 vault1 kernel: ffff81012ebafea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:36 vault1 kernel: 0000000000000000 0000000000000000 0000000000000006 ffff81012f46e080
May 15 05:27:36 vault1 kernel: ffff81012fac80c0 000000000000267d ffff81012f46e288 000000022f46e080
May 15 05:27:36 vault1 kernel: Call Trace:
May 15 05:27:36 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:36 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:36 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:36 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:36 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:36 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:36 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:36 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:36 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:36 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:36 vault1 kernel:
May 15 05:27:36 vault1 kernel: ata/3 S ffff81012f5bd6b8 0 1367 19 (L-TLB)
May 15 05:27:36 vault1 kernel: ffff81012faf9ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:36 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff81012f0950c0
May 15 05:27:36 vault1 kernel: ffff8101308a9080 00000000000021a6 ffff81012f0952c8 000000032f0950c0
May 15 05:27:36 vault1 kernel: Call Trace:
May 15 05:27:36 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:36 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:36 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:37 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:37 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:37 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:37 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:37 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:37 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:37 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:37 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:37 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:37 vault1 kernel:
May 15 05:27:37 vault1 kernel: ata_aux S ffff810130840058 0 1368 19 (L-TLB)
May 15 05:27:37 vault1 kernel: ffff81012ebb1ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:37 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff81012f501040
May 15 05:27:37 vault1 kernel: ffff81012f871040 0000000000001202 ffff81012f501248 000000002f501040
May 15 05:27:37 vault1 kernel: Call Trace:
May 15 05:27:37 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:37 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:37 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:37 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:37 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:37 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:37 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:37 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:37 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:37 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:37 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:37 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:37 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:37 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:37 vault1 kernel:
May 15 05:27:37 vault1 kernel: scsi_eh_5 S ffff810130840918 0 1675 19 (L-TLB)
May 15 05:27:37 vault1 kernel: ffff81012e7b1e70 0000000000000046 ffff81012f0b4040 ffffffff802638fc
May 15 05:27:37 vault1 kernel: ffff81012f898000 0000000000000000 0000000000000003 ffff81012f0b4040
May 15 05:27:37 vault1 kernel: ffff810230d72080 0000000000015241 ffff81012f0b4248 0000000100000282
May 15 05:27:37 vault1 kernel: Call Trace:
May 15 05:27:37 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:37 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 15 05:27:37 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:37 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:27:37 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:37 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:37 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:27:37 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:37 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:37 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:37 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:37 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:37 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:37 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:37 vault1 kernel:
May 15 05:27:37 vault1 kernel: scsi_eh_6 S ffff81012eb76998 0 1677 19 (L-TLB)
May 15 05:27:37 vault1 kernel: ffff81012e2cfe70 0000000000000046 ffff81012fdea0c0 ffffffff802638fc
May 15 05:27:37 vault1 kernel: ffff81012ff60000 0000000000000000 0000000000000006 ffff81012fdea0c0
May 15 05:27:37 vault1 kernel: ffff81012f7dc0c0 00000000000060c6 ffff81012fdea2c8 0000000000000282
May 15 05:27:37 vault1 kernel: Call Trace:
May 15 05:27:37 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:37 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 15 05:27:37 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:37 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:27:37 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:37 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:37 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:27:37 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:37 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:37 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:37 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:37 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:37 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:37 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:37 vault1 kernel:
May 15 05:27:37 vault1 kernel: scsi_eh_7 S ffff81022fe50e38 0 2076 19 (L-TLB)
May 15 05:27:37 vault1 kernel: ffff81012df47e70 0000000000000046 ffff81012efdc040 ffffffff802638fc
May 15 05:27:37 vault1 kernel: ffff81012df20000 0000000000000000 0000000000000005 ffff81012efdc040
May 15 05:27:37 vault1 kernel: ffff81012f7dc0c0 0000000000005d45 ffff81012efdc248 0000000000000282
May 15 05:27:37 vault1 kernel: Call Trace:
May 15 05:27:37 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:37 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 15 05:27:37 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:27:38 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:38 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:27:38 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:38 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:38 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:38 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:38 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:38 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:38 vault1 kernel:
May 15 05:27:38 vault1 kernel: scsi_eh_8 S ffff8102300764b8 0 2077 19 (L-TLB)
May 15 05:27:38 vault1 kernel: ffff81012df1fe70 0000000000000046 ffff81012eea30c0 ffffffff802638fc
May 15 05:27:38 vault1 kernel: ffff81012e5e4000 0000000000000000 0000000000000008 ffff81012eea30c0
May 15 05:27:38 vault1 kernel: ffff81012f7dc0c0 0000000000004d90 ffff81012eea32c8 0000000000000282
May 15 05:27:38 vault1 kernel: Call Trace:
May 15 05:27:38 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:38 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 15 05:27:38 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:27:38 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:38 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:27:38 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:38 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:38 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:38 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:38 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:38 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:38 vault1 kernel:
May 15 05:27:38 vault1 kernel: kmpathd/0 S ffff810230cee998 0 2515 19 (L-TLB)
May 15 05:27:38 vault1 kernel: ffff81012d897ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:38 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81012f70f040
May 15 05:27:38 vault1 kernel: ffff810130a13080 0000000000000d99 ffff81012f70f248 000000002f70f040
May 15 05:27:38 vault1 kernel: Call Trace:
May 15 05:27:38 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:38 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:38 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:38 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:38 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:38 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:38 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:38 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:38 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:38 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:38 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:38 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:38 vault1 kernel:
May 15 05:27:38 vault1 kernel: kmpathd/1 S ffff810230f85a98 0 2516 19 (L-TLB)
May 15 05:27:38 vault1 kernel: ffff81012d899ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:38 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff810130f6a0c0
May 15 05:27:38 vault1 kernel: ffff810136b92040 0000000000001abe ffff810130f6a2c8 0000000130f6a0c0
May 15 05:27:38 vault1 kernel: Call Trace:
May 15 05:27:38 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:38 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:38 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:38 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:38 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:38 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:38 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:38 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:38 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:38 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:38 vault1 kernel:
May 15 05:27:38 vault1 kernel: kmpathd/2 S ffff81012f856e38 0 2517 19 (L-TLB)
May 15 05:27:38 vault1 kernel: ffff81012d89bea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:38 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff8101308a9080
May 15 05:27:38 vault1 kernel: ffff81013025a080 0000000000002088 ffff8101308a9288 00000002308a9080
May 15 05:27:38 vault1 kernel: Call Trace:
May 15 05:27:38 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:38 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:38 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:38 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:38 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:38 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:39 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:39 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:39 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:39 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:39 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:39 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:39 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:39 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:39 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:39 vault1 kernel:
May 15 05:27:39 vault1 kernel: kmpathd/3 S ffff810230cee998 0 2518 19 (L-TLB)
May 15 05:27:39 vault1 kernel: ffff81012d89dea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:39 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012f2030c0
May 15 05:27:39 vault1 kernel: ffff810230c4c0c0 0000000000003006 ffff81012f2032c8 000000032f2030c0
May 15 05:27:39 vault1 kernel: Call Trace:
May 15 05:27:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:39 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:39 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:39 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:39 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:39 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:39 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:39 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:39 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:39 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:39 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:39 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:39 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:39 vault1 kernel:
May 15 05:27:39 vault1 kernel: kmirrord S ffff810230d4f6f8 0 2533 19 (L-TLB)
May 15 05:27:39 vault1 kernel: ffff81012dfd9ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:39 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff810130ee0080
May 15 05:27:39 vault1 kernel: ffff810230c4c0c0 00000000000019d6 ffff810130ee0288 0000000330ee0080
May 15 05:27:39 vault1 kernel: Call Trace:
May 15 05:27:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:39 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:39 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:39 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:39 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:39 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:39 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:39 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:39 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:39 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:39 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:39 vault1 kernel:
May 15 05:27:39 vault1 kernel: jfsIO S ffff810130b704b8 0 2571 19 (L-TLB)
May 15 05:27:39 vault1 kernel: ffff81012c877f00 0000000000000046 0000000000000000 0000000000000001
May 15 05:27:39 vault1 kernel: ffff81012c876000 ffff8101b644cd00 000000000000000a ffff810130dba0c0
May 15 05:27:39 vault1 kernel: ffff810230c4c0c0 000000000006370c ffff810130dba2c8 0000000300000001
May 15 05:27:39 vault1 kernel: Call Trace:
May 15 05:27:39 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:39 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:39 vault1 kernel: [<ffffffff88338385>] :jfs:jfsIOWait+0x0/0x96
May 15 05:27:39 vault1 kernel: [<ffffffff883383fe>] :jfs:jfsIOWait+0x79/0x96
May 15 05:27:39 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:39 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:39 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:39 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:39 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:39 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:39 vault1 kernel:
May 15 05:27:39 vault1 kernel: jfsCommit S ffff810230f84058 0 2572 19 (L-TLB)
May 15 05:27:39 vault1 kernel: ffff81012c879eb0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:39 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012f22d080
May 15 05:27:39 vault1 kernel: ffff810136be9080 0000000000000776 ffff81012f22d288 000000022f22d080
May 15 05:27:39 vault1 kernel: Call Trace:
May 15 05:27:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:39 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:39 vault1 kernel: [<ffffffff8833b108>] :jfs:jfs_lazycommit+0x1d9/0x212
May 15 05:27:39 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:39 vault1 kernel: [<ffffffff8833af2f>] :jfs:jfs_lazycommit+0x0/0x212
May 15 05:27:39 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:39 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:39 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:39 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:39 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:39 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:39 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:39 vault1 kernel:
May 15 05:27:39 vault1 kernel: jfsCommit S ffff810230f84058 0 2573 19 (L-TLB)
May 15 05:27:39 vault1 kernel: ffff81012c87beb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:39 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012f722080
May 15 05:27:39 vault1 kernel: ffff81012f6be0c0 000000000000bbdd ffff81012f722288 000000022f722080
May 15 05:27:39 vault1 kernel: Call Trace:
May 15 05:27:39 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:39 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:39 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:39 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:39 vault1 kernel: [<ffffffff8833b108>] :jfs:jfs_lazycommit+0x1d9/0x212
May 15 05:27:39 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:39 vault1 kernel: [<ffffffff8833af2f>] :jfs:jfs_lazycommit+0x0/0x212
May 15 05:27:40 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:40 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:40 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:40 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:40 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:40 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:40 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:40 vault1 kernel:
May 15 05:27:40 vault1 kernel: jfsCommit S ffff810230f84058 0 2574 19 (L-TLB)
May 15 05:27:40 vault1 kernel: ffff81012c87deb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:40 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff810130dda080
May 15 05:27:40 vault1 kernel: ffff81012f22d080 0000000000000976 ffff810130dda288 0000000230dda080
May 15 05:27:40 vault1 kernel: Call Trace:
May 15 05:27:40 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:40 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:40 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:40 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:40 vault1 kernel: [<ffffffff8833b108>] :jfs:jfs_lazycommit+0x1d9/0x212
May 15 05:27:40 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:40 vault1 kernel: [<ffffffff8833af2f>] :jfs:jfs_lazycommit+0x0/0x212
May 15 05:27:40 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:40 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:40 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:40 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:40 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:40 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:40 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:40 vault1 kernel:
May 15 05:27:40 vault1 kernel: jfsCommit S ffff810230f84058 0 2575 19 (L-TLB)
May 15 05:27:40 vault1 kernel: ffff81012c87feb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:40 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012f6be0c0
May 15 05:27:40 vault1 kernel: ffff810130dda080 0000000000000a3e ffff81012f6be2c8 000000022f6be0c0
May 15 05:27:40 vault1 kernel: Call Trace:
May 15 05:27:40 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:40 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:40 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:40 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:40 vault1 kernel: [<ffffffff8833b108>] :jfs:jfs_lazycommit+0x1d9/0x212
May 15 05:27:40 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:40 vault1 kernel: [<ffffffff8833af2f>] :jfs:jfs_lazycommit+0x0/0x212
May 15 05:27:40 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:40 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:40 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:40 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:40 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:40 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:40 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:40 vault1 kernel:
May 15 05:27:40 vault1 kernel: jfsSync S ffff81012f856e38 0 2576 19 (L-TLB)
May 15 05:27:40 vault1 kernel: ffff81012c881ee0 0000000000000046 0000000000000000 ffff81012ef5a040
May 15 05:27:40 vault1 kernel: 00000000000001a9 0000000000000001 0000000000000008 ffff81012ef5a040
May 15 05:27:40 vault1 kernel: ffff810136be9080 00000000000122f8 ffff81012ef5a248 0000000200000246
May 15 05:27:40 vault1 kernel: Call Trace:
May 15 05:27:40 vault1 kernel: [<ffffffff8833a8e3>] :jfs:jfs_sync+0x181/0x1ac
May 15 05:27:40 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:40 vault1 kernel: [<ffffffff8833a762>] :jfs:jfs_sync+0x0/0x1ac
May 15 05:27:40 vault1 kernel: [<ffffffff8833a8e8>] :jfs:jfs_sync+0x186/0x1ac
May 15 05:27:40 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:40 vault1 kernel: [<ffffffff8833a762>] :jfs:jfs_sync+0x0/0x1ac
May 15 05:27:40 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:40 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:40 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:40 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:40 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:40 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:40 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:40 vault1 kernel:
May 15 05:27:40 vault1 kernel: kjournald S ffff810230e444f8 0 2578 19 (L-TLB)
May 15 05:27:40 vault1 kernel: ffff81012d563ec0 0000000000000046 0000000000000000 0000000000000000
May 15 05:27:40 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff81012f871040
May 15 05:27:40 vault1 kernel: ffff810136b92040 0000000000001be6 ffff81012f871248 000000018029fc0f
May 15 05:27:40 vault1 kernel: Call Trace:
May 15 05:27:41 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 15 05:27:41 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:41 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 15 05:27:41 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:41 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:41 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 15 05:27:41 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:41 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:41 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:41 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:41 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:41 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:41 vault1 kernel:
May 15 05:27:41 vault1 kernel: kjournald S ffff810230f84058 0 2580 19 (L-TLB)
May 15 05:27:41 vault1 kernel: ffff81012d3cdec0 0000000000000046 0000000000000000 0000000000000000
May 15 05:27:41 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff81012fac80c0
May 15 05:27:41 vault1 kernel: ffff810136b92040 0000000000002f6b ffff81012fac82c8 000000018029fc0f
May 15 05:27:41 vault1 kernel: Call Trace:
May 15 05:27:41 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 15 05:27:41 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:41 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 15 05:27:41 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:41 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:41 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 15 05:27:41 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:41 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:41 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:41 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:41 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:41 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:41 vault1 kernel:
May 15 05:27:41 vault1 kernel: kondemand/0 S ffff81012f450db8 0 2712 19 (L-TLB)
May 15 05:27:41 vault1 kernel: ffff81012d509ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:41 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012e9e5040
May 15 05:27:41 vault1 kernel: ffffffff8058a6e0 000000000000101c ffff81012e9e5248 000000002e9e5040
May 15 05:27:41 vault1 kernel: Call Trace:
May 15 05:27:41 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:41 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:41 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:41 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:41 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:41 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:41 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:41 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:41 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:41 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:41 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:41 vault1 kernel:
May 15 05:27:41 vault1 kernel: kondemand/1 S ffff81012f450db8 0 2713 19 (L-TLB)
May 15 05:27:41 vault1 kernel: ffff81012c68fea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:41 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff81012f238040
May 15 05:27:41 vault1 kernel: ffff810136b92040 00000000000021a8 ffff81012f238248 000000012f238040
May 15 05:27:41 vault1 kernel: Call Trace:
May 15 05:27:41 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:41 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:41 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:41 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:41 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:41 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:41 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:41 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:41 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:41 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:41 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:41 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:41 vault1 kernel:
May 15 05:27:41 vault1 kernel: kondemand/2 S ffff81012eb76998 0 2714 19 (L-TLB)
May 15 05:27:41 vault1 kernel: ffff81012c595ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:41 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff8101301b8080
May 15 05:27:41 vault1 kernel: ffff810136be9080 0000000000000d81 ffff8101301b8288 00000002301b8080
May 15 05:27:41 vault1 kernel: Call Trace:
May 15 05:27:41 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:41 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:41 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:41 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:41 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:41 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:41 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:41 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:41 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:41 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:41 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:41 vault1 kernel:
May 15 05:27:41 vault1 kernel: kondemand/3 S ffff81012f856e38 0 2715 19 (L-TLB)
May 15 05:27:41 vault1 kernel: ffff81012c599ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:41 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff81012e46b0c0
May 15 05:27:41 vault1 kernel: ffff81013025a080 0000000000002efb ffff81012e46b2c8 000000032e46b0c0
May 15 05:27:41 vault1 kernel: Call Trace:
May 15 05:27:41 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:41 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:41 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:41 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:41 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:41 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:42 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:42 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:42 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:42 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:42 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:42 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:42 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:42 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:42 vault1 kernel:
May 15 05:27:42 vault1 kernel: dhclient S ffff81012eb77b18 0 3260 1 (NOTLB)
May 15 05:27:42 vault1 kernel: ffff810130dfba18 0000000000000046 0000000000000000 ffff81012f3b0040
May 15 05:27:42 vault1 kernel: ffff81012f3b0040 ffffffff802638fc 000000000000000a ffff81012f3b0040
May 15 05:27:42 vault1 kernel: ffff810136b92040 00000000000022f6 ffff81012f3b0248 0000000130c84000
May 15 05:27:42 vault1 kernel: Call Trace:
May 15 05:27:42 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:42 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 15 05:27:42 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 15 05:27:42 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:27:42 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:42 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:42 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:42 vault1 kernel: [<ffffffff802078c3>] kmem_cache_free+0x257/0x267
May 15 05:27:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:42 vault1 kernel: [<ffffffff804435a0>] udp_recvmsg+0x1ee/0x270
May 15 05:27:42 vault1 kernel: [<ffffffff80232600>] sock_common_recvmsg+0x30/0x45
May 15 05:27:42 vault1 kernel: [<ffffffff80230bc6>] sock_recvmsg+0xf8/0x117
May 15 05:27:42 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:42 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:27:42 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:27:42 vault1 kernel: [<ffffffff8040bc77>] move_addr_to_user+0x5d/0x78
May 15 05:27:42 vault1 kernel: [<ffffffff8022c605>] sys_recvfrom+0x104/0x130
May 15 05:27:42 vault1 kernel: [<ffffffff80216b98>] sys_select+0x15a/0x183
May 15 05:27:42 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:42 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:42 vault1 kernel:
May 15 05:27:42 vault1 kernel: syslogd D ffff81012fe03b18 0 3374 1 (NOTLB)
May 15 05:27:42 vault1 kernel: ffff81012bd7fd88 0000000000000046 0000000000000000 0000000000000000
May 15 05:27:42 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff81012f0670c0
May 15 05:27:42 vault1 kernel: ffff8101301b70c0 0000000000019980 ffff81012f0672c8 000000008029fc0f
May 15 05:27:42 vault1 kernel: Call Trace:
May 15 05:27:42 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:42 vault1 kernel: [<ffffffff88026c13>] :jbd:log_wait_commit+0xa4/0xf7
May 15 05:27:42 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:42 vault1 kernel: [<ffffffff88021730>] :jbd:journal_stop+0x1e5/0x21a
May 15 05:27:42 vault1 kernel: [<ffffffff802304c1>] __writeback_single_inode+0x1fa/0x338
May 15 05:27:42 vault1 kernel: [<ffffffff802e2073>] sync_inode+0x19/0x39
May 15 05:27:42 vault1 kernel: [<ffffffff802e2073>] sync_inode+0x19/0x39
May 15 05:27:42 vault1 kernel: [<ffffffff802e207e>] sync_inode+0x24/0x39
May 15 05:27:42 vault1 kernel: [<ffffffff88037332>] :ext3:ext3_sync_file+0xb2/0xc4
May 15 05:27:42 vault1 kernel: [<ffffffff8024e527>] do_fsync+0x52/0xa4
May 15 05:27:42 vault1 kernel: [<ffffffff802e4478>] __do_fsync+0x23/0x36
May 15 05:27:42 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:42 vault1 kernel:
May 15 05:27:42 vault1 kernel: klogd S ffff81012f450db8 0 3377 1 (NOTLB)
May 15 05:27:42 vault1 kernel: ffff810127185be8 0000000000000046 0000000000000000 ffff8101301b70c0
May 15 05:27:42 vault1 kernel: 0000000000000001 ffffffff802a0de5 0000000000000009 ffff8101301b70c0
May 15 05:27:42 vault1 kernel: ffffffff8058a6e0 00000000000038ca ffff8101301b72c8 0000000000000002
May 15 05:27:42 vault1 kernel: Call Trace:
May 15 05:27:42 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:42 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:42 vault1 kernel: [<ffffffff80460f4b>] unix_wait_for_peer+0x86/0xb0
May 15 05:27:42 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:42 vault1 kernel: [<ffffffff804614fb>] unix_dgram_sendmsg+0x3d3/0x4cd
May 15 05:27:42 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 15 05:27:42 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 15 05:27:42 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 15 05:27:42 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:42 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:42 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:42 vault1 kernel: [<ffffffff80216cda>] vfs_write+0xe1/0x177
May 15 05:27:42 vault1 kernel: [<ffffffff802176bf>] sys_write+0x45/0x6e
May 15 05:27:42 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:42 vault1 kernel:
May 15 05:27:42 vault1 kernel: irqbalance S ffff81012fe020d8 0 3393 1 (NOTLB)
May 15 05:27:42 vault1 kernel: ffff81012bd75eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 15 05:27:42 vault1 kernel: ffff8100062e79f8 ffff8100062e79f0 0000000000000007 ffff81012eea00c0
May 15 05:27:42 vault1 kernel: ffff810136b92040 00000000001ab791 ffff81012eea02c8 0000000100000292
May 15 05:27:42 vault1 kernel: Call Trace:
May 15 05:27:42 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:42 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 15 05:27:42 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 15 05:27:42 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 15 05:27:42 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 15 05:27:42 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:42 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 15 05:27:42 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:42 vault1 kernel:
May 15 05:27:42 vault1 kernel: portmap S ffff810130b70058 0 3428 1 (NOTLB)
May 15 05:27:42 vault1 kernel: ffff81012c6bdb28 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:27:43 vault1 kernel: 0000000180248a3b 0000000000000002 000000000000000a ffff8101309f4080
May 15 05:27:43 vault1 kernel: ffff810136b92040 000000000001dcc0 ffff8101309f4288 0000000100000000
May 15 05:27:43 vault1 kernel: Call Trace:
May 15 05:27:43 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:43 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:43 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:43 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 15 05:27:43 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:27:43 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:43 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:43 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:43 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:43 vault1 kernel: [<ffffffff8029ff51>] debug_check_no_locks_freed+0x11d/0x129
May 15 05:27:43 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:43 vault1 kernel: [<ffffffff8020b848>] free_hot_cold_page+0x184/0x190
May 15 05:27:43 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:43 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:43 vault1 kernel: [<ffffffff8040b777>] sys_sendmsg+0x217/0x28a
May 15 05:27:43 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:43 vault1 kernel: [<ffffffff8023089b>] __up_write+0x27/0xf2
May 15 05:27:43 vault1 kernel: [<ffffffff8023089b>] __up_write+0x27/0xf2
May 15 05:27:43 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:43 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:27:43 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:43 vault1 kernel:
May 15 05:27:43 vault1 kernel: rpc.statd S ffff810130b70918 0 3453 1 (NOTLB)
May 15 05:27:43 vault1 kernel: ffff81012b387a18 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:27:43 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000006 ffff81013017c040
May 15 05:27:43 vault1 kernel: ffffffff8058a6e0 000000000000295e ffff81013017c248 0000000000000000
May 15 05:27:43 vault1 kernel: Call Trace:
May 15 05:27:43 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:43 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:43 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:43 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 15 05:27:43 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:27:43 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:43 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:43 vault1 last message repeated 2 times
May 15 05:27:43 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 15 05:27:43 vault1 kernel: [<ffffffff8046155d>] unix_dgram_sendmsg+0x435/0x4cd
May 15 05:27:43 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 15 05:27:43 vault1 kernel: [<ffffffff802530fc>] sock_sendmsg+0xea/0x107
May 15 05:27:43 vault1 kernel: [<ffffffff8020ed97>] link_path_walk+0xce/0xe0
May 15 05:27:43 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:43 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:27:43 vault1 kernel: [<ffffffff8022d806>] mntput_no_expire+0x1c/0x96
May 15 05:27:43 vault1 kernel: [<ffffffff8040ba92>] sys_sendto+0x128/0x151
May 15 05:27:43 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 15 05:27:43 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:43 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:43 vault1 kernel:
May 15 05:27:43 vault1 kernel: dbus-daemon S ffff81012eb760d8 0 3481 1 (NOTLB)
May 15 05:27:43 vault1 kernel: ffff81012b4d7b28 0000000000000046 0000000000000000 ffff81012f7dc0c0
May 15 05:27:43 vault1 kernel: 0000000000000000 ffffffff802a0de5 000000000000000a ffff81012f7dc0c0
May 15 05:27:43 vault1 kernel: ffffffff8058a6e0 0000000000000a36 ffff81012f7dc2c8 0000000000000002
May 15 05:27:43 vault1 kernel: Call Trace:
May 15 05:27:43 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:43 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:43 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:27:43 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:43 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:43 vault1 last message repeated 2 times
May 15 05:27:43 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 15 05:27:43 vault1 kernel: [<ffffffff8024660c>] sock_aio_write+0x0/0x128
May 15 05:27:43 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:27:43 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:43 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:43 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:43 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:43 vault1 kernel: [<ffffffff802d3dfa>] do_readv_writev+0x199/0x1ae
May 15 05:27:43 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 15 05:27:43 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:43 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:27:43 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:43 vault1 kernel:
May 15 05:27:43 vault1 kernel: hidd S ffff81012e41c9d8 0 3518 1 (NOTLB)
May 15 05:27:43 vault1 kernel: ffff81012b439ad8 0000000000000046 0000000000000000 ffff81012eea2040
May 15 05:27:43 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000001 ffff81012eea2040
May 15 05:27:43 vault1 kernel: ffffffff8058a6e0 00000000000085b1 ffff81012eea2248 0000000000000002
May 15 05:27:43 vault1 kernel: Call Trace:
May 15 05:27:43 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:44 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:27:44 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:44 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 15 05:27:44 vault1 kernel: [<ffffffff8046155d>] unix_dgram_sendmsg+0x435/0x4cd
May 15 05:27:44 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 15 05:27:44 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 15 05:27:44 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 15 05:27:44 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 15 05:27:44 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:27:44 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:44 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff8021e12c>] sigprocmask+0x28/0xc1
May 15 05:27:44 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:44 vault1 kernel: [<ffffffff802d861f>] sys_ppoll+0xf1/0x1ff
May 15 05:27:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:44 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:44 vault1 kernel:
May 15 05:27:44 vault1 kernel: acpid S ffff81012f8569d8 0 3535 1 (NOTLB)
May 15 05:27:44 vault1 kernel: ffff81012b4e5b28 0000000000000046 0000000000000612 ffff81012e53b080
May 15 05:27:44 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000007 ffff81012e53b080
May 15 05:27:44 vault1 kernel: ffff8102300e9040 000000000001ef10 ffff81012e53b288 0000000000000002
May 15 05:27:44 vault1 kernel: Call Trace:
May 15 05:27:44 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:44 vault1 kernel: [<ffffffff88021755>] :jbd:journal_stop+0x20a/0x21a
May 15 05:27:44 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:44 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:27:44 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:44 vault1 kernel: [<ffffffff8020e533>] current_fs_time+0x3b/0x40
May 15 05:27:44 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff80216881>] __generic_file_aio_write_nolock+0x378/0x3eb
May 15 05:27:44 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 15 05:27:44 vault1 kernel: [<ffffffff80261ece>] __mutex_unlock_slowpath+0x128/0x133
May 15 05:27:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:44 vault1 kernel: [<ffffffff802224af>] generic_file_aio_write+0x6c/0xc1
May 15 05:27:44 vault1 kernel: [<ffffffff88037192>] :ext3:ext3_file_write+0x16/0x94
May 15 05:27:44 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 15 05:27:44 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:44 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:44 vault1 kernel: [<ffffffff8040bfa2>] sys_accept+0x19e/0x1fb
May 15 05:27:44 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:44 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:27:44 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:44 vault1 kernel:
May 15 05:27:44 vault1 kernel: sshd S ffff81012ec1da98 0 3553 1 (NOTLB)
May 15 05:27:44 vault1 kernel: ffff81012739da18 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:27:44 vault1 kernel: 000000018023bee1 0000000000000002 0000000000000007 ffff81012ff09040
May 15 05:27:44 vault1 kernel: ffff810136b92040 00000000000044e3 ffff81012ff09248 0000000100000000
May 15 05:27:44 vault1 kernel: Call Trace:
May 15 05:27:44 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:44 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:44 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 15 05:27:44 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:27:44 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:44 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff802133c4>] sock_def_readable+0x18/0x6c
May 15 05:27:44 vault1 kernel: [<ffffffff802133c4>] sock_def_readable+0x18/0x6c
May 15 05:27:44 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 15 05:27:44 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:44 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 15 05:27:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:44 vault1 kernel: [<ffffffff802c070d>] zone_statistics+0x3f/0x60
May 15 05:27:44 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff802118a0>] do_wp_page+0x497/0x60b
May 15 05:27:44 vault1 kernel: [<ffffffff80211a03>] do_wp_page+0x5fa/0x60b
May 15 05:27:44 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:27:44 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:44 vault1 kernel: [<ffffffff8020c6a9>] _atomic_dec_and_lock+0x39/0x58
May 15 05:27:44 vault1 kernel: [<ffffffff80212fc9>] poison_obj+0x26/0x2f
May 15 05:27:44 vault1 kernel: [<ffffffff802078c3>] kmem_cache_free+0x257/0x267
May 15 05:27:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:44 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 15 05:27:44 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:44 vault1 kernel: [<ffffffff80239ba9>] generic_drop_inode+0x0/0x15c
May 15 05:27:45 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:45 vault1 kernel:
May 15 05:27:45 vault1 kernel: rpc.idmapd S ffff810130b704b8 0 3679 1 (NOTLB)
May 15 05:27:45 vault1 kernel: ffff8101267a1e78 0000000000000046 0000000000000000 ffff81012f5ba0c0
May 15 05:27:45 vault1 kernel: ffff81012f5ba0c0 ffffffff802638fc 000000000000000a ffff81012f5ba0c0
May 15 05:27:45 vault1 kernel: ffff810136b92040 0000000000004042 ffff81012f5ba2c8 0000000130c84000
May 15 05:27:45 vault1 kernel: Call Trace:
May 15 05:27:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:45 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 15 05:27:45 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 15 05:27:45 vault1 kernel: [<ffffffff802ead19>] sys_epoll_wait+0x193/0x426
May 15 05:27:45 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:45 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:45 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:45 vault1 kernel:
May 15 05:27:45 vault1 kernel: crond S ffff81012f0bbb58 0 3694 1 (NOTLB)
May 15 05:27:45 vault1 kernel: ffff810125d63eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 15 05:27:45 vault1 kernel: ffff8100062e79f8 ffff8100062e79f0 0000000000000009 ffff81012e84d080
May 15 05:27:45 vault1 kernel: ffff810136b92040 0000000000006656 ffff81012e84d288 0000000100000292
May 15 05:27:45 vault1 kernel: Call Trace:
May 15 05:27:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:45 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 15 05:27:45 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 15 05:27:45 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 15 05:27:45 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 15 05:27:45 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:45 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 15 05:27:45 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:45 vault1 kernel:
May 15 05:27:45 vault1 kernel: atd S ffff81012f451678 0 3709 1 (NOTLB)
May 15 05:27:45 vault1 kernel: ffff810126653eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 15 05:27:45 vault1 kernel: ffff8100062e79f8 ffff8100062e79f0 0000000000000007 ffff81012f0050c0
May 15 05:27:45 vault1 kernel: ffff810136b92040 00000000000166a6 ffff81012f0052c8 0000000100000292
May 15 05:27:45 vault1 kernel: Call Trace:
May 15 05:27:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:45 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 15 05:27:45 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 15 05:27:45 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 15 05:27:45 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 15 05:27:45 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:45 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 15 05:27:45 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:45 vault1 kernel:
May 15 05:27:45 vault1 kernel: hald S ffff81012f451218 0 3724 1 (NOTLB)
May 15 05:27:45 vault1 kernel: ffff810125dfbb28 0000000000000046 0000000000000000 ffff81012f4f3040
May 15 05:27:45 vault1 kernel: ffff81012f4f3040 ffffffff802638fc 000000000000000a ffff81012f4f3040
May 15 05:27:45 vault1 kernel: ffff810136b92040 0000000000053fac ffff81012f4f3248 0000000130c84000
May 15 05:27:45 vault1 kernel: Call Trace:
May 15 05:27:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:45 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 15 05:27:45 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 15 05:27:45 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:27:45 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:45 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:45 vault1 last message repeated 9 times
May 15 05:27:45 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:45 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:45 vault1 kernel: [<ffffffff8028dc04>] _local_bh_enable+0xbb/0xd0
May 15 05:27:45 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:45 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:27:45 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:45 vault1 kernel:
May 15 05:27:45 vault1 kernel: hald-runner S ffff8102303edb18 0 3725 3724 (NOTLB)
May 15 05:27:45 vault1 kernel: ffff810230d6bb28 0000000000000046 0000000000000000 ffff8102300e9040
May 15 05:27:45 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000001 ffff8102300e9040
May 15 05:27:45 vault1 kernel: ffffffff8058a6e0 000000000000076b ffff8102300e9248 0000000000000002
May 15 05:27:45 vault1 kernel: Call Trace:
May 15 05:27:45 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:45 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:45 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:27:45 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:45 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:45 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 15 05:27:45 vault1 kernel: [<ffffffff8026212e>] __mutex_lock_slowpath+0x250/0x25b
May 15 05:27:45 vault1 kernel: [<ffffffff8024660c>] sock_aio_write+0x0/0x128
May 15 05:27:45 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:27:45 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:45 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:45 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:45 vault1 kernel: [<ffffffff802d3dfa>] do_readv_writev+0x199/0x1ae
May 15 05:27:45 vault1 kernel: [<ffffffff80229266>] do_wait+0x9a7/0xa75
May 15 05:27:45 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:45 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:27:45 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:45 vault1 kernel:
May 15 05:27:45 vault1 kernel: hald-addon-ke S ffff810130840058 0 3790 3725 (NOTLB)
May 15 05:27:45 vault1 kernel: ffff810125a6fea8 0000000000000046 0000000000000000 0000000000000000
May 15 05:27:45 vault1 kernel: 0000000000000000 0000000000000046 0000000000000007 ffff81012f209080
May 15 05:27:45 vault1 kernel: ffffffff8058a6e0 000000000008f29f ffff81012f209288 000000002f209080
May 15 05:27:45 vault1 kernel: Call Trace:
May 15 05:27:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:45 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 15 05:27:46 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:46 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:27:46 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:27:46 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:46 vault1 kernel:
May 15 05:27:46 vault1 kernel: hald-addon-ac S ffff810230e44098 0 3797 3725 (NOTLB)
May 15 05:27:46 vault1 kernel: ffff810125b4bbe8 0000000000000046 0000000000000612 ffff8101308d4040
May 15 05:27:46 vault1 kernel: 0000000000000001 ffffffff802a0de5 0000000000000008 ffff8101308d4040
May 15 05:27:46 vault1 kernel: ffff81012e53b080 000000000005f23b ffff8101308d4248 0000000000000002
May 15 05:27:46 vault1 kernel: Call Trace:
May 15 05:27:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:46 vault1 kernel: [<ffffffff8023ad89>] prepare_to_wait+0x1b/0x5c
May 15 05:27:46 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:46 vault1 kernel: [<ffffffff80249a11>] unix_stream_recvmsg+0x233/0x548
May 15 05:27:46 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:46 vault1 kernel: [<ffffffff8040ae7a>] sock_aio_read+0x10c/0x124
May 15 05:27:46 vault1 kernel: [<ffffffff8020cf53>] do_sync_read+0xc9/0x10c
May 15 05:27:46 vault1 kernel: [<ffffffff8046210a>] unix_stream_connect+0x389/0x3ff
May 15 05:27:46 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:46 vault1 kernel: [<ffffffff8020e38c>] do_mmap_pgoff+0x65d/0x7c9
May 15 05:27:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:46 vault1 kernel: [<ffffffff8020b5b0>] vfs_read+0xde/0x173
May 15 05:27:46 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:27:46 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:46 vault1 kernel:
May 15 05:27:46 vault1 kernel: hald-addon-cp S ffff81012ec1cd78 0 3798 3725 (NOTLB)
May 15 05:27:46 vault1 kernel: ffff810125ba9b28 0000000000000046 0000000000000000 ffff81012f3dc040
May 15 05:27:46 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000003 ffff81012f3dc040
May 15 05:27:46 vault1 kernel: ffffffff8058a6e0 0000000000034e85 ffff81012f3dc248 0000000000000002
May 15 05:27:46 vault1 kernel: Call Trace:
May 15 05:27:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:46 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:46 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:27:46 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:46 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:46 vault1 kernel: [<ffffffff80249bf3>] unix_stream_recvmsg+0x415/0x548
May 15 05:27:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:46 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 15 05:27:46 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 15 05:27:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:46 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 15 05:27:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:46 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:27:46 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:27:46 vault1 kernel: [<ffffffff80208d64>] __handle_mm_fault+0x514/0xccf
May 15 05:27:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:46 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:46 vault1 kernel: [<ffffffff80265bc5>] do_page_fault+0x490/0x7cf
May 15 05:27:46 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:46 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:27:46 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:46 vault1 kernel:
May 15 05:27:46 vault1 kernel: hald-addon-ke S ffff81012f856118 0 3799 3725 (NOTLB)
May 15 05:27:46 vault1 kernel: ffff810125be9ea8 0000000000000046 0000000000000000 0000000000000000
May 15 05:27:46 vault1 kernel: 0000000000000000 0000000000000046 0000000000000001 ffff81012f7ef0c0
May 15 05:27:46 vault1 kernel: ffff810136b92040 0000000000069995 ffff81012f7ef2c8 000000012f7ef0c0
May 15 05:27:46 vault1 kernel: Call Trace:
May 15 05:27:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:46 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 15 05:27:46 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:46 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:27:46 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:27:46 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:46 vault1 kernel:
May 15 05:27:46 vault1 kernel: hald-addon-ke S ffff810230cee0d8 0 3802 3725 (NOTLB)
May 15 05:27:46 vault1 kernel: ffff8101259a3ea8 0000000000000046 0000000000000000 0000000000000000
May 15 05:27:46 vault1 kernel: 0000000000000000 0000000000000046 0000000000000001 ffff810130097040
May 15 05:27:46 vault1 kernel: ffff810230c4c0c0 00000000000a3b0d ffff810130097248 0000000330097040
May 15 05:27:47 vault1 kernel: Call Trace:
May 15 05:27:47 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:47 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 15 05:27:47 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:47 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:47 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:27:47 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:27:47 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:47 vault1 kernel:
May 15 05:27:47 vault1 kernel: hald-addon-st S ffff810230e444f8 0 3812 3725 (NOTLB)
May 15 05:27:47 vault1 kernel: ffff8101254bdeb8 0000000000000046 0000000000000000 ffffffff802638fc
May 15 05:27:47 vault1 kernel: ffff8100062e79f8 ffff8100062e79f0 0000000000000009 ffff81012ea19040
May 15 05:27:47 vault1 kernel: ffff810136b92040 0000000000001a9b ffff81012ea19248 0000000100000292
May 15 05:27:47 vault1 kernel: Call Trace:
May 15 05:27:47 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:47 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 15 05:27:47 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 15 05:27:47 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 15 05:27:47 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 15 05:27:47 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:47 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 15 05:27:47 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:47 vault1 kernel:
May 15 05:27:47 vault1 kernel: smartd S ffff81012ec1c4b8 0 3876 1 (NOTLB)
May 15 05:27:47 vault1 kernel: ffff81012662beb8 0000000000000046 0000000000000000 ffffffff802638fc
May 15 05:27:47 vault1 kernel: ffff8100062dcbf8 ffff8100062dcbf0 0000000000000007 ffff81012fdbb040
May 15 05:27:47 vault1 kernel: ffffffff8058a6e0 0000000000002dd0 ffff81012fdbb248 0000000000000292
May 15 05:27:47 vault1 kernel: Call Trace:
May 15 05:27:47 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:47 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 15 05:27:47 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 15 05:27:47 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 15 05:27:47 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 15 05:27:47 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:47 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 15 05:27:47 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:47 vault1 kernel:
May 15 05:27:47 vault1 kernel: mingetty S ffff81012f0ba578 0 3879 1 (NOTLB)
May 15 05:27:47 vault1 kernel: ffff810126425dc8 0000000000000046 0000000000000000 ffff81012f9307f0
May 15 05:27:47 vault1 kernel: 0000000000000612 ffff81012f930080 0000000000000007 ffff81012f930080
May 15 05:27:47 vault1 kernel: ffff81012f342040 0000000000025b03 ffff81012f930288 0000000100000046
May 15 05:27:47 vault1 kernel: Call Trace:
May 15 05:27:47 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:47 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:47 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:27:47 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:47 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:47 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:27:47 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:27:47 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:27:47 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:47 vault1 kernel:
May 15 05:27:47 vault1 kernel: mingetty S ffff81012ec1d638 0 3880 1 (NOTLB)
May 15 05:27:47 vault1 kernel: ffff8101253a1dc8 0000000000000046 0000000000000002 0000000000000000
May 15 05:27:47 vault1 kernel: ffff81012f38f7f8 ffff81012f38f0c0 0000000000000006 ffff81012f38f0c0
May 15 05:27:47 vault1 kernel: ffff81012f930080 00000000001e63ba ffff81012f38f2c8 0000000100000046
May 15 05:27:47 vault1 kernel: Call Trace:
May 15 05:27:47 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:47 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:27:47 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:47 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:47 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:27:47 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:27:47 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:27:47 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:47 vault1 kernel:
May 15 05:27:47 vault1 kernel: mingetty S ffff81012e41d298 0 3881 1 (NOTLB)
May 15 05:27:47 vault1 kernel: ffff810125223dc8 0000000000000046 0000000000000000 ffff81012f3427b0
May 15 05:27:47 vault1 kernel: 0000000000000612 ffff81012f342040 0000000000000005 ffff81012f342040
May 15 05:27:47 vault1 kernel: ffff81012f9940c0 000000000016c887 ffff81012f342248 0000000100000046
May 15 05:27:47 vault1 kernel: Call Trace:
May 15 05:27:47 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:47 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:47 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:27:47 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:47 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:47 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:27:47 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:27:47 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:27:47 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:47 vault1 kernel:
May 15 05:27:48 vault1 kernel: mingetty S ffff81012effedb8 0 3882 1 (NOTLB)
May 15 05:27:48 vault1 kernel: ffff810125453dc8 0000000000000046 0000000000000000 ffff81012f994830
May 15 05:27:48 vault1 kernel: 0000000000000612 ffff81012f9940c0 0000000000000004 ffff81012f9940c0
May 15 05:27:48 vault1 kernel: ffff8101308d20c0 00000000000032cb ffff81012f9942c8 0000000100000046
May 15 05:27:48 vault1 kernel: Call Trace:
May 15 05:27:48 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:48 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:48 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:48 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:27:48 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:48 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:48 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:27:48 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:27:48 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:27:48 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:48 vault1 kernel:
May 15 05:27:48 vault1 kernel: mingetty S ffff81012f5bd258 0 3883 1 (NOTLB)
May 15 05:27:48 vault1 kernel: ffff81012d1bfdc8 0000000000000046 0000000000000000 ffff8101308d2830
May 15 05:27:48 vault1 kernel: 0000000000000612 ffff8101308d20c0 0000000000000003 ffff8101308d20c0
May 15 05:27:48 vault1 kernel: ffff81013006d0c0 00000000000c79ab ffff8101308d22c8 0000000100000046
May 15 05:27:48 vault1 kernel: Call Trace:
May 15 05:27:48 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:48 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:48 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:48 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:27:48 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:48 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:48 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:27:48 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:27:48 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:27:48 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:48 vault1 kernel:
May 15 05:27:48 vault1 kernel: mingetty S ffff81012f5bd6b8 0 3884 1 (NOTLB)
May 15 05:27:48 vault1 kernel: ffff810125513dc8 0000000000000046 0000000000000000 ffff81013006d830
May 15 05:27:48 vault1 kernel: 0000000000000612 ffff81013006d0c0 0000000000000002 ffff81013006d0c0
May 15 05:27:48 vault1 kernel: ffff810136b92040 000000000012884d ffff81013006d2c8 0000000100000046
May 15 05:27:48 vault1 kernel: Call Trace:
May 15 05:27:48 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:48 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:48 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:27:48 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:48 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:48 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:27:48 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:27:48 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:27:48 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:48 vault1 kernel:
May 15 05:27:48 vault1 kernel: sshd S ffff810230f84058 0 3943 3553 (NOTLB)
May 15 05:27:48 vault1 kernel: ffff8101256b9a18 0000000000000046 0000000000000000 0000000000000046
May 15 05:27:48 vault1 kernel: ffffffff8022edfd 0000000000000046 000000000000000a ffff810127218080
May 15 05:27:48 vault1 kernel: ffff810125ab70c0 000000000000a175 ffff810127218288 0000000127218080
May 15 05:27:48 vault1 kernel: Call Trace:
May 15 05:27:48 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 15 05:27:48 vault1 kernel: [<ffffffff8023a80a>] tty_ldisc_deref+0x62/0x75
May 15 05:27:48 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:48 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:48 vault1 kernel: [<ffffffff80226c98>] tty_poll+0x5f/0x6d
May 15 05:27:48 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:27:48 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:48 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:48 vault1 last message repeated 3 times
May 15 05:27:48 vault1 kernel: [<ffffffff802313cd>] release_sock+0x2f/0xd6
May 15 05:27:48 vault1 kernel: [<ffffffff8028de66>] local_bh_enable_ip+0xe4/0xea
May 15 05:27:48 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:48 vault1 kernel: [<ffffffff80227651>] tcp_sendmsg+0x9ab/0xab5
May 15 05:27:48 vault1 last message repeated 2 times
May 15 05:27:48 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 15 05:27:48 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:48 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:27:48 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 15 05:27:48 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:48 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:48 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:48 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 15 05:27:48 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:48 vault1 kernel: [<ffffffff80227651>] tcp_sendmsg+0x9ab/0xab5
May 15 05:27:48 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:48 vault1 kernel:
May 15 05:27:48 vault1 kernel: rpc.rquotad S ffff81012f0ba118 0 4049 1 (NOTLB)
May 15 05:27:48 vault1 kernel: ffff81012c6b5b28 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:27:49 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000001 ffff81012f00f080
May 15 05:27:49 vault1 kernel: ffff810136be9080 00000000000360c4 ffff81012f00f288 0000000200000000
May 15 05:27:49 vault1 kernel: Call Trace:
May 15 05:27:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:49 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:49 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:49 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 15 05:27:49 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:27:49 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:49 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:49 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:49 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 15 05:27:49 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:49 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 15 05:27:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:49 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 15 05:27:49 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 15 05:27:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:49 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 15 05:27:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:49 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:27:49 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:27:49 vault1 kernel: [<ffffffff80208d64>] __handle_mm_fault+0x514/0xccf
May 15 05:27:49 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:49 vault1 kernel: [<ffffffff80265bc5>] do_page_fault+0x490/0x7cf
May 15 05:27:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:49 vault1 kernel: [<ffffffff8021e472>] sys_close+0x2b/0xd1
May 15 05:27:49 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:49 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:27:49 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:49 vault1 kernel:
May 15 05:27:49 vault1 kernel: lockd S ffff81022eda4958 0 4052 1 (L-TLB)
May 15 05:27:49 vault1 kernel: ffff81022ff83e10 0000000000000046 0000000000000000 ffff81022f5c07f0
May 15 05:27:49 vault1 kernel: 0000000000000612 ffff81022f5c0080 0000000000000001 ffff81022f5c0080
May 15 05:27:49 vault1 kernel: ffffffff8058a6e0 00000000000041f8 ffff81022f5c0288 0000000000000046
May 15 05:27:49 vault1 kernel: Call Trace:
May 15 05:27:49 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:49 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:49 vault1 kernel: [<ffffffff883de24e>] :sunrpc:svc_recv+0x2b5/0x44f
May 15 05:27:49 vault1 kernel: [<ffffffff883de257>] :sunrpc:svc_recv+0x2be/0x44f
May 15 05:27:49 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:49 vault1 kernel: [<ffffffff8840b556>] :lockd:lockd+0x139/0x263
May 15 05:27:49 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:49 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:49 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:49 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:49 vault1 kernel: [<ffffffff802078c3>] kmem_cache_free+0x257/0x267
May 15 05:27:49 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:49 vault1 kernel: [<ffffffff8840b41d>] :lockd:lockd+0x0/0x263
May 15 05:27:49 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:49 vault1 kernel:
May 15 05:27:49 vault1 kernel: rpciod/0 S ffff81022eda4958 0 4053 19 (L-TLB)
May 15 05:27:49 vault1 kernel: ffff810124befea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:27:49 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130851080
May 15 05:27:49 vault1 kernel: ffff81012f485080 0000000000000b30 ffff810130851288 0000000030851080
May 15 05:27:49 vault1 kernel: Call Trace:
May 15 05:27:49 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:27:49 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:49 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:49 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:49 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:49 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:49 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:49 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:49 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:49 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:49 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:49 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:49 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:49 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:49 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:49 vault1 kernel:
May 15 05:27:49 vault1 kernel: rpciod/1 S ffff810130b71a98 0 4054 19 (L-TLB)
May 15 05:27:49 vault1 kernel: ffff810124bc3ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:49 vault1 kernel: 0000000000000000 0000000000000000 0000000000000008 ffff81012f485080
May 15 05:27:49 vault1 kernel: ffff810136b92040 0000000000001672 ffff81012f485288 000000012f485080
May 15 05:27:49 vault1 kernel: Call Trace:
May 15 05:27:49 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:49 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:49 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:49 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:49 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:49 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:49 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:49 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:49 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:49 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:50 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:50 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:50 vault1 kernel:
May 15 05:27:50 vault1 kernel: rpciod/2 S ffff810130b70058 0 4055 19 (L-TLB)
May 15 05:27:50 vault1 kernel: ffff810124c43ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:50 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012552e040
May 15 05:27:50 vault1 kernel: ffff810136be9080 0000000000001a1e ffff81012552e248 000000022552e040
May 15 05:27:50 vault1 kernel: Call Trace:
May 15 05:27:50 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:50 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:50 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:50 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:50 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:50 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:50 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:50 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:50 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:50 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:50 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:50 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:50 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:50 vault1 kernel:
May 15 05:27:50 vault1 kernel: rpciod/3 S ffff81022eda4958 0 4056 19 (L-TLB)
May 15 05:27:50 vault1 kernel: ffff8101249edea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:50 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff8101252b5040
May 15 05:27:50 vault1 kernel: ffff810230c4c0c0 00000000000019be ffff8101252b5248 00000003252b5040
May 15 05:27:50 vault1 kernel: Call Trace:
May 15 05:27:50 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:50 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:50 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:50 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:50 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:50 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:50 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:50 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:50 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:50 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:50 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:27:50 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:50 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:50 vault1 kernel:
May 15 05:27:50 vault1 kernel: nfsd4 S ffff81012eb76998 0 4057 19 (L-TLB)
May 15 05:27:50 vault1 kernel: ffff810124bffea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:50 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff810130eae0c0
May 15 05:27:50 vault1 kernel: ffff810136be9080 00000000000018fd ffff810130eae2c8 0000000230eae0c0
May 15 05:27:50 vault1 kernel: Call Trace:
May 15 05:27:50 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:50 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:27:50 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:27:50 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:27:50 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:50 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:50 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:27:50 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:50 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:50 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:50 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:27:50 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:50 vault1 kernel:
May 15 05:27:50 vault1 kernel: nfsd D ffff810230f85a98 0 4058 1 (L-TLB)
May 15 05:27:50 vault1 kernel: ffff81022ff77420 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:50 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81022fa12040
May 15 05:27:50 vault1 kernel: ffffffff8058a6e0 00000000006928e3 ffff81022fa12248 000000002fa12858
May 15 05:27:50 vault1 kernel: Call Trace:
May 15 05:27:50 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:50 vault1 kernel: [<ffffffff802616cf>] io_schedule+0x3f/0x67
May 15 05:27:50 vault1 kernel: [<ffffffff8833680a>] :jfs:lbmAllocate+0x8a/0x114
May 15 05:27:50 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:50 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:50 vault1 kernel: [<ffffffff8833723a>] :jfs:lmNextPage+0xea/0x127
May 15 05:27:50 vault1 kernel: [<ffffffff88337508>] :jfs:lmWriteRecord+0x291/0x2d1
May 15 05:27:50 vault1 kernel: [<ffffffff88338340>] :jfs:lmLog+0x125/0x16a
May 15 05:27:50 vault1 kernel: [<ffffffff8833a17b>] :jfs:txCommit+0xcdd/0xef2
May 15 05:27:50 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 15 05:27:50 vault1 kernel: [<ffffffff88322175>] :jfs:jfs_commit_inode+0xab/0xf5
May 15 05:27:50 vault1 kernel: [<ffffffff88335683>] :jfs:extAlloc+0x3ce/0x423
May 15 05:27:50 vault1 kernel: [<ffffffff8832265c>] :jfs:jfs_get_block+0x167/0x237
May 15 05:27:50 vault1 kernel: [<ffffffff802e5c6c>] nobh_prepare_write+0xd3/0x458
May 15 05:27:50 vault1 kernel: [<ffffffff883224f5>] :jfs:jfs_get_block+0x0/0x237
May 15 05:27:50 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:50 vault1 kernel: [<ffffffff8020c722>] add_to_page_cache+0x2d/0x8c
May 15 05:27:50 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:27:50 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:51 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:27:51 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:51 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:27:51 vault1 kernel: [<ffffffff80210354>] generic_file_buffered_write+0x27b/0x77b
May 15 05:27:51 vault1 kernel: [<ffffffff8020e533>] current_fs_time+0x3b/0x40
May 15 05:27:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:51 vault1 kernel: [<ffffffff80216881>] __generic_file_aio_write_nolock+0x378/0x3eb
May 15 05:27:51 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 15 05:27:51 vault1 kernel: [<ffffffff8026212e>] __mutex_lock_slowpath+0x250/0x25b
May 15 05:27:51 vault1 kernel: [<ffffffff802224a4>] generic_file_aio_write+0x61/0xc1
May 15 05:27:51 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:27:51 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:27:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:51 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:51 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:27:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:51 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:27:51 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:27:51 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:27:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:51 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:27:51 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:27:51 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:27:51 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:27:51 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:27:51 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:27:51 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:27:51 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:27:51 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:27:51 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:51 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:51 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:51 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:51 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:27:51 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:51 vault1 kernel:
May 15 05:27:51 vault1 kernel: nfsd D ffff810230e444f8 0 4059 1 (L-TLB)
May 15 05:27:51 vault1 kernel: ffff81022f7b5520 0000000000000046 0000000000000000 0000000000000002
May 15 05:27:51 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff81022fd6e0c0
May 15 05:27:51 vault1 kernel: ffff810136b92040 000000000001166f ffff81022fd6e2c8 0000000100000046
May 15 05:27:51 vault1 kernel: Call Trace:
May 15 05:27:51 vault1 kernel: [<ffffffff802616cf>] io_schedule+0x3f/0x67
May 15 05:27:51 vault1 kernel: [<ffffffff88338540>] :jfs:lmGroupCommit+0x125/0x177
May 15 05:27:51 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:51 vault1 kernel: [<ffffffff8833a1eb>] :jfs:txCommit+0xd4d/0xef2
May 15 05:27:51 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 15 05:27:51 vault1 kernel: [<ffffffff88322175>] :jfs:jfs_commit_inode+0xab/0xf5
May 15 05:27:51 vault1 kernel: [<ffffffff88335683>] :jfs:extAlloc+0x3ce/0x423
May 15 05:27:51 vault1 kernel: [<ffffffff8832265c>] :jfs:jfs_get_block+0x167/0x237
May 15 05:27:51 vault1 kernel: [<ffffffff802e5c6c>] nobh_prepare_write+0xd3/0x458
May 15 05:27:51 vault1 kernel: [<ffffffff883224f5>] :jfs:jfs_get_block+0x0/0x237
May 15 05:27:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:51 vault1 kernel: [<ffffffff8020c722>] add_to_page_cache+0x2d/0x8c
May 15 05:27:51 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:27:51 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:51 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:27:51 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:51 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:27:51 vault1 kernel: [<ffffffff80210354>] generic_file_buffered_write+0x27b/0x77b
May 15 05:27:51 vault1 kernel: [<ffffffff8020e533>] current_fs_time+0x3b/0x40
May 15 05:27:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:51 vault1 kernel: [<ffffffff80216881>] __generic_file_aio_write_nolock+0x378/0x3eb
May 15 05:27:51 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 15 05:27:51 vault1 kernel: [<ffffffff8026212e>] __mutex_lock_slowpath+0x250/0x25b
May 15 05:27:51 vault1 kernel: [<ffffffff802224a4>] generic_file_aio_write+0x61/0xc1
May 15 05:27:51 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:27:51 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:27:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:51 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:51 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:27:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:51 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:27:51 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:27:51 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:27:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:51 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:27:51 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:27:51 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:27:51 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:27:51 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:27:51 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:27:52 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:27:52 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:27:52 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:27:52 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:52 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:52 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:52 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:52 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:27:52 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:52 vault1 kernel:
May 15 05:27:52 vault1 kernel: nfsd D ffff810230e444f8 0 4060 1 (L-TLB)
May 15 05:27:52 vault1 kernel: ffff8102302aba60 0000000000000046 0000000000000000 ffff81022fd3b040
May 15 05:27:52 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81022fd3b040
May 15 05:27:52 vault1 kernel: ffff810136b92040 000000000025e9d4 ffff81022fd3b248 000000012fd3b7b0
May 15 05:27:52 vault1 kernel: Call Trace:
May 15 05:27:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:52 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:52 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:27:52 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:27:52 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:52 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:27:52 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:27:52 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:52 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:52 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:27:52 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:52 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:27:52 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:27:52 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:27:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:52 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:52 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:52 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:27:52 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:27:52 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:27:52 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:27:52 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:27:52 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:27:52 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:27:52 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:27:52 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:27:52 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:52 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:52 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:52 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:52 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:27:52 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:52 vault1 kernel:
May 15 05:27:52 vault1 kernel: nfsd D ffff810230f84058 0 4061 1 (L-TLB)
May 15 05:27:52 vault1 kernel: ffff81022f37fa60 0000000000000046 0000000000000000 ffff81022f726080
May 15 05:27:52 vault1 kernel: 0000000000000001 ffffffff802a0de5 0000000000000009 ffff81022f726080
May 15 05:27:52 vault1 kernel: ffff810136be9080 00000000000302d3 ffff81022f726288 000000022f7267f0
May 15 05:27:52 vault1 kernel: Call Trace:
May 15 05:27:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:52 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:52 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:27:52 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:27:52 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:52 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:27:52 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:27:52 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:52 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:52 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:27:52 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:52 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:27:52 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:27:52 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:27:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:52 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:52 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:52 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:27:52 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:27:52 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:27:52 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:27:52 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:27:52 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:27:52 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:27:52 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:27:52 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:52 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:52 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:52 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:52 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:27:52 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:52 vault1 kernel:
May 15 05:27:52 vault1 kernel: nfsd D ffff810230f85a98 0 4062 1 (L-TLB)
May 15 05:27:52 vault1 kernel: ffff81022f42ba60 0000000000000046 000000000000052b ffff81022f5760c0
May 15 05:27:52 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81022f5760c0
May 15 05:27:52 vault1 kernel: ffff81022f1930c0 0000000000022d38 ffff81022f5762c8 000000002f576830
May 15 05:27:53 vault1 kernel: Call Trace:
May 15 05:27:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:53 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 15 05:27:53 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:53 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 15 05:27:53 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:27:53 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:27:53 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:53 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:27:53 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:27:53 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:53 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:53 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:27:53 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:53 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:27:53 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:27:53 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:27:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:53 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:53 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:53 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:27:53 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:27:53 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:27:53 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:27:53 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:27:53 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:27:53 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:27:53 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:27:53 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:27:53 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:53 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:53 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:53 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:27:53 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:53 vault1 kernel:
May 15 05:27:53 vault1 kernel: nfsd D ffff810230f85a98 0 4063 1 (L-TLB)
May 15 05:27:53 vault1 kernel: ffff81022fb05a60 0000000000000046 0000000000000000 ffff81022f615040
May 15 05:27:53 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81022f615040
May 15 05:27:53 vault1 kernel: ffffffff8058a6e0 00000000001cf979 ffff81022f615248 000000002f6157b0
May 15 05:27:53 vault1 kernel: Call Trace:
May 15 05:27:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:53 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:53 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:27:53 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:27:53 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:53 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:27:53 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:27:53 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:53 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:53 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:27:53 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:53 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:27:53 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:27:53 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:27:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:53 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:53 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:53 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:27:53 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:27:53 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:27:53 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:27:53 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:27:53 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:27:53 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:27:53 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:27:53 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:27:53 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:53 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:53 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:53 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:27:53 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:53 vault1 kernel:
May 15 05:27:53 vault1 kernel: nfsd D ffff810130b704b8 0 4064 1 (L-TLB)
May 15 05:27:53 vault1 kernel: ffff81022f0c1a60 0000000000000046 0000000000000000 ffff81022fa2c080
May 15 05:27:53 vault1 kernel: 0000000000000001 ffffffff802a0de5 0000000000000009 ffff81022fa2c080
May 15 05:27:53 vault1 kernel: ffff810230c4c0c0 000000000001d4dc ffff81022fa2c288 000000032fa2c7f0
May 15 05:27:53 vault1 kernel: Call Trace:
May 15 05:27:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:54 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:54 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:27:54 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:27:54 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:54 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:27:54 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:27:54 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:54 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:54 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:27:54 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:54 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:27:54 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:27:54 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:27:54 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:54 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:54 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:54 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:27:54 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:27:54 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:27:54 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:27:54 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:27:54 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:27:54 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:27:54 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:27:54 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:54 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:54 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:27:54 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:54 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:27:54 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:54 vault1 kernel:
May 15 05:27:54 vault1 kernel: nfsd D ffff810230f85a98 0 4065 1 (L-TLB)
May 15 05:27:54 vault1 kernel: ffff81022f19da60 0000000000000046 0000000000000000 ffff81022f1930c0
May 15 05:27:54 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81022f1930c0
May 15 05:27:54 vault1 kernel: ffffffff8058a6e0 0000000000018984 ffff81022f1932c8 000000002f193830
May 15 05:27:54 vault1 kernel: Call Trace:
May 15 05:27:54 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:54 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:54 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:27:54 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:27:54 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:54 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:27:54 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:27:54 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:54 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:27:54 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:27:54 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:27:54 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:27:54 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:27:54 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:27:54 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:54 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:54 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:27:54 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:27:54 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:27:54 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:27:54 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:27:54 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:27:54 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:27:54 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:27:54 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:27:54 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:27:54 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:54 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:27:54 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:27:54 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:27:54 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:27:54 vault1 kernel:
May 15 05:27:54 vault1 kernel: rpc.mountd S ffff81012f0ba9d8 0 4068 1 (NOTLB)
May 15 05:27:54 vault1 kernel: ffff810124c41a18 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:27:54 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000009 ffff81012efa30c0
May 15 05:27:54 vault1 kernel: ffff810136b92040 000000000001006a ffff81012efa32c8 0000000100000000
May 15 05:27:54 vault1 kernel: Call Trace:
May 15 05:27:54 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:54 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:27:54 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:27:54 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 15 05:27:54 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:27:54 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:27:54 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:27:55 vault1 last message repeated 4 times
May 15 05:27:55 vault1 kernel: [<ffffffff8020b406>] kfree+0x1a8/0x274
May 15 05:27:55 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 15 05:27:55 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 15 05:27:55 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:55 vault1 kernel: [<ffffffff883e158b>] :sunrpc:cache_clean+0x12/0x208
May 15 05:27:55 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 15 05:27:55 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:27:55 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 15 05:27:55 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:55 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:27:55 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:55 vault1 kernel: [<ffffffff883e2363>] :sunrpc:cache_write+0x45/0xc0
May 15 05:27:55 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:27:55 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:55 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:27:55 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:27:55 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 15 05:27:55 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:27:55 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:27:55 vault1 kernel:
May 15 05:27:55 vault1 kernel:
May 15 05:27:55 vault1 kernel: Showing all locks held in the system:
May 15 05:27:55 vault1 kernel: 2 locks held by pdflush/299:
May 15 05:27:55 vault1 kernel: #0: (&type->s_umount_key#20){----}, at: [<ffffffff8024f2e2>] writeback_inodes+0x73/0xee
May 15 05:27:55 vault1 kernel: #1: (&jfs_ip->rdwrlock#2){----}, at: [<ffffffff8832253a>] jfs_get_block+0x45/0x237 [jfs]
May 15 05:27:55 vault1 kernel: 1 lock held by syslogd/3374:
May 15 05:27:55 vault1 kernel: #0: (&inode->i_mutex){--..}, at: [<ffffffff8024e516>] do_fsync+0x41/0xa4
May 15 05:27:55 vault1 kernel: 1 lock held by mingetty/3879:
May 15 05:27:55 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:27:55 vault1 kernel: 1 lock held by mingetty/3880:
May 15 05:27:55 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:27:55 vault1 kernel: 1 lock held by mingetty/3881:
May 15 05:27:55 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:27:55 vault1 kernel: 1 lock held by mingetty/3882:
May 15 05:27:55 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:27:55 vault1 kernel: 1 lock held by mingetty/3883:
May 15 05:27:55 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:27:55 vault1 kernel: 1 lock held by mingetty/3884:
May 15 05:27:55 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:27:55 vault1 kernel: 2 locks held by bash/3945:
May 15 05:27:55 vault1 kernel: #0: (sysrq_key_table_lock){....}, at: [<ffffffff80395b2a>] __handle_sysrq+0x22/0x128
May 15 05:27:55 vault1 kernel: #1: (tasklist_lock){..-?}, at: [<ffffffff8029ebd4>] debug_show_all_locks+0x3e/0x152
May 15 05:27:55 vault1 kernel: 5 locks held by nfsd/4058:
May 15 05:27:55 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:27:55 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:55 vault1 kernel: #2: (&jfs_ip->rdwrlock#2){----}, at: [<ffffffff8832253a>] jfs_get_block+0x45/0x237 [jfs]
May 15 05:27:55 vault1 kernel: #3: (&jfs_ip->commit_mutex){--..}, at: [<ffffffff88322126>] jfs_commit_inode+0x5c/0xf5 [jfs]
May 15 05:27:55 vault1 kernel: #4: (&(log)->loglock){--..}, at: [<ffffffff8833824a>] lmLog+0x2f/0x16a [jfs]
May 15 05:27:55 vault1 kernel: 4 locks held by nfsd/4059:
May 15 05:27:55 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:27:55 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:55 vault1 kernel: #2: (&jfs_ip->rdwrlock#2){----}, at: [<ffffffff8832253a>] jfs_get_block+0x45/0x237 [jfs]
May 15 05:27:55 vault1 kernel: #3: (&jfs_ip->commit_mutex){--..}, at: [<ffffffff88322126>] jfs_commit_inode+0x5c/0xf5 [jfs]
May 15 05:27:55 vault1 kernel: 2 locks held by nfsd/4060:
May 15 05:27:55 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:27:55 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:55 vault1 kernel: 2 locks held by nfsd/4061:
May 15 05:27:55 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:27:55 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:55 vault1 kernel: 2 locks held by nfsd/4062:
May 15 05:27:55 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:27:55 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:55 vault1 kernel: 2 locks held by nfsd/4063:
May 15 05:27:55 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:27:55 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:55 vault1 kernel: 2 locks held by nfsd/4064:
May 15 05:27:55 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:27:55 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:55 vault1 kernel: 2 locks held by nfsd/4065:
May 15 05:27:55 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:27:55 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:27:55 vault1 kernel:
May 15 05:27:55 vault1 kernel: =============================================
May 15 05:27:55 vault1 kernel:
May 15 05:33:53 vault1 kernel: SysRq : Show State
May 15 05:33:53 vault1 kernel:
May 15 05:33:53 vault1 kernel: free sibling
May 15 05:33:53 vault1 kernel: task PC stack pid father child younger older
May 15 05:33:53 vault1 kernel: init S ffff810230f85a98 0 1 0 (NOTLB)
May 15 05:33:53 vault1 kernel: ffff810136ba3a18 0000000000000046 0000000000000000 ffff810130c70040
May 15 05:33:53 vault1 kernel: ffff810130c70040 ffffffff802638fc 000000000000000a ffff810130c70040
May 15 05:33:53 vault1 kernel: ffffffff8058a6e0 0000000000009ac3 ffff810130c70248 0000000080733000
May 15 05:33:53 vault1 kernel: Call Trace:
May 15 05:33:53 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:53 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 15 05:33:53 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 15 05:33:53 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:33:53 vault1 kernel: ventd_create_kthread+0x0/0x6d
May 15 05:33:53 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:53 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:53 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:53 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:53 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:53 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:53 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:53 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:53 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:53 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:53 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:53 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:53 vault1 kernel:
May 15 05:33:53 vault1 kernel: cqueue/3 S ffffffff8058a280 0 238 19 (L-TLB)
May 15 05:33:53 vault1 kernel: ffff810230e97ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:53 vault1 kernel: 0000000000000000 0000000000000000 0000000000000005 ffff810230f44040
May 15 05:33:53 vault1 kernel: ffff810230c4c0c0 00000000000014a6 ffff810230f44248 0000000330f44040
May 15 05:33:53 vault1 kernel: Call Trace:
May 15 05:33:53 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:53 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:53 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:53 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:53 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:53 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:53 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:53 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:53 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:53 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:53 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:53 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:54 vault1 kernel:
May 15 05:33:54 vault1 kernel: ksuspend_usbd S ffffffff8058a280 0 239 19 (L-TLB)
May 15 05:33:54 vault1 kernel: ffff810230eb9ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:54 vault1 kernel: 0000000000000000 0000000000000000 0000000000000006 ffff810230f450c0
May 15 05:33:54 vault1 kernel: ffff810136be9080 0000000000001012 ffff810230f452c8 0000000230f450c0
May 15 05:33:54 vault1 kernel: Call Trace:
May 15 05:33:54 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:54 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:54 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:54 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:54 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:54 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:54 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:54 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:54 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:54 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:54 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:54 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:54 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:54 vault1 kernel:
May 15 05:33:54 vault1 kernel: khubd S ffff810130b71638 0 242 19 (L-TLB)
May 15 05:33:54 vault1 kernel: ffff8102309c1e20 0000000000000046 0000000000000000 0000000000000000
May 15 05:33:54 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810230f4d080
May 15 05:33:54 vault1 kernel: ffff810136be9080 0000000000011d07 ffff810230f4d288 0000000230f4d080
May 15 05:33:54 vault1 kernel: Call Trace:
May 15 05:33:54 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:54 vault1 kernel: [<ffffffff803d6dc8>] hub_thread+0xac0/0xb3c
May 15 05:33:54 vault1 kernel: [<ffffffff80260f58>] thread_return+0x88/0x12a
May 15 05:33:54 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:33:54 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:54 vault1 kernel: [<ffffffff803d6308>] hub_thread+0x0/0xb3c
May 15 05:33:54 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:54 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:54 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:54 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:54 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:54 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:54 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:54 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:54 vault1 kernel:
May 15 05:33:54 vault1 kernel: kseriod S ffffffff8058a280 0 244 19 (L-TLB)
May 15 05:33:54 vault1 kernel: ffff8102309c7ec0 0000000000000046 0000000000000000 0000000000000000
May 15 05:33:54 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810230f610c0
May 15 05:33:54 vault1 kernel: ffffffff8058a6e0 0000000000005165 ffff810230f612c8 0000000030f610c0
May 15 05:33:54 vault1 kernel: Call Trace:
May 15 05:33:54 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:54 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:54 vault1 kernel: [<ffffffff803eaaea>] serio_thread+0x2a5/0x2f8
May 15 05:33:54 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:33:54 vault1 kernel: [<ffffffff803ea845>] serio_thread+0x0/0x2f8
May 15 05:33:54 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:54 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:54 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:54 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:54 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:54 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:54 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:54 vault1 kernel:
May 15 05:33:54 vault1 kernel: pdflush S ffff810230f84058 0 298 19 (L-TLB)
May 15 05:33:54 vault1 kernel: ffff8102308f1ed0 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:33:54 vault1 kernel: 0000000180598080 0000000000000002 000000000000000a ffff810230f20080
May 15 05:33:54 vault1 kernel: ffff810136be9080 0000000000004416 ffff810230f20288 0000000200000000
May 15 05:33:54 vault1 kernel: Call Trace:
May 15 05:33:54 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:33:54 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:54 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:54 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:54 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 15 05:33:54 vault1 kernel: [<ffffffff80254e21>] pdflush+0xcf/0x1e7
May 15 05:33:54 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:54 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:54 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:54 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:54 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:54 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:54 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:54 vault1 kernel:
May 15 05:33:54 vault1 kernel: pdflush D ffff810230f84058 0 299 19 (L-TLB)
May 15 05:33:54 vault1 kernel: ffff8102308f3a20 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:33:54 vault1 kernel: 000000018020c209 0000000000000002 000000000000000a ffff810230f1f0c0
May 15 05:33:54 vault1 kernel: ffff810136be9080 00000000000024d0 ffff810230f1f2c8 0000000200000001
May 15 05:33:54 vault1 kernel: Call Trace:
May 15 05:33:54 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:33:54 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:33:54 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:54 vault1 kernel: [<ffffffff80262fa6>] __down_write_nested+0x7e/0x96
May 15 05:33:54 vault1 kernel: [<ffffffff8832253a>] :jfs:jfs_get_block+0x45/0x237
May 15 05:33:55 vault1 kernel: [<ffffffff8021c67c>] generic_make_request+0x207/0x21e
May 15 05:33:55 vault1 kernel: [<ffffffff802e9478>] __mpage_writepage+0x1f5/0x55d
May 15 05:33:55 vault1 kernel: [<ffffffff883224f5>] :jfs:jfs_get_block+0x0/0x237
May 15 05:33:55 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 15 05:33:55 vault1 kernel: [<ffffffff802c0835>] dec_zone_page_state+0x33/0x3a
May 15 05:33:55 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:33:55 vault1 kernel: [<ffffffff802c0835>] dec_zone_page_state+0x33/0x3a
May 15 05:33:55 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:55 vault1 kernel: [<ffffffff8021d493>] mpage_writepages+0x228/0x375
May 15 05:33:55 vault1 kernel: [<ffffffff883220bb>] :jfs:jfs_writepage+0x0/0xf
May 15 05:33:55 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:33:55 vault1 kernel: [<ffffffff883224f5>] :jfs:jfs_get_block+0x0/0x237
May 15 05:33:55 vault1 kernel: [<ffffffff802593b8>] do_writepages+0x23/0x30
May 15 05:33:55 vault1 kernel: [<ffffffff80230487>] __writeback_single_inode+0x1c0/0x338
May 15 05:33:55 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:55 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 15 05:33:55 vault1 kernel: [<ffffffff80221c26>] sync_sb_inodes+0x1a6/0x267
May 15 05:33:55 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:55 vault1 kernel: [<ffffffff8024f304>] writeback_inodes+0x95/0xee
May 15 05:33:55 vault1 kernel: [<ffffffff802be735>] wb_kupdate+0x9e/0x112
May 15 05:33:55 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 15 05:33:55 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 15 05:33:55 vault1 kernel: [<ffffffff80254e8e>] pdflush+0x13c/0x1e7
May 15 05:33:55 vault1 kernel: [<ffffffff802be697>] wb_kupdate+0x0/0x112
May 15 05:33:55 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:55 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:55 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:55 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:55 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:55 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:55 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:55 vault1 kernel:
May 15 05:33:55 vault1 kernel: kswapd0 S ffff810230e444f8 0 300 19 (L-TLB)
May 15 05:33:55 vault1 kernel: ffff8102308f5e20 0000000000000046 0000000000000000 0000000000000000
May 15 05:33:55 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810230f1e040
May 15 05:33:55 vault1 kernel: ffffffff8058a6e0 00000000000a6796 ffff810230f1e248 0000000030f1e040
May 15 05:33:55 vault1 kernel: Call Trace:
May 15 05:33:55 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:55 vault1 kernel: [<ffffffff802560c6>] kswapd+0xea/0x469
May 15 05:33:55 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:33:55 vault1 kernel: [<ffffffff80255fdc>] kswapd+0x0/0x469
May 15 05:33:55 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:55 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:55 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:55 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:55 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:55 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:55 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:55 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:55 vault1 kernel:
May 15 05:33:55 vault1 kernel: kswapd1 S ffff81012f857b58 0 301 19 (L-TLB)
May 15 05:33:55 vault1 kernel: ffff8102308f7e20 0000000000000046 0000000000000000 0000000000000000
May 15 05:33:55 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810230f1d080
May 15 05:33:55 vault1 kernel: ffff810136be9080 00000000000a9722 ffff810230f1d288 0000000230f1d080
May 15 05:33:55 vault1 kernel: Call Trace:
May 15 05:33:55 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:55 vault1 kernel: [<ffffffff802560c6>] kswapd+0xea/0x469
May 15 05:33:55 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:33:55 vault1 kernel: [<ffffffff80255fdc>] kswapd+0x0/0x469
May 15 05:33:55 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:55 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:55 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:55 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:55 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:55 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:55 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:55 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:55 vault1 kernel:
May 15 05:33:55 vault1 kernel: aio/0 S ffffffff8058a280 0 302 19 (L-TLB)
May 15 05:33:55 vault1 kernel: ffff8102308f9ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:55 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff810230f1c0c0
May 15 05:33:55 vault1 kernel: ffffffff8058a6e0 00000000000016fc ffff810230f1c2c8 0000000030f1c0c0
May 15 05:33:55 vault1 kernel: Call Trace:
May 15 05:33:55 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:55 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:55 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:55 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:55 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:55 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:55 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:55 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:55 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:55 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:55 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:55 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:55 vault1 kernel:
May 15 05:33:55 vault1 kernel: aio/1 S ffffffff8058a280 0 303 19 (L-TLB)
May 15 05:33:55 vault1 kernel: ffff8102308fbea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:56 vault1 kernel: 0000000000000000 0000000000000000 0000000000000006 ffff810230f1b040
May 15 05:33:56 vault1 kernel: ffff810136b92040 0000000000001bb0 ffff810230f1b248 0000000130f1b040
May 15 05:33:56 vault1 kernel: Call Trace:
May 15 05:33:56 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:56 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:56 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:56 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:56 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:56 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:56 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:56 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:56 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:56 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:56 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:56 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:56 vault1 kernel:
May 15 05:33:56 vault1 kernel: aio/2 S ffffffff8058a280 0 304 19 (L-TLB)
May 15 05:33:56 vault1 kernel: ffff8102308fdea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:33:56 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff810230f19080
May 15 05:33:56 vault1 kernel: ffff810130a13080 0000000000001be7 ffff810230f19288 0000000230f19080
May 15 05:33:56 vault1 kernel: Call Trace:
May 15 05:33:56 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:33:56 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:56 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:56 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:56 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:56 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:56 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:56 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:56 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:56 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:56 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:56 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:56 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:56 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:56 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:56 vault1 kernel:
May 15 05:33:56 vault1 kernel: aio/3 S ffffffff8058a280 0 305 19 (L-TLB)
May 15 05:33:56 vault1 kernel: ffff8102308c1ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:56 vault1 kernel: 0000000000000000 0000000000000000 0000000000000008 ffff810230f32080
May 15 05:33:56 vault1 kernel: ffff810230c4c0c0 0000000000001498 ffff810230f32288 0000000330f32080
May 15 05:33:56 vault1 kernel: Call Trace:
May 15 05:33:56 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:56 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:56 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:56 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:56 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:56 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:56 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:56 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:56 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:56 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:56 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:56 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:56 vault1 kernel:
May 15 05:33:56 vault1 kernel: kpsmoused S ffffffff8058a280 0 461 19 (L-TLB)
May 15 05:33:56 vault1 kernel: ffff810130b5dea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:56 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130f640c0
May 15 05:33:56 vault1 kernel: ffffffff8058a6e0 0000000000000fc7 ffff810130f642c8 0000000030f640c0
May 15 05:33:56 vault1 kernel: Call Trace:
May 15 05:33:56 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:56 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:56 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:56 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:56 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:56 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:56 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:56 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:56 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:56 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:56 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:56 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:56 vault1 kernel:
May 15 05:33:56 vault1 kernel: scsi_eh_0 S ffff810130b71638 0 520 19 (L-TLB)
May 15 05:33:56 vault1 kernel: ffff810130203e70 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:56 vault1 kernel: 0000000000000000 0000000000000046 0000000000000004 ffff8101309050c0
May 15 05:33:56 vault1 kernel: ffff810136be9080 00000000000009fd ffff8101309052c8 0000000280263894
May 15 05:33:56 vault1 kernel: Call Trace:
May 15 05:33:56 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:56 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:56 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:33:56 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:33:56 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:56 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:56 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:56 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:56 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:56 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:57 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:57 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:57 vault1 kernel:
May 15 05:33:57 vault1 kernel: scsi_eh_1 S ffff810130b71638 0 536 19 (L-TLB)
May 15 05:33:57 vault1 kernel: ffff81013029de70 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:57 vault1 kernel: 0000000000000000 0000000000000046 0000000000000006 ffff81013007e0c0
May 15 05:33:57 vault1 kernel: ffff810136be9080 00000000000006ce ffff81013007e2c8 0000000280263894
May 15 05:33:57 vault1 kernel: Call Trace:
May 15 05:33:57 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:57 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:57 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:33:57 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:33:57 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:57 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:57 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:57 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:57 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:57 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:57 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:57 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:57 vault1 kernel:
May 15 05:33:57 vault1 kernel: qla2xxx_1_dpc S ffff810230e444f8 0 537 19 (L-TLB)
May 15 05:33:57 vault1 kernel: ffff810130f75ee0 0000000000000046 0000000000000000 ffffffff802913d9
May 15 05:33:57 vault1 kernel: ffff810130e5c7e8 ffff810130917040 000000000000000a ffff810130917040
May 15 05:33:57 vault1 kernel: ffff810136b92040 00000000000008a0 ffff810130917248 000000018029fe09
May 15 05:33:57 vault1 kernel: Call Trace:
May 15 05:33:57 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 15 05:33:57 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:57 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:33:57 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 15 05:33:57 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:57 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:33:57 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:57 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:57 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:57 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:57 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:57 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:57 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:57 vault1 kernel:
May 15 05:33:57 vault1 kernel: scsi_wq_1 S ffff810130b71638 0 538 19 (L-TLB)
May 15 05:33:57 vault1 kernel: ffff810130f73ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:57 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130ff8040
May 15 05:33:57 vault1 kernel: ffff810136be9080 00000000000014b7 ffff810130ff8248 0000000230ff8040
May 15 05:33:57 vault1 kernel: Call Trace:
May 15 05:33:57 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:57 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:57 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:57 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:57 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:57 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:57 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:57 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:57 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:57 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:57 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:57 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:57 vault1 kernel:
May 15 05:33:57 vault1 kernel: fc_wq_1 S ffff810130b71638 0 539 19 (L-TLB)
May 15 05:33:57 vault1 kernel: ffff810130fafea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:57 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130fff080
May 15 05:33:57 vault1 kernel: ffff810136be9080 00000000000012d2 ffff810130fff288 0000000230fff080
May 15 05:33:57 vault1 kernel: Call Trace:
May 15 05:33:57 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:57 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:57 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:57 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:57 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:57 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:57 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:57 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:57 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:57 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:57 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:57 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:57 vault1 kernel:
May 15 05:33:57 vault1 kernel: fc_dl_1 S ffff810130b71a98 0 540 19 (L-TLB)
May 15 05:33:57 vault1 kernel: ffff810130137ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:57 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81013007f0c0
May 15 05:33:57 vault1 kernel: ffffffff8058a6e0 0000000000001268 ffff81013007f2c8 000000003007f0c0
May 15 05:33:58 vault1 kernel: Call Trace:
May 15 05:33:58 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:58 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:58 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:58 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:58 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:58 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:58 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:58 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:58 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:58 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:58 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:58 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:58 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:58 vault1 kernel:
May 15 05:33:58 vault1 kernel: scsi_eh_2 S ffff810130b71638 0 541 19 (L-TLB)
May 15 05:33:58 vault1 kernel: ffff81013091de70 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:58 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff810130c5a040
May 15 05:33:58 vault1 kernel: ffff810136be9080 00000000000007e9 ffff810130c5a248 0000000280263894
May 15 05:33:58 vault1 kernel: Call Trace:
May 15 05:33:58 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:58 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:58 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:33:58 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:33:58 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:58 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:58 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:58 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:58 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:58 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:58 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:58 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:58 vault1 kernel:
May 15 05:33:58 vault1 kernel: qla2xxx_2_dpc S ffff810230e444f8 0 542 19 (L-TLB)
May 15 05:33:58 vault1 kernel: ffff810130fcbee0 0000000000000046 0000000000000000 ffff810130be1080
May 15 05:33:58 vault1 kernel: ffff810130be1080 ffffffff802638fc 000000000000000a ffff810130be1080
May 15 05:33:58 vault1 kernel: ffff810136b92040 0000000000000604 ffff810130be1288 00000001880ba30f
May 15 05:33:58 vault1 kernel: Call Trace:
May 15 05:33:58 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:58 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:33:58 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:58 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:33:58 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 15 05:33:58 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:58 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:33:58 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:58 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:58 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:58 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:58 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:58 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:58 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:58 vault1 kernel:
May 15 05:33:58 vault1 kernel: scsi_wq_2 S ffff810130b71638 0 543 19 (L-TLB)
May 15 05:33:58 vault1 kernel: ffff810130911ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:58 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff8101308170c0
May 15 05:33:58 vault1 kernel: ffff810136be9080 0000000000001242 ffff8101308172c8 00000002308170c0
May 15 05:33:58 vault1 kernel: Call Trace:
May 15 05:33:58 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:58 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:58 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:58 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:58 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:58 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:58 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:58 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:58 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:58 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:58 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:58 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:58 vault1 kernel:
May 15 05:33:58 vault1 kernel: fc_wq_2 S ffff810130b71638 0 544 19 (L-TLB)
May 15 05:33:58 vault1 kernel: ffff810130fc1ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:58 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130852040
May 15 05:33:58 vault1 kernel: ffff810136be9080 0000000000000fdf ffff810130852248 0000000230852040
May 15 05:33:58 vault1 kernel: Call Trace:
May 15 05:33:58 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:58 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:58 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:58 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:58 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:58 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:58 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:58 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:58 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:58 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:58 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:58 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:58 vault1 kernel:
May 15 05:33:58 vault1 kernel: fc_dl_2 S ffff810130b71638 0 545 19 (L-TLB)
May 15 05:33:58 vault1 kernel: ffff81013009dea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:58 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130853080
May 15 05:33:58 vault1 kernel: ffff810136be9080 00000000000011d4 ffff810130853288 0000000230853080
May 15 05:33:59 vault1 kernel: Call Trace:
May 15 05:33:59 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:59 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:59 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:59 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:59 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:59 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:59 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:59 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:59 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:59 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:59 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:59 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:59 vault1 kernel:
May 15 05:33:59 vault1 kernel: scsi_eh_3 S ffff810130b71638 0 546 19 (L-TLB)
May 15 05:33:59 vault1 kernel: ffff8101308e5e70 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:59 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81013081c0c0
May 15 05:33:59 vault1 kernel: ffff810136be9080 000000000000074f ffff81013081c2c8 0000000280263894
May 15 05:33:59 vault1 kernel: Call Trace:
May 15 05:33:59 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:33:59 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:59 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:33:59 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:33:59 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:59 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:59 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:59 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:59 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:59 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:59 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:59 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:59 vault1 kernel:
May 15 05:33:59 vault1 kernel: qla2xxx_3_dpc S ffff81012fe02998 0 547 19 (L-TLB)
May 15 05:33:59 vault1 kernel: ffff8101309f7ee0 0000000000000046 0000000000000000 ffff810130843040
May 15 05:33:59 vault1 kernel: ffff810130843040 ffffffff802638fc 000000000000000a ffff810130843040
May 15 05:33:59 vault1 kernel: ffff810136b92040 0000000000006120 ffff810130843248 00000001880ba30f
May 15 05:33:59 vault1 kernel: Call Trace:
May 15 05:33:59 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:59 vault1 kernel: [<ffffffff880bcb27>] :qla2xxx:qla2x00_fw_ready+0x84/0x1a2
May 15 05:33:59 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:33:59 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:59 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:33:59 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 15 05:33:59 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:59 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:33:59 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:59 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:59 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:59 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:59 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:59 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:59 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:59 vault1 kernel:
May 15 05:33:59 vault1 kernel: scsi_wq_3 S ffff810230f851d8 0 548 19 (L-TLB)
May 15 05:33:59 vault1 kernel: ffff8101309e9ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:59 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff810130be5080
May 15 05:33:59 vault1 kernel: ffffffff8058a6e0 0000000000042b71 ffff810130be5288 0000000030be5080
May 15 05:33:59 vault1 kernel: Call Trace:
May 15 05:33:59 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:59 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:59 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:59 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:59 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:59 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:59 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:59 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:59 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:59 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:59 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:33:59 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:33:59 vault1 kernel:
May 15 05:33:59 vault1 kernel: fc_wq_3 S ffff810130b71638 0 549 19 (L-TLB)
May 15 05:33:59 vault1 kernel: ffff810130293ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:33:59 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff8101300810c0
May 15 05:33:59 vault1 kernel: ffff810136be9080 0000000000001055 ffff8101300812c8 00000002300810c0
May 15 05:33:59 vault1 kernel: Call Trace:
May 15 05:33:59 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:33:59 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:33:59 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:33:59 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:33:59 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:33:59 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:33:59 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:33:59 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:33:59 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:33:59 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:33:59 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:00 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:00 vault1 kernel:
May 15 05:34:00 vault1 kernel: fc_dl_3 S ffff810130b71638 0 550 19 (L-TLB)
May 15 05:34:00 vault1 kernel: ffff810130909ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:00 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81013011d040
May 15 05:34:00 vault1 kernel: ffff810136be9080 0000000000001376 ffff81013011d248 000000023011d040
May 15 05:34:00 vault1 kernel: Call Trace:
May 15 05:34:00 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:00 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:00 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:00 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:00 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:00 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:00 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:00 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:00 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:00 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:34:00 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:00 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:00 vault1 kernel:
May 15 05:34:00 vault1 kernel: scsi_eh_4 S ffff810130b71638 0 551 19 (L-TLB)
May 15 05:34:00 vault1 kernel: ffff810130915e70 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:00 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81013008b080
May 15 05:34:00 vault1 kernel: ffff810136be9080 000000000000086b ffff81013008b288 0000000280263894
May 15 05:34:00 vault1 kernel: Call Trace:
May 15 05:34:00 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:00 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:00 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:34:00 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:34:00 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:00 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:00 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:00 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:00 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:00 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:00 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:00 vault1 kernel:
May 15 05:34:00 vault1 kernel: qla2xxx_4_dpc S ffff81012f451218 0 552 19 (L-TLB)
May 15 05:34:00 vault1 kernel: ffff81013014bee0 0000000000000046 0000000000000000 ffff8101300740c0
May 15 05:34:00 vault1 kernel: ffff8101300740c0 ffffffff802638fc 000000000000000a ffff8101300740c0
May 15 05:34:00 vault1 kernel: ffff81012f4f3040 0000000000008a77 ffff8101300742c8 00000000880ba30f
May 15 05:34:00 vault1 kernel: Call Trace:
May 15 05:34:00 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:00 vault1 kernel: [<ffffffff880bcb27>] :qla2xxx:qla2x00_fw_ready+0x84/0x1a2
May 15 05:34:00 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:34:00 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:00 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:34:00 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 15 05:34:00 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:00 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 15 05:34:00 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:00 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:00 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:00 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:00 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:00 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:00 vault1 kernel:
May 15 05:34:00 vault1 kernel: scsi_wq_4 S ffff81012f451218 0 553 19 (L-TLB)
May 15 05:34:00 vault1 kernel: ffff81013016dea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:00 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff8101301e2040
May 15 05:34:00 vault1 kernel: ffff81012eea1080 00000000000671cb ffff8101301e2248 00000000301e2040
May 15 05:34:00 vault1 kernel: Call Trace:
May 15 05:34:00 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:00 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:00 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:00 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:00 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:00 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:00 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:00 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:00 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:00 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:00 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:00 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:00 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:00 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:00 vault1 kernel:
May 15 05:34:00 vault1 kernel: fc_wq_4 S ffff810130b71638 0 554 19 (L-TLB)
May 15 05:34:00 vault1 kernel: ffff810130099ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:00 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130185080
May 15 05:34:01 vault1 kernel: ffff810136be9080 0000000000000fd3 ffff810130185288 0000000230185080
May 15 05:34:01 vault1 kernel: Call Trace:
May 15 05:34:01 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:01 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:01 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:01 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:01 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:01 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:01 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:01 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:01 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:01 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:01 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:01 vault1 kernel:
May 15 05:34:01 vault1 kernel: fc_dl_4 S ffff810130b71638 0 555 19 (L-TLB)
May 15 05:34:01 vault1 kernel: ffff81013009bea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:01 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff8101301840c0
May 15 05:34:01 vault1 kernel: ffff810136be9080 0000000000001084 ffff8101301842c8 00000002301840c0
May 15 05:34:01 vault1 kernel: Call Trace:
May 15 05:34:01 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:01 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:01 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:01 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:01 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:01 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:01 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:01 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:01 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:01 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:01 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:01 vault1 kernel:
May 15 05:34:01 vault1 kernel: kjournald S ffff810230e45678 0 556 19 (L-TLB)
May 15 05:34:01 vault1 kernel: ffff81013017bec0 0000000000000046 0000000000000000 0000000000000000
May 15 05:34:01 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810130168040
May 15 05:34:01 vault1 kernel: ffff810125693040 00000000000061ad ffff810130168248 000000018029fc0f
May 15 05:34:01 vault1 kernel: Call Trace:
May 15 05:34:01 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 15 05:34:01 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:01 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 15 05:34:01 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:01 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:01 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 15 05:34:01 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:01 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:01 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:01 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:01 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:01 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:01 vault1 kernel:
May 15 05:34:01 vault1 kernel: kauditd S ffff81012f8576f8 0 583 19 (L-TLB)
May 15 05:34:01 vault1 kernel: ffff810130117ec0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:01 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81012fd850c0
May 15 05:34:01 vault1 kernel: ffff81012ff09040 0000000000007ed7 ffff81012fd852c8 000000002fd850c0
May 15 05:34:01 vault1 kernel: Call Trace:
May 15 05:34:01 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:01 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:01 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:01 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:01 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:01 vault1 kernel: [<ffffffff802b2b39>] kauditd_thread+0x111/0x148
May 15 05:34:01 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:01 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:01 vault1 kernel: [<ffffffff802b2a28>] kauditd_thread+0x0/0x148
May 15 05:34:01 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:01 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:01 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:01 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:01 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:01 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:01 vault1 kernel:
May 15 05:34:01 vault1 kernel: udevd S ffff81012f856e38 0 617 1 (NOTLB)
May 15 05:34:01 vault1 kernel: ffff810130acba18 0000000000000046 0000000000000000 0000000000000046
May 15 05:34:01 vault1 kernel: ffffffff802ea6a4 ffff81013025a7b8 0000000000000005 ffff81013025a080
May 15 05:34:01 vault1 kernel: ffffffff8058a6e0 0000000000024d6c ffff81013025a288 0000000080262123
May 15 05:34:01 vault1 kernel: Call Trace:
May 15 05:34:01 vault1 kernel: [<ffffffff802ea6a4>] inotify_poll+0x31/0x56
May 15 05:34:01 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:01 vault1 kernel: [<ffffffff802ea6c2>] inotify_poll+0x4f/0x56
May 15 05:34:01 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:34:01 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:01 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:01 vault1 last message repeated 3 times
May 15 05:34:01 vault1 kernel: [<ffffffff8023e153>] lookup_mnt+0x19/0x4c
May 15 05:34:01 vault1 kernel: [<ffffffff8023e17e>] lookup_mnt+0x44/0x4c
May 15 05:34:02 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 15 05:34:02 vault1 kernel: [<ffffffff80219f0b>] __follow_mount+0x26/0x7b
May 15 05:34:02 vault1 kernel: [<ffffffff8020cff9>] do_lookup+0x63/0x1ae
May 15 05:34:02 vault1 kernel: [<ffffffff802119b8>] do_wp_page+0x5af/0x60b
May 15 05:34:02 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:34:02 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:34:02 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:02 vault1 kernel: [<ffffffff8023bee1>] remove_wait_queue+0x10/0x2c
May 15 05:34:02 vault1 kernel: [<ffffffff8023bee1>] remove_wait_queue+0x10/0x2c
May 15 05:34:02 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:02 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:02 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 15 05:34:02 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:02 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:02 vault1 kernel:
May 15 05:34:02 vault1 kernel: khpsbpkt S ffff81012f856e38 0 1220 19 (L-TLB)
May 15 05:34:02 vault1 kernel: ffff81012f317f00 0000000000000046 ffffffff80246747 0000000000000002
May 15 05:34:02 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012f7f0040
May 15 05:34:02 vault1 kernel: ffff81013025a080 0000000000001ce5 ffff81012f7f0248 0000000230191d30
May 15 05:34:02 vault1 kernel: Call Trace:
May 15 05:34:02 vault1 kernel: [<ffffffff80246747>] skb_dequeue+0x13/0x50
May 15 05:34:02 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:02 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:02 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:02 vault1 kernel: [<ffffffff8811e239>] :ieee1394:hpsbpkt_thread+0x0/0x8d
May 15 05:34:02 vault1 kernel: [<ffffffff8811e2a9>] :ieee1394:hpsbpkt_thread+0x70/0x8d
May 15 05:34:02 vault1 kernel: [<ffffffff8811e239>] :ieee1394:hpsbpkt_thread+0x0/0x8d
May 15 05:34:02 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:02 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:02 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:02 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:02 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:02 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:02 vault1 kernel:
May 15 05:34:02 vault1 kernel: knodemgrd_0 S ffff810130b71638 0 1271 19 (L-TLB)
May 15 05:34:02 vault1 kernel: ffff81012f21be70 0000000000000046 ffff810130d82080 ffffffff80261ece
May 15 05:34:02 vault1 kernel: 0000000000000246 ffff810230d3bc88 0000000000000009 ffff810130d82080
May 15 05:34:02 vault1 kernel: ffff81012f3dc040 0000000000098704 ffff810130d82288 0000000080261ece
May 15 05:34:02 vault1 kernel: Call Trace:
May 15 05:34:02 vault1 kernel: [<ffffffff80261ece>] __mutex_unlock_slowpath+0x128/0x133
May 15 05:34:02 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:02 vault1 kernel: [<ffffffff88122d00>] :ieee1394:nodemgr_host_thread+0x87/0xa58
May 15 05:34:02 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:02 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:02 vault1 kernel: [<ffffffff88122c79>] :ieee1394:nodemgr_host_thread+0x0/0xa58
May 15 05:34:02 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:02 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:02 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:02 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:02 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:02 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:02 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:02 vault1 kernel:
May 15 05:34:02 vault1 kernel: ata/0 S ffff810230f85a98 0 1364 19 (L-TLB)
May 15 05:34:02 vault1 kernel: ffff81012eba9ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:02 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff81012f6d2080
May 15 05:34:02 vault1 kernel: ffff810130a13080 0000000000001a45 ffff81012f6d2288 000000002f6d2080
May 15 05:34:02 vault1 kernel: Call Trace:
May 15 05:34:02 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:02 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:02 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:02 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:02 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:02 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:02 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:02 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:02 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:02 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:02 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:02 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:02 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:02 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:02 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:02 vault1 kernel:
May 15 05:34:02 vault1 kernel: ata/1 S ffff81012f856e38 0 1365 19 (L-TLB)
May 15 05:34:02 vault1 kernel: ffff81012ebabea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:02 vault1 kernel: 0000000000000000 0000000000000000 0000000000000006 ffff81012f791040
May 15 05:34:02 vault1 kernel: ffff81013025a080 000000000000208c ffff81012f791248 000000012f791040
May 15 05:34:02 vault1 kernel: Call Trace:
May 15 05:34:02 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:02 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:02 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:02 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:02 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:02 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:02 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:02 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:02 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:02 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:02 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:02 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:02 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:02 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:02 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:02 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:02 vault1 kernel:
May 15 05:34:02 vault1 kernel: ata/2 S ffff810230f84058 0 1366 19 (L-TLB)
May 15 05:34:02 vault1 kernel: ffff81012ebafea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:02 vault1 kernel: 0000000000000000 0000000000000000 0000000000000006 ffff81012f46e080
May 15 05:34:02 vault1 kernel: ffff81012fac80c0 000000000000267d ffff81012f46e288 000000022f46e080
May 15 05:34:02 vault1 kernel: Call Trace:
May 15 05:34:02 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:03 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:03 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:03 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:03 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:03 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:03 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:03 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:03 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:03 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:03 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:03 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:03 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:03 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:03 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:03 vault1 kernel:
May 15 05:34:03 vault1 kernel: ata/3 S ffff81012f5bd6b8 0 1367 19 (L-TLB)
May 15 05:34:03 vault1 kernel: ffff81012faf9ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:03 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff81012f0950c0
May 15 05:34:03 vault1 kernel: ffff8101308a9080 00000000000021a6 ffff81012f0952c8 000000032f0950c0
May 15 05:34:03 vault1 kernel: Call Trace:
May 15 05:34:03 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:03 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:03 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:03 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:03 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:03 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:03 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:03 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:03 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:03 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:03 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:03 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:03 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:03 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:03 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:03 vault1 kernel:
May 15 05:34:03 vault1 kernel: ata_aux S ffff810130840058 0 1368 19 (L-TLB)
May 15 05:34:03 vault1 kernel: ffff81012ebb1ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:03 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff81012f501040
May 15 05:34:03 vault1 kernel: ffff81012f871040 0000000000001202 ffff81012f501248 000000002f501040
May 15 05:34:03 vault1 kernel: Call Trace:
May 15 05:34:03 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:03 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:03 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:03 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:03 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:03 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:03 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:03 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:03 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:03 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:03 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:03 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:03 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:03 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:03 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:03 vault1 kernel:
May 15 05:34:03 vault1 kernel: scsi_eh_5 S ffff810130840918 0 1675 19 (L-TLB)
May 15 05:34:03 vault1 kernel: ffff81012e7b1e70 0000000000000046 ffff81012f0b4040 ffffffff802638fc
May 15 05:34:03 vault1 kernel: ffff81012f898000 0000000000000000 0000000000000003 ffff81012f0b4040
May 15 05:34:03 vault1 kernel: ffff810230d72080 0000000000015241 ffff81012f0b4248 0000000100000282
May 15 05:34:03 vault1 kernel: Call Trace:
May 15 05:34:03 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:03 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 15 05:34:03 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:03 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:34:03 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:03 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:03 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:34:03 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:03 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:03 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:03 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:03 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:03 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:03 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:03 vault1 kernel:
May 15 05:34:03 vault1 kernel: scsi_eh_6 S ffff81012eb76998 0 1677 19 (L-TLB)
May 15 05:34:03 vault1 kernel: ffff81012e2cfe70 0000000000000046 ffff81012fdea0c0 ffffffff802638fc
May 15 05:34:03 vault1 kernel: ffff81012ff60000 0000000000000000 0000000000000006 ffff81012fdea0c0
May 15 05:34:04 vault1 kernel: ffff81012f7dc0c0 00000000000060c6 ffff81012fdea2c8 0000000000000282
May 15 05:34:04 vault1 kernel: Call Trace:
May 15 05:34:04 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:04 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:04 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:04 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:04 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:04 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:04 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:04 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:04 vault1 kernel:
May 15 05:34:04 vault1 kernel: scsi_eh_7 S ffff81022fe50e38 0 2076 19 (L-TLB)
May 15 05:34:04 vault1 kernel: ffff81012df47e70 0000000000000046 ffff81012efdc040 ffffffff802638fc
May 15 05:34:04 vault1 kernel: ffff81012df20000 0000000000000000 0000000000000005 ffff81012efdc040
May 15 05:34:04 vault1 kernel: ffff81012f7dc0c0 0000000000005d45 ffff81012efdc248 0000000000000282
May 15 05:34:04 vault1 kernel: Call Trace:
May 15 05:34:04 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:04 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:04 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:04 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:04 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:04 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:04 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:04 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:04 vault1 kernel:
May 15 05:34:04 vault1 kernel: scsi_eh_8 S ffff8102300764b8 0 2077 19 (L-TLB)
May 15 05:34:04 vault1 kernel: ffff81012df1fe70 0000000000000046 ffff81012eea30c0 ffffffff802638fc
May 15 05:34:04 vault1 kernel: ffff81012e5e4000 0000000000000000 0000000000000008 ffff81012eea30c0
May 15 05:34:04 vault1 kernel: ffff81012f7dc0c0 0000000000004d90 ffff81012eea32c8 0000000000000282
May 15 05:34:04 vault1 kernel: Call Trace:
May 15 05:34:04 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:04 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:04 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:04 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:04 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:04 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:04 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:04 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:04 vault1 kernel:
May 15 05:34:04 vault1 kernel: kmpathd/0 S ffff810230cee998 0 2515 19 (L-TLB)
May 15 05:34:04 vault1 kernel: ffff81012d897ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:04 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81012f70f040
May 15 05:34:04 vault1 kernel: ffff810130a13080 0000000000000d99 ffff81012f70f248 000000002f70f040
May 15 05:34:04 vault1 kernel: Call Trace:
May 15 05:34:04 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:04 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:04 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:04 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:04 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:04 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:04 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:04 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:04 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:04 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:04 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:04 vault1 kernel:
May 15 05:34:04 vault1 kernel: kmpathd/1 S ffff810230f85a98 0 2516 19 (L-TLB)
May 15 05:34:04 vault1 kernel: ffff81012d899ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:04 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff810130f6a0c0
May 15 05:34:04 vault1 kernel: ffff810136b92040 0000000000001abe ffff810130f6a2c8 0000000130f6a0c0
May 15 05:34:04 vault1 kernel: Call Trace:
May 15 05:34:04 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:04 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:04 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:04 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:04 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:04 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:04 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:05 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:05 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:05 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:05 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:05 vault1 kernel:
May 15 05:34:05 vault1 kernel: kmpathd/2 S ffff81012f856e38 0 2517 19 (L-TLB)
May 15 05:34:05 vault1 kernel: ffff81012d89bea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:05 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff8101308a9080
May 15 05:34:05 vault1 kernel: ffff81013025a080 0000000000002088 ffff8101308a9288 00000002308a9080
May 15 05:34:05 vault1 kernel: Call Trace:
May 15 05:34:05 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:05 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:05 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:05 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:05 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:05 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:05 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:05 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:05 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:05 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:05 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:05 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:05 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:34:05 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:05 vault1 kernel:
May 15 05:34:05 vault1 kernel: kmpathd/3 S ffff810230cee998 0 2518 19 (L-TLB)
May 15 05:34:05 vault1 kernel: ffff81012d89dea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:05 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012f2030c0
May 15 05:34:05 vault1 kernel: ffff810230c4c0c0 0000000000003006 ffff81012f2032c8 000000032f2030c0
May 15 05:34:05 vault1 kernel: Call Trace:
May 15 05:34:05 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:05 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:05 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:05 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:05 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:05 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:05 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:05 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:05 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:05 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:34:05 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:05 vault1 kernel:
May 15 05:34:05 vault1 kernel: kmirrord S ffff810230d4f6f8 0 2533 19 (L-TLB)
May 15 05:34:05 vault1 kernel: ffff81012dfd9ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:05 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff810130ee0080
May 15 05:34:05 vault1 kernel: ffff810230c4c0c0 00000000000019d6 ffff810130ee0288 0000000330ee0080
May 15 05:34:05 vault1 kernel: Call Trace:
May 15 05:34:05 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:05 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:05 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:05 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:05 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:05 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:05 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:05 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:05 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:05 vault1 kernel:
May 15 05:34:05 vault1 kernel: jfsIO S ffff810130b704b8 0 2571 19 (L-TLB)
May 15 05:34:05 vault1 kernel: ffff81012c877f00 0000000000000046 0000000000000000 0000000000000001
May 15 05:34:05 vault1 kernel: ffff81012c876000 ffff8101b644cd00 000000000000000a ffff810130dba0c0
May 15 05:34:05 vault1 kernel: ffff810230c4c0c0 000000000006370c ffff810130dba2c8 0000000300000001
May 15 05:34:05 vault1 kernel: Call Trace:
May 15 05:34:05 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:05 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:05 vault1 kernel: [<ffffffff88338385>] :jfs:jfsIOWait+0x0/0x96
May 15 05:34:05 vault1 kernel: [<ffffffff883383fe>] :jfs:jfsIOWait+0x79/0x96
May 15 05:34:05 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:05 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:05 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:05 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:05 vault1 kernel:
May 15 05:34:05 vault1 kernel: jfsCommit S ffff810230f84058 0 2572 19 (L-TLB)
May 15 05:34:05 vault1 kernel: ffff81012c879eb0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:05 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012f22d080
May 15 05:34:05 vault1 kernel: ffff810136be9080 0000000000000776 ffff81012f22d288 000000022f22d080
May 15 05:34:05 vault1 kernel: Call Trace:
May 15 05:34:05 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:05 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:05 vault1 kernel: [<ffffffff8833b108>] :jfs:jfs_lazycommit+0x1d9/0x212
May 15 05:34:05 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:05 vault1 kernel: [<ffffffff8833af2f>] :jfs:jfs_lazycommit+0x0/0x212
May 15 05:34:05 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:05 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:05 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:05 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:34:05 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:05 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:05 vault1 kernel:
May 15 05:34:05 vault1 kernel: jfsCommit S ffff810230f84058 0 2573 19 (L-TLB)
May 15 05:34:06 vault1 kernel: ffff81012c87beb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:06 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012f722080
May 15 05:34:06 vault1 kernel: ffff81012f6be0c0 000000000000bbdd ffff81012f722288 000000022f722080
May 15 05:34:06 vault1 kernel: Call Trace:
May 15 05:34:06 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:06 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:06 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:06 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:06 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:06 vault1 kernel: [<ffffffff8833b108>] :jfs:jfs_lazycommit+0x1d9/0x212
May 15 05:34:06 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:06 vault1 kernel: [<ffffffff8833af2f>] :jfs:jfs_lazycommit+0x0/0x212
May 15 05:34:06 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:06 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:06 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:06 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:06 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:34:06 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:06 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:06 vault1 kernel:
May 15 05:34:06 vault1 kernel: jfsCommit S ffff810230f84058 0 2574 19 (L-TLB)
May 15 05:34:06 vault1 kernel: ffff81012c87deb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:06 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff810130dda080
May 15 05:34:06 vault1 kernel: ffff81012f22d080 0000000000000976 ffff810130dda288 0000000230dda080
May 15 05:34:06 vault1 kernel: Call Trace:
May 15 05:34:06 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:06 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:06 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:06 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:06 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:06 vault1 kernel: [<ffffffff8833b108>] :jfs:jfs_lazycommit+0x1d9/0x212
May 15 05:34:06 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:06 vault1 kernel: [<ffffffff8833af2f>] :jfs:jfs_lazycommit+0x0/0x212
May 15 05:34:06 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:06 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:06 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:06 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:06 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:34:06 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:06 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:06 vault1 kernel:
May 15 05:34:06 vault1 kernel: jfsCommit S ffff810230f84058 0 2575 19 (L-TLB)
May 15 05:34:06 vault1 kernel: ffff81012c87feb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:06 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012f6be0c0
May 15 05:34:06 vault1 kernel: ffff810130dda080 0000000000000a3e ffff81012f6be2c8 000000022f6be0c0
May 15 05:34:06 vault1 kernel: Call Trace:
May 15 05:34:06 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:06 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:06 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:06 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:06 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:06 vault1 kernel: [<ffffffff8833b108>] :jfs:jfs_lazycommit+0x1d9/0x212
May 15 05:34:06 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:06 vault1 kernel: [<ffffffff8833af2f>] :jfs:jfs_lazycommit+0x0/0x212
May 15 05:34:06 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:06 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:06 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:06 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:06 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:34:06 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:06 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:06 vault1 kernel:
May 15 05:34:06 vault1 kernel: jfsSync S ffff81012f856e38 0 2576 19 (L-TLB)
May 15 05:34:06 vault1 kernel: ffff81012c881ee0 0000000000000046 0000000000000000 ffff81012ef5a040
May 15 05:34:06 vault1 kernel: 00000000000001a9 0000000000000001 0000000000000008 ffff81012ef5a040
May 15 05:34:06 vault1 kernel: ffff810136be9080 00000000000122f8 ffff81012ef5a248 0000000200000246
May 15 05:34:06 vault1 kernel: Call Trace:
May 15 05:34:06 vault1 kernel: [<ffffffff8833a8e3>] :jfs:jfs_sync+0x181/0x1ac
May 15 05:34:06 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:06 vault1 kernel: [<ffffffff8833a762>] :jfs:jfs_sync+0x0/0x1ac
May 15 05:34:06 vault1 kernel: [<ffffffff8833a8e8>] :jfs:jfs_sync+0x186/0x1ac
May 15 05:34:06 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:06 vault1 kernel: [<ffffffff8833a762>] :jfs:jfs_sync+0x0/0x1ac
May 15 05:34:06 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:07 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:07 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:07 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:07 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:34:07 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:07 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:07 vault1 kernel:
May 15 05:34:07 vault1 kernel: kjournald S ffff810230e444f8 0 2578 19 (L-TLB)
May 15 05:34:07 vault1 kernel: ffff81012d563ec0 0000000000000046 0000000000000000 0000000000000000
May 15 05:34:07 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff81012f871040
May 15 05:34:07 vault1 kernel: ffff810136b92040 0000000000001be6 ffff81012f871248 000000018029fc0f
May 15 05:34:07 vault1 kernel: Call Trace:
May 15 05:34:07 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 15 05:34:07 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:07 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 15 05:34:07 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:07 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:07 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 15 05:34:07 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:07 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:07 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:07 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:07 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:07 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:07 vault1 kernel:
May 15 05:34:07 vault1 kernel: kjournald S ffff810230e45678 0 2580 19 (L-TLB)
May 15 05:34:07 vault1 kernel: ffff81012d3cdec0 0000000000000046 0000000000000000 0000000000000000
May 15 05:34:07 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff81012fac80c0
May 15 05:34:07 vault1 kernel: ffff810136b92040 0000000000004e14 ffff81012fac82c8 000000018029fc0f
May 15 05:34:07 vault1 kernel: Call Trace:
May 15 05:34:07 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 15 05:34:07 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:07 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 15 05:34:07 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:07 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:07 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 15 05:34:07 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:07 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:07 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:07 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:07 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:07 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:07 vault1 kernel:
May 15 05:34:07 vault1 kernel: kondemand/0 S ffff810230e444f8 0 2712 19 (L-TLB)
May 15 05:34:07 vault1 kernel: ffff81012d509ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:07 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012e9e5040
May 15 05:34:07 vault1 kernel: ffffffff8058a6e0 0000000000001127 ffff81012e9e5248 000000002e9e5040
May 15 05:34:07 vault1 kernel: Call Trace:
May 15 05:34:07 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:07 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:07 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:07 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:07 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:07 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:07 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:07 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:07 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:07 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:07 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:07 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:07 vault1 kernel:
May 15 05:34:07 vault1 kernel: kondemand/1 S ffff81012f450db8 0 2713 19 (L-TLB)
May 15 05:34:07 vault1 kernel: ffff81012c68fea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:07 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff81012f238040
May 15 05:34:07 vault1 kernel: ffff810136b92040 00000000000021a8 ffff81012f238248 000000012f238040
May 15 05:34:07 vault1 kernel: Call Trace:
May 15 05:34:07 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:07 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:07 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:07 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:07 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:07 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:07 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:07 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:07 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:07 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:07 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:07 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:07 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:07 vault1 kernel:
May 15 05:34:07 vault1 kernel: kondemand/2 S ffff81012f857b58 0 2714 19 (L-TLB)
May 15 05:34:07 vault1 kernel: ffff81012c595ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:07 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff8101301b8080
May 15 05:34:07 vault1 kernel: ffff810136be9080 0000000000000fee ffff8101301b8288 00000002301b8080
May 15 05:34:07 vault1 kernel: Call Trace:
May 15 05:34:07 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:07 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:07 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:07 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:07 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:07 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:07 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:07 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:07 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:07 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:08 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:08 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:08 vault1 kernel:
May 15 05:34:08 vault1 kernel: kondemand/3 S ffff81012f856e38 0 2715 19 (L-TLB)
May 15 05:34:08 vault1 kernel: ffff81012c599ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:08 vault1 kernel: 0000000000000000 0000000000000000 0000000000000007 ffff81012e46b0c0
May 15 05:34:08 vault1 kernel: ffff81013025a080 0000000000002efb ffff81012e46b2c8 000000032e46b0c0
May 15 05:34:08 vault1 kernel: Call Trace:
May 15 05:34:08 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:08 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:08 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:08 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:08 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:08 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:08 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:08 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:08 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:08 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:08 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:08 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:08 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:08 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:08 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:08 vault1 kernel:
May 15 05:34:08 vault1 kernel: dhclient S ffff81012eb77b18 0 3260 1 (NOTLB)
May 15 05:34:08 vault1 kernel: ffff810130dfba18 0000000000000046 0000000000000000 ffff81012f3b0040
May 15 05:34:08 vault1 kernel: ffff81012f3b0040 ffffffff802638fc 000000000000000a ffff81012f3b0040
May 15 05:34:08 vault1 kernel: ffff810136b92040 00000000000022f6 ffff81012f3b0248 0000000130c84000
May 15 05:34:08 vault1 kernel: Call Trace:
May 15 05:34:08 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:08 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 15 05:34:08 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 15 05:34:08 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:34:08 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:08 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:08 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:08 vault1 kernel: [<ffffffff802078c3>] kmem_cache_free+0x257/0x267
May 15 05:34:08 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:08 vault1 kernel: [<ffffffff804435a0>] udp_recvmsg+0x1ee/0x270
May 15 05:34:08 vault1 kernel: [<ffffffff80232600>] sock_common_recvmsg+0x30/0x45
May 15 05:34:08 vault1 kernel: [<ffffffff80230bc6>] sock_recvmsg+0xf8/0x117
May 15 05:34:08 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:08 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:34:08 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:34:08 vault1 kernel: [<ffffffff8040bc77>] move_addr_to_user+0x5d/0x78
May 15 05:34:08 vault1 kernel: [<ffffffff8022c605>] sys_recvfrom+0x104/0x130
May 15 05:34:08 vault1 kernel: [<ffffffff80216b98>] sys_select+0x15a/0x183
May 15 05:34:08 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:08 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:08 vault1 kernel:
May 15 05:34:08 vault1 kernel: syslogd D ffff81012fe03b18 0 3374 1 (NOTLB)
May 15 05:34:08 vault1 kernel: ffff81012bd7fd88 0000000000000046 0000000000000000 0000000000000000
May 15 05:34:08 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff81012f0670c0
May 15 05:34:08 vault1 kernel: ffff8101301b70c0 0000000000020f2a ffff81012f0672c8 000000008029fc0f
May 15 05:34:08 vault1 kernel: Call Trace:
May 15 05:34:08 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:08 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:08 vault1 kernel: [<ffffffff88026c13>] :jbd:log_wait_commit+0xa4/0xf7
May 15 05:34:08 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:08 vault1 kernel: [<ffffffff88021730>] :jbd:journal_stop+0x1e5/0x21a
May 15 05:34:08 vault1 kernel: [<ffffffff802304c1>] __writeback_single_inode+0x1fa/0x338
May 15 05:34:08 vault1 kernel: [<ffffffff802e2073>] sync_inode+0x19/0x39
May 15 05:34:08 vault1 kernel: [<ffffffff802e2073>] sync_inode+0x19/0x39
May 15 05:34:08 vault1 kernel: [<ffffffff802e207e>] sync_inode+0x24/0x39
May 15 05:34:08 vault1 kernel: [<ffffffff88037332>] :ext3:ext3_sync_file+0xb2/0xc4
May 15 05:34:08 vault1 kernel: [<ffffffff8024e527>] do_fsync+0x52/0xa4
May 15 05:34:08 vault1 kernel: [<ffffffff802e4478>] __do_fsync+0x23/0x36
May 15 05:34:08 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:08 vault1 kernel:
May 15 05:34:08 vault1 kernel: klogd S ffff81012f450db8 0 3377 1 (NOTLB)
May 15 05:34:08 vault1 kernel: ffff810127185be8 0000000000000046 0000000000000000 ffff8101301b70c0
May 15 05:34:08 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff8101301b70c0
May 15 05:34:08 vault1 kernel: ffffffff8058a6e0 0000000000009e90 ffff8101301b72c8 0000000000000002
May 15 05:34:08 vault1 kernel: Call Trace:
May 15 05:34:08 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:08 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:08 vault1 kernel: [<ffffffff80460f4b>] unix_wait_for_peer+0x86/0xb0
May 15 05:34:08 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:08 vault1 kernel: [<ffffffff804614fb>] unix_dgram_sendmsg+0x3d3/0x4cd
May 15 05:34:08 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 15 05:34:08 vault1 kernel: [<ffffffff80260e98>] __sched_text_start+0x928/0x960
May 15 05:34:08 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 15 05:34:08 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:08 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:08 vault1 kernel: [<ffffffff80216cda>] vfs_write+0xe1/0x177
May 15 05:34:08 vault1 kernel: [<ffffffff802176bf>] sys_write+0x45/0x6e
May 15 05:34:08 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:08 vault1 kernel:
May 15 05:34:08 vault1 kernel: irqbalance S ffff81012fe020d8 0 3393 1 (NOTLB)
May 15 05:34:08 vault1 kernel: ffff81012bd75eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 15 05:34:08 vault1 kernel: ffff8100062dcbf8 ffff8100062dcbf0 0000000000000007 ffff81012eea00c0
May 15 05:34:09 vault1 kernel: ffffffff8058a6e0 0000000000202cde ffff81012eea02c8 0000000000000292
May 15 05:34:09 vault1 kernel: Call Trace:
May 15 05:34:09 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:09 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 15 05:34:09 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 15 05:34:09 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 15 05:34:09 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 15 05:34:09 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:09 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 15 05:34:09 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:09 vault1 kernel:
May 15 05:34:09 vault1 kernel: portmap S ffff810130b70058 0 3428 1 (NOTLB)
May 15 05:34:09 vault1 kernel: ffff81012c6bdb28 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:34:09 vault1 kernel: 0000000180248a3b 0000000000000002 000000000000000a ffff8101309f4080
May 15 05:34:09 vault1 kernel: ffff810136b92040 000000000001dcc0 ffff8101309f4288 0000000100000000
May 15 05:34:09 vault1 kernel: Call Trace:
May 15 05:34:09 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:09 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:09 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:09 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 15 05:34:09 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:34:09 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:09 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:09 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:09 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:09 vault1 kernel: [<ffffffff8029ff51>] debug_check_no_locks_freed+0x11d/0x129
May 15 05:34:09 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:09 vault1 kernel: [<ffffffff8020b848>] free_hot_cold_page+0x184/0x190
May 15 05:34:09 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:09 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:09 vault1 kernel: [<ffffffff8040b777>] sys_sendmsg+0x217/0x28a
May 15 05:34:09 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:09 vault1 kernel: [<ffffffff8023089b>] __up_write+0x27/0xf2
May 15 05:34:09 vault1 kernel: [<ffffffff8023089b>] __up_write+0x27/0xf2
May 15 05:34:09 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:09 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:34:09 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:09 vault1 kernel:
May 15 05:34:09 vault1 kernel: rpc.statd S ffff810130b70918 0 3453 1 (NOTLB)
May 15 05:34:09 vault1 kernel: ffff81012b387a18 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:34:09 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000006 ffff81013017c040
May 15 05:34:09 vault1 kernel: ffffffff8058a6e0 000000000000295e ffff81013017c248 0000000000000000
May 15 05:34:09 vault1 kernel: Call Trace:
May 15 05:34:09 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:09 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:09 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:09 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 15 05:34:09 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:34:09 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:09 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:09 vault1 last message repeated 2 times
May 15 05:34:09 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 15 05:34:09 vault1 kernel: [<ffffffff8046155d>] unix_dgram_sendmsg+0x435/0x4cd
May 15 05:34:09 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 15 05:34:09 vault1 kernel: [<ffffffff802530fc>] sock_sendmsg+0xea/0x107
May 15 05:34:09 vault1 kernel: [<ffffffff8020ed97>] link_path_walk+0xce/0xe0
May 15 05:34:09 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:09 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:34:09 vault1 kernel: [<ffffffff8022d806>] mntput_no_expire+0x1c/0x96
May 15 05:34:09 vault1 kernel: [<ffffffff8040ba92>] sys_sendto+0x128/0x151
May 15 05:34:09 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 15 05:34:09 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:09 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:09 vault1 kernel:
May 15 05:34:09 vault1 kernel: dbus-daemon S ffff81012eb760d8 0 3481 1 (NOTLB)
May 15 05:34:09 vault1 kernel: ffff81012b4d7b28 0000000000000046 0000000000000000 ffff81012f7dc0c0
May 15 05:34:09 vault1 kernel: 0000000000000000 ffffffff802a0de5 000000000000000a ffff81012f7dc0c0
May 15 05:34:09 vault1 kernel: ffffffff8058a6e0 0000000000000a36 ffff81012f7dc2c8 0000000000000002
May 15 05:34:09 vault1 kernel: Call Trace:
May 15 05:34:09 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:09 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:09 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:34:09 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:09 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:09 vault1 last message repeated 2 times
May 15 05:34:09 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 15 05:34:09 vault1 kernel: [<ffffffff8024660c>] sock_aio_write+0x0/0x128
May 15 05:34:09 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:34:09 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:09 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:09 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:09 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:09 vault1 kernel: [<ffffffff802d3dfa>] do_readv_writev+0x199/0x1ae
May 15 05:34:10 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 15 05:34:10 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:10 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:34:10 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:10 vault1 kernel:
May 15 05:34:10 vault1 kernel: hidd S ffff81012e41c9d8 0 3518 1 (NOTLB)
May 15 05:34:10 vault1 kernel: ffff81012b439ad8 0000000000000046 0000000000000000 ffff81012eea2040
May 15 05:34:10 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000001 ffff81012eea2040
May 15 05:34:10 vault1 kernel: ffffffff8058a6e0 00000000000085b1 ffff81012eea2248 0000000000000002
May 15 05:34:10 vault1 kernel: Call Trace:
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:10 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:34:10 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:10 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:10 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:10 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 15 05:34:10 vault1 kernel: [<ffffffff8046155d>] unix_dgram_sendmsg+0x435/0x4cd
May 15 05:34:10 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 15 05:34:10 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:34:10 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff8021e12c>] sigprocmask+0x28/0xc1
May 15 05:34:10 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:10 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:10 vault1 kernel: [<ffffffff802d861f>] sys_ppoll+0xf1/0x1ff
May 15 05:34:10 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:10 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:10 vault1 kernel:
May 15 05:34:10 vault1 kernel: acpid S ffff81012f8569d8 0 3535 1 (NOTLB)
May 15 05:34:10 vault1 kernel: ffff81012b4e5b28 0000000000000046 0000000000000612 ffff81012e53b080
May 15 05:34:10 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000007 ffff81012e53b080
May 15 05:34:10 vault1 kernel: ffff8102300e9040 000000000001ef10 ffff81012e53b288 0000000000000002
May 15 05:34:10 vault1 kernel: Call Trace:
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:10 vault1 kernel: [<ffffffff88021755>] :jbd:journal_stop+0x20a/0x21a
May 15 05:34:10 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:10 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:34:10 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:10 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:10 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:10 vault1 kernel: [<ffffffff8020e533>] current_fs_time+0x3b/0x40
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff80216881>] __generic_file_aio_write_nolock+0x378/0x3eb
May 15 05:34:10 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 15 05:34:10 vault1 kernel: [<ffffffff80261ece>] __mutex_unlock_slowpath+0x128/0x133
May 15 05:34:10 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:10 vault1 kernel: [<ffffffff802224af>] generic_file_aio_write+0x6c/0xc1
May 15 05:34:10 vault1 kernel: [<ffffffff88037192>] :ext3:ext3_file_write+0x16/0x94
May 15 05:34:10 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:10 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:10 vault1 kernel: [<ffffffff8040bfa2>] sys_accept+0x19e/0x1fb
May 15 05:34:10 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:10 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:34:10 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:10 vault1 kernel:
May 15 05:34:10 vault1 kernel: sshd S ffff81012ec1da98 0 3553 1 (NOTLB)
May 15 05:34:10 vault1 kernel: ffff81012739da18 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:34:10 vault1 kernel: 000000018023bee1 0000000000000002 0000000000000007 ffff81012ff09040
May 15 05:34:10 vault1 kernel: ffffffff8058a6e0 0000000000005619 ffff81012ff09248 0000000000000000
May 15 05:34:10 vault1 kernel: Call Trace:
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:10 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:10 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 15 05:34:10 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:34:10 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:10 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:10 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff802133c4>] sock_def_readable+0x18/0x6c
May 15 05:34:10 vault1 kernel: [<ffffffff802133c4>] sock_def_readable+0x18/0x6c
May 15 05:34:10 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 15 05:34:10 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:10 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 15 05:34:10 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:10 vault1 kernel: [<ffffffff802c070d>] zone_statistics+0x3f/0x60
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff802118a0>] do_wp_page+0x497/0x60b
May 15 05:34:10 vault1 kernel: [<ffffffff80211a03>] do_wp_page+0x5fa/0x60b
May 15 05:34:10 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:34:10 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:10 vault1 kernel: [<ffffffff8020c6a9>] _atomic_dec_and_lock+0x39/0x58
May 15 05:34:10 vault1 kernel: [<ffffffff80212fc9>] poison_obj+0x26/0x2f
May 15 05:34:10 vault1 kernel: [<ffffffff802078c3>] kmem_cache_free+0x257/0x267
May 15 05:34:10 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:10 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 15 05:34:10 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:11 vault1 kernel: [<ffffffff80239ba9>] generic_drop_inode+0x0/0x15c
May 15 05:34:11 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:11 vault1 kernel:
May 15 05:34:11 vault1 kernel: rpc.idmapd S ffff810130b704b8 0 3679 1 (NOTLB)
May 15 05:34:11 vault1 kernel: ffff8101267a1e78 0000000000000046 0000000000000000 ffff81012f5ba0c0
May 15 05:34:11 vault1 kernel: ffff81012f5ba0c0 ffffffff802638fc 000000000000000a ffff81012f5ba0c0
May 15 05:34:11 vault1 kernel: ffff810136b92040 00000000000046db ffff81012f5ba2c8 0000000130c84000
May 15 05:34:11 vault1 kernel: Call Trace:
May 15 05:34:11 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:11 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 15 05:34:11 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 15 05:34:11 vault1 kernel: [<ffffffff802ead19>] sys_epoll_wait+0x193/0x426
May 15 05:34:11 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:11 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:11 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:11 vault1 kernel:
May 15 05:34:11 vault1 kernel: crond S ffff81012f0bbb58 0 3694 1 (NOTLB)
May 15 05:34:11 vault1 kernel: ffff810125d63eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 15 05:34:11 vault1 kernel: ffff8100062e79f8 ffff8100062e79f0 000000000000000a ffff81012e84d080
May 15 05:34:11 vault1 kernel: ffff810136b92040 000000000002dd2a ffff81012e84d288 0000000100000292
May 15 05:34:11 vault1 kernel: Call Trace:
May 15 05:34:11 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:11 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 15 05:34:11 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 15 05:34:11 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 15 05:34:11 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 15 05:34:11 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:11 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 15 05:34:11 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:11 vault1 kernel:
May 15 05:34:11 vault1 kernel: atd S ffff81012f451678 0 3709 1 (NOTLB)
May 15 05:34:11 vault1 kernel: ffff810126653eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 15 05:34:11 vault1 kernel: ffff8100062e79f8 ffff8100062e79f0 0000000000000007 ffff81012f0050c0
May 15 05:34:11 vault1 kernel: ffff810136b92040 0000000000018405 ffff81012f0052c8 0000000100000292
May 15 05:34:11 vault1 kernel: Call Trace:
May 15 05:34:11 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:11 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 15 05:34:11 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 15 05:34:11 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 15 05:34:11 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 15 05:34:11 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:11 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 15 05:34:11 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:11 vault1 kernel:
May 15 05:34:11 vault1 kernel: hald S ffff81012f451218 0 3724 1 (NOTLB)
May 15 05:34:11 vault1 kernel: ffff810125dfbb28 0000000000000046 ffff810130c84018 ffff81012f4f3040
May 15 05:34:11 vault1 kernel: ffff81012f4f3040 ffffffff802638fc 000000000000000a ffff81012f4f3040
May 15 05:34:11 vault1 kernel: ffff810125693040 0000000000061014 ffff81012f4f3248 0000000130c84000
May 15 05:34:11 vault1 kernel: Call Trace:
May 15 05:34:11 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:11 vault1 kernel: [<ffffffff8021d0d9>] __mod_timer+0xa7/0xb5
May 15 05:34:11 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 15 05:34:11 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 15 05:34:11 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:34:11 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:11 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:11 vault1 last message repeated 9 times
May 15 05:34:11 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:11 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:11 vault1 kernel: [<ffffffff8028dc04>] _local_bh_enable+0xbb/0xd0
May 15 05:34:11 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:11 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:34:11 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:11 vault1 kernel:
May 15 05:34:11 vault1 kernel: hald-runner S ffff8102303edb18 0 3725 3724 (NOTLB)
May 15 05:34:11 vault1 kernel: ffff810230d6bb28 0000000000000046 0000000000000000 ffff8102300e9040
May 15 05:34:11 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000001 ffff8102300e9040
May 15 05:34:11 vault1 kernel: ffffffff8058a6e0 000000000000076b ffff8102300e9248 0000000000000002
May 15 05:34:11 vault1 kernel: Call Trace:
May 15 05:34:11 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:11 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:11 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:34:11 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:11 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:11 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 15 05:34:11 vault1 kernel: [<ffffffff8026212e>] __mutex_lock_slowpath+0x250/0x25b
May 15 05:34:11 vault1 kernel: [<ffffffff8024660c>] sock_aio_write+0x0/0x128
May 15 05:34:11 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:34:11 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:11 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:11 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:11 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:11 vault1 kernel: [<ffffffff802d3dfa>] do_readv_writev+0x199/0x1ae
May 15 05:34:11 vault1 kernel: [<ffffffff80229266>] do_wait+0x9a7/0xa75
May 15 05:34:12 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:12 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:34:12 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:12 vault1 kernel:
May 15 05:34:12 vault1 kernel: hald-addon-ke S ffff810130840058 0 3790 3725 (NOTLB)
May 15 05:34:12 vault1 kernel: ffff810125a6fea8 0000000000000046 0000000000000000 0000000000000000
May 15 05:34:12 vault1 kernel: 0000000000000000 0000000000000046 0000000000000007 ffff81012f209080
May 15 05:34:12 vault1 kernel: ffffffff8058a6e0 000000000008f29f ffff81012f209288 000000002f209080
May 15 05:34:12 vault1 kernel: Call Trace:
May 15 05:34:12 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:12 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 15 05:34:12 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:12 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:12 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:34:12 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:34:12 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:12 vault1 kernel:
May 15 05:34:12 vault1 kernel: hald-addon-ac S ffff810230e44098 0 3797 3725 (NOTLB)
May 15 05:34:12 vault1 kernel: ffff810125b4bbe8 0000000000000046 0000000000000612 ffff8101308d4040
May 15 05:34:12 vault1 kernel: 0000000000000001 ffffffff802a0de5 0000000000000008 ffff8101308d4040
May 15 05:34:12 vault1 kernel: ffff81012e53b080 000000000005f23b ffff8101308d4248 0000000000000002
May 15 05:34:12 vault1 kernel: Call Trace:
May 15 05:34:12 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:12 vault1 kernel: [<ffffffff8023ad89>] prepare_to_wait+0x1b/0x5c
May 15 05:34:12 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:12 vault1 kernel: [<ffffffff80249a11>] unix_stream_recvmsg+0x233/0x548
May 15 05:34:12 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:12 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:12 vault1 kernel: [<ffffffff8040ae7a>] sock_aio_read+0x10c/0x124
May 15 05:34:12 vault1 kernel: [<ffffffff8020cf53>] do_sync_read+0xc9/0x10c
May 15 05:34:12 vault1 kernel: [<ffffffff8046210a>] unix_stream_connect+0x389/0x3ff
May 15 05:34:12 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:12 vault1 kernel: [<ffffffff8020e38c>] do_mmap_pgoff+0x65d/0x7c9
May 15 05:34:12 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:12 vault1 kernel: [<ffffffff8020b5b0>] vfs_read+0xde/0x173
May 15 05:34:12 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:34:12 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:12 vault1 kernel:
May 15 05:34:12 vault1 kernel: hald-addon-cp S ffff81012ec1cd78 0 3798 3725 (NOTLB)
May 15 05:34:12 vault1 kernel: ffff810125ba9b28 0000000000000046 0000000000000000 ffff81012f3dc040
May 15 05:34:12 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000003 ffff81012f3dc040
May 15 05:34:12 vault1 kernel: ffffffff8058a6e0 0000000000034e85 ffff81012f3dc248 0000000000000002
May 15 05:34:12 vault1 kernel: Call Trace:
May 15 05:34:12 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:12 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:12 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:34:12 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:12 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:12 vault1 kernel: [<ffffffff80249bf3>] unix_stream_recvmsg+0x415/0x548
May 15 05:34:12 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:12 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 15 05:34:12 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 15 05:34:12 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:12 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 15 05:34:12 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:12 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:34:12 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:34:12 vault1 kernel: [<ffffffff80208d64>] __handle_mm_fault+0x514/0xccf
May 15 05:34:12 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:12 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:12 vault1 kernel: [<ffffffff80265bc5>] do_page_fault+0x490/0x7cf
May 15 05:34:12 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:12 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:34:12 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:12 vault1 kernel:
May 15 05:34:12 vault1 kernel: hald-addon-ke S ffff81012f856118 0 3799 3725 (NOTLB)
May 15 05:34:12 vault1 kernel: ffff810125be9ea8 0000000000000046 0000000000000000 0000000000000000
May 15 05:34:12 vault1 kernel: 0000000000000000 0000000000000046 0000000000000001 ffff81012f7ef0c0
May 15 05:34:12 vault1 kernel: ffff810136b92040 0000000000069995 ffff81012f7ef2c8 000000012f7ef0c0
May 15 05:34:12 vault1 kernel: Call Trace:
May 15 05:34:12 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:12 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 15 05:34:12 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:12 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:12 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:34:12 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:34:12 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:12 vault1 kernel:
May 15 05:34:12 vault1 kernel: hald-addon-ke S ffff810230cee0d8 0 3802 3725 (NOTLB)
May 15 05:34:12 vault1 kernel: ffff8101259a3ea8 0000000000000046 0000000000000000 0000000000000000
May 15 05:34:12 vault1 kernel: 0000000000000000 0000000000000046 0000000000000001 ffff810130097040
May 15 05:34:12 vault1 kernel: ffff810230c4c0c0 00000000000a3b0d ffff810130097248 0000000330097040
May 15 05:34:12 vault1 kernel: Call Trace:
May 15 05:34:12 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:12 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 15 05:34:12 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:13 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:13 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:34:13 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:34:13 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:13 vault1 kernel:
May 15 05:34:13 vault1 kernel: hald-addon-st S ffff810230e444f8 0 3812 3725 (NOTLB)
May 15 05:34:13 vault1 kernel: ffff8101254bdeb8 0000000000000046 0000000000000000 ffffffff802638fc
May 15 05:34:13 vault1 kernel: ffff8100062dcbf8 ffff8100062dcbf0 0000000000000009 ffff81012ea19040
May 15 05:34:13 vault1 kernel: ffffffff8058a6e0 0000000000001e57 ffff81012ea19248 0000000000000292
May 15 05:34:13 vault1 kernel: Call Trace:
May 15 05:34:13 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:13 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 15 05:34:13 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 15 05:34:13 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 15 05:34:13 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 15 05:34:13 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:13 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 15 05:34:13 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:13 vault1 kernel:
May 15 05:34:13 vault1 kernel: smartd S ffff81012ec1c4b8 0 3876 1 (NOTLB)
May 15 05:34:13 vault1 kernel: ffff81012662beb8 0000000000000046 0000000000000000 ffffffff802638fc
May 15 05:34:13 vault1 kernel: ffff8100062dcbf8 ffff8100062dcbf0 0000000000000007 ffff81012fdbb040
May 15 05:34:13 vault1 kernel: ffffffff8058a6e0 0000000000002dd0 ffff81012fdbb248 0000000000000292
May 15 05:34:13 vault1 kernel: Call Trace:
May 15 05:34:13 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:13 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 15 05:34:13 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 15 05:34:13 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 15 05:34:13 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 15 05:34:13 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:13 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 15 05:34:13 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:13 vault1 kernel:
May 15 05:34:13 vault1 kernel: mingetty S ffff81012f0ba578 0 3879 1 (NOTLB)
May 15 05:34:13 vault1 kernel: ffff810126425dc8 0000000000000046 0000000000000000 ffff81012f9307f0
May 15 05:34:13 vault1 kernel: 0000000000000612 ffff81012f930080 0000000000000007 ffff81012f930080
May 15 05:34:13 vault1 kernel: ffff81012f342040 0000000000025b03 ffff81012f930288 0000000100000046
May 15 05:34:13 vault1 kernel: Call Trace:
May 15 05:34:13 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:13 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:13 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:13 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:34:13 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:13 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:13 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:34:13 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:34:13 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:34:13 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:13 vault1 kernel:
May 15 05:34:13 vault1 kernel: mingetty S ffff81012ec1d638 0 3880 1 (NOTLB)
May 15 05:34:13 vault1 kernel: ffff8101253a1dc8 0000000000000046 0000000000000002 0000000000000000
May 15 05:34:13 vault1 kernel: ffff81012f38f7f8 ffff81012f38f0c0 0000000000000006 ffff81012f38f0c0
May 15 05:34:13 vault1 kernel: ffff81012f930080 00000000001e63ba ffff81012f38f2c8 0000000100000046
May 15 05:34:13 vault1 kernel: Call Trace:
May 15 05:34:13 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:13 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:13 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:34:13 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:13 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:13 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:34:13 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:34:13 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:34:13 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:13 vault1 kernel:
May 15 05:34:13 vault1 kernel: mingetty S ffff81012e41d298 0 3881 1 (NOTLB)
May 15 05:34:13 vault1 kernel: ffff810125223dc8 0000000000000046 0000000000000000 ffff81012f3427b0
May 15 05:34:13 vault1 kernel: 0000000000000612 ffff81012f342040 0000000000000005 ffff81012f342040
May 15 05:34:13 vault1 kernel: ffff81012f9940c0 000000000016c887 ffff81012f342248 0000000100000046
May 15 05:34:13 vault1 kernel: Call Trace:
May 15 05:34:13 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:13 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:13 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:13 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:34:13 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:13 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:13 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:34:13 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:34:13 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:34:13 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:13 vault1 kernel:
May 15 05:34:13 vault1 kernel: mingetty S ffff81012effedb8 0 3882 1 (NOTLB)
May 15 05:34:13 vault1 kernel: ffff810125453dc8 0000000000000046 0000000000000000 ffff81012f994830
May 15 05:34:13 vault1 kernel: 0000000000000612 ffff81012f9940c0 0000000000000004 ffff81012f9940c0
May 15 05:34:14 vault1 kernel: ffff8101308d20c0 00000000000032cb ffff81012f9942c8 0000000100000046
May 15 05:34:14 vault1 kernel: Call Trace:
May 15 05:34:14 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:14 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:14 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:14 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:34:14 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:14 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:14 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:34:14 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:34:14 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:34:14 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:14 vault1 kernel:
May 15 05:34:14 vault1 kernel: mingetty S ffff81012f5bd258 0 3883 1 (NOTLB)
May 15 05:34:14 vault1 kernel: ffff81012d1bfdc8 0000000000000046 0000000000000000 ffff8101308d2830
May 15 05:34:14 vault1 kernel: 0000000000000612 ffff8101308d20c0 0000000000000003 ffff8101308d20c0
May 15 05:34:14 vault1 kernel: ffff81013006d0c0 00000000000c79ab ffff8101308d22c8 0000000100000046
May 15 05:34:14 vault1 kernel: Call Trace:
May 15 05:34:14 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:14 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:14 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:14 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:34:14 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:14 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:14 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:34:14 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:34:14 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:34:14 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:14 vault1 kernel:
May 15 05:34:14 vault1 kernel: mingetty S ffff81012f5bd6b8 0 3884 1 (NOTLB)
May 15 05:34:14 vault1 kernel: ffff810125513dc8 0000000000000046 0000000000000000 ffff81013006d830
May 15 05:34:14 vault1 kernel: 0000000000000612 ffff81013006d0c0 0000000000000002 ffff81013006d0c0
May 15 05:34:14 vault1 kernel: ffff810136b92040 000000000012884d ffff81013006d2c8 0000000100000046
May 15 05:34:14 vault1 kernel: Call Trace:
May 15 05:34:14 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:14 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:14 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 15 05:34:14 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:14 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:14 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 15 05:34:14 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:34:14 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:34:14 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:14 vault1 kernel:
May 15 05:34:14 vault1 kernel: sshd S ffff810230f84058 0 3943 3553 (NOTLB)
May 15 05:34:14 vault1 kernel: ffff8101256b9a18 0000000000000046 0000000000000000 0000000000000046
May 15 05:34:14 vault1 kernel: ffffffff8022edfd 0000000000000046 000000000000000a ffff810127218080
May 15 05:34:14 vault1 kernel: ffff810136b92040 0000000000015052 ffff810127218288 0000000127218080
May 15 05:34:14 vault1 kernel: Call Trace:
May 15 05:34:14 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 15 05:34:14 vault1 kernel: [<ffffffff8023a80a>] tty_ldisc_deref+0x62/0x75
May 15 05:34:14 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:14 vault1 kernel: [<ffffffff80226c98>] tty_poll+0x5f/0x6d
May 15 05:34:14 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:34:14 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:14 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:14 vault1 last message repeated 3 times
May 15 05:34:14 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:14 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 15 05:34:14 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:14 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 15 05:34:14 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 15 05:34:14 vault1 kernel: [<ffffffff8023a80a>] tty_ldisc_deref+0x62/0x75
May 15 05:34:14 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:14 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:34:14 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 15 05:34:14 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:14 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:14 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:14 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 15 05:34:14 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:14 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:14 vault1 kernel:
May 15 05:34:14 vault1 kernel: bash S ffff810230e44958 0 3945 3943 (NOTLB)
May 15 05:34:14 vault1 kernel: ffff81012571bec8 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:34:14 vault1 kernel: 0000000000000000 0000000200000002 0000000000000009 ffff810125ab70c0
May 15 05:34:14 vault1 kernel: ffffffff8058a6e0 00000000000a8ebb ffff810125ab72c8 0000000000000000
May 15 05:34:14 vault1 kernel: Call Trace:
May 15 05:34:14 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:14 vault1 kernel: [<ffffffff8022920b>] do_wait+0x94c/0xa75
May 15 05:34:14 vault1 kernel: [<ffffffff8022922e>] do_wait+0x96f/0xa75
May 15 05:34:14 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:15 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:15 vault1 kernel:
May 15 05:34:15 vault1 kernel: rpc.rquotad S ffff81012f0ba118 0 4049 1 (NOTLB)
May 15 05:34:15 vault1 kernel: ffff81012c6b5b28 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:34:15 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000001 ffff81012f00f080
May 15 05:34:15 vault1 kernel: ffff810136be9080 00000000000360c4 ffff81012f00f288 0000000200000000
May 15 05:34:15 vault1 kernel: Call Trace:
May 15 05:34:15 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:15 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:15 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:15 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 15 05:34:15 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 15 05:34:15 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:15 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:15 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:15 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 15 05:34:15 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:15 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 15 05:34:15 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:15 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 15 05:34:15 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 15 05:34:15 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:15 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 15 05:34:15 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:15 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:34:15 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 15 05:34:15 vault1 kernel: [<ffffffff80208d64>] __handle_mm_fault+0x514/0xccf
May 15 05:34:15 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:15 vault1 kernel: [<ffffffff80265bc5>] do_page_fault+0x490/0x7cf
May 15 05:34:15 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:15 vault1 kernel: [<ffffffff8021e472>] sys_close+0x2b/0xd1
May 15 05:34:15 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:15 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 15 05:34:15 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:15 vault1 kernel:
May 15 05:34:15 vault1 kernel: lockd S ffff81022eda4958 0 4052 1 (L-TLB)
May 15 05:34:15 vault1 kernel: ffff81022ff83e10 0000000000000046 0000000000000000 ffff81022f5c07f0
May 15 05:34:15 vault1 kernel: 0000000000000612 ffff81022f5c0080 0000000000000001 ffff81022f5c0080
May 15 05:34:15 vault1 kernel: ffffffff8058a6e0 00000000000041f8 ffff81022f5c0288 0000000000000046
May 15 05:34:15 vault1 kernel: Call Trace:
May 15 05:34:15 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:15 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:15 vault1 kernel: [<ffffffff883de24e>] :sunrpc:svc_recv+0x2b5/0x44f
May 15 05:34:15 vault1 kernel: [<ffffffff883de257>] :sunrpc:svc_recv+0x2be/0x44f
May 15 05:34:15 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:15 vault1 kernel: [<ffffffff8840b556>] :lockd:lockd+0x139/0x263
May 15 05:34:15 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:15 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:15 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:15 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:15 vault1 kernel: [<ffffffff802078c3>] kmem_cache_free+0x257/0x267
May 15 05:34:15 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:15 vault1 kernel: [<ffffffff8840b41d>] :lockd:lockd+0x0/0x263
May 15 05:34:15 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:15 vault1 kernel:
May 15 05:34:15 vault1 kernel: rpciod/0 S ffff81022eda4958 0 4053 19 (L-TLB)
May 15 05:34:15 vault1 kernel: ffff810124befea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 15 05:34:15 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810130851080
May 15 05:34:15 vault1 kernel: ffff81012f485080 0000000000000b30 ffff810130851288 0000000030851080
May 15 05:34:15 vault1 kernel: Call Trace:
May 15 05:34:15 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 15 05:34:15 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:15 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:15 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:15 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:15 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:15 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:15 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:15 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:15 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:15 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:15 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:15 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:15 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:15 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:15 vault1 kernel:
May 15 05:34:15 vault1 kernel: rpciod/1 S ffff810130b71a98 0 4054 19 (L-TLB)
May 15 05:34:15 vault1 kernel: ffff810124bc3ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:15 vault1 kernel: 0000000000000000 0000000000000000 0000000000000008 ffff81012f485080
May 15 05:34:15 vault1 kernel: ffff810136b92040 0000000000001672 ffff81012f485288 000000012f485080
May 15 05:34:15 vault1 kernel: Call Trace:
May 15 05:34:15 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:15 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:15 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:15 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:15 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:16 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:16 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:16 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:16 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:16 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:16 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:16 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:16 vault1 kernel:
May 15 05:34:16 vault1 kernel: rpciod/2 S ffff810130b70058 0 4055 19 (L-TLB)
May 15 05:34:16 vault1 kernel: ffff810124c43ea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:16 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81012552e040
May 15 05:34:16 vault1 kernel: ffff810136be9080 0000000000001a1e ffff81012552e248 000000022552e040
May 15 05:34:16 vault1 kernel: Call Trace:
May 15 05:34:16 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:16 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:16 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:16 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:16 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:16 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:16 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:16 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:16 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:16 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:16 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:34:16 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:16 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:16 vault1 kernel:
May 15 05:34:16 vault1 kernel: rpciod/3 S ffff81022eda4958 0 4056 19 (L-TLB)
May 15 05:34:16 vault1 kernel: ffff8101249edea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:16 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff8101252b5040
May 15 05:34:16 vault1 kernel: ffff810230c4c0c0 00000000000019be ffff8101252b5248 00000003252b5040
May 15 05:34:16 vault1 kernel: Call Trace:
May 15 05:34:16 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:16 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:16 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:16 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:16 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:16 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:16 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:16 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:16 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:16 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:16 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 15 05:34:16 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:16 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:16 vault1 kernel:
May 15 05:34:16 vault1 kernel: nfsd4 S ffff81012f5bc998 0 4057 19 (L-TLB)
May 15 05:34:16 vault1 kernel: ffff810124bffea0 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:16 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff810130eae0c0
May 15 05:34:16 vault1 kernel: ffff810136be9080 0000000000001a74 ffff810130eae2c8 0000000230eae0c0
May 15 05:34:16 vault1 kernel: Call Trace:
May 15 05:34:16 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:16 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 15 05:34:16 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 15 05:34:16 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 15 05:34:16 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:16 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:16 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 15 05:34:16 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:16 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:16 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:16 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 15 05:34:16 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:16 vault1 kernel:
May 15 05:34:16 vault1 kernel: nfsd D ffff810230f85a98 0 4058 1 (L-TLB)
May 15 05:34:16 vault1 kernel: ffff81022ff77420 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:16 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81022fa12040
May 15 05:34:16 vault1 kernel: ffffffff8058a6e0 00000000006928e3 ffff81022fa12248 000000002fa12858
May 15 05:34:16 vault1 kernel: Call Trace:
May 15 05:34:16 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:16 vault1 kernel: [<ffffffff802616cf>] io_schedule+0x3f/0x67
May 15 05:34:16 vault1 kernel: [<ffffffff8833680a>] :jfs:lbmAllocate+0x8a/0x114
May 15 05:34:16 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:16 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:16 vault1 kernel: [<ffffffff8833723a>] :jfs:lmNextPage+0xea/0x127
May 15 05:34:16 vault1 kernel: [<ffffffff88337508>] :jfs:lmWriteRecord+0x291/0x2d1
May 15 05:34:16 vault1 kernel: [<ffffffff88338340>] :jfs:lmLog+0x125/0x16a
May 15 05:34:16 vault1 kernel: [<ffffffff8833a17b>] :jfs:txCommit+0xcdd/0xef2
May 15 05:34:16 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 15 05:34:16 vault1 kernel: [<ffffffff88322175>] :jfs:jfs_commit_inode+0xab/0xf5
May 15 05:34:16 vault1 kernel: [<ffffffff88335683>] :jfs:extAlloc+0x3ce/0x423
May 15 05:34:16 vault1 kernel: [<ffffffff8832265c>] :jfs:jfs_get_block+0x167/0x237
May 15 05:34:16 vault1 kernel: [<ffffffff802e5c6c>] nobh_prepare_write+0xd3/0x458
May 15 05:34:16 vault1 kernel: [<ffffffff883224f5>] :jfs:jfs_get_block+0x0/0x237
May 15 05:34:16 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:16 vault1 kernel: [<ffffffff8020c722>] add_to_page_cache+0x2d/0x8c
May 15 05:34:16 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:34:17 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:17 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:34:17 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:17 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:34:17 vault1 kernel: [<ffffffff80210354>] generic_file_buffered_write+0x27b/0x77b
May 15 05:34:17 vault1 kernel: [<ffffffff8020e533>] current_fs_time+0x3b/0x40
May 15 05:34:17 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:17 vault1 kernel: [<ffffffff80216881>] __generic_file_aio_write_nolock+0x378/0x3eb
May 15 05:34:17 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 15 05:34:17 vault1 kernel: [<ffffffff8026212e>] __mutex_lock_slowpath+0x250/0x25b
May 15 05:34:17 vault1 kernel: [<ffffffff802224a4>] generic_file_aio_write+0x61/0xc1
May 15 05:34:17 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:34:17 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:34:17 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:17 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:17 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:34:17 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:17 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:34:17 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:34:17 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:34:17 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:17 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:17 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:17 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:34:17 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:34:17 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:34:17 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:34:17 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:34:17 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:34:17 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:34:17 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:34:17 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:34:17 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:17 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:17 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:17 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:17 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:34:17 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:17 vault1 kernel:
May 15 05:34:17 vault1 kernel: nfsd D ffff810230e444f8 0 4059 1 (L-TLB)
May 15 05:34:17 vault1 kernel: ffff81022f7b5520 0000000000000046 0000000000000000 0000000000000002
May 15 05:34:17 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff81022fd6e0c0
May 15 05:34:17 vault1 kernel: ffff810136b92040 000000000001166f ffff81022fd6e2c8 0000000100000046
May 15 05:34:17 vault1 kernel: Call Trace:
May 15 05:34:17 vault1 kernel: [<ffffffff802616cf>] io_schedule+0x3f/0x67
May 15 05:34:17 vault1 kernel: [<ffffffff88338540>] :jfs:lmGroupCommit+0x125/0x177
May 15 05:34:17 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:17 vault1 kernel: [<ffffffff8833a1eb>] :jfs:txCommit+0xd4d/0xef2
May 15 05:34:17 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 15 05:34:17 vault1 kernel: [<ffffffff88322175>] :jfs:jfs_commit_inode+0xab/0xf5
May 15 05:34:17 vault1 kernel: [<ffffffff88335683>] :jfs:extAlloc+0x3ce/0x423
May 15 05:34:17 vault1 kernel: [<ffffffff8832265c>] :jfs:jfs_get_block+0x167/0x237
May 15 05:34:17 vault1 kernel: [<ffffffff802e5c6c>] nobh_prepare_write+0xd3/0x458
May 15 05:34:17 vault1 kernel: [<ffffffff883224f5>] :jfs:jfs_get_block+0x0/0x237
May 15 05:34:17 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:17 vault1 kernel: [<ffffffff8020c722>] add_to_page_cache+0x2d/0x8c
May 15 05:34:17 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:34:17 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:17 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:34:17 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:17 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 15 05:34:17 vault1 kernel: [<ffffffff80210354>] generic_file_buffered_write+0x27b/0x77b
May 15 05:34:17 vault1 kernel: [<ffffffff8020e533>] current_fs_time+0x3b/0x40
May 15 05:34:17 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:17 vault1 kernel: [<ffffffff80216881>] __generic_file_aio_write_nolock+0x378/0x3eb
May 15 05:34:17 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 15 05:34:17 vault1 kernel: [<ffffffff8026212e>] __mutex_lock_slowpath+0x250/0x25b
May 15 05:34:17 vault1 kernel: [<ffffffff802224a4>] generic_file_aio_write+0x61/0xc1
May 15 05:34:17 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:34:17 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:34:17 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:17 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:17 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:34:17 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:17 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:34:17 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:34:17 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:34:17 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:17 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:17 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:17 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:34:17 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:34:18 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:34:18 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:34:18 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:34:18 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:34:18 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:34:18 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:34:18 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:34:18 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:18 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:18 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:18 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:18 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:34:18 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:18 vault1 kernel:
May 15 05:34:18 vault1 kernel: nfsd D ffff810230e444f8 0 4060 1 (L-TLB)
May 15 05:34:18 vault1 kernel: ffff8102302aba60 0000000000000046 0000000000000000 ffff81022fd3b040
May 15 05:34:18 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81022fd3b040
May 15 05:34:18 vault1 kernel: ffff810136b92040 000000000025e9d4 ffff81022fd3b248 000000012fd3b7b0
May 15 05:34:18 vault1 kernel: Call Trace:
May 15 05:34:18 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:18 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:18 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:34:18 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:34:18 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:18 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:34:18 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:34:18 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:18 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:18 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:34:18 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:18 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:34:18 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:34:18 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:34:18 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:18 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:18 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:18 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:34:18 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:34:18 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:34:18 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:34:18 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:34:18 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:34:18 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:34:18 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:34:18 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:34:18 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:18 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:18 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:18 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:18 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:34:18 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:18 vault1 kernel:
May 15 05:34:18 vault1 kernel: nfsd D ffff810230f84058 0 4061 1 (L-TLB)
May 15 05:34:18 vault1 kernel: ffff81022f37fa60 0000000000000046 0000000000000000 ffff81022f726080
May 15 05:34:18 vault1 kernel: 0000000000000001 ffffffff802a0de5 0000000000000009 ffff81022f726080
May 15 05:34:18 vault1 kernel: ffff810136be9080 00000000000302d3 ffff81022f726288 000000022f7267f0
May 15 05:34:18 vault1 kernel: Call Trace:
May 15 05:34:18 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:18 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:18 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:34:18 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:34:18 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:18 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:34:18 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:34:18 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:18 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:18 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:34:18 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:18 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:34:18 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:34:18 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:34:18 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:18 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:18 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:18 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:34:18 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:34:18 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:34:18 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:34:18 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:34:18 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:34:18 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:34:18 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:34:18 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:18 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:18 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:18 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:18 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:34:18 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:18 vault1 kernel:
May 15 05:34:19 vault1 kernel: nfsd D ffff810230f85a98 0 4062 1 (L-TLB)
May 15 05:34:19 vault1 kernel: ffff81022f42ba60 0000000000000046 000000000000052b ffff81022f5760c0
May 15 05:34:19 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81022f5760c0
May 15 05:34:19 vault1 kernel: ffff81022f1930c0 0000000000022d38 ffff81022f5762c8 000000002f576830
May 15 05:34:19 vault1 kernel: Call Trace:
May 15 05:34:19 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:19 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 15 05:34:19 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:19 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 15 05:34:19 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:34:19 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:34:19 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:19 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:34:19 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:34:19 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:19 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:19 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:34:19 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:19 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:34:19 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:34:19 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:34:19 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:19 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:19 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:19 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:34:19 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:34:19 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:34:19 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:34:19 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:34:19 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:34:19 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:34:19 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:34:19 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:34:19 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:19 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:19 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:19 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:19 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:34:19 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:19 vault1 kernel:
May 15 05:34:19 vault1 kernel: nfsd D ffff810230f85a98 0 4063 1 (L-TLB)
May 15 05:34:19 vault1 kernel: ffff81022fb05a60 0000000000000046 0000000000000000 ffff81022f615040
May 15 05:34:19 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81022f615040
May 15 05:34:19 vault1 kernel: ffffffff8058a6e0 00000000001cf979 ffff81022f615248 000000002f6157b0
May 15 05:34:19 vault1 kernel: Call Trace:
May 15 05:34:19 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:19 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:19 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:34:19 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:34:19 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:19 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:34:19 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:34:19 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:19 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:19 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:34:19 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:19 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:34:19 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:34:19 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:34:19 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:19 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:19 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:19 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:34:19 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:34:19 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:34:19 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:34:19 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:34:19 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:34:19 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:34:19 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:34:19 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:34:19 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:19 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:19 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:19 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:19 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:34:19 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:19 vault1 kernel:
May 15 05:34:19 vault1 kernel: nfsd D ffff810130b704b8 0 4064 1 (L-TLB)
May 15 05:34:19 vault1 kernel: ffff81022f0c1a60 0000000000000046 0000000000000000 ffff81022fa2c080
May 15 05:34:19 vault1 kernel: 0000000000000001 ffffffff802a0de5 0000000000000009 ffff81022fa2c080
May 15 05:34:19 vault1 kernel: ffff810230c4c0c0 000000000001d4dc ffff81022fa2c288 000000032fa2c7f0
May 15 05:34:19 vault1 kernel: Call Trace:
May 15 05:34:19 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:19 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:20 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:34:20 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:34:20 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:20 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:34:20 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:34:20 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:20 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:20 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:34:20 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:20 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:34:20 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:34:20 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:34:20 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:20 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:20 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:20 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:34:20 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:34:20 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:34:20 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:34:20 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:34:20 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:34:20 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:34:20 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:34:20 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:20 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:20 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:20 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:20 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:34:20 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:20 vault1 kernel:
May 15 05:34:20 vault1 kernel: nfsd D ffff810230f85a98 0 4065 1 (L-TLB)
May 15 05:34:20 vault1 kernel: ffff81022f19da60 0000000000000046 0000000000000000 ffff81022f1930c0
May 15 05:34:20 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81022f1930c0
May 15 05:34:20 vault1 kernel: ffffffff8058a6e0 0000000000018984 ffff81022f1932c8 000000002f193830
May 15 05:34:20 vault1 kernel: Call Trace:
May 15 05:34:20 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:20 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:20 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:34:20 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 15 05:34:20 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:20 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 15 05:34:20 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 15 05:34:20 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:20 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:20 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 15 05:34:20 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 15 05:34:20 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 15 05:34:20 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 15 05:34:20 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 15 05:34:20 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:20 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:20 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 15 05:34:20 vault1 kernel: [<ffffffff88425c8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 15 05:34:20 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 15 05:34:20 vault1 kernel: [<ffffffff883e29af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 15 05:34:20 vault1 kernel: [<ffffffff88426305>] :nfsd:nfsd_open+0x13c/0x16f
May 15 05:34:20 vault1 kernel: [<ffffffff8842653b>] :nfsd:nfsd_write+0xc5/0xe2
May 15 05:34:20 vault1 kernel: [<ffffffff8842d611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 15 05:34:20 vault1 kernel: [<ffffffff88422244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 15 05:34:20 vault1 kernel: [<ffffffff883db353>] :sunrpc:svc_process+0x3ea/0x6f3
May 15 05:34:20 vault1 kernel: [<ffffffff884227fe>] :nfsd:nfsd+0x194/0x2b2
May 15 05:34:20 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:20 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 15 05:34:20 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 15 05:34:20 vault1 kernel: [<ffffffff8842266a>] :nfsd:nfsd+0x0/0x2b2
May 15 05:34:20 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 15 05:34:20 vault1 kernel:
May 15 05:34:20 vault1 kernel: rpc.mountd S ffff81012f0ba9d8 0 4068 1 (NOTLB)
May 15 05:34:20 vault1 kernel: ffff810124c41a18 0000000000000046 0000000000000000 ffffffff802a0de5
May 15 05:34:20 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000009 ffff81012efa30c0
May 15 05:34:20 vault1 kernel: ffff810136b92040 000000000001006a ffff81012efa32c8 0000000100000000
May 15 05:34:20 vault1 kernel: Call Trace:
May 15 05:34:20 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:20 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:20 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:20 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 15 05:34:20 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:34:20 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:20 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:20 vault1 last message repeated 4 times
May 15 05:34:20 vault1 kernel: [<ffffffff8020b406>] kfree+0x1a8/0x274
May 15 05:34:21 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 15 05:34:21 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 15 05:34:21 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:21 vault1 kernel: [<ffffffff883e158b>] :sunrpc:cache_clean+0x12/0x208
May 15 05:34:21 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 15 05:34:21 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:21 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 15 05:34:21 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:21 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:34:21 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:21 vault1 kernel: [<ffffffff883e2363>] :sunrpc:cache_write+0x45/0xc0
May 15 05:34:21 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:21 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:21 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:21 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:21 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 15 05:34:21 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:21 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:21 vault1 kernel:
May 15 05:34:21 vault1 kernel: cat D ffff810230e45678 0 4862 3945 (NOTLB)
May 15 05:34:21 vault1 kernel: ffff810124e75bd8 0000000000000046 0000000000000000 0000000000000046
May 15 05:34:21 vault1 kernel: ffffffff80258fc6 ffff810125693040 0000000000000007 ffff810125693040
May 15 05:34:21 vault1 kernel: ffff810136b92040 0000000000075674 ffff810125693248 000000018029fe09
May 15 05:34:21 vault1 kernel: Call Trace:
May 15 05:34:21 vault1 kernel: [<ffffffff80258fc6>] generic_unplug_device+0x18/0x25
May 15 05:34:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 15 05:34:21 vault1 kernel: [<ffffffff80229a50>] sync_page+0x0/0x45
May 15 05:34:21 vault1 kernel: [<ffffffff802616cf>] io_schedule+0x3f/0x67
May 15 05:34:21 vault1 kernel: [<ffffffff80229a91>] sync_page+0x41/0x45
May 15 05:34:21 vault1 kernel: [<ffffffff80261813>] __wait_on_bit_lock+0x36/0x66
May 15 05:34:21 vault1 kernel: [<ffffffff80240387>] __lock_page+0x5e/0x64
May 15 05:34:21 vault1 kernel: [<ffffffff8029a120>] wake_bit_function+0x0/0x23
May 15 05:34:21 vault1 kernel: [<ffffffff8020c43e>] do_generic_mapping_read+0x1d9/0x40b
May 15 05:34:21 vault1 kernel: [<ffffffff8020d21f>] file_read_actor+0x0/0x18a
May 15 05:34:21 vault1 kernel: [<ffffffff8021713c>] generic_file_aio_read+0x15e/0x1a0
May 15 05:34:21 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:21 vault1 kernel: [<ffffffff8020cf53>] do_sync_read+0xc9/0x10c
May 15 05:34:21 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 15 05:34:21 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:21 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 15 05:34:21 vault1 kernel: [<ffffffff8020b86d>] fget_light+0x19/0x7b
May 15 05:34:21 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 15 05:34:21 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:21 vault1 kernel:
May 15 05:34:21 vault1 kernel: mkdir D ffff81012e41c118 0 4864 3945 (NOTLB)
May 15 05:34:21 vault1 kernel: ffff81016c38da18 0000000000000046 0000000000000000 ffff8101716db0c0
May 15 05:34:21 vault1 kernel: ffff8101716db908 ffffffff802a0de5 0000000000000007 ffff8101716db0c0
May 15 05:34:21 vault1 kernel: ffff810136b92040 000000000000b5af ffff8101716db2c8 00000001716db910
May 15 05:34:21 vault1 kernel: Call Trace:
May 15 05:34:21 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:21 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 15 05:34:21 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 15 05:34:21 vault1 kernel: [<ffffffff88335bd6>] :jfs:release_metapage+0x15b/0x167
May 15 05:34:21 vault1 kernel: [<ffffffff8833824a>] :jfs:lmLog+0x2f/0x16a
May 15 05:34:21 vault1 kernel: [<ffffffff883397fc>] :jfs:txCommit+0x35e/0xef2
May 15 05:34:21 vault1 kernel: [<ffffffff802078c3>] kmem_cache_free+0x257/0x267
May 15 05:34:21 vault1 kernel: [<ffffffff8832a524>] :jfs:diAllocAG+0x34e/0x9eb
May 15 05:34:21 vault1 kernel: [<ffffffff8026212e>] __mutex_lock_slowpath+0x250/0x25b
May 15 05:34:21 vault1 kernel: [<ffffffff8832b4c0>] :jfs:diAlloc+0x5d4/0x728
May 15 05:34:21 vault1 kernel: [<ffffffff88335028>] :jfs:ialloc+0x59/0x2a1
May 15 05:34:21 vault1 kernel: [<ffffffff88324138>] :jfs:jfs_mkdir+0x55/0x336
May 15 05:34:21 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:21 vault1 kernel: [<ffffffff802237ca>] d_alloc+0x170/0x1b8
May 15 05:34:21 vault1 kernel: [<ffffffff80223807>] d_alloc+0x1ad/0x1b8
May 15 05:34:21 vault1 kernel: [<ffffffff80223807>] d_alloc+0x1ad/0x1b8
May 15 05:34:21 vault1 kernel: [<ffffffff802d74ed>] vfs_mkdir+0xdb/0x14b
May 15 05:34:21 vault1 kernel: [<ffffffff802d79b1>] sys_mkdirat+0x97/0xd8
May 15 05:34:21 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:21 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:21 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:21 vault1 kernel:
May 15 05:34:21 vault1 kernel: sshd S ffff810230ceedf8 0 4865 3553 (NOTLB)
May 15 05:34:21 vault1 kernel: ffff810159291a18 0000000000000046 0000000000000000 0000000000000046
May 15 05:34:21 vault1 kernel: ffffffff8022edfd 0000000000000046 000000000000000a ffff810158cdf040
May 15 05:34:21 vault1 kernel: ffff810056f10040 000000000000d67f ffff810158cdf248 0000000158cdf040
May 15 05:34:21 vault1 kernel: Call Trace:
May 15 05:34:21 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 15 05:34:21 vault1 kernel: [<ffffffff8023a80a>] tty_ldisc_deref+0x62/0x75
May 15 05:34:21 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 15 05:34:21 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 15 05:34:21 vault1 kernel: [<ffffffff80226c98>] tty_poll+0x5f/0x6d
May 15 05:34:21 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 15 05:34:21 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 15 05:34:21 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 15 05:34:21 vault1 last message repeated 3 times
May 15 05:34:21 vault1 kernel: [<ffffffff802313cd>] release_sock+0x2f/0xd6
May 15 05:34:21 vault1 kernel: [<ffffffff8028de66>] local_bh_enable_ip+0xe4/0xea
May 15 05:34:21 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 15 05:34:21 vault1 kernel: [<ffffffff80227651>] tcp_sendmsg+0x9ab/0xab5
May 15 05:34:21 vault1 kernel: [<ffffffff80227651>] tcp_sendmsg+0x9ab/0xab5
May 15 05:34:21 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 15 05:34:21 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:21 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 15 05:34:21 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 15 05:34:21 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 15 05:34:21 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:21 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 15 05:34:21 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 15 05:34:21 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 15 05:34:21 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 15 05:34:22 vault1 kernel:
May 15 05:34:22 vault1 kernel:
May 15 05:34:22 vault1 kernel: Showing all locks held in the system:
May 15 05:34:22 vault1 kernel: 2 locks held by pdflush/299:
May 15 05:34:22 vault1 kernel: #0: (&type->s_umount_key#20){----}, at: [<ffffffff8024f2e2>] writeback_inodes+0x73/0xee
May 15 05:34:22 vault1 kernel: #1: (&jfs_ip->rdwrlock#2){----}, at: [<ffffffff8832253a>] jfs_get_block+0x45/0x237 [jfs]
May 15 05:34:22 vault1 kernel: 1 lock held by syslogd/3374:
May 15 05:34:22 vault1 kernel: #0: (&inode->i_mutex){--..}, at: [<ffffffff8024e516>] do_fsync+0x41/0xa4
May 15 05:34:22 vault1 kernel: 1 lock held by mingetty/3879:
May 15 05:34:22 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:34:22 vault1 kernel: 1 lock held by mingetty/3880:
May 15 05:34:22 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:34:22 vault1 kernel: 1 lock held by mingetty/3881:
May 15 05:34:22 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:34:22 vault1 kernel: 1 lock held by mingetty/3882:
May 15 05:34:22 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:34:22 vault1 kernel: 1 lock held by mingetty/3883:
May 15 05:34:22 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:34:22 vault1 kernel: 1 lock held by mingetty/3884:
May 15 05:34:22 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 15 05:34:22 vault1 kernel: 5 locks held by nfsd/4058:
May 15 05:34:22 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:34:22 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:22 vault1 kernel: #2: (&jfs_ip->rdwrlock#2){----}, at: [<ffffffff8832253a>] jfs_get_block+0x45/0x237 [jfs]
May 15 05:34:22 vault1 kernel: #3: (&jfs_ip->commit_mutex){--..}, at: [<ffffffff88322126>] jfs_commit_inode+0x5c/0xf5 [jfs]
May 15 05:34:22 vault1 kernel: #4: (&(log)->loglock){--..}, at: [<ffffffff8833824a>] lmLog+0x2f/0x16a [jfs]
May 15 05:34:22 vault1 kernel: 4 locks held by nfsd/4059:
May 15 05:34:22 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:34:22 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:22 vault1 kernel: #2: (&jfs_ip->rdwrlock#2){----}, at: [<ffffffff8832253a>] jfs_get_block+0x45/0x237 [jfs]
May 15 05:34:22 vault1 kernel: #3: (&jfs_ip->commit_mutex){--..}, at: [<ffffffff88322126>] jfs_commit_inode+0x5c/0xf5 [jfs]
May 15 05:34:22 vault1 kernel: 2 locks held by nfsd/4060:
May 15 05:34:22 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:34:22 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:22 vault1 kernel: 2 locks held by nfsd/4061:
May 15 05:34:22 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:34:22 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:22 vault1 kernel: 2 locks held by nfsd/4062:
May 15 05:34:22 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:34:22 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:22 vault1 kernel: 2 locks held by nfsd/4063:
May 15 05:34:22 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:34:22 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:22 vault1 kernel: 2 locks held by nfsd/4064:
May 15 05:34:22 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:34:22 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:22 vault1 kernel: 2 locks held by nfsd/4065:
May 15 05:34:22 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff884227e7>] nfsd+0x17d/0x2b2 [nfsd]
May 15 05:34:22 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 15 05:34:22 vault1 kernel: 6 locks held by mkdir/4864:
May 15 05:34:22 vault1 kernel: #0: (&inode->i_mutex/1){--..}, at: [<ffffffff80255f09>] lookup_create+0x23/0x84
May 15 05:34:22 vault1 kernel: #1: (&(imap->im_aglock[index])){--..}, at: [<ffffffff8832afd8>] diAlloc+0xec/0x728 [jfs]
May 15 05:34:22 vault1 kernel: #2: (&imap->im_freelock){--..}, at: [<ffffffff8832a295>] diAllocAG+0xbf/0x9eb [jfs]
May 15 05:34:22 vault1 kernel: #3: (&jfs_ip->rdwrlock/1){----}, at: [<ffffffff8832a2ca>] diAllocAG+0xf4/0x9eb [jfs]
May 15 05:34:22 vault1 kernel: #4: (&jfs_ip->commit_mutex){--..}, at: [<ffffffff8832a3b8>] diAllocAG+0x1e2/0x9eb [jfs]
May 15 05:34:22 vault1 kernel: #5: (&(log)->loglock){--..}, at: [<ffffffff8833824a>] lmLog+0x2f/0x16a [jfs]
May 15 05:34:22 vault1 kernel: 2 locks held by bash/4867:
May 15 05:34:22 vault1 kernel: #0: (sysrq_key_table_lock){....}, at: [<ffffffff80395b2a>] __handle_sysrq+0x22/0x128
May 15 05:34:22 vault1 kernel: #1: (tasklist_lock){..-?}, at: [<ffffffff8029ebd4>] debug_show_all_locks+0x3e/0x152
May 15 05:34:22 vault1 kernel:
May 15 05:34:22 vault1 kernel: =============================================
May 15 05:34:22 vault1 kernel:
[-- Attachment #3: Type: text/plain, Size: 286 bytes --]
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
[-- Attachment #4: Type: text/plain, Size: 140 bytes --]
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Apparent Deadlock with nfsd/jfs on 2.6.21.1 under bonnie.
[not found] ` <pan.2007.05.15.22.03.07.359081@linux.vnet.ibm.com>
@ 2007-05-15 22:06 ` Roger Heflin
2007-05-17 14:37 ` Roger Heflin
1 sibling, 0 replies; 9+ messages in thread
From: Roger Heflin @ 2007-05-15 22:06 UTC (permalink / raw)
To: Dave Kleikamp; +Cc: nfs, linux-kernel
Dave Kleikamp wrote:
> Sorry if I'm missing anyone on the reply, but my mail feed is messed up
> and I'm replying from the gmane archive.
>
> On Tue, 15 May 2007 09:08:25 -0500, Roger Heflin wrote:
>
>> Hello,
>>
>> Running 2.6.21.1 (FC6 Dist), with a RHEL client (client
>> appears to not be having issues) I am getting what I believe
>> is a deadlock on the server end. This is with JFS and
>> NFSD, I have not tested yet with a non-JFS filesystem,
>> though our customer indicated that they have duplicated it with
>> the ext3 filesystem.
>
> I don't have an answer to an ext3 deadlock, but this looks like a jfs
> problem that was recently fixed in linux-2.6.22-rc1. I had intended to
> send it to the stable kernel after it was picked up in mainline, but
> hadn't gotten to it yet.
>
> The patch is here:
> http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=05ec9e26be1f668ccba4ca54d9a4966c6208c611
>
Ok.
My customer reported that he though he had a ext3, so far I have
not been able to duplicate the ext3 hang.
If ext3 survives until tomorrow, I will retest unpatched jfs, and then
patch it and test again.
>> The basic setup is:
>> fiber channel array -> qlogic fiber card -> /dev/sdx -> LVM stripe ->
>> jfs -> nfs.
>>
>> Running bonnie on a NFS share has apparently produced a deadlock. I
>> have ran bonnie several times without having any issues, I don't believe
>> this is a HW issue, we have a couple of other machines configured with
>> slightly different HW and are also able to duplicate this problem on
>> those machines. There are no abnormal messages in dmesg or in the
>> messages file.
>>
>> After having the apparent deadlock I started a dd of a on the deadlocked
>> filesystem and according to vmstat 1 that was actually working, I then
>> did a "mkdir junk" on the deadlocked filesystem and that apparently put
>> the cat into a permanent "D" state. I will include the sysrq -t from
>> before the cat/mkdir and after the cat/mkdir.
>>
>> I believe I can duplicate this again, and other than the processes going
>> into the "D" state everything else seems to work. Other filesytems
>> appear to be functional, I can still login to the machine.
>>
>> Right now the machine is in the deadlocked state, and I will wait for
>> any suggestions of more data to collect or other tests to try.
>
> I haven't tried it on a locked-up system, but you may try waking up the
> [jfsIO] kernel thread with a signal. I'm not sure what signals may get
> through, since the thread doesn't specifically act on a signal.
>
I will try on the next lockup.
Roger
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Apparent Deadlock with nfsd/jfs on 2.6.21.1 under bonnie.
@ 2007-05-15 22:06 ` Roger Heflin
0 siblings, 0 replies; 9+ messages in thread
From: Roger Heflin @ 2007-05-15 22:06 UTC (permalink / raw)
To: Dave Kleikamp; +Cc: linux-kernel, nfs
Dave Kleikamp wrote:
> Sorry if I'm missing anyone on the reply, but my mail feed is messed up
> and I'm replying from the gmane archive.
>
> On Tue, 15 May 2007 09:08:25 -0500, Roger Heflin wrote:
>
>> Hello,
>>
>> Running 2.6.21.1 (FC6 Dist), with a RHEL client (client
>> appears to not be having issues) I am getting what I believe
>> is a deadlock on the server end. This is with JFS and
>> NFSD, I have not tested yet with a non-JFS filesystem,
>> though our customer indicated that they have duplicated it with
>> the ext3 filesystem.
>
> I don't have an answer to an ext3 deadlock, but this looks like a jfs
> problem that was recently fixed in linux-2.6.22-rc1. I had intended to
> send it to the stable kernel after it was picked up in mainline, but
> hadn't gotten to it yet.
>
> The patch is here:
> http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=05ec9e26be1f668ccba4ca54d9a4966c6208c611
>
Ok.
My customer reported that he though he had a ext3, so far I have
not been able to duplicate the ext3 hang.
If ext3 survives until tomorrow, I will retest unpatched jfs, and then
patch it and test again.
>> The basic setup is:
>> fiber channel array -> qlogic fiber card -> /dev/sdx -> LVM stripe ->
>> jfs -> nfs.
>>
>> Running bonnie on a NFS share has apparently produced a deadlock. I
>> have ran bonnie several times without having any issues, I don't believe
>> this is a HW issue, we have a couple of other machines configured with
>> slightly different HW and are also able to duplicate this problem on
>> those machines. There are no abnormal messages in dmesg or in the
>> messages file.
>>
>> After having the apparent deadlock I started a dd of a on the deadlocked
>> filesystem and according to vmstat 1 that was actually working, I then
>> did a "mkdir junk" on the deadlocked filesystem and that apparently put
>> the cat into a permanent "D" state. I will include the sysrq -t from
>> before the cat/mkdir and after the cat/mkdir.
>>
>> I believe I can duplicate this again, and other than the processes going
>> into the "D" state everything else seems to work. Other filesytems
>> appear to be functional, I can still login to the machine.
>>
>> Right now the machine is in the deadlocked state, and I will wait for
>> any suggestions of more data to collect or other tests to try.
>
> I haven't tried it on a locked-up system, but you may try waking up the
> [jfsIO] kernel thread with a signal. I'm not sure what signals may get
> through, since the thread doesn't specifically act on a signal.
>
I will try on the next lockup.
Roger
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Apparent Deadlock with nfsd/jfs on 2.6.21.1 under bonnie.
[not found] ` <pan.2007.05.15.22.03.07.359081@linux.vnet.ibm.com>
2007-05-15 22:06 ` Roger Heflin
@ 2007-05-17 14:37 ` Roger Heflin
2007-05-17 14:48 ` Dave Kleikamp
1 sibling, 1 reply; 9+ messages in thread
From: Roger Heflin @ 2007-05-17 14:37 UTC (permalink / raw)
To: Dave Kleikamp; +Cc: linux-kernel, nfs
Dave Kleikamp wrote:
>
> I don't have an answer to an ext3 deadlock, but this looks like a jfs
> problem that was recently fixed in linux-2.6.22-rc1. I had intended to
> send it to the stable kernel after it was picked up in mainline, but
> hadn't gotten to it yet.
>
> The patch is here:
> http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=05ec9e26be1f668ccba4ca54d9a4966c6208c611
>
Dave,
That appears to have fixed the JFS hangup.
MTBF before was about 1 hour, under the same test I am over 20 hours
and things appear to still be holding together.
Roger
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Apparent Deadlock with nfsd/jfs on 2.6.21.1 under bonnie.
2007-05-17 14:37 ` Roger Heflin
@ 2007-05-17 14:48 ` Dave Kleikamp
2007-05-29 17:16 ` Roger Heflin
0 siblings, 1 reply; 9+ messages in thread
From: Dave Kleikamp @ 2007-05-17 14:48 UTC (permalink / raw)
To: Roger Heflin; +Cc: linux-kernel, nfs
On Thu, 2007-05-17 at 09:37 -0500, Roger Heflin wrote:
> Dave Kleikamp wrote:
>
> >
> > I don't have an answer to an ext3 deadlock, but this looks like a jfs
> > problem that was recently fixed in linux-2.6.22-rc1. I had intended to
> > send it to the stable kernel after it was picked up in mainline, but
> > hadn't gotten to it yet.
> >
> > The patch is here:
> > http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=05ec9e26be1f668ccba4ca54d9a4966c6208c611
> >
>
> Dave,
>
> That appears to have fixed the JFS hangup.
>
> MTBF before was about 1 hour, under the same test I am over 20 hours
> and things appear to still be holding together.
Great. The patch is queued for the 2.6.21 stable tree now.
Thanks,
Shaggy
--
David Kleikamp
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Apparent Deadlock with nfsd/jfs on 2.6.21.1 under bonnie.
2007-05-17 14:48 ` Dave Kleikamp
@ 2007-05-29 17:16 ` Roger Heflin
2007-05-29 18:10 ` Dave Kleikamp
0 siblings, 1 reply; 9+ messages in thread
From: Roger Heflin @ 2007-05-29 17:16 UTC (permalink / raw)
To: Dave Kleikamp; +Cc: linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]
Dave Kleikamp wrote:
> On Thu, 2007-05-17 at 09:37 -0500, Roger Heflin wrote:
>> Dave Kleikamp wrote:
>>
>>> I don't have an answer to an ext3 deadlock, but this looks like a jfs
>>> problem that was recently fixed in linux-2.6.22-rc1. I had intended to
>>> send it to the stable kernel after it was picked up in mainline, but
>>> hadn't gotten to it yet.
>>>
>>> The patch is here:
>>> http://git.kernel.org/gitweb.cgi?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=05ec9e26be1f668ccba4ca54d9a4966c6208c611
>>>
>> Dave,
>>
>> That appears to have fixed the JFS hangup.
>>
>> MTBF before was about 1 hour, under the same test I am over 20 hours
>> and things appear to still be holding together.
>
> Great. The patch is queued for the 2.6.21 stable tree now.
>
> Thanks,
> Shaggy
Dave,
Apparently there appears to be another different similar lockup,
The MTBF has risen from 1-2 hours without that patch to >100 hours,
so I am fairly sure the patch did correct the original lockup, or
at the very least make it a lot less likely.
I hit the machine across NFS for 5 days before it deadlocked, before
the patch I could only get an hour or two (2-4 different tries).
Given that pdflush is "D" it does not appear to be an NFS issue.
Included is the sysrq-t.
This is with 2.6.21.1 + the JFSIO patch.
[-- Attachment #2: jdb.lockup.txt --]
[-- Type: text/plain, Size: 233700 bytes --]
May 29 11:58:20 vault1 kernel: SysRq : Show State
May 29 11:58:20 vault1 kernel:
May 29 11:58:20 vault1 kernel: free sibling
May 29 11:58:20 vault1 kernel: task PC stack pid father child younger older
May 29 11:58:20 vault1 kernel: init S ffff81011f96fa98 0 1 0 (NOTLB)
May 29 11:58:20 vault1 kernel: ffff81011fc89a18 0000000000000046 0000000000000000 ffff810037e10040
May 29 11:58:20 vault1 kernel: ffff810037e10040 ffffffff802638fc 000000000000000a ffff810037e10040
May 29 11:58:20 vault1 kernel: ffff81011fc78040 0000000000006e1f ffff810037e10248 0000000137e28000
May 29 11:58:20 vault1 kernel: Call Trace:
May 29 11:58:20 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:20 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 29 11:58:20 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:20 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:20 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:20 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:20 vault1 kernel: [<ffffffff80209596>] __d_lookup+0x77/0x110
May 29 11:58:20 vault1 kernel: [<ffffffff802095f2>] __d_lookup+0xd3/0x110
May 29 11:58:20 vault1 kernel: [<ffffffff802635ce>] _spin_unlock+0x17/0x20
May 29 11:58:20 vault1 kernel: [<ffffffff802095f2>] __d_lookup+0xd3/0x110
May 29 11:58:20 vault1 kernel: [<ffffffff8020cff9>] do_lookup+0x63/0x1ae
May 29 11:58:20 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 29 11:58:20 vault1 kernel: [<ffffffff8020a2b6>] __link_path_walk+0xc87/0xde9
May 29 11:58:20 vault1 kernel: [<ffffffff8028dc04>] _local_bh_enable+0xbb/0xd0
May 29 11:58:20 vault1 kernel: [<ffffffff8022d806>] mntput_no_expire+0x1c/0x96
May 29 11:58:20 vault1 kernel: [<ffffffff8020ed97>] link_path_walk+0xce/0xe0
May 29 11:58:20 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:20 vault1 kernel: [<ffffffff8022d806>] mntput_no_expire+0x1c/0x96
May 29 11:58:20 vault1 kernel: [<ffffffff8021b2bb>] cp_new_stat+0xe5/0xfd
May 29 11:58:20 vault1 kernel: [<ffffffff80216b98>] sys_select+0x15a/0x183
May 29 11:58:20 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:20 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:20 vault1 kernel:
May 29 11:58:20 vault1 kernel: migration/0 S ffff81011f2b9218 0 2 1 (L-TLB)
May 29 11:58:20 vault1 kernel: ffff810037e1bec0 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:20 vault1 kernel: 0000000100000000 0000000000000002 0000000000000001 ffff81011fc6f040
May 29 11:58:20 vault1 kernel: ffffffff8058a6e0 0000000000000f63 ffff81011fc6f248 0000000000000000
May 29 11:58:20 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff80243daa>] migration_thread+0x1a2/0x23f
May 29 11:58:21 vault1 kernel: [<ffffffff80243c08>] migration_thread+0x0/0x23f
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:21 vault1 kernel: [<ffffffff80356044>] vgacon_cursor+0x0/0x1a5
May 29 11:58:21 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:21 vault1 kernel:
May 29 11:58:21 vault1 kernel: ksoftirqd/0 S ffff81007fe18098 0 3 1 (L-TLB)
May 29 11:58:21 vault1 kernel: ffff810037e1ff00 0000000000000046 0000000000000000 ffff81011fc72080
May 29 11:58:21 vault1 kernel: ffff81011fc72080 00000000000721d2 0000000000000009 ffff81011fc72080
May 29 11:58:21 vault1 kernel: ffffffff8058a6e0 000000000043167f ffff81011fc72288 00000000805ea150
May 29 11:58:21 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff8026b5d6>] do_softirq+0x90/0x94
May 29 11:58:21 vault1 kernel: [<ffffffff8028deb7>] ksoftirqd+0x0/0xbf
May 29 11:58:21 vault1 kernel: [<ffffffff8028def2>] ksoftirqd+0x3b/0xbf
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:21 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:21 vault1 kernel:
May 29 11:58:21 vault1 kernel: watchdog/0 S ffff81007fe18098 0 4 1 (L-TLB)
May 29 11:58:21 vault1 kernel: ffff810037e21ef0 0000000000000046 0000000000000000 ffffffff8028dc04
May 29 11:58:21 vault1 kernel: 0000000000000007 0000000000000001 0000000000000001 ffff81011fc730c0
May 29 11:58:21 vault1 kernel: ffffffff8058a6e0 000000000000034a ffff81011fc732c8 0000000000000000
May 29 11:58:21 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff8028dc04>] _local_bh_enable+0xbb/0xd0
May 29 11:58:21 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff802b7437>] watchdog+0x4f/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:21 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:21 vault1 kernel:
May 29 11:58:21 vault1 kernel: migration/1 S ffff81011f2b9218 0 5 1 (L-TLB)
May 29 11:58:21 vault1 kernel: ffff810037e23ec0 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:21 vault1 kernel: 0000000100000000 0000000000000002 0000000000000001 ffff81011fc74040
May 29 11:58:21 vault1 kernel: ffff81011fc78040 00000000000013c6 ffff81011fc74248 0000000100000000
May 29 11:58:21 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff80243daa>] migration_thread+0x1a2/0x23f
May 29 11:58:21 vault1 kernel: [<ffffffff80243c08>] migration_thread+0x0/0x23f
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:21 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:21 vault1 kernel:
May 29 11:58:21 vault1 kernel: ksoftirqd/1 S ffff81007f1149d8 0 6 1 (L-TLB)
May 29 11:58:21 vault1 kernel: ffff81000428bf00 0000000000000046 0000000000000000 ffff81011fc75080
May 29 11:58:21 vault1 kernel: ffff81011fc75080 00000000000ca563 000000000000000a ffff81011fc75080
May 29 11:58:21 vault1 kernel: ffff81011fc78040 0000000000000c5d ffff81011fc75288 00000001806bc060
May 29 11:58:21 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff8026b5d6>] do_softirq+0x90/0x94
May 29 11:58:21 vault1 kernel: [<ffffffff8028deb7>] ksoftirqd+0x0/0xbf
May 29 11:58:21 vault1 kernel: [<ffffffff8028def2>] ksoftirqd+0x3b/0xbf
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:21 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:21 vault1 kernel:
May 29 11:58:21 vault1 kernel: watchdog/1 S ffff81007f1149d8 0 7 1 (L-TLB)
May 29 11:58:21 vault1 kernel: ffff81000428def0 0000000000000046 0000000000000000 0000000000000001
May 29 11:58:21 vault1 kernel: ffff81000428c000 ffff81008354ee00 0000000000000008 ffff81011fc760c0
May 29 11:58:21 vault1 kernel: ffff81011fc78040 0000000000000336 ffff81011fc762c8 0000000100000001
May 29 11:58:21 vault1 kernel: Call Trace:
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff802b7437>] watchdog+0x4f/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:21 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:21 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:21 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:21 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff802c038d>] next_zone+0x23/0x24
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:22 vault1 kernel: migration/2 S ffff81011f96e058 0 8 1 (L-TLB)
May 29 11:58:22 vault1 kernel: ffff8100042b1ec0 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:22 vault1 kernel: 0000000100000000 0000000000000002 0000000000000001 ffff81011fc7a080
May 29 11:58:22 vault1 kernel: ffff81007ff1d080 000000000000190d ffff81011fc7a288 0000000200000000
May 29 11:58:22 vault1 kernel: Call Trace:
May 29 11:58:22 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff80243daa>] migration_thread+0x1a2/0x23f
May 29 11:58:22 vault1 kernel: [<ffffffff80243c08>] migration_thread+0x0/0x23f
May 29 11:58:22 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:22 vault1 kernel: ksoftirqd/2 S ffff81007e8de9d8 0 9 1 (L-TLB)
May 29 11:58:22 vault1 kernel: ffff8100042b3f00 0000000000000046 0000000000000000 ffff81011fccd0c0
May 29 11:58:22 vault1 kernel: ffff81011fccd0c0 00000000000d838b 000000000000000a ffff81011fccd0c0
May 29 11:58:22 vault1 kernel: ffff81011fccf080 0000000000000a83 ffff81011fccd2c8 00000002042b3ef0
May 29 11:58:22 vault1 kernel: Call Trace:
May 29 11:58:22 vault1 kernel: [<ffffffff8026b5d6>] do_softirq+0x90/0x94
May 29 11:58:22 vault1 kernel: [<ffffffff8028deb7>] ksoftirqd+0x0/0xbf
May 29 11:58:22 vault1 kernel: [<ffffffff8028def2>] ksoftirqd+0x3b/0xbf
May 29 11:58:22 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:22 vault1 kernel: watchdog/2 S ffff81007fe19218 0 10 1 (L-TLB)
May 29 11:58:22 vault1 kernel: ffff8100042b5ef0 0000000000000046 0000000000000000 0000000000000001
May 29 11:58:22 vault1 kernel: ffff8100042b4000 ffff81010255df00 0000000000000007 ffff81011fcce040
May 29 11:58:22 vault1 kernel: ffff81011fccf080 000000000000024f ffff81011fcce248 0000000200000001
May 29 11:58:22 vault1 kernel: Call Trace:
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:22 vault1 kernel: [<ffffffff802b7437>] watchdog+0x4f/0x5f
May 29 11:58:22 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:22 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff802c038d>] next_zone+0x23/0x24
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:22 vault1 kernel: migration/3 S ffff81007ebd24b8 0 11 1 (L-TLB)
May 29 11:58:22 vault1 kernel: ffff8100042dbec0 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:22 vault1 kernel: 0000000100000000 0000000000000002 0000000000000001 ffff81011fcd10c0
May 29 11:58:22 vault1 kernel: ffff81011fd360c0 0000000000001383 ffff81011fcd12c8 0000000300000000
May 29 11:58:22 vault1 kernel: Call Trace:
May 29 11:58:22 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff80243daa>] migration_thread+0x1a2/0x23f
May 29 11:58:22 vault1 kernel: [<ffffffff80243c08>] migration_thread+0x0/0x23f
May 29 11:58:22 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:22 vault1 kernel: ksoftirqd/3 S ffff81007e8de9d8 0 12 1 (L-TLB)
May 29 11:58:22 vault1 kernel: ffff8100042ddf00 0000000000000046 0000000000000000 ffff81011fd34040
May 29 11:58:22 vault1 kernel: ffff81007fc350c0 000000000022a171 000000000000000a ffff81011fd34040
May 29 11:58:22 vault1 kernel: ffff81011fd360c0 00000000000007f8 ffff81011fd34248 00000003042ddfd8
May 29 11:58:22 vault1 kernel: Call Trace:
May 29 11:58:22 vault1 kernel: [<ffffffff8026b5d6>] do_softirq+0x90/0x94
May 29 11:58:22 vault1 kernel: [<ffffffff8028deb7>] ksoftirqd+0x0/0xbf
May 29 11:58:22 vault1 kernel: [<ffffffff8028def2>] ksoftirqd+0x3b/0xbf
May 29 11:58:22 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:22 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:22 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:22 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:22 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:22 vault1 kernel:
May 29 11:58:23 vault1 kernel: watchdog/3 S ffff810037d4ba98 0 13 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff8100042e1ef0 0000000000000046 0000000000000000 0000000000000001
May 29 11:58:23 vault1 kernel: ffff8100042e0000 ffff810102568d00 0000000000000001 ffff81011fd35080
May 29 11:58:23 vault1 kernel: ffff81011fd360c0 00000000000002b1 ffff81011fd35288 0000000300000001
May 29 11:58:23 vault1 kernel: Call Trace:
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:23 vault1 kernel: [<ffffffff802b7437>] watchdog+0x4f/0x5f
May 29 11:58:23 vault1 kernel: [<ffffffff802b73e8>] watchdog+0x0/0x5f
May 29 11:58:23 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:23 vault1 kernel: [<ffffffff802c038d>] next_zone+0x23/0x24
May 29 11:58:23 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:23 vault1 kernel:
May 29 11:58:23 vault1 kernel: events/0 S ffff81007e8de118 0 14 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff81007fdfbea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:23 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fa61040
May 29 11:58:23 vault1 kernel: ffffffff8058a6e0 0000000000009929 ffff81011fa61248 000000001fa61040
May 29 11:58:23 vault1 kernel: Call Trace:
May 29 11:58:23 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:23 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:23 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:23 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:23 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:23 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:23 vault1 kernel:
May 29 11:58:23 vault1 kernel: events/1 S ffff81007e837b18 0 15 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff810004303ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:23 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fa63080
May 29 11:58:23 vault1 kernel: ffff81011fc78040 000000000000aa1b ffff81011fa63288 000000011fa63080
May 29 11:58:23 vault1 kernel: Call Trace:
May 29 11:58:23 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:23 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:23 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:23 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:23 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:23 vault1 kernel:
May 29 11:58:23 vault1 kernel: events/2 S ffff81007ebd3638 0 16 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff81007fde9ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:23 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fa640c0
May 29 11:58:23 vault1 kernel: ffff81011fccf080 000000000001434a ffff81011fa642c8 000000021fa640c0
May 29 11:58:23 vault1 kernel: Call Trace:
May 29 11:58:23 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:23 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:23 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:23 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:23 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:23 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:23 vault1 kernel:
May 29 11:58:23 vault1 kernel: events/3 S ffff810037d4ba98 0 17 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff8100042bfea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:23 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fa65040
May 29 11:58:23 vault1 kernel: ffff81011fd360c0 000000000000f2ba ffff81011fa65248 000000031fa65040
May 29 11:58:23 vault1 kernel: Call Trace:
May 29 11:58:23 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:23 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:23 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:23 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:23 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:23 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:23 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:23 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:23 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:23 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:23 vault1 kernel:
May 29 11:58:23 vault1 kernel: khelper S ffff81007f114118 0 18 1 (L-TLB)
May 29 11:58:23 vault1 kernel: ffff810037d39ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:23 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fa66080
May 29 11:58:23 vault1 kernel: ffff81011f2ce080 0000000000007797 ffff81011fa66288 000000001fa66080
May 29 11:58:24 vault1 kernel: Call Trace:
May 29 11:58:24 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:24 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:24 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:24 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:24 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:24 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:24 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:24 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:24 vault1 kernel:
May 29 11:58:24 vault1 kernel: kthread S ffff81007f1149d8 0 19 1 (L-TLB)
May 29 11:58:24 vault1 kernel: ffff81011fa6fea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:24 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007fdf3080
May 29 11:58:24 vault1 kernel: ffff81011fc78040 000000000000199a ffff81007fdf3288 000000017fdf3080
May 29 11:58:24 vault1 kernel: Call Trace:
May 29 11:58:24 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:24 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:24 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:24 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:24 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:24 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:24 vault1 kernel:
May 29 11:58:24 vault1 kernel: kblockd/0 S ffff81011f2b8958 0 59 19 (L-TLB)
May 29 11:58:24 vault1 kernel: ffff81011fe7fea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:24 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fe24040
May 29 11:58:24 vault1 kernel: ffffffff8058a6e0 000000000000324c ffff81011fe24248 000000001fe24040
May 29 11:58:24 vault1 kernel: Call Trace:
May 29 11:58:24 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:24 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:24 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:24 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:24 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:24 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:24 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:24 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:24 vault1 kernel:
May 29 11:58:24 vault1 kernel: kblockd/1 S ffff81011f2b8958 0 60 19 (L-TLB)
May 29 11:58:24 vault1 kernel: ffff81011fb81ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:24 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fe23080
May 29 11:58:24 vault1 kernel: ffff81011fc78040 000000000000334b ffff81011fe23288 000000011fe23080
May 29 11:58:24 vault1 kernel: Call Trace:
May 29 11:58:24 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:24 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:24 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:24 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:24 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:24 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:24 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:24 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:24 vault1 kernel:
May 29 11:58:24 vault1 kernel: kblockd/2 S ffff81007e8de9d8 0 61 19 (L-TLB)
May 29 11:58:24 vault1 kernel: ffff81011fb83ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:24 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011fe220c0
May 29 11:58:24 vault1 kernel: ffff81007e4c5080 000000000001120f ffff81011fe222c8 000000021fe220c0
May 29 11:58:24 vault1 kernel: Call Trace:
May 29 11:58:24 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:24 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:24 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:24 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:24 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:24 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:24 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:24 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:24 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:24 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:24 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:24 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:24 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:24 vault1 kernel:
May 29 11:58:24 vault1 kernel: kblockd/3 S ffff81007e8de9d8 0 62 19 (L-TLB)
May 29 11:58:25 vault1 kernel: ffff81007fde5ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:25 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff810037e3e0c0
May 29 11:58:25 vault1 kernel: ffff81011fd360c0 0000000000007674 ffff810037e3e2c8 0000000337e3e0c0
May 29 11:58:25 vault1 kernel: Call Trace:
May 29 11:58:25 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:25 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:25 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:25 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:25 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:25 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:25 vault1 kernel:
May 29 11:58:25 vault1 kernel: kacpid S ffffffff8058a280 0 63 19 (L-TLB)
May 29 11:58:25 vault1 kernel: ffff8100042d1ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:25 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff810004281040
May 29 11:58:25 vault1 kernel: ffff81011fccf080 0000000000000ff3 ffff810004281248 0000000204281040
May 29 11:58:25 vault1 kernel: Call Trace:
May 29 11:58:25 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:25 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:25 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:25 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:25 vault1 kernel: [<ffffffff8031ff4a>] selinux_d_instantiate+0x0/0x14
May 29 11:58:25 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:25 vault1 kernel:
May 29 11:58:25 vault1 kernel: cqueue/0 S ffffffff8058a280 0 217 19 (L-TLB)
May 29 11:58:25 vault1 kernel: ffff81007fdd5ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:25 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007fdca080
May 29 11:58:25 vault1 kernel: ffff81007fdf3080 0000000000001336 ffff81007fdca288 000000007fdca080
May 29 11:58:25 vault1 kernel: Call Trace:
May 29 11:58:25 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:25 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:25 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:25 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:25 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:25 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:25 vault1 kernel:
May 29 11:58:25 vault1 kernel: cqueue/1 S ffffffff8058a280 0 218 19 (L-TLB)
May 29 11:58:25 vault1 kernel: ffff81007fdd7ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:25 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007fdcb0c0
May 29 11:58:25 vault1 kernel: ffff81011fc78040 00000000000014e8 ffff81007fdcb2c8 000000017fdcb0c0
May 29 11:58:25 vault1 kernel: Call Trace:
May 29 11:58:25 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:25 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:25 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:25 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:25 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:25 vault1 kernel:
May 29 11:58:25 vault1 kernel: cqueue/2 S ffffffff8058a280 0 219 19 (L-TLB)
May 29 11:58:25 vault1 kernel: ffff81007fdd9ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:25 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007fdcd040
May 29 11:58:25 vault1 kernel: ffff81011fccf080 00000000000012c0 ffff81007fdcd248 000000027fdcd040
May 29 11:58:25 vault1 kernel: Call Trace:
May 29 11:58:25 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:25 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:25 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:25 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:25 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:26 vault1 kernel: cqueue/3 S ffffffff8058a280 0 220 19 (L-TLB)
May 29 11:58:26 vault1 kernel: ffff81007fdddea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:26 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007fdce080
May 29 11:58:26 vault1 kernel: ffff81011fd360c0 0000000000001847 ffff81007fdce288 000000037fdce080
May 29 11:58:26 vault1 kernel: Call Trace:
May 29 11:58:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:26 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:26 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:26 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:26 vault1 kernel: ksuspend_usbd S ffffffff8058a280 0 221 19 (L-TLB)
May 29 11:58:26 vault1 kernel: ffff81007fddfea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:26 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007fdd00c0
May 29 11:58:26 vault1 kernel: ffff81011fccf080 00000000000011b5 ffff81007fdd02c8 000000027fdd00c0
May 29 11:58:26 vault1 kernel: Call Trace:
May 29 11:58:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:26 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:26 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:26 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:26 vault1 kernel: khubd S ffff81011f96fa98 0 224 19 (L-TLB)
May 29 11:58:26 vault1 kernel: ffff810004361e20 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:26 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff810004342040
May 29 11:58:26 vault1 kernel: ffff81011fc72080 00000000007498c5 ffff810004342248 0000000004342040
May 29 11:58:26 vault1 kernel: Call Trace:
May 29 11:58:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:26 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:26 vault1 kernel: [<ffffffff803d6dc8>] hub_thread+0xac0/0xb3c
May 29 11:58:26 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:26 vault1 kernel: [<ffffffff803d6308>] hub_thread+0x0/0xb3c
May 29 11:58:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:26 vault1 kernel: kseriod S ffff810037d4ba98 0 226 19 (L-TLB)
May 29 11:58:26 vault1 kernel: ffff810004365ec0 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:26 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810004344080
May 29 11:58:26 vault1 kernel: ffffffff8058a6e0 000000000000270d ffff810004344288 0000000004344080
May 29 11:58:26 vault1 kernel: Call Trace:
May 29 11:58:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:26 vault1 kernel: [<ffffffff803eaaea>] serio_thread+0x2a5/0x2f8
May 29 11:58:26 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:26 vault1 kernel: [<ffffffff803ea845>] serio_thread+0x0/0x2f8
May 29 11:58:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff8031ff4a>] selinux_d_instantiate+0x0/0x14
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:26 vault1 kernel: kswapd0 S ffff81007e836538 0 283 19 (L-TLB)
May 29 11:58:26 vault1 kernel: ffff810037e6fe20 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:26 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810037fe8080
May 29 11:58:26 vault1 kernel: ffff81006b4ed040 00000000000b1784 ffff810037fe8288 0000000037fe8080
May 29 11:58:26 vault1 kernel: Call Trace:
May 29 11:58:26 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:26 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:26 vault1 kernel: [<ffffffff802560c6>] kswapd+0xea/0x469
May 29 11:58:26 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:26 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:26 vault1 kernel: [<ffffffff80255fdc>] kswapd+0x0/0x469
May 29 11:58:26 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:26 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:26 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:26 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:26 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:26 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:26 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:26 vault1 kernel:
May 29 11:58:27 vault1 kernel: kswapd1 S ffff81011febc958 0 284 19 (L-TLB)
May 29 11:58:27 vault1 kernel: ffff81007fce9e20 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:27 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810037fe70c0
May 29 11:58:27 vault1 kernel: ffff81007af01080 00000000000debec ffff810037fe72c8 0000000237fe70c0
May 29 11:58:27 vault1 kernel: Call Trace:
May 29 11:58:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff802560c6>] kswapd+0xea/0x469
May 29 11:58:27 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff80255fdc>] kswapd+0x0/0x469
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:27 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:27 vault1 kernel:
May 29 11:58:27 vault1 kernel: aio/0 S ffffffff8058a280 0 285 19 (L-TLB)
May 29 11:58:27 vault1 kernel: ffff81007fcedea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000003 ffff810037fe6040
May 29 11:58:27 vault1 kernel: ffff81007fdf3080 0000000000001520 ffff810037fe6248 0000000037fe6040
May 29 11:58:27 vault1 kernel: Call Trace:
May 29 11:58:27 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:27 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:27 vault1 kernel:
May 29 11:58:27 vault1 kernel: aio/1 S ffffffff8058a280 0 286 19 (L-TLB)
May 29 11:58:27 vault1 kernel: ffff810037f41ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000002 ffff810037fe4080
May 29 11:58:27 vault1 kernel: ffff81011fc78040 0000000000001723 ffff810037fe4288 0000000137fe4080
May 29 11:58:27 vault1 kernel: Call Trace:
May 29 11:58:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:27 vault1 kernel:
May 29 11:58:27 vault1 kernel: aio/2 S ffffffff8058a280 0 287 19 (L-TLB)
May 29 11:58:27 vault1 kernel: ffff810037f43ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000002 ffff810037fe30c0
May 29 11:58:27 vault1 kernel: ffff81011fccf080 0000000000001b18 ffff810037fe32c8 0000000237fe30c0
May 29 11:58:27 vault1 kernel: Call Trace:
May 29 11:58:27 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:27 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:27 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:27 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:27 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:27 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:27 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:27 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:27 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:27 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:27 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:27 vault1 kernel:
May 29 11:58:27 vault1 kernel: aio/3 S ffffffff8058a280 0 288 19 (L-TLB)
May 29 11:58:27 vault1 kernel: ffff810037f45ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:27 vault1 kernel: 0000000000000000 0000000000000000 0000000000000003 ffff810037fe2040
May 29 11:58:28 vault1 kernel: ffff81011fd360c0 0000000000001406 ffff810037fe2248 0000000337fe2040
May 29 11:58:28 vault1 kernel: Call Trace:
May 29 11:58:28 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:28 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:28 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:28 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:28 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:28 vault1 kernel:
May 29 11:58:28 vault1 kernel: kpsmoused S ffffffff8058a280 0 446 19 (L-TLB)
May 29 11:58:28 vault1 kernel: ffff810037d59ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:28 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff810037f81080
May 29 11:58:28 vault1 kernel: ffffffff8058a6e0 0000000000000d43 ffff810037f81288 0000000037f81080
May 29 11:58:28 vault1 kernel: Call Trace:
May 29 11:58:28 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:28 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:28 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:28 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:28 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:28 vault1 kernel:
May 29 11:58:28 vault1 kernel: scsi_eh_0 S ffff810037d4b638 0 499 19 (L-TLB)
May 29 11:58:28 vault1 kernel: ffff81007f41fe70 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:28 vault1 kernel: 0000000000000000 0000000000000046 0000000000000004 ffff810037f69040
May 29 11:58:28 vault1 kernel: ffff81011fccf080 0000000000000c44 ffff810037f69248 0000000280263894
May 29 11:58:28 vault1 kernel: Call Trace:
May 29 11:58:28 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:28 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:28 vault1 kernel:
May 29 11:58:28 vault1 kernel: kjournald S ffff81007f1149d8 0 535 19 (L-TLB)
May 29 11:58:28 vault1 kernel: ffff81007f79bec0 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:28 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810037d50080
May 29 11:58:28 vault1 kernel: ffff81011fc78040 0000000000000b58 ffff810037d50288 000000018029fc0f
May 29 11:58:28 vault1 kernel: Call Trace:
May 29 11:58:28 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 29 11:58:28 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 29 11:58:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:28 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:28 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:28 vault1 kernel:
May 29 11:58:28 vault1 kernel: kauditd S ffff810037d4a4b8 0 557 19 (L-TLB)
May 29 11:58:28 vault1 kernel: ffff81007f3a9ec0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:28 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81007f19b040
May 29 11:58:28 vault1 kernel: ffff81007f32f0c0 000000000000750a ffff81007f19b248 000000007f19b040
May 29 11:58:28 vault1 kernel: Call Trace:
May 29 11:58:28 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:28 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff802b2b39>] kauditd_thread+0x111/0x148
May 29 11:58:28 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:28 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:28 vault1 kernel: [<ffffffff802b2a28>] kauditd_thread+0x0/0x148
May 29 11:58:28 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:28 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:28 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:29 vault1 kernel:
May 29 11:58:29 vault1 kernel: udevd S ffff81007f343638 0 586 1 (NOTLB)
May 29 11:58:29 vault1 kernel: ffff81007fe1ba18 0000000000000046 0000000000000000 0000000000000046
May 29 11:58:29 vault1 kernel: ffffffff802ea6a4 ffff81007fe107f8 0000000000000005 ffff81007fe100c0
May 29 11:58:29 vault1 kernel: ffff81011fc78040 00000000000282bc ffff81007fe102c8 0000000180262123
May 29 11:58:29 vault1 kernel: Call Trace:
May 29 11:58:29 vault1 kernel: [<ffffffff802ea6a4>] inotify_poll+0x31/0x56
May 29 11:58:29 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:29 vault1 kernel: [<ffffffff802ea6c2>] inotify_poll+0x4f/0x56
May 29 11:58:29 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:29 vault1 kernel: [<ffffffff80217eea>] cache_grow+0x31f/0x378
May 29 11:58:29 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:29 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:29 vault1 last message repeated 3 times
May 29 11:58:29 vault1 kernel: [<ffffffff8023e153>] lookup_mnt+0x19/0x4c
May 29 11:58:29 vault1 kernel: [<ffffffff8023e17e>] lookup_mnt+0x44/0x4c
May 29 11:58:29 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 29 11:58:29 vault1 kernel: [<ffffffff80219f0b>] __follow_mount+0x26/0x7b
May 29 11:58:29 vault1 kernel: [<ffffffff8020cff9>] do_lookup+0x63/0x1ae
May 29 11:58:29 vault1 kernel: [<ffffffff802119b8>] do_wp_page+0x5af/0x60b
May 29 11:58:29 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:29 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:29 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:29 vault1 kernel: [<ffffffff8023bee1>] remove_wait_queue+0x10/0x2c
May 29 11:58:29 vault1 kernel: [<ffffffff8023bee1>] remove_wait_queue+0x10/0x2c
May 29 11:58:29 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:29 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:29 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:29 vault1 kernel:
May 29 11:58:29 vault1 kernel: khpsbpkt S ffff810037d4a4b8 0 1142 19 (L-TLB)
May 29 11:58:29 vault1 kernel: ffff81007e7a5f00 0000000000000046 ffffffff80246747 0000000000000002
May 29 11:58:29 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007fd63040
May 29 11:58:29 vault1 kernel: ffff81007e1680c0 0000000000001b49 ffff81007fd63248 000000001f3cbd30
May 29 11:58:29 vault1 kernel: Call Trace:
May 29 11:58:29 vault1 kernel: [<ffffffff80246747>] skb_dequeue+0x13/0x50
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff88105239>] :ieee1394:hpsbpkt_thread+0x0/0x8d
May 29 11:58:29 vault1 kernel: [<ffffffff881052a9>] :ieee1394:hpsbpkt_thread+0x70/0x8d
May 29 11:58:29 vault1 kernel: [<ffffffff88105239>] :ieee1394:hpsbpkt_thread+0x0/0x8d
May 29 11:58:29 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:29 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:29 vault1 kernel:
May 29 11:58:29 vault1 kernel: knodemgrd_0 S ffff81007e4304f8 0 1204 19 (L-TLB)
May 29 11:58:29 vault1 kernel: ffff81007e1f5e70 0000000000000046 ffff81007e883080 ffffffff80261ece
May 29 11:58:29 vault1 kernel: 0000000000000246 ffff81007e7e4378 0000000000000009 ffff81007e883080
May 29 11:58:29 vault1 kernel: ffff810037f5d040 00000000000e6dc7 ffff81007e883288 0000000180261ece
May 29 11:58:29 vault1 kernel: Call Trace:
May 29 11:58:29 vault1 kernel: [<ffffffff80261ece>] __mutex_unlock_slowpath+0x128/0x133
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff88109d00>] :ieee1394:nodemgr_host_thread+0x87/0xa58
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff88109c79>] :ieee1394:nodemgr_host_thread+0x0/0xa58
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:29 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:29 vault1 kernel:
May 29 11:58:29 vault1 kernel: ata/0 S ffff81007ebd24b8 0 1289 19 (L-TLB)
May 29 11:58:29 vault1 kernel: ffff81011e58fea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:29 vault1 kernel: 0000000000000000 0000000000000000 0000000000000004 ffff81011fe3d080
May 29 11:58:29 vault1 kernel: ffff81007e917040 0000000000002176 ffff81011fe3d288 000000001fe3d080
May 29 11:58:29 vault1 kernel: Call Trace:
May 29 11:58:29 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:29 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:29 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:29 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:29 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:29 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:29 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:29 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:29 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:29 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:29 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:29 vault1 kernel:
May 29 11:58:30 vault1 kernel: ata/1 S ffff81007f343638 0 1290 19 (L-TLB)
May 29 11:58:30 vault1 kernel: ffff81011e505ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:30 vault1 kernel: 0000000000000000 0000000000000000 0000000000000004 ffff81011fe34080
May 29 11:58:30 vault1 kernel: ffff81007fe100c0 00000000000020c5 ffff81011fe34288 000000011fe34080
May 29 11:58:30 vault1 kernel: Call Trace:
May 29 11:58:30 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:30 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:30 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:30 vault1 kernel:
May 29 11:58:30 vault1 kernel: ata/2 S ffff81011fe0b1d8 0 1291 19 (L-TLB)
May 29 11:58:30 vault1 kernel: ffff81011e69fea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:30 vault1 kernel: 0000000000000000 0000000000000000 0000000000000004 ffff81011f35c080
May 29 11:58:30 vault1 kernel: ffff81011edcd0c0 00000000000012dc ffff81011f35c288 000000021f35c080
May 29 11:58:30 vault1 kernel: Call Trace:
May 29 11:58:30 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:30 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:30 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:30 vault1 kernel:
May 29 11:58:30 vault1 kernel: ata/3 S ffff81011edb5b58 0 1292 19 (L-TLB)
May 29 11:58:30 vault1 kernel: ffff81011e6abea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:30 vault1 kernel: 0000000000000000 0000000000000000 0000000000000003 ffff81011edcd0c0
May 29 11:58:30 vault1 kernel: ffff81011f0330c0 00000000000026f9 ffff81011edcd2c8 000000031edcd0c0
May 29 11:58:30 vault1 kernel: Call Trace:
May 29 11:58:30 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:30 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:30 vault1 kernel:
May 29 11:58:30 vault1 kernel: ata_aux S ffff81011fe0b1d8 0 1293 19 (L-TLB)
May 29 11:58:30 vault1 kernel: ffff81011e6d3ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:30 vault1 kernel: 0000000000000000 0000000000000000 0000000000000004 ffff81011fe3e040
May 29 11:58:30 vault1 kernel: ffff81011eec3080 0000000000001260 ffff81011fe3e248 000000021fe3e040
May 29 11:58:30 vault1 kernel: Call Trace:
May 29 11:58:30 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:30 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:30 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:30 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:30 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:30 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:30 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:30 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:30 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:30 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:30 vault1 kernel:
May 29 11:58:30 vault1 kernel: scsi_eh_5 S ffff81007e8df6f8 0 1315 19 (L-TLB)
May 29 11:58:30 vault1 kernel: ffff81011ee35e70 0000000000000046 ffff81011f2e3040 ffffffff802638fc
May 29 11:58:30 vault1 kernel: ffff81007f3a4000 0000000000000000 0000000000000005 ffff81011f2e3040
May 29 11:58:30 vault1 kernel: ffff81011fe2c0c0 00000000009d77e9 ffff81011f2e3248 0000000000000282
May 29 11:58:31 vault1 kernel: Call Trace:
May 29 11:58:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:31 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:31 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:31 vault1 kernel:
May 29 11:58:31 vault1 kernel: scsi_eh_6 S ffff81007f343638 0 1316 19 (L-TLB)
May 29 11:58:31 vault1 kernel: ffff81011fecbe70 0000000000000046 ffff81011ec6d080 ffffffff802638fc
May 29 11:58:31 vault1 kernel: ffff81007e2ec000 0000000000000000 0000000000000008 ffff81011ec6d080
May 29 11:58:31 vault1 kernel: ffff81011fd4a040 00000000009d51e7 ffff81011ec6d288 0000000200000282
May 29 11:58:31 vault1 kernel: Call Trace:
May 29 11:58:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:31 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:31 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:31 vault1 kernel:
May 29 11:58:31 vault1 kernel: scsi_eh_7 S ffff81007fe19ad8 0 1741 19 (L-TLB)
May 29 11:58:31 vault1 kernel: ffff81011e8dde70 0000000000000046 ffff81011ec7f0c0 ffffffff802638fc
May 29 11:58:31 vault1 kernel: ffff81007fd5c000 0000000000000000 0000000000000008 ffff81011ec7f0c0
May 29 11:58:31 vault1 kernel: ffff81011fd4a040 00000000009d3031 ffff81011ec7f2c8 0000000200000282
May 29 11:58:31 vault1 kernel: Call Trace:
May 29 11:58:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:31 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:31 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:31 vault1 kernel:
May 29 11:58:31 vault1 kernel: scsi_eh_8 S ffff81011f1a5b18 0 1742 19 (L-TLB)
May 29 11:58:31 vault1 kernel: ffff81011e4ffe70 0000000000000046 ffff81011ff03080 ffffffff802638fc
May 29 11:58:31 vault1 kernel: ffff81007e4ac000 0000000000000000 000000000000000a ffff81011ff03080
May 29 11:58:31 vault1 kernel: ffff81011fd4a040 00000000009d292b ffff81011ff03288 0000000200000282
May 29 11:58:31 vault1 kernel: Call Trace:
May 29 11:58:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:31 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:31 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:31 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:31 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:31 vault1 kernel:
May 29 11:58:31 vault1 kernel: scsi_eh_9 S ffff81007f343638 0 2015 19 (L-TLB)
May 29 11:58:31 vault1 kernel: ffff81011f3f9e70 0000000000000046 ffff81011f030080 ffffffff802638fc
May 29 11:58:31 vault1 kernel: ffff81007e120000 0000000000000000 000000000000000a ffff81011f030080
May 29 11:58:31 vault1 kernel: ffff81011fd4a040 00000000009d3f30 ffff81011f030288 0000000200000282
May 29 11:58:31 vault1 kernel: Call Trace:
May 29 11:58:31 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:31 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:31 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:31 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:31 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:31 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:31 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:32 vault1 kernel:
May 29 11:58:32 vault1 kernel: scsi_eh_10 S ffff810037d4ad78 0 2016 19 (L-TLB)
May 29 11:58:32 vault1 kernel: ffff81011e7fde70 0000000000000046 ffff81011ffe00c0 ffffffff802638fc
May 29 11:58:32 vault1 kernel: ffff81007e288000 0000000000000000 000000000000000a ffff81011ffe00c0
May 29 11:58:32 vault1 kernel: ffff81007f22b080 00000000009c774d ffff81011ffe02c8 0000000000000282
May 29 11:58:32 vault1 kernel: Call Trace:
May 29 11:58:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:32 vault1 kernel: [<ffffffff88060287>] :scsi_mod:__scsi_iterate_devices+0x5c/0x75
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:32 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:32 vault1 kernel:
May 29 11:58:32 vault1 kernel: kmpathd/0 S ffff81011f96e918 0 2243 19 (L-TLB)
May 29 11:58:32 vault1 kernel: ffff81007dc5dea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:32 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007e125040
May 29 11:58:32 vault1 kernel: ffff81007fdf3080 0000000000000d28 ffff81007e125248 000000007e125040
May 29 11:58:32 vault1 kernel: Call Trace:
May 29 11:58:32 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:32 vault1 kernel:
May 29 11:58:32 vault1 kernel: kmpathd/1 S ffff81007e836998 0 2244 19 (L-TLB)
May 29 11:58:32 vault1 kernel: ffff81007dc5fea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:32 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007e962040
May 29 11:58:32 vault1 kernel: ffff81011fc78040 0000000000001a16 ffff81007e962248 000000017e962040
May 29 11:58:32 vault1 kernel: Call Trace:
May 29 11:58:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:32 vault1 kernel:
May 29 11:58:32 vault1 kernel: kmpathd/2 S ffff81007f343638 0 2245 19 (L-TLB)
May 29 11:58:32 vault1 kernel: ffff81007dea1ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:32 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007e5ae080
May 29 11:58:32 vault1 kernel: ffff81007fe100c0 0000000000002189 ffff81007e5ae288 000000027e5ae080
May 29 11:58:32 vault1 kernel: Call Trace:
May 29 11:58:32 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:32 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:32 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:32 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:32 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:32 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:32 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:32 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:32 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:32 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:32 vault1 kernel:
May 29 11:58:32 vault1 kernel: kmpathd/3 S ffff81011f96e918 0 2246 19 (L-TLB)
May 29 11:58:32 vault1 kernel: ffff81007dea3ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:32 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007f22b080
May 29 11:58:32 vault1 kernel: ffff81011fd360c0 00000000000032a3 ffff81007f22b288 000000037f22b080
May 29 11:58:32 vault1 kernel: Call Trace:
May 29 11:58:32 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:32 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:33 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:33 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:33 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:33 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:33 vault1 kernel:
May 29 11:58:33 vault1 kernel: kmirrord S ffff81007e837b18 0 2261 19 (L-TLB)
May 29 11:58:33 vault1 kernel: ffff81007dfbfea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:33 vault1 kernel: 0000000000000000 0000000000000000 0000000000000001 ffff81007e9c7040
May 29 11:58:33 vault1 kernel: ffff81007e9970c0 0000000000000c37 ffff81007e9c7248 000000007e9c7040
May 29 11:58:33 vault1 kernel: Call Trace:
May 29 11:58:33 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:33 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:33 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:33 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:33 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:33 vault1 kernel:
May 29 11:58:33 vault1 kernel: jfsIO S ffff81007e8de9d8 0 2299 19 (L-TLB)
May 29 11:58:33 vault1 kernel: ffff81007c859f00 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:33 vault1 kernel: 000000017c858000 0000000000000002 000000000000000a ffff81007e54c040
May 29 11:58:33 vault1 kernel: ffff81011fccf080 00000000000074c3 ffff81007e54c248 0000000200000000
May 29 11:58:33 vault1 kernel: Call Trace:
May 29 11:58:33 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff88242385>] :jfs:jfsIOWait+0x0/0xa2
May 29 11:58:33 vault1 kernel: [<ffffffff8824240a>] :jfs:jfsIOWait+0x85/0xa2
May 29 11:58:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:33 vault1 kernel:
May 29 11:58:33 vault1 kernel: jfsCommit S ffff81007e8de9d8 0 2300 19 (L-TLB)
May 29 11:58:33 vault1 kernel: ffff81007c85beb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:33 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007e1680c0
May 29 11:58:33 vault1 kernel: ffff81007e4620c0 000000000000e4f1 ffff81007e1682c8 000000027e1680c0
May 29 11:58:33 vault1 kernel: Call Trace:
May 29 11:58:33 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:33 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:33 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:33 vault1 kernel: [<ffffffff88245114>] :jfs:jfs_lazycommit+0x1d9/0x212
May 29 11:58:33 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:33 vault1 kernel: [<ffffffff88244f3b>] :jfs:jfs_lazycommit+0x0/0x212
May 29 11:58:33 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:33 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:33 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:33 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:33 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:33 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:33 vault1 kernel:
May 29 11:58:33 vault1 kernel: jfsCommit S ffff81007e8de9d8 0 2301 19 (L-TLB)
May 29 11:58:33 vault1 kernel: ffff81007c85deb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:34 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007e4620c0
May 29 11:58:34 vault1 kernel: ffff81007ef410c0 000000000000097b ffff81007e4622c8 000000027e4620c0
May 29 11:58:34 vault1 kernel: Call Trace:
May 29 11:58:34 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:34 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff88245114>] :jfs:jfs_lazycommit+0x1d9/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:34 vault1 kernel: [<ffffffff88244f3b>] :jfs:jfs_lazycommit+0x0/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:34 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:34 vault1 kernel:
May 29 11:58:34 vault1 kernel: jfsCommit S ffff81007e8de9d8 0 2302 19 (L-TLB)
May 29 11:58:34 vault1 kernel: ffff81007c85feb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:34 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007e5e50c0
May 29 11:58:34 vault1 kernel: ffff81007e0c80c0 0000000000000719 ffff81007e5e52c8 000000027e5e50c0
May 29 11:58:34 vault1 kernel: Call Trace:
May 29 11:58:34 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:34 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff88245114>] :jfs:jfs_lazycommit+0x1d9/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:34 vault1 kernel: [<ffffffff88244f3b>] :jfs:jfs_lazycommit+0x0/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:34 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:34 vault1 kernel:
May 29 11:58:34 vault1 kernel: jfsCommit S ffff81007e8de9d8 0 2303 19 (L-TLB)
May 29 11:58:34 vault1 kernel: ffff81007c861eb0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:34 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81007ef410c0
May 29 11:58:34 vault1 kernel: ffff81007e5e50c0 000000000000071c ffff81007ef412c8 000000027ef410c0
May 29 11:58:34 vault1 kernel: Call Trace:
May 29 11:58:34 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:34 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff88245114>] :jfs:jfs_lazycommit+0x1d9/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:34 vault1 kernel: [<ffffffff88244f3b>] :jfs:jfs_lazycommit+0x0/0x212
May 29 11:58:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:34 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:34 vault1 kernel:
May 29 11:58:34 vault1 kernel: jfsSync S ffff81007f343638 0 2304 19 (L-TLB)
May 29 11:58:34 vault1 kernel: ffff81007c863ee0 0000000000000046 0000000000000000 ffff81000438c040
May 29 11:58:34 vault1 kernel: 000000000000018e 0000000000000001 0000000000000007 ffff81000438c040
May 29 11:58:34 vault1 kernel: ffff81011fccf080 000000000000d778 ffff81000438c248 0000000200000246
May 29 11:58:34 vault1 kernel: Call Trace:
May 29 11:58:34 vault1 kernel: [<ffffffff882448ef>] :jfs:jfs_sync+0x181/0x1ac
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff8824476e>] :jfs:jfs_sync+0x0/0x1ac
May 29 11:58:34 vault1 kernel: [<ffffffff882448f4>] :jfs:jfs_sync+0x186/0x1ac
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff8824476e>] :jfs:jfs_sync+0x0/0x1ac
May 29 11:58:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:34 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:34 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:34 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:34 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:34 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:34 vault1 kernel:
May 29 11:58:34 vault1 kernel: kjournald S ffff81007f1149d8 0 2306 19 (L-TLB)
May 29 11:58:34 vault1 kernel: ffff81007d54fec0 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:34 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810037f5d040
May 29 11:58:34 vault1 kernel: ffff81011fc78040 0000000000004c8c ffff810037f5d248 000000018029fc0f
May 29 11:58:34 vault1 kernel: Call Trace:
May 29 11:58:34 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 29 11:58:34 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:34 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:34 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 29 11:58:34 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:35 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:35 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:35 vault1 kernel:
May 29 11:58:35 vault1 kernel: kjournald S ffff81011f96fa98 0 2308 19 (L-TLB)
May 29 11:58:35 vault1 kernel: ffff81007d541ec0 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:35 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff810037f60040
May 29 11:58:35 vault1 kernel: ffff81011fc78040 000000000000287b ffff810037f60248 000000018029fc0f
May 29 11:58:35 vault1 kernel: Call Trace:
May 29 11:58:35 vault1 kernel: [<ffffffff880279be>] :jbd:kjournald+0x18e/0x21d
May 29 11:58:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:35 vault1 kernel: [<ffffffff880279c3>] :jbd:kjournald+0x193/0x21d
May 29 11:58:35 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:35 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 29 11:58:35 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:35 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:35 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:35 vault1 kernel:
May 29 11:58:35 vault1 kernel: kjournald D ffff81007e837b18 0 2310 19 (L-TLB)
May 29 11:58:35 vault1 kernel: ffff81007d0efd40 0000000000000046 0000000000000000 ffff81007e9970c0
May 29 11:58:35 vault1 kernel: ffff81011fc78040 00000000000055bc ffff81007e9972c8 000000017e9970c0
May 29 11:58:35 vault1 kernel: ffff81011fc78040 0000000000004b2c ffffffffffffffff ffff810082c32700
May 29 11:58:35 vault1 kernel: Call Trace:
May 29 11:58:35 vault1 kernel: [<ffffffff80258fc6>] generic_unplug_device+0x18/0x25
May 29 11:58:35 vault1 kernel: [<ffffffff80258fc6>] generic_unplug_device+0x18/0x25
May 29 11:58:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:35 vault1 kernel: [<ffffffff8021576b>] sync_buffer+0x0/0x43
May 29 11:58:35 vault1 kernel: [<ffffffff8021576b>] sync_buffer+0x0/0x43
May 29 11:58:35 vault1 kernel: [<ffffffff802157a9>] sync_buffer+0x3e/0x43
May 29 11:58:35 vault1 kernel: [<ffffffff802618fb>] __wait_on_bit+0x40/0x6e
May 29 11:58:35 vault1 kernel: [<ffffffff8021576b>] sync_buffer+0x0/0x43
May 29 11:58:35 vault1 kernel: [<ffffffff8021576b>] sync_buffer+0x0/0x43
May 29 11:58:35 vault1 kernel: [<ffffffff80261995>] out_of_line_wait_on_bit+0x6c/0x78
May 29 11:58:35 vault1 kernel: [<ffffffff8029a120>] wake_bit_function+0x0/0x23
May 29 11:58:35 vault1 kernel: [<ffffffff8023aa60>] sync_dirty_buffer+0xae/0xe2
May 29 11:58:35 vault1 kernel: [<ffffffff88023eff>] :jbd:journal_commit_transaction+0xbc7/0x109c
May 29 11:58:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:35 vault1 kernel: [<ffffffff880278ea>] :jbd:kjournald+0xba/0x21d
May 29 11:58:35 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:35 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:35 vault1 kernel: [<ffffffff88027830>] :jbd:kjournald+0x0/0x21d
May 29 11:58:35 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:35 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:35 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:35 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:35 vault1 kernel:
May 29 11:58:35 vault1 kernel: dhclient S ffff81007f1156f8 0 2686 1 (NOTLB)
May 29 11:58:35 vault1 kernel: ffff81007747da18 0000000000000046 0000000000000000 ffff81007e9fb0c0
May 29 11:58:35 vault1 kernel: ffff81007e9fb0c0 ffffffff802638fc 0000000000000007 ffff81007e9fb0c0
May 29 11:58:35 vault1 kernel: ffff81011fc78040 0000000000002776 ffff81007e9fb2c8 0000000137e28000
May 29 11:58:35 vault1 kernel: Call Trace:
May 29 11:58:35 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:35 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 29 11:58:35 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:35 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:35 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:35 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:35 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:35 vault1 kernel: [<ffffffff802078c3>] kmem_cache_free+0x257/0x267
May 29 11:58:35 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:35 vault1 kernel: [<ffffffff804435a0>] udp_recvmsg+0x1ee/0x270
May 29 11:58:35 vault1 kernel: [<ffffffff80232600>] sock_common_recvmsg+0x30/0x45
May 29 11:58:35 vault1 kernel: [<ffffffff80230bc6>] sock_recvmsg+0xf8/0x117
May 29 11:58:35 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:35 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:35 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:35 vault1 kernel: [<ffffffff8040bc77>] move_addr_to_user+0x5d/0x78
May 29 11:58:35 vault1 kernel: [<ffffffff8022c605>] sys_recvfrom+0x104/0x130
May 29 11:58:35 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:35 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:35 vault1 kernel: [<ffffffff80216b98>] sys_select+0x15a/0x183
May 29 11:58:35 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:35 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:35 vault1 kernel:
May 29 11:58:35 vault1 kernel: syslogd D ffff81007e837b18 0 2963 1 (NOTLB)
May 29 11:58:35 vault1 kernel: ffff81007869fd88 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:35 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81007e1b3080
May 29 11:58:36 vault1 kernel: ffff81007e9970c0 0000000000014ad9 ffff81007e1b3288 000000018029fc0f
May 29 11:58:36 vault1 kernel: Call Trace:
May 29 11:58:36 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:36 vault1 kernel: [<ffffffff88026c13>] :jbd:log_wait_commit+0xa4/0xf7
May 29 11:58:36 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:36 vault1 kernel: [<ffffffff88021730>] :jbd:journal_stop+0x1e5/0x21a
May 29 11:58:36 vault1 kernel: [<ffffffff802304c1>] __writeback_single_inode+0x1fa/0x338
May 29 11:58:36 vault1 kernel: [<ffffffff802e2073>] sync_inode+0x19/0x39
May 29 11:58:36 vault1 kernel: [<ffffffff802e2073>] sync_inode+0x19/0x39
May 29 11:58:36 vault1 kernel: [<ffffffff802e207e>] sync_inode+0x24/0x39
May 29 11:58:36 vault1 kernel: [<ffffffff88037332>] :ext3:ext3_sync_file+0xb2/0xc4
May 29 11:58:36 vault1 kernel: [<ffffffff8024e527>] do_fsync+0x52/0xa4
May 29 11:58:36 vault1 kernel: [<ffffffff802e4478>] __do_fsync+0x23/0x36
May 29 11:58:36 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:36 vault1 kernel:
May 29 11:58:36 vault1 kernel: klogd S ffff81007ebd3638 0 2966 1 (NOTLB)
May 29 11:58:36 vault1 kernel: ffff810077c2fbe8 0000000000000046 0000000000000000 ffff81007e0fe080
May 29 11:58:36 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007e0fe080
May 29 11:58:36 vault1 kernel: ffff81011fccf080 000000000000343a ffff81007e0fe288 0000000200000002
May 29 11:58:36 vault1 kernel: Call Trace:
May 29 11:58:36 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:36 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:36 vault1 kernel: [<ffffffff80460f4b>] unix_wait_for_peer+0x86/0xb0
May 29 11:58:36 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:36 vault1 kernel: [<ffffffff804614fb>] unix_dgram_sendmsg+0x3d3/0x4cd
May 29 11:58:36 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 29 11:58:36 vault1 kernel: [<ffffffff80260e98>] __sched_text_start+0x928/0x960
May 29 11:58:36 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 29 11:58:36 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:36 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:36 vault1 kernel: [<ffffffff80216cda>] vfs_write+0xe1/0x177
May 29 11:58:36 vault1 kernel: [<ffffffff8020b86d>] fget_light+0x19/0x7b
May 29 11:58:36 vault1 kernel: [<ffffffff802176bf>] sys_write+0x45/0x6e
May 29 11:58:36 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:36 vault1 kernel:
May 29 11:58:36 vault1 kernel: irqbalance S ffff81007fe18098 0 2977 1 (NOTLB)
May 29 11:58:36 vault1 kernel: ffff8100774e9eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 29 11:58:36 vault1 kernel: ffff810003c2cbf8 ffff810003c2cbf0 0000000000000007 ffff810037e90040
May 29 11:58:36 vault1 kernel: ffffffff8058a6e0 00000000001be12b ffff810037e90248 0000000000000292
May 29 11:58:36 vault1 kernel: Call Trace:
May 29 11:58:36 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:36 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 29 11:58:36 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 29 11:58:36 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 29 11:58:36 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 29 11:58:36 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:36 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 29 11:58:36 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:36 vault1 kernel:
May 29 11:58:36 vault1 kernel: portmap S ffff81007fe18db8 0 2997 1 (NOTLB)
May 29 11:58:36 vault1 kernel: ffff8100774d5b28 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:36 vault1 kernel: 0000000180248a3b 0000000000000002 000000000000000a ffff810037f6e080
May 29 11:58:36 vault1 kernel: ffff81011fd34040 0000000000b33289 ffff810037f6e288 0000000300000000
May 29 11:58:36 vault1 kernel: Call Trace:
May 29 11:58:36 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:36 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:36 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:36 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:36 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 29 11:58:36 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:36 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:36 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:36 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:36 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:36 vault1 kernel: [<ffffffff8029ff51>] debug_check_no_locks_freed+0x11d/0x129
May 29 11:58:36 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:36 vault1 kernel: [<ffffffff8020b848>] free_hot_cold_page+0x184/0x190
May 29 11:58:36 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:36 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:36 vault1 kernel: [<ffffffff8040b777>] sys_sendmsg+0x217/0x28a
May 29 11:58:36 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:36 vault1 kernel: [<ffffffff8023089b>] __up_write+0x27/0xf2
May 29 11:58:36 vault1 kernel: [<ffffffff8023089b>] __up_write+0x27/0xf2
May 29 11:58:36 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:36 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:36 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:36 vault1 kernel:
May 29 11:58:36 vault1 kernel: rpc.statd S ffff81007e1bb258 0 3017 1 (NOTLB)
May 29 11:58:37 vault1 kernel: ffff810077419a18 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:37 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000001 ffff810037f5e080
May 29 11:58:37 vault1 kernel: ffff81007e917040 0000000000002c39 ffff810037f5e288 0000000000000000
May 29 11:58:37 vault1 kernel: Call Trace:
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:37 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:37 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:37 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 29 11:58:37 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:37 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 last message repeated 2 times
May 29 11:58:37 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 29 11:58:37 vault1 kernel: [<ffffffff8046155d>] unix_dgram_sendmsg+0x435/0x4cd
May 29 11:58:37 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 29 11:58:37 vault1 kernel: [<ffffffff802530fc>] sock_sendmsg+0xea/0x107
May 29 11:58:37 vault1 kernel: [<ffffffff8020ed97>] link_path_walk+0xce/0xe0
May 29 11:58:37 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:37 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:37 vault1 kernel: [<ffffffff8022d806>] mntput_no_expire+0x1c/0x96
May 29 11:58:37 vault1 kernel: [<ffffffff8040ba92>] sys_sendto+0x128/0x151
May 29 11:58:37 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:37 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:37 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:37 vault1 kernel:
May 29 11:58:37 vault1 kernel: dbus-daemon S ffff81007f115b58 0 3035 1 (NOTLB)
May 29 11:58:37 vault1 kernel: ffff81007773bb28 0000000000000046 0000000000000000 ffff81007f1d5080
May 29 11:58:37 vault1 kernel: 0000000000000000 ffffffff802a0de5 000000000000000a ffff81007f1d5080
May 29 11:58:37 vault1 kernel: ffff81011fc78040 0000000000009ca4 ffff81007f1d5288 0000000100000002
May 29 11:58:37 vault1 kernel: Call Trace:
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:37 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:37 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 last message repeated 2 times
May 29 11:58:37 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 29 11:58:37 vault1 kernel: [<ffffffff8040ae7a>] sock_aio_read+0x10c/0x124
May 29 11:58:37 vault1 kernel: [<ffffffff8020cf53>] do_sync_read+0xc9/0x10c
May 29 11:58:37 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:37 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:37 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:37 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:37 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:37 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:37 vault1 kernel:
May 29 11:58:37 vault1 kernel: hidd S ffff810037d4ad78 0 3067 1 (NOTLB)
May 29 11:58:37 vault1 kernel: ffff81007734bad8 0000000000000046 0000000000000000 ffff81007f0fb0c0
May 29 11:58:37 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000001 ffff81007f0fb0c0
May 29 11:58:37 vault1 kernel: ffff81011fccf080 0000000000008764 ffff81007f0fb2c8 0000000200000002
May 29 11:58:37 vault1 kernel: Call Trace:
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:37 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:37 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 29 11:58:37 vault1 kernel: [<ffffffff8046155d>] unix_dgram_sendmsg+0x435/0x4cd
May 29 11:58:37 vault1 kernel: [<ffffffff8020d542>] dput+0x3c/0x12b
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 29 11:58:37 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:37 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff8021e12c>] sigprocmask+0x28/0xc1
May 29 11:58:37 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:37 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:37 vault1 kernel: [<ffffffff802d861f>] sys_ppoll+0xf1/0x1ff
May 29 11:58:37 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:37 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:37 vault1 kernel:
May 29 11:58:37 vault1 kernel: acpid S ffff81007ebd2d78 0 3079 1 (NOTLB)
May 29 11:58:37 vault1 kernel: ffff8100775ffb28 0000000000000046 0000000000000627 ffff81007e740080
May 29 11:58:37 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000007 ffff81007e740080
May 29 11:58:37 vault1 kernel: ffff81007e932080 0000000000017592 ffff81007e740288 0000000000000002
May 29 11:58:37 vault1 kernel: Call Trace:
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:37 vault1 kernel: [<ffffffff88021755>] :jbd:journal_stop+0x20a/0x21a
May 29 11:58:37 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:37 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:37 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:37 vault1 kernel: [<ffffffff8020e533>] current_fs_time+0x3b/0x40
May 29 11:58:37 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:37 vault1 kernel: [<ffffffff80216881>] __generic_file_aio_write_nolock+0x378/0x3eb
May 29 11:58:38 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 29 11:58:38 vault1 kernel: [<ffffffff80261ece>] __mutex_unlock_slowpath+0x128/0x133
May 29 11:58:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:38 vault1 kernel: [<ffffffff802224af>] generic_file_aio_write+0x6c/0xc1
May 29 11:58:38 vault1 kernel: [<ffffffff88037192>] :ext3:ext3_file_write+0x16/0x94
May 29 11:58:38 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 29 11:58:38 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:38 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:38 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:38 vault1 kernel: [<ffffffff8040bfa2>] sys_accept+0x19e/0x1fb
May 29 11:58:38 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:38 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:38 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:38 vault1 kernel:
May 29 11:58:38 vault1 kernel: sshd S ffff81007f342918 0 3092 1 (NOTLB)
May 29 11:58:38 vault1 kernel: ffff810076dbda18 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:38 vault1 kernel: 000000018023bee1 0000000000000002 000000000000000a ffff81007eaf3080
May 29 11:58:38 vault1 kernel: ffff81011fccf080 0000000000007534 ffff81007eaf3288 0000000200000000
May 29 11:58:38 vault1 kernel: Call Trace:
May 29 11:58:38 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:38 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:38 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:38 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 29 11:58:38 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:38 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:38 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:38 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:38 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:38 vault1 kernel: [<ffffffff802133c4>] sock_def_readable+0x18/0x6c
May 29 11:58:38 vault1 kernel: [<ffffffff802133c4>] sock_def_readable+0x18/0x6c
May 29 11:58:38 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 29 11:58:38 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:38 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 29 11:58:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:38 vault1 kernel: [<ffffffff802c070d>] zone_statistics+0x3f/0x60
May 29 11:58:38 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:38 vault1 kernel: [<ffffffff802118a0>] do_wp_page+0x497/0x60b
May 29 11:58:38 vault1 kernel: [<ffffffff80211a03>] do_wp_page+0x5fa/0x60b
May 29 11:58:38 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:38 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:38 vault1 kernel: [<ffffffff8020c6a9>] _atomic_dec_and_lock+0x39/0x58
May 29 11:58:38 vault1 kernel: [<ffffffff80212fc9>] poison_obj+0x26/0x2f
May 29 11:58:38 vault1 kernel: [<ffffffff802078c3>] kmem_cache_free+0x257/0x267
May 29 11:58:38 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:38 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:38 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:38 vault1 kernel: [<ffffffff80239ba9>] generic_drop_inode+0x0/0x15c
May 29 11:58:38 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:38 vault1 kernel:
May 29 11:58:38 vault1 kernel: rpc.idmapd S ffff81007e8de118 0 3193 1 (NOTLB)
May 29 11:58:38 vault1 kernel: ffff810075fabe78 0000000000000046 0000000000000000 ffff81007ee64040
May 29 11:58:38 vault1 kernel: ffff81007ee64040 ffffffff802638fc 000000000000000a ffff81007ee64040
May 29 11:58:38 vault1 kernel: ffffffff8058a6e0 00000000000027d9 ffff81007ee64248 0000000080733000
May 29 11:58:38 vault1 kernel: Call Trace:
May 29 11:58:38 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:38 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 29 11:58:38 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:38 vault1 kernel: [<ffffffff802ead19>] sys_epoll_wait+0x193/0x426
May 29 11:58:38 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:38 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:38 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:38 vault1 kernel:
May 29 11:58:38 vault1 kernel: crond S ffff81007f114118 0 3203 1 (NOTLB)
May 29 11:58:38 vault1 kernel: ffff810075c15eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 29 11:58:38 vault1 kernel: ffff810003c379f8 ffff810003c379f0 000000000000000a ffff81007e1b80c0
May 29 11:58:38 vault1 kernel: ffff81011fc78040 0000000000027cfa ffff81007e1b82c8 0000000100000292
May 29 11:58:38 vault1 kernel: Call Trace:
May 29 11:58:38 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:38 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 29 11:58:38 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 29 11:58:38 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 29 11:58:38 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 29 11:58:38 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:38 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 29 11:58:38 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:38 vault1 kernel:
May 29 11:58:38 vault1 kernel: atd S ffff81007fe184f8 0 3213 1 (NOTLB)
May 29 11:58:39 vault1 kernel: ffff810075c55eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 29 11:58:39 vault1 kernel: ffff810003c379f8 ffff810003c379f0 0000000000000007 ffff810037ec00c0
May 29 11:58:39 vault1 kernel: ffff81011fc78040 0000000000014409 ffff810037ec02c8 0000000100000292
May 29 11:58:39 vault1 kernel: Call Trace:
May 29 11:58:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:39 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 29 11:58:39 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 29 11:58:39 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 29 11:58:39 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 29 11:58:39 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:39 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 29 11:58:39 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:39 vault1 kernel:
May 29 11:58:39 vault1 kernel: hald S ffff81007f3424b8 0 3223 1 (NOTLB)
May 29 11:58:39 vault1 kernel: ffff810075dabb28 0000000000000046 0000000000000000 ffff81007f4a2080
May 29 11:58:39 vault1 kernel: ffff81007f4a2080 ffffffff802638fc 000000000000000a ffff81007f4a2080
May 29 11:58:39 vault1 kernel: ffffffff8058a6e0 000000000005abda ffff81007f4a2288 0000000080733000
May 29 11:58:39 vault1 kernel: Call Trace:
May 29 11:58:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:39 vault1 kernel: [<ffffffff80261790>] schedule_timeout+0x8a/0xad
May 29 11:58:39 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:39 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:39 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:39 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:39 vault1 last message repeated 9 times
May 29 11:58:39 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:39 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:39 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:39 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:39 vault1 kernel:
May 29 11:58:39 vault1 kernel: hald-runner S ffff81011f3c9678 0 3224 3223 (NOTLB)
May 29 11:58:39 vault1 kernel: ffff81011ecafb28 0000000000000046 0000000000000000 ffff81011f39d080
May 29 11:58:39 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000008 ffff81011f39d080
May 29 11:58:39 vault1 kernel: ffff81011fc78040 00000000000006d2 ffff81011f39d288 0000000100000002
May 29 11:58:39 vault1 kernel: Call Trace:
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:39 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:39 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:39 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:39 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 29 11:58:39 vault1 kernel: [<ffffffff8026212e>] __mutex_lock_slowpath+0x250/0x25b
May 29 11:58:39 vault1 kernel: [<ffffffff8024660c>] sock_aio_write+0x0/0x128
May 29 11:58:39 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:39 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:39 vault1 kernel: [<ffffffff802d3dfa>] do_readv_writev+0x199/0x1ae
May 29 11:58:39 vault1 kernel: [<ffffffff80229266>] do_wait+0x9a7/0xa75
May 29 11:58:39 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:39 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:39 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:39 vault1 kernel:
May 29 11:58:39 vault1 kernel: hald-addon-ac S ffff81007f114e38 0 3230 3224 (NOTLB)
May 29 11:58:39 vault1 kernel: ffff810075593be8 0000000000000046 0000000000000627 ffff81007e932080
May 29 11:58:39 vault1 kernel: 0000000000000001 ffffffff802a0de5 0000000000000003 ffff81007e932080
May 29 11:58:39 vault1 kernel: ffff81007f4a2080 0000000000002a2c ffff81007e932288 0000000000000002
May 29 11:58:39 vault1 kernel: Call Trace:
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff8023ad89>] prepare_to_wait+0x1b/0x5c
May 29 11:58:39 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:39 vault1 kernel: [<ffffffff80249a11>] unix_stream_recvmsg+0x233/0x548
May 29 11:58:39 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff8040ae7a>] sock_aio_read+0x10c/0x124
May 29 11:58:39 vault1 kernel: [<ffffffff8020cf53>] do_sync_read+0xc9/0x10c
May 29 11:58:39 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:39 vault1 kernel: [<ffffffff8020e38c>] do_mmap_pgoff+0x65d/0x7c9
May 29 11:58:39 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:39 vault1 kernel: [<ffffffff8020b5b0>] vfs_read+0xde/0x173
May 29 11:58:39 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:39 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:39 vault1 kernel:
May 29 11:58:39 vault1 kernel: hald-addon-cp S ffff81007e8360d8 0 3231 3224 (NOTLB)
May 29 11:58:39 vault1 kernel: ffff810075563b28 0000000000000046 0000000000000627 ffff81007e9fc080
May 29 11:58:39 vault1 kernel: 0000000000000000 ffffffff802a0de5 0000000000000003 ffff81007e9fc080
May 29 11:58:39 vault1 kernel: ffff81007f4a2080 000000000001e385 ffff81007e9fc288 0000000000000002
May 29 11:58:39 vault1 kernel: Call Trace:
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:39 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:39 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:39 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:39 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:39 vault1 kernel: [<ffffffff80249bf3>] unix_stream_recvmsg+0x415/0x548
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 29 11:58:39 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 29 11:58:39 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:39 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 29 11:58:40 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:40 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:40 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:40 vault1 kernel: [<ffffffff80208d64>] __handle_mm_fault+0x514/0xccf
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:40 vault1 kernel: [<ffffffff80265bc5>] do_page_fault+0x490/0x7cf
May 29 11:58:40 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:40 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: hald-addon-ke S ffff81007fe19678 0 3235 3224 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff8100756adea8 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:40 vault1 kernel: 0000000000000000 0000000000000046 0000000000000004 ffff81007fe9d040
May 29 11:58:40 vault1 kernel: ffffffff8058a6e0 00000000000b8af2 ffff81007fe9d248 000000007fe9d040
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 29 11:58:40 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:40 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: hald-addon-ke S ffff81011ed8f638 0 3245 3224 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff810075c2bea8 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:40 vault1 kernel: 0000000000000000 0000000000000046 000000000000000a ffff81007eb3f040
May 29 11:58:40 vault1 kernel: ffff81011fccf080 000000000000243f ffff81007eb3f248 000000027eb3f040
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 29 11:58:40 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:40 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:40 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: hald-addon-ke S ffff81007e1ba0d8 0 3252 3224 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff810075f9dea8 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:40 vault1 kernel: 0000000000000000 0000000000000046 0000000000000001 ffff81007f2710c0
May 29 11:58:40 vault1 kernel: ffff81011fc78040 00000000000be1fa ffff81007f2712c8 000000017f2710c0
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 29 11:58:40 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:40 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: hald-addon-ke S ffff81007fe18958 0 3255 3224 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff810076715ea8 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:40 vault1 kernel: 0000000000000000 0000000000000046 0000000000000001 ffff810037f5f0c0
May 29 11:58:40 vault1 kernel: ffff81011fc78040 00000000000bd9f3 ffff810037f5f2c8 0000000137f5f0c0
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff803f1497>] evdev_read+0x112/0x235
May 29 11:58:40 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:40 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: hald-addon-st S ffff81007e8de9d8 0 3263 3224 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff810075027eb8 0000000000000046 0000000000000000 ffffffff802638fc
May 29 11:58:40 vault1 kernel: ffff810003c379f8 ffff810003c379f0 0000000000000009 ffff81007e864040
May 29 11:58:40 vault1 kernel: ffff81011fc78040 0000000000002aae ffff81007e864248 0000000100000292
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 29 11:58:40 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 29 11:58:40 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 29 11:58:40 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 29 11:58:40 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:40 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 29 11:58:40 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:40 vault1 kernel:
May 29 11:58:40 vault1 kernel: smartd S ffff81011febcdb8 0 3307 1 (NOTLB)
May 29 11:58:40 vault1 kernel: ffff81011ed2deb8 0000000000000046 0000000000000000 ffffffff802638fc
May 29 11:58:40 vault1 kernel: ffff810003c379f8 ffff810003c379f0 0000000000000007 ffff81011ee23040
May 29 11:58:40 vault1 kernel: ffff81011fc78040 0000000000002d86 ffff81011ee23248 0000000100000292
May 29 11:58:40 vault1 kernel: Call Trace:
May 29 11:58:40 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:40 vault1 kernel: [<ffffffff8023c6d0>] hrtimer_start+0xfd/0x110
May 29 11:58:40 vault1 kernel: [<ffffffff802626ef>] do_nanosleep+0x46/0x77
May 29 11:58:40 vault1 kernel: [<ffffffff80258b18>] hrtimer_nanosleep+0x58/0x11e
May 29 11:58:40 vault1 kernel: [<ffffffff8029c17e>] hrtimer_wakeup+0x0/0x22
May 29 11:58:40 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:41 vault1 kernel: [<ffffffff80252bb1>] sys_nanosleep+0x4c/0x62
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: login S ffff81007f115298 0 3310 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810075cd3ec8 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:41 vault1 kernel: 0000000000000000 0000000200000002 0000000000000009 ffff81007fd33040
May 29 11:58:41 vault1 kernel: ffff81011fd360c0 000000000007dc88 ffff81007fd33248 0000000300000000
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:41 vault1 kernel: [<ffffffff8022920b>] do_wait+0x94c/0xa75
May 29 11:58:41 vault1 kernel: [<ffffffff8022922e>] do_wait+0x96f/0xa75
May 29 11:58:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: login S ffff81007e8de578 0 3311 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810076361ec8 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:41 vault1 kernel: 0000000000000000 0000000200000002 0000000000000007 ffff81007fd3e0c0
May 29 11:58:41 vault1 kernel: ffff81011fc78040 0000000000087012 ffff81007fd3e2c8 0000000100000000
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:41 vault1 kernel: [<ffffffff8022920b>] do_wait+0x94c/0xa75
May 29 11:58:41 vault1 kernel: [<ffffffff8022922e>] do_wait+0x96f/0xa75
May 29 11:58:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: mingetty S ffff81011f3c8958 0 3312 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810075fd5dc8 0000000000000046 0000000000026a68 ffff81011fd1da10
May 29 11:58:41 vault1 kernel: 0000000000000296 ffffffff80263178 0000000000000004 ffff81007fc0e040
May 29 11:58:41 vault1 kernel: ffff81007ea550c0 00000000000441eb ffff81007fc0e248 0000000375fd4000
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:41 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:41 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:41 vault1 kernel: [<ffffffff802638fe>] _spin_unlock_irqrestore+0x40/0x44
May 29 11:58:41 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:41 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:41 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:41 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: mingetty S ffff81011f2b9ad8 0 3313 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810075f07dc8 0000000000000046 0000000000000000 ffff81007f4a37b0
May 29 11:58:41 vault1 kernel: 0000000000000627 ffff81007f4a3040 0000000000000004 ffff81007f4a3040
May 29 11:58:41 vault1 kernel: ffff81007e6b4080 000000000002f310 ffff81007f4a3248 0000000200000046
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:41 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:41 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:41 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:41 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:41 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:41 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:41 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: mingetty S ffff81011f2b8098 0 3314 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810075d57dc8 0000000000000046 0000000000000000 ffff81007ea55830
May 29 11:58:41 vault1 kernel: 0000000000000627 ffff81007ea550c0 0000000000000004 ffff81007ea550c0
May 29 11:58:41 vault1 kernel: ffff81011fd360c0 0000000000002997 ffff81007ea552c8 0000000300000046
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:41 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:41 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:41 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:41 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:41 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:41 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:41 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:41 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:41 vault1 kernel:
May 29 11:58:41 vault1 kernel: mingetty S ffff81011f3c8db8 0 3315 1 (NOTLB)
May 29 11:58:41 vault1 kernel: ffff810075c95dc8 0000000000000046 0000000000000000 ffff81007e4167f0
May 29 11:58:41 vault1 kernel: 0000000000000627 ffff81007e416080 0000000000000004 ffff81007e416080
May 29 11:58:41 vault1 kernel: ffff81007e6b4080 0000000000002b22 ffff81007e416288 0000000200000046
May 29 11:58:41 vault1 kernel: Call Trace:
May 29 11:58:41 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:42 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:42 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:42 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:42 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:42 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:42 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:42 vault1 kernel:
May 29 11:58:42 vault1 kernel: mingetty S ffff81011edb4578 0 3316 1 (NOTLB)
May 29 11:58:42 vault1 kernel: ffff81007e0bddc8 0000000000000046 0000000000000000 ffff81007e6b47f0
May 29 11:58:42 vault1 kernel: 0000000000000627 ffff81007e6b4080 0000000000000003 ffff81007e6b4080
May 29 11:58:42 vault1 kernel: ffff81011fccf080 000000000001b72f ffff81007e6b4288 0000000200000046
May 29 11:58:42 vault1 kernel: Call Trace:
May 29 11:58:42 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:42 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:42 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:42 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:42 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:42 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:42 vault1 kernel:
May 29 11:58:42 vault1 kernel: bash S ffff81007f114578 0 3377 3310 (NOTLB)
May 29 11:58:42 vault1 kernel: ffff81011f889dc8 0000000000000046 0000000000000000 ffff81011f3fc830
May 29 11:58:42 vault1 kernel: 0000000000000627 ffff81011f3fc0c0 0000000000000009 ffff81011f3fc0c0
May 29 11:58:42 vault1 kernel: ffff81011fc78040 000000000000af71 ffff81011f3fc2c8 0000000100000046
May 29 11:58:42 vault1 kernel: Call Trace:
May 29 11:58:42 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff8022a7cf>] read_chan+0x384/0x609
May 29 11:58:42 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:42 vault1 kernel: [<ffffffff80232d53>] tty_read+0x72/0xc7
May 29 11:58:42 vault1 kernel: [<ffffffff8020b59d>] vfs_read+0xcb/0x173
May 29 11:58:42 vault1 kernel: [<ffffffff80211fc7>] sys_read+0x45/0x6e
May 29 11:58:42 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:42 vault1 kernel:
May 29 11:58:42 vault1 kernel: scsi_eh_11 S ffff81011f2b84f8 0 3447 19 (L-TLB)
May 29 11:58:42 vault1 kernel: ffff81011e1bbe70 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:42 vault1 kernel: 0000000000000000 0000000000000046 0000000000000008 ffff81011eeb50c0
May 29 11:58:42 vault1 kernel: ffff81011fd360c0 0000000000000758 ffff81011eeb52c8 0000000380263894
May 29 11:58:42 vault1 kernel: Call Trace:
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:42 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:42 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:42 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:42 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:42 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:42 vault1 kernel:
May 29 11:58:42 vault1 kernel: qla2xxx_11_dp S ffff81011f96fa98 0 3455 19 (L-TLB)
May 29 11:58:42 vault1 kernel: ffff81011e775ee0 0000000000000046 0000000000000000 0000000000200200
May 29 11:58:42 vault1 kernel: 0000000100032ce3 ffffffff802913d9 000000000000000a ffff81011e452080
May 29 11:58:42 vault1 kernel: ffff81011fc78040 0000000000000923 ffff81011e452288 0000000100000102
May 29 11:58:42 vault1 kernel: Call Trace:
May 29 11:58:42 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:42 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:42 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:42 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:42 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:42 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:42 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:42 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:42 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:42 vault1 kernel:
May 29 11:58:42 vault1 kernel: scsi_wq_11 S ffff81011ed8e058 0 3456 19 (L-TLB)
May 29 11:58:42 vault1 kernel: ffff81011e777ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:42 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011e5d5080
May 29 11:58:42 vault1 kernel: ffff81011fd360c0 00000000000014e9 ffff81011e5d5288 000000031e5d5080
May 29 11:58:42 vault1 kernel: Call Trace:
May 29 11:58:42 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:42 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:42 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:42 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:42 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:42 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:42 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:42 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:42 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:43 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:43 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:43 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:43 vault1 kernel:
May 29 11:58:43 vault1 kernel: fc_wq_11 S ffff81011ed8e058 0 3457 19 (L-TLB)
May 29 11:58:43 vault1 kernel: ffff81011e779ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:43 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011fff7040
May 29 11:58:43 vault1 kernel: ffff81011fd360c0 0000000000000e63 ffff81011fff7248 000000031fff7040
May 29 11:58:43 vault1 kernel: Call Trace:
May 29 11:58:43 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:43 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:43 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:43 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:43 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:43 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:43 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:43 vault1 kernel:
May 29 11:58:43 vault1 kernel: fc_dl_11 S ffff81011f2b84f8 0 3458 19 (L-TLB)
May 29 11:58:43 vault1 kernel: ffff81011e77bea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:43 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011e96a080
May 29 11:58:43 vault1 kernel: ffff810037d460c0 00000000000022c2 ffff81011e96a288 000000011e96a080
May 29 11:58:43 vault1 kernel: Call Trace:
May 29 11:58:43 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:43 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:43 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:43 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:43 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:43 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:43 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:43 vault1 kernel:
May 29 11:58:43 vault1 kernel: scsi_eh_12 S ffff81011ed8e058 0 3465 19 (L-TLB)
May 29 11:58:43 vault1 kernel: ffff81011ee2be70 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:43 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81011e50e040
May 29 11:58:43 vault1 kernel: ffff81011fd360c0 0000000000000913 ffff81011e50e248 0000000380263894
May 29 11:58:43 vault1 kernel: Call Trace:
May 29 11:58:43 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:43 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:43 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:43 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:43 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:43 vault1 kernel:
May 29 11:58:43 vault1 kernel: qla2xxx_12_dp S ffff81007f3424b8 0 3466 19 (L-TLB)
May 29 11:58:43 vault1 kernel: ffff81011f20bee0 0000000000000046 0000000000000000 0000000000200200
May 29 11:58:43 vault1 kernel: 00000001000341a9 ffffffff802913d9 000000000000000a ffff81011e80d0c0
May 29 11:58:43 vault1 kernel: ffffffff8058a6e0 000000000000080f ffff81011e80d2c8 0000000000000102
May 29 11:58:43 vault1 kernel: Call Trace:
May 29 11:58:43 vault1 kernel: [<ffffffff802913d9>] process_timeout+0x0/0x5
May 29 11:58:43 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:43 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 29 11:58:43 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:43 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:43 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:43 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:43 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:43 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:43 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:44 vault1 kernel:
May 29 11:58:44 vault1 kernel: scsi_wq_12 S ffff81011ed8e058 0 3467 19 (L-TLB)
May 29 11:58:44 vault1 kernel: ffff81011f257ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:44 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011f03b040
May 29 11:58:44 vault1 kernel: ffff81011fd360c0 0000000000001d8d ffff81011f03b248 000000031f03b040
May 29 11:58:44 vault1 kernel: Call Trace:
May 29 11:58:44 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:44 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:44 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:44 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:44 vault1 kernel:
May 29 11:58:44 vault1 kernel: fc_wq_12 S ffff81011ed8e058 0 3468 19 (L-TLB)
May 29 11:58:44 vault1 kernel: ffff81011e77dea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:44 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011eeb4080
May 29 11:58:44 vault1 kernel: ffff81011fd360c0 000000000000127d ffff81011eeb4288 000000031eeb4080
May 29 11:58:44 vault1 kernel: Call Trace:
May 29 11:58:44 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:44 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:44 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:44 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:44 vault1 kernel:
May 29 11:58:44 vault1 kernel: fc_dl_12 S ffff81011ed8e058 0 3469 19 (L-TLB)
May 29 11:58:44 vault1 kernel: ffff81011e7bdea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:44 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011e949040
May 29 11:58:44 vault1 kernel: ffff81011fd360c0 0000000000001255 ffff81011e949248 000000031e949040
May 29 11:58:44 vault1 kernel: Call Trace:
May 29 11:58:44 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:44 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:44 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:44 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:44 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:44 vault1 kernel:
May 29 11:58:44 vault1 kernel: scsi_eh_13 S ffff81011ed8e058 0 3476 19 (L-TLB)
May 29 11:58:44 vault1 kernel: ffff81011e71be70 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:44 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81011e869080
May 29 11:58:44 vault1 kernel: ffff81011fd360c0 00000000000008a3 ffff81011e869288 0000000380263894
May 29 11:58:44 vault1 kernel: Call Trace:
May 29 11:58:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:44 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:44 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:44 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:44 vault1 kernel:
May 29 11:58:44 vault1 kernel: qla2xxx_13_dp S ffff81007e8de9d8 0 3477 19 (L-TLB)
May 29 11:58:44 vault1 kernel: ffff81011e90fee0 0000000000000046 0000000000000000 0000000000000282
May 29 11:58:44 vault1 kernel: ffff81011e90fe40 ffff81011e90fe40 0000000000000009 ffff81011fbcc0c0
May 29 11:58:44 vault1 kernel: ffff81011fc78040 0000000000000b5a ffff81011fbcc2c8 0000000100000000
May 29 11:58:44 vault1 kernel: Call Trace:
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:44 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 29 11:58:44 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:44 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:44 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:44 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:44 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:44 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:44 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:44 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:45 vault1 kernel:
May 29 11:58:45 vault1 kernel: scsi_wq_13 S ffff81011f2b84f8 0 3478 19 (L-TLB)
May 29 11:58:45 vault1 kernel: ffff81011e77fea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:45 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011ed7f040
May 29 11:58:45 vault1 kernel: ffff81011fc78040 000000000032574d ffff81011ed7f248 000000011ed7f040
May 29 11:58:45 vault1 kernel: Call Trace:
May 29 11:58:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:45 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:45 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:45 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:45 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:45 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:45 vault1 kernel:
May 29 11:58:45 vault1 kernel: fc_wq_13 S ffff81011ed8e058 0 3479 19 (L-TLB)
May 29 11:58:45 vault1 kernel: ffff81011e079ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:45 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011ec4b040
May 29 11:58:45 vault1 kernel: ffff81011fd360c0 0000000000000d40 ffff81011ec4b248 000000031ec4b040
May 29 11:58:45 vault1 kernel: Call Trace:
May 29 11:58:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:45 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:45 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:45 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:45 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:45 vault1 kernel:
May 29 11:58:45 vault1 kernel: fc_dl_13 S ffff81011ed8e058 0 3480 19 (L-TLB)
May 29 11:58:45 vault1 kernel: ffff81011feafea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:45 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011fcb7080
May 29 11:58:45 vault1 kernel: ffff81011fd360c0 0000000000000c43 ffff81011fcb7288 000000031fcb7080
May 29 11:58:45 vault1 kernel: Call Trace:
May 29 11:58:45 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:45 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:45 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:45 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:45 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:45 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:45 vault1 kernel:
May 29 11:58:45 vault1 kernel: scsi_eh_14 S ffff81011ed8e058 0 3493 19 (L-TLB)
May 29 11:58:45 vault1 kernel: ffff81011e873e70 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:45 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81011e6030c0
May 29 11:58:45 vault1 kernel: ffff81011fd360c0 00000000000007e0 ffff81011e6032c8 0000000380263894
May 29 11:58:45 vault1 kernel: Call Trace:
May 29 11:58:45 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff88063f8e>] :scsi_mod:scsi_error_handler+0x6a/0x4dd
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:45 vault1 kernel: [<ffffffff88063f24>] :scsi_mod:scsi_error_handler+0x0/0x4dd
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:45 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:45 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:45 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:45 vault1 kernel:
May 29 11:58:45 vault1 kernel: qla2xxx_14_dp S ffff81007f343638 0 3501 19 (L-TLB)
May 29 11:58:45 vault1 kernel: ffff81011e111ee0 0000000000000046 ffff810074d4c7d0 0000000000000282
May 29 11:58:45 vault1 kernel: ffff81011e111e40 ffff81011e111e40 000000000000000a ffff81011e122080
May 29 11:58:45 vault1 kernel: ffff81007fe100c0 0000000000000afb ffff81011e122288 0000000000000000
May 29 11:58:45 vault1 kernel: Call Trace:
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:45 vault1 kernel: [<ffffffff880ba348>] :qla2xxx:qla2x00_do_dpc+0x39/0x439
May 29 11:58:45 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:45 vault1 kernel: [<ffffffff880ba30f>] :qla2xxx:qla2x00_do_dpc+0x0/0x439
May 29 11:58:45 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:45 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:45 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:45 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:45 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:46 vault1 kernel:
May 29 11:58:46 vault1 kernel: scsi_wq_14 S ffff81007ebd3638 0 3502 19 (L-TLB)
May 29 11:58:46 vault1 kernel: ffff81011e049ea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:46 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011e0570c0
May 29 11:58:46 vault1 kernel: ffff81007e0fe080 000000000036e7a4 ffff81011e0572c8 000000001e0570c0
May 29 11:58:46 vault1 kernel: Call Trace:
May 29 11:58:46 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:46 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:46 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:46 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:46 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:46 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:46 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:46 vault1 kernel:
May 29 11:58:46 vault1 kernel: fc_wq_14 S ffff81011ed8e058 0 3503 19 (L-TLB)
May 29 11:58:46 vault1 kernel: ffff81011f0b3ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:46 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011e117040
May 29 11:58:46 vault1 kernel: ffff81011fd360c0 0000000000000dc9 ffff81011e117248 000000031e117040
May 29 11:58:46 vault1 kernel: Call Trace:
May 29 11:58:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:46 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:46 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:46 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:46 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:46 vault1 kernel:
May 29 11:58:46 vault1 kernel: fc_dl_14 S ffff81011ed8e058 0 3504 19 (L-TLB)
May 29 11:58:46 vault1 kernel: ffff81011e189ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:46 vault1 kernel: 0000000000000000 0000000000000000 0000000000000009 ffff81011f9ea080
May 29 11:58:46 vault1 kernel: ffff81011fd360c0 0000000000000dda ffff81011f9ea288 000000031f9ea080
May 29 11:58:46 vault1 kernel: Call Trace:
May 29 11:58:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:46 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:46 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:46 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:46 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:46 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:46 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:46 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:46 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:46 vault1 kernel:
May 29 11:58:46 vault1 kernel: rpc.rquotad S ffff81011febdad8 0 3740 1 (NOTLB)
May 29 11:58:46 vault1 kernel: ffff81011dd5fb28 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:46 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000001 ffff81011ddca080
May 29 11:58:46 vault1 kernel: ffffffff8058a6e0 000000000002fcdc ffff81011ddca288 0000000000000000
May 29 11:58:46 vault1 kernel: Call Trace:
May 29 11:58:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:46 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:46 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 29 11:58:46 vault1 kernel: [<ffffffff8022ff1d>] do_sys_poll+0x276/0x35f
May 29 11:58:46 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:46 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:46 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:46 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 29 11:58:46 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:46 vault1 kernel: [<ffffffff8020a6c3>] get_page_from_freelist+0x2ab/0x534
May 29 11:58:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:46 vault1 kernel: [<ffffffff802078e8>] find_get_page+0x15/0x4b
May 29 11:58:46 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 29 11:58:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:46 vault1 kernel: [<ffffffff80208c1f>] __handle_mm_fault+0x3cf/0xccf
May 29 11:58:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:46 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:46 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:46 vault1 kernel: [<ffffffff80208d64>] __handle_mm_fault+0x514/0xccf
May 29 11:58:46 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:46 vault1 kernel: [<ffffffff80265bc5>] do_page_fault+0x490/0x7cf
May 29 11:58:46 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:47 vault1 kernel: [<ffffffff8021e472>] sys_close+0x2b/0xd1
May 29 11:58:47 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:47 vault1 kernel: [<ffffffff8024b1fc>] sys_poll+0x32/0x3b
May 29 11:58:47 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:47 vault1 kernel:
May 29 11:58:47 vault1 kernel: lockd S ffff81007fe18db8 0 3743 1 (L-TLB)
May 29 11:58:47 vault1 kernel: ffff81007620de10 0000000000000046 0000000000000000 ffff81007ffe3830
May 29 11:58:47 vault1 kernel: 0000000000000627 ffff81007ffe30c0 0000000000000001 ffff81007ffe30c0
May 29 11:58:47 vault1 kernel: ffff81011fccf080 0000000000003c35 ffff81007ffe32c8 0000000200000046
May 29 11:58:47 vault1 kernel: Call Trace:
May 29 11:58:47 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff882e824e>] :sunrpc:svc_recv+0x2b5/0x44f
May 29 11:58:47 vault1 kernel: [<ffffffff882e8257>] :sunrpc:svc_recv+0x2be/0x44f
May 29 11:58:47 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:47 vault1 kernel: [<ffffffff88315556>] :lockd:lockd+0x139/0x263
May 29 11:58:47 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:47 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:47 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:47 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:47 vault1 kernel: [<ffffffff8831541d>] :lockd:lockd+0x0/0x263
May 29 11:58:47 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:47 vault1 kernel:
May 29 11:58:47 vault1 kernel: rpciod/0 S ffff81011f96e4b8 0 3744 19 (L-TLB)
May 29 11:58:47 vault1 kernel: ffff81011d85dea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:47 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011e7ae0c0
May 29 11:58:47 vault1 kernel: ffffffff8058a6e0 0000000000001725 ffff81011e7ae2c8 000000001e7ae0c0
May 29 11:58:47 vault1 kernel: Call Trace:
May 29 11:58:47 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:47 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:47 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:47 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:47 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:47 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:47 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:47 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:47 vault1 kernel:
May 29 11:58:47 vault1 kernel: rpciod/1 S ffff81007e836df8 0 3745 19 (L-TLB)
May 29 11:58:47 vault1 kernel: ffff81011e8ebea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:47 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011dd85040
May 29 11:58:47 vault1 kernel: ffff81011eca90c0 0000000000001573 ffff81011dd85248 000000011dd85040
May 29 11:58:47 vault1 kernel: Call Trace:
May 29 11:58:47 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:47 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:47 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:47 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:47 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:47 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:47 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:47 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:47 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:47 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:47 vault1 kernel:
May 29 11:58:47 vault1 kernel: rpciod/2 S ffff81007e836df8 0 3746 19 (L-TLB)
May 29 11:58:47 vault1 kernel: ffff81011e2ebea0 0000000000000046 ffffffff80248a3b 0000000000000002
May 29 11:58:47 vault1 kernel: 0000000000000000 0000000000000000 0000000000000005 ffff81011dd83080
May 29 11:58:47 vault1 kernel: ffff81007ffe30c0 00000000000013c6 ffff81011dd83288 000000021dd83080
May 29 11:58:47 vault1 kernel: Call Trace:
May 29 11:58:47 vault1 kernel: [<ffffffff80248a3b>] add_wait_queue+0x13/0x34
May 29 11:58:47 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:47 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:47 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:47 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:47 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:47 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:47 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:47 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:47 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:47 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:47 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:47 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:48 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:48 vault1 kernel:
May 29 11:58:48 vault1 kernel: rpciod/3 S ffff81011ed8e058 0 3747 19 (L-TLB)
May 29 11:58:48 vault1 kernel: ffff81011d86fea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:48 vault1 kernel: 0000000000000000 0000000000000000 0000000000000005 ffff81011d8d10c0
May 29 11:58:48 vault1 kernel: ffff81011fd360c0 000000000000117b ffff81011d8d12c8 000000031d8d10c0
May 29 11:58:48 vault1 kernel: Call Trace:
May 29 11:58:48 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:48 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:48 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:48 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:48 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:48 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:48 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:48 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:48 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:48 vault1 kernel: [<ffffffff80276ee4>] physflat_send_IPI_mask+0x0/0x82
May 29 11:58:48 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:48 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:48 vault1 kernel:
May 29 11:58:48 vault1 kernel: nfsd4 S ffff81007f3424b8 0 3748 19 (L-TLB)
May 29 11:58:48 vault1 kernel: ffff81011e229ea0 0000000000000046 0000000000000000 0000000000000002
May 29 11:58:48 vault1 kernel: 0000000000000000 0000000000000000 000000000000000a ffff81011d8d0040
May 29 11:58:48 vault1 kernel: ffffffff8058a6e0 00000000000018ad ffff81011d8d0248 000000001d8d0040
May 29 11:58:48 vault1 kernel: Call Trace:
May 29 11:58:48 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:48 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:48 vault1 kernel: [<ffffffff80248f5b>] worker_thread+0x0/0x150
May 29 11:58:48 vault1 kernel: [<ffffffff80249045>] worker_thread+0xea/0x150
May 29 11:58:48 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:48 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:48 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:48 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:48 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:48 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:48 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:48 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:48 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:48 vault1 kernel:
May 29 11:58:48 vault1 kernel: nfsd D ffff81007e8de9d8 0 3749 1 (L-TLB)
May 29 11:58:48 vault1 kernel: ffff810075cf3a60 0000000000000046 0000000000000540 ffff810074f460c0
May 29 11:58:48 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff810074f460c0
May 29 11:58:48 vault1 kernel: ffff81007e0c80c0 000000000003c1a3 ffff810074f462c8 0000000274f46830
May 29 11:58:48 vault1 kernel: Call Trace:
May 29 11:58:48 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:48 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:48 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:48 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:48 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:48 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:48 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:48 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:48 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:48 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:48 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:48 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:48 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:48 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:48 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:48 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:48 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:48 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:48 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:48 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:48 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:48 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:48 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:48 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:48 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:48 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:48 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:48 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:48 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:48 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:48 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:48 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:48 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:48 vault1 kernel:
May 29 11:58:48 vault1 kernel: nfsd D ffff81007e8de118 0 3750 1 (L-TLB)
May 29 11:58:48 vault1 kernel: ffff8100765fba60 0000000000000046 0000000000000000 ffff81007ea050c0
May 29 11:58:48 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007ea050c0
May 29 11:58:48 vault1 kernel: ffffffff8058a6e0 000000000004b18a ffff81007ea052c8 000000007ea05830
May 29 11:58:48 vault1 kernel: Call Trace:
May 29 11:58:48 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:48 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:48 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:48 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:48 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:48 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:48 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:48 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:48 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:48 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:48 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:48 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:48 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:49 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:49 vault1 kernel: [<ffffffff882ec9af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 29 11:58:49 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:49 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:49 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:49 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:49 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:49 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:49 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:49 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:49 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:49 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:49 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:49 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:49 vault1 kernel:
May 29 11:58:49 vault1 kernel: nfsd D ffff81007e8de9d8 0 3751 1 (L-TLB)
May 29 11:58:49 vault1 kernel: ffff81007618fa60 0000000000000046 0000000000000000 ffff81007e9fd040
May 29 11:58:49 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007e9fd040
May 29 11:58:49 vault1 kernel: ffff81011fccf080 000000000005db1b ffff81007e9fd248 000000027e9fd7b0
May 29 11:58:49 vault1 kernel: Call Trace:
May 29 11:58:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:49 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:49 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:49 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:49 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:49 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:49 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:49 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:49 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:49 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:49 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:49 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:49 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:49 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:49 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:49 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:49 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:49 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:49 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:49 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:49 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:49 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:49 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:49 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:49 vault1 kernel:
May 29 11:58:49 vault1 kernel: nfsd D ffff81007e8de118 0 3752 1 (L-TLB)
May 29 11:58:49 vault1 kernel: ffff81007699fa60 0000000000000046 0000000000000540 ffff81007e4c5080
May 29 11:58:49 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007e4c5080
May 29 11:58:49 vault1 kernel: ffff81007e1b9040 000000000001b7e6 ffff81007e4c5288 000000007e4c57f0
May 29 11:58:49 vault1 kernel: Call Trace:
May 29 11:58:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:49 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:49 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:49 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:49 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:49 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:49 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:49 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:49 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:49 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:49 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:49 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:49 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:49 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:49 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:49 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:49 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:49 vault1 kernel: [<ffffffff882ec9af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 29 11:58:49 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:49 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:49 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:49 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:49 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:49 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:50 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:50 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:50 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:50 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:50 vault1 kernel:
May 29 11:58:50 vault1 kernel: nfsd D ffff81007e8de118 0 3753 1 (L-TLB)
May 29 11:58:50 vault1 kernel: ffff81007663ba60 0000000000000046 0000000000000000 ffff81007e1b9040
May 29 11:58:50 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007e1b9040
May 29 11:58:50 vault1 kernel: ffffffff8058a6e0 00000000000396b7 ffff81007e1b9248 000000007e1b97b0
May 29 11:58:50 vault1 kernel: Call Trace:
May 29 11:58:50 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:50 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:50 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:50 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:50 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:50 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:50 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:50 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:50 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:50 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:50 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:50 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:50 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:50 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:50 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:50 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:50 vault1 kernel: [<ffffffff882ec9af>] :sunrpc:sunrpc_cache_lookup+0x30/0x131
May 29 11:58:50 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:50 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:50 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:50 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:50 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:50 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:50 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:50 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:50 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:50 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:50 vault1 kernel:
May 29 11:58:50 vault1 kernel: nfsd D ffff81007e8de9d8 0 3754 1 (L-TLB)
May 29 11:58:50 vault1 kernel: ffff810076203a60 0000000000000046 0000000000000540 ffff81007e0c80c0
May 29 11:58:50 vault1 kernel: 0000000000000001 ffffffff802a0de5 0000000000000009 ffff81007e0c80c0
May 29 11:58:50 vault1 kernel: ffff81007e9fd040 0000000000051705 ffff81007e0c82c8 000000027e0c8830
May 29 11:58:50 vault1 kernel: Call Trace:
May 29 11:58:50 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:50 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:50 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:50 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:50 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:50 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:50 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:50 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:50 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:50 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:50 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:50 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:50 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:50 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:50 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:50 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:50 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:50 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:50 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:50 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:50 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:50 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:50 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:50 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:50 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:50 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:50 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:50 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:50 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:50 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:50 vault1 kernel:
May 29 11:58:50 vault1 kernel: nfsd D ffff81007e8de9d8 0 3755 1 (L-TLB)
May 29 11:58:50 vault1 kernel: ffff810071643a60 0000000000000046 0000000000000540 ffff81007fc350c0
May 29 11:58:50 vault1 kernel: 0000000000000001 ffffffff802a0de5 000000000000000a ffff81007fc350c0
May 29 11:58:51 vault1 kernel: ffff81011fd34040 000000000001ead4 ffff81007fc352c8 000000037fc35830
May 29 11:58:51 vault1 kernel: Call Trace:
May 29 11:58:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:51 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:51 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:51 vault1 kernel: [<ffffffff80262019>] __mutex_lock_slowpath+0x13b/0x25b
May 29 11:58:51 vault1 kernel: [<ffffffff80262021>] __mutex_lock_slowpath+0x143/0x25b
May 29 11:58:51 vault1 kernel: [<ffffffff8024e63b>] d_find_alias+0x3a/0x42
May 29 11:58:51 vault1 kernel: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:51 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:51 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:51 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:51 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:51 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:51 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:51 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:51 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:51 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:51 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:51 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:51 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:51 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:51 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:51 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:51 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:51 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:51 vault1 kernel:
May 29 11:58:51 vault1 kernel: nfsd D ffff81007e8de9d8 0 3756 1 (L-TLB)
May 29 11:58:51 vault1 kernel: ffff8100761f3620 0000000000000046 0000000000000002 0000000000000000
May 29 11:58:51 vault1 kernel: 0000000000000000 0000000000000046 0000000000000009 ffff81007ea200c0
May 29 11:58:51 vault1 kernel: ffff81007e0c80c0 000000000000c278 ffff81007ea202c8 000000037ea208a0
May 29 11:58:51 vault1 kernel: Call Trace:
May 29 11:58:51 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:51 vault1 kernel: [<ffffffff802616cf>] io_schedule+0x3f/0x67
May 29 11:58:51 vault1 kernel: [<ffffffff882456d9>] :jfs:txBeginAnon+0x85/0x13c
May 29 11:58:51 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:51 vault1 kernel: [<ffffffff8823f2f2>] :jfs:extAlloc+0x3d/0x423
May 29 11:58:51 vault1 kernel: [<ffffffff8823f786>] :jfs:extHint+0xae/0xfc
May 29 11:58:51 vault1 kernel: [<ffffffff8822c65c>] :jfs:jfs_get_block+0x167/0x237
May 29 11:58:51 vault1 kernel: [<ffffffff802e5c6c>] nobh_prepare_write+0xd3/0x458
May 29 11:58:51 vault1 kernel: [<ffffffff8822c4f5>] :jfs:jfs_get_block+0x0/0x237
May 29 11:58:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:51 vault1 kernel: [<ffffffff8020c722>] add_to_page_cache+0x2d/0x8c
May 29 11:58:51 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:51 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:51 vault1 kernel: [<ffffffff80263540>] _write_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff80210354>] generic_file_buffered_write+0x27b/0x77b
May 29 11:58:51 vault1 kernel: [<ffffffff802606f4>] __sched_text_start+0x184/0x960
May 29 11:58:51 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:51 vault1 kernel: [<ffffffff8020e533>] current_fs_time+0x3b/0x40
May 29 11:58:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:51 vault1 kernel: [<ffffffff80216881>] __generic_file_aio_write_nolock+0x378/0x3eb
May 29 11:58:51 vault1 kernel: [<ffffffff80262123>] __mutex_lock_slowpath+0x245/0x25b
May 29 11:58:51 vault1 kernel: [<ffffffff8026212e>] __mutex_lock_slowpath+0x250/0x25b
May 29 11:58:51 vault1 kernel: [<ffffffff802224a4>] generic_file_aio_write+0x61/0xc1
May 29 11:58:51 vault1 kernel: [<ffffffff80222443>] generic_file_aio_write+0x0/0xc1
May 29 11:58:51 vault1 kernel: [<ffffffff802d38c7>] do_sync_readv_writev+0xc0/0x107
May 29 11:58:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff8029a0f2>] autoremove_wake_function+0x0/0x2e
May 29 11:58:51 vault1 kernel: [<ffffffff8020c870>] cache_alloc_debugcheck_after+0xef/0x139
May 29 11:58:51 vault1 kernel: [<ffffffff802d3776>] rw_copy_check_uvector+0x4b/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff802cf576>] __kmalloc+0x131/0x13d
May 29 11:58:51 vault1 kernel: [<ffffffff802d3797>] rw_copy_check_uvector+0x6c/0xdc
May 29 11:58:51 vault1 kernel: [<ffffffff802d3d38>] do_readv_writev+0xd7/0x1ae
May 29 11:58:51 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:51 vault1 kernel: [<ffffffff8022400d>] file_move+0x1d/0x49
May 29 11:58:51 vault1 kernel: [<ffffffff8832fc8d>] :nfsd:nfsd_vfs_write+0xe9/0x2dd
May 29 11:58:51 vault1 kernel: [<ffffffff8021f168>] __dentry_open+0x104/0x1b0
May 29 11:58:51 vault1 kernel: [<ffffffff88330305>] :nfsd:nfsd_open+0x13c/0x16f
May 29 11:58:51 vault1 kernel: [<ffffffff8833053b>] :nfsd:nfsd_write+0xc5/0xe2
May 29 11:58:51 vault1 kernel: [<ffffffff88337611>] :nfsd:nfsd3_proc_write+0xfb/0x11a
May 29 11:58:51 vault1 kernel: [<ffffffff8832c244>] :nfsd:nfsd_dispatch+0xdd/0x19e
May 29 11:58:51 vault1 kernel: [<ffffffff882e5353>] :sunrpc:svc_process+0x3ea/0x6f3
May 29 11:58:51 vault1 kernel: [<ffffffff8832c7fe>] :nfsd:nfsd+0x194/0x2b2
May 29 11:58:51 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:51 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:51 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:51 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:51 vault1 kernel: [<ffffffff8832c66a>] :nfsd:nfsd+0x0/0x2b2
May 29 11:58:51 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:51 vault1 kernel:
May 29 11:58:51 vault1 kernel: rpc.mountd S ffff81007e8dee38 0 3759 1 (NOTLB)
May 29 11:58:51 vault1 kernel: ffff8100766c5a18 0000000000000046 0000000000000000 0000000000000000
May 29 11:58:51 vault1 kernel: 0000000180248a3b 0000000000000002 0000000000000009 ffff81007e618040
May 29 11:58:51 vault1 kernel: ffff81011fc78040 00000000000120f1 ffff81007e618248 0000000100000000
May 29 11:58:51 vault1 kernel: Call Trace:
May 29 11:58:51 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:52 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:52 vault1 kernel: [<ffffffff80213f07>] tcp_poll+0x1f/0x133
May 29 11:58:52 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:52 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:52 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:52 vault1 last message repeated 4 times
May 29 11:58:52 vault1 kernel: [<ffffffff8020b406>] kfree+0x1a8/0x274
May 29 11:58:52 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff8020b406>] kfree+0x1a8/0x274
May 29 11:58:52 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 29 11:58:52 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:52 vault1 kernel: [<ffffffff8020b4c4>] kfree+0x266/0x274
May 29 11:58:52 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:52 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff882ec363>] :sunrpc:cache_write+0x45/0xc0
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:52 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:52 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:52 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:52 vault1 kernel:
May 29 11:58:52 vault1 kernel: bash S ffff81011fff8998 0 4891 3311 (NOTLB)
May 29 11:58:52 vault1 kernel: ffff8100339c3ec8 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:52 vault1 kernel: 0000000000000000 0000000200000002 0000000000000009 ffff81004a1760c0
May 29 11:58:52 vault1 kernel: ffff81011fccf080 00000000000a29d2 ffff81004a1762c8 0000000200000000
May 29 11:58:52 vault1 kernel: Call Trace:
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff8022920b>] do_wait+0x94c/0xa75
May 29 11:58:52 vault1 kernel: [<ffffffff8022922e>] do_wait+0x96f/0xa75
May 29 11:58:52 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:52 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:52 vault1 kernel:
May 29 11:58:52 vault1 kernel: ssh S ffff81007f343a98 0 4918 4891 (NOTLB)
May 29 11:58:52 vault1 kernel: ffff810116ffba18 0000000000000046 0000000000000000 0000000000000046
May 29 11:58:52 vault1 kernel: ffffffff8022edfd 0000000000000046 000000000000000a ffff810091f5d0c0
May 29 11:58:52 vault1 kernel: ffff81011fd360c0 0000000000073aab ffff810091f5d2c8 0000000391f5d0c0
May 29 11:58:52 vault1 kernel: Call Trace:
May 29 11:58:52 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 29 11:58:52 vault1 kernel: [<ffffffff8023a80a>] tty_ldisc_deref+0x62/0x75
May 29 11:58:52 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:52 vault1 kernel: [<ffffffff80226c98>] tty_poll+0x5f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:52 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:52 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:52 vault1 last message repeated 4 times
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff802174d3>] release_console_sem+0x44/0x1eb
May 29 11:58:52 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:52 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:52 vault1 kernel: [<ffffffff802638fc>] _spin_unlock_irqrestore+0x3e/0x44
May 29 11:58:52 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:52 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 29 11:58:52 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:52 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:52 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:52 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:52 vault1 kernel:
May 29 11:58:52 vault1 kernel: sshd S ffff81007f1149d8 0 6871 3092 (NOTLB)
May 29 11:58:52 vault1 kernel: ffff81011e20da18 0000000000000046 0000000000000000 0000000000000046
May 29 11:58:52 vault1 kernel: ffffffff8022edfd 0000000000000046 000000000000000a ffff81010bea70c0
May 29 11:58:52 vault1 kernel: ffff81011fc78040 0000000000012ee2 ffff81010bea72c8 000000010bea70c0
May 29 11:58:52 vault1 kernel: Call Trace:
May 29 11:58:52 vault1 kernel: [<ffffffff8022edfd>] __wake_up+0x22/0x4e
May 29 11:58:52 vault1 kernel: [<ffffffff8023a80a>] tty_ldisc_deref+0x62/0x75
May 29 11:58:52 vault1 kernel: [<ffffffff80261724>] schedule_timeout+0x1e/0xad
May 29 11:58:52 vault1 kernel: [<ffffffff80226c98>] tty_poll+0x5f/0x6d
May 29 11:58:52 vault1 kernel: [<ffffffff80211e12>] do_select+0x3fe/0x462
May 29 11:58:52 vault1 kernel: [<ffffffff8021f267>] __pollwait+0x0/0xe1
May 29 11:58:52 vault1 kernel: [<ffffffff80287a44>] default_wake_function+0x0/0xe
May 29 11:58:53 vault1 last message repeated 3 times
May 29 11:58:53 vault1 kernel: [<ffffffff802313cd>] release_sock+0x2f/0xd6
May 29 11:58:53 vault1 kernel: [<ffffffff8028de66>] local_bh_enable_ip+0xe4/0xea
May 29 11:58:53 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:53 vault1 kernel: [<ffffffff80227651>] tcp_sendmsg+0x9ab/0xab5
May 29 11:58:53 vault1 kernel: [<ffffffff80227651>] tcp_sendmsg+0x9ab/0xab5
May 29 11:58:53 vault1 kernel: [<ffffffff8024671c>] sock_aio_write+0x110/0x128
May 29 11:58:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:53 vault1 kernel: [<ffffffff802d81fe>] core_sys_select+0x1bb/0x264
May 29 11:58:53 vault1 kernel: [<ffffffff80218507>] do_sync_write+0xc9/0x10c
May 29 11:58:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:53 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff8020cc19>] dnotify_parent+0x1f/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff80216aff>] sys_select+0xc1/0x183
May 29 11:58:53 vault1 kernel: [<ffffffff80263178>] trace_hardirqs_on_thunk+0x35/0x37
May 29 11:58:53 vault1 kernel: [<ffffffff8025c11e>] system_call+0x7e/0x83
May 29 11:58:53 vault1 kernel:
May 29 11:58:53 vault1 kernel: pdflush D ffff81007e8de9d8 0 8692 19 (L-TLB)
May 29 11:58:53 vault1 kernel: ffff81005aba3a20 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:53 vault1 kernel: 0000000137d30000 0000000000000002 000000000000000a ffff81004687d0c0
May 29 11:58:53 vault1 kernel: ffff81011fd360c0 0000000000003dcf ffff81004687d2c8 0000000300000001
May 29 11:58:53 vault1 kernel: Call Trace:
May 29 11:58:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:53 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:53 vault1 kernel: [<ffffffff80262fa6>] __down_write_nested+0x7e/0x96
May 29 11:58:53 vault1 kernel: [<ffffffff8822c53a>] :jfs:jfs_get_block+0x45/0x237
May 29 11:58:53 vault1 kernel: [<ffffffff8022307f>] __up_read+0x19/0x7f
May 29 11:58:53 vault1 kernel: [<ffffffff802e9478>] __mpage_writepage+0x1f5/0x55d
May 29 11:58:53 vault1 kernel: [<ffffffff8822c4f5>] :jfs:jfs_get_block+0x0/0x237
May 29 11:58:53 vault1 kernel: [<ffffffff802638bb>] _read_unlock_irq+0x24/0x27
May 29 11:58:53 vault1 kernel: [<ffffffff802c0835>] dec_zone_page_state+0x33/0x3a
May 29 11:58:53 vault1 kernel: [<ffffffff8029fc0f>] mark_held_locks+0x4a/0x6c
May 29 11:58:53 vault1 kernel: [<ffffffff802c0835>] dec_zone_page_state+0x33/0x3a
May 29 11:58:53 vault1 kernel: [<ffffffff8029fe09>] trace_hardirqs_on+0x124/0x14f
May 29 11:58:53 vault1 kernel: [<ffffffff8021d493>] mpage_writepages+0x228/0x375
May 29 11:58:53 vault1 kernel: [<ffffffff8822c0bb>] :jfs:jfs_writepage+0x0/0xf
May 29 11:58:53 vault1 kernel: [<ffffffff8822c4f5>] :jfs:jfs_get_block+0x0/0x237
May 29 11:58:53 vault1 kernel: [<ffffffff802593b8>] do_writepages+0x23/0x30
May 29 11:58:53 vault1 kernel: [<ffffffff80230487>] __writeback_single_inode+0x1c0/0x338
May 29 11:58:53 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff80263584>] _read_unlock+0x17/0x20
May 29 11:58:53 vault1 kernel: [<ffffffff80221c26>] sync_sb_inodes+0x1a6/0x267
May 29 11:58:53 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff8024f304>] writeback_inodes+0x95/0xee
May 29 11:58:53 vault1 kernel: [<ffffffff802be735>] wb_kupdate+0x9e/0x112
May 29 11:58:53 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 29 11:58:53 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 29 11:58:53 vault1 kernel: [<ffffffff80254e8e>] pdflush+0x13c/0x1e7
May 29 11:58:53 vault1 kernel: [<ffffffff802be697>] wb_kupdate+0x0/0x112
May 29 11:58:53 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:53 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:53 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:53 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:53 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:53 vault1 kernel:
May 29 11:58:53 vault1 kernel: pdflush S ffff81007e8de9d8 0 10661 19 (L-TLB)
May 29 11:58:53 vault1 kernel: ffff81007e775ed0 0000000000000046 0000000000000000 ffffffff802a0de5
May 29 11:58:53 vault1 kernel: 0000000180598080 0000000000000002 000000000000000a ffff810055df5040
May 29 11:58:53 vault1 kernel: ffff81011fccf080 0000000000003d7f ffff810055df5248 0000000200000000
May 29 11:58:53 vault1 kernel: Call Trace:
May 29 11:58:53 vault1 kernel: [<ffffffff802a0de5>] __lock_acquire+0xb4a/0xbdf
May 29 11:58:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:53 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff80299f53>] keventd_create_kthread+0x0/0x6d
May 29 11:58:53 vault1 kernel: [<ffffffff80254d52>] pdflush+0x0/0x1e7
May 29 11:58:53 vault1 kernel: [<ffffffff80254e21>] pdflush+0xcf/0x1e7
May 29 11:58:53 vault1 kernel: [<ffffffff8023312e>] kthread+0xd0/0xff
May 29 11:58:53 vault1 kernel: [<ffffffff8025cfa8>] child_rip+0xa/0x12
May 29 11:58:53 vault1 kernel: [<ffffffff80263894>] _spin_unlock_irq+0x24/0x27
May 29 11:58:53 vault1 kernel: [<ffffffff8025c6bc>] restore_args+0x0/0x30
May 29 11:58:53 vault1 kernel: [<ffffffff8023305e>] kthread+0x0/0xff
May 29 11:58:53 vault1 kernel: [<ffffffff8025cf9e>] child_rip+0x0/0x12
May 29 11:58:53 vault1 kernel:
May 29 11:58:53 vault1 kernel:
May 29 11:58:53 vault1 kernel: Showing all locks held in the system:
May 29 11:58:53 vault1 kernel: 1 lock held by syslogd/2963:
May 29 11:58:53 vault1 kernel: #0: (&inode->i_mutex){--..}, at: [<ffffffff8024e516>] do_fsync+0x41/0xa4
May 29 11:58:53 vault1 kernel: 1 lock held by mingetty/3312:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 1 lock held by mingetty/3313:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 1 lock held by mingetty/3314:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 1 lock held by mingetty/3315:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 1 lock held by mingetty/3316:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 1 lock held by bash/3377:
May 29 11:58:53 vault1 kernel: #0: (&tty->atomic_read_lock){--..}, at: [<ffffffff8022a64c>] read_chan+0x201/0x609
May 29 11:58:53 vault1 kernel: 2 locks held by nfsd/3749:
May 29 11:58:53 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3750:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3751:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3752:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3753:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3754:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 2 locks held by nfsd/3755:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: 3 locks held by nfsd/3756:
May 29 11:58:54 vault1 kernel: #0: (hash_sem){----}, at: [<ffffffff8832c7e7>] nfsd+0x17d/0x2b2 [nfsd]
May 29 11:58:54 vault1 kernel: #1: (&inode->i_mutex){--..}, at: [<ffffffff8022248f>] generic_file_aio_write+0x4c/0xc1
May 29 11:58:54 vault1 kernel: #2: (&jfs_ip->rdwrlock#2){----}, at: [<ffffffff8822c53a>] jfs_get_block+0x45/0x237 [jfs]
May 29 11:58:54 vault1 kernel: 2 locks held by bash/6873:
May 29 11:58:54 vault1 kernel: #0: (sysrq_key_table_lock){....}, at: [<ffffffff80395b2a>] __handle_sysrq+0x22/0x128
May 29 11:58:54 vault1 kernel: #1: (tasklist_lock){..-?}, at: [<ffffffff8029ebd4>] debug_show_all_locks+0x3e/0x152
May 29 11:58:54 vault1 kernel: 2 locks held by pdflush/8692:
May 29 11:58:54 vault1 kernel: #0: (&type->s_umount_key#20){----}, at: [<ffffffff8024f2e2>] writeback_inodes+0x73/0xee
May 29 11:58:54 vault1 kernel: #1: (&jfs_ip->rdwrlock#2){----}, at: [<ffffffff8822c53a>] jfs_get_block+0x45/0x237 [jfs]
May 29 11:58:54 vault1 kernel:
May 29 11:58:54 vault1 kernel: =============================================
May 29 11:58:54 vault1 kernel:
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Apparent Deadlock with nfsd/jfs on 2.6.21.1 under bonnie.
2007-05-29 17:16 ` Roger Heflin
@ 2007-05-29 18:10 ` Dave Kleikamp
2007-05-29 18:14 ` Roger Heflin
0 siblings, 1 reply; 9+ messages in thread
From: Dave Kleikamp @ 2007-05-29 18:10 UTC (permalink / raw)
To: Roger Heflin; +Cc: linux-kernel
On Tue, 2007-05-29 at 12:16 -0500, Roger Heflin wrote:
> Dave,
>
> Apparently there appears to be another different similar lockup,
> The MTBF has risen from 1-2 hours without that patch to >100 hours,
> so I am fairly sure the patch did correct the original lockup, or
> at the very least make it a lot less likely.
>
> I hit the machine across NFS for 5 days before it deadlocked, before
> the patch I could only get an hour or two (2-4 different tries).
>
> Given that pdflush is "D" it does not appear to be an NFS issue.
>
> Included is the sysrq-t.
>
> This is with 2.6.21.1 + the JFSIO patch.
Is the system still in this state? Can you cat /proc/fs/jfs/TxAnchor
(if CONFIG_JFS_DEBUG is defined) and /proc/fs/jfs/txstats (if
CONFIG_JFS_STATISTICS is defined)?
Thanks,
Shaggy
--
David Kleikamp
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Apparent Deadlock with nfsd/jfs on 2.6.21.1 under bonnie.
2007-05-29 18:10 ` Dave Kleikamp
@ 2007-05-29 18:14 ` Roger Heflin
2007-05-29 18:30 ` Dave Kleikamp
0 siblings, 1 reply; 9+ messages in thread
From: Roger Heflin @ 2007-05-29 18:14 UTC (permalink / raw)
To: Dave Kleikamp; +Cc: linux-kernel
Dave Kleikamp wrote:
> On Tue, 2007-05-29 at 12:16 -0500, Roger Heflin wrote:
>
>> Dave,
>>
>> Apparently there appears to be another different similar lockup,
>> The MTBF has risen from 1-2 hours without that patch to >100 hours,
>> so I am fairly sure the patch did correct the original lockup, or
>> at the very least make it a lot less likely.
>>
>> I hit the machine across NFS for 5 days before it deadlocked, before
>> the patch I could only get an hour or two (2-4 different tries).
>>
>> Given that pdflush is "D" it does not appear to be an NFS issue.
>>
>> Included is the sysrq-t.
>>
>> This is with 2.6.21.1 + the JFSIO patch.
>
> Is the system still in this state? Can you cat /proc/fs/jfs/TxAnchor
> (if CONFIG_JFS_DEBUG is defined) and /proc/fs/jfs/txstats (if
> CONFIG_JFS_STATISTICS is defined)?
>
> Thanks,
> Shaggy
Yes, the machine is still in that state.
Apparently I don't have either of those configured.
Anything else that we can collect before I rebuild the kernel with
those options setup?
Roger
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Apparent Deadlock with nfsd/jfs on 2.6.21.1 under bonnie.
2007-05-29 18:14 ` Roger Heflin
@ 2007-05-29 18:30 ` Dave Kleikamp
0 siblings, 0 replies; 9+ messages in thread
From: Dave Kleikamp @ 2007-05-29 18:30 UTC (permalink / raw)
To: Roger Heflin; +Cc: linux-kernel
On Tue, 2007-05-29 at 13:14 -0500, Roger Heflin wrote:
> Dave Kleikamp wrote:
> > On Tue, 2007-05-29 at 12:16 -0500, Roger Heflin wrote:
> >
> >> Dave,
> >>
> >> Apparently there appears to be another different similar lockup,
> >> The MTBF has risen from 1-2 hours without that patch to >100 hours,
> >> so I am fairly sure the patch did correct the original lockup, or
> >> at the very least make it a lot less likely.
> >>
> >> I hit the machine across NFS for 5 days before it deadlocked, before
> >> the patch I could only get an hour or two (2-4 different tries).
> >>
> >> Given that pdflush is "D" it does not appear to be an NFS issue.
> >>
> >> Included is the sysrq-t.
> >>
> >> This is with 2.6.21.1 + the JFSIO patch.
> >
> > Is the system still in this state? Can you cat /proc/fs/jfs/TxAnchor
> > (if CONFIG_JFS_DEBUG is defined) and /proc/fs/jfs/txstats (if
> > CONFIG_JFS_STATISTICS is defined)?
> >
> > Thanks,
> > Shaggy
>
> Yes, the machine is still in that state.
>
> Apparently I don't have either of those configured.
>
> Anything else that we can collect before I rebuild the kernel with
> those options setup?
No. I think I may have found something to explain the hang. I need to
look a bit closer. Go ahead and rebuild the kernel with those options
in case I'm mistaken.
Thanks,
Shaggy
>
> Roger
>
--
David Kleikamp
IBM Linux Technology Center
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2007-05-29 18:30 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-05-15 14:08 Apparent Deadlock with nfsd/jfs on 2.6.21.1 under bonnie Roger Heflin
[not found] ` <pan.2007.05.15.22.03.07.359081@linux.vnet.ibm.com>
2007-05-15 22:06 ` Roger Heflin
2007-05-15 22:06 ` Roger Heflin
2007-05-17 14:37 ` Roger Heflin
2007-05-17 14:48 ` Dave Kleikamp
2007-05-29 17:16 ` Roger Heflin
2007-05-29 18:10 ` Dave Kleikamp
2007-05-29 18:14 ` Roger Heflin
2007-05-29 18:30 ` Dave Kleikamp
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.