* [bug] stuck localhost TCP connections, v2.6.26-rc3+
@ 2008-05-26 11:56 Ingo Molnar
2008-05-26 13:28 ` Ilpo Järvinen
` (3 more replies)
0 siblings, 4 replies; 96+ messages in thread
From: Ingo Molnar @ 2008-05-26 11:56 UTC (permalink / raw)
To: linux-kernel; +Cc: netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
in an overnight -tip testruns that is based on recent -git i got two
stuck TCP connections:
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED
tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015 ESTABLISHED
on a previously reliable machine. That connection has been stuck for 9
hours so it does not time out, etc. - and the distcc run that goes over
that connection is stuck as well.
kernel config is attached.
in terms of debugging there's not much i can do i'm afraid. It's not
possible to get a tcpdump of this incident, given the extreme amount of
load these testboxes handle. This problem started sometime around rc3
and it occured on two boxes (on a laptop and on a desktop), both are SMP
Core2Duo based systems. I never saw this problem before on thousands of
similar bootups, so i'm 99.9% sure the bug is either new or became
easier to trigger.
It's not possible to bisect it as it needs up to 12 hours of heavy
workload to trigger. The incident happened about 5 times since the first
incident a couple of days ago - 4 times on one box and once on another
box. The first failing head i became aware of was 78b58e549a3098. (-tip
has other changes beyond -git but changes nothing in networking.)
One clue (which might or might not matter) is that distcc is one of the
very few applications that makes use of sendfile().
Ingo
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.26-rc3
# Sat May 24 04:28:24 2008
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_DEFCONFIG_LIST="arch/x86/configs/x86_64_defconfig"
# CONFIG_GENERIC_LOCKBREAK is not set
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
# CONFIG_GENERIC_GPIO is not set
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
# CONFIG_ARCH_HAS_ILOG2_U32 is not set
# CONFIG_ARCH_HAS_ILOG2_U64 is not set
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_AOUT=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
# CONFIG_KTIME_SCALAR is not set
CONFIG_BOOTPARAM_SUPPORT=y
#
# General setup
#
CONFIG_EXPERIMENTAL=y
# CONFIG_BROKEN_BOOT_ALLOWED3 is not set
# CONFIG_BROKEN_BOOT_EUROPE is not set
# CONFIG_BROKEN_BOOT_TITAN is not set
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_SWAP=y
# CONFIG_SYSVIPC is not set
# CONFIG_POSIX_MQUEUE is not set
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
# CONFIG_AUDIT is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=20
# CONFIG_CGROUPS is not set
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_GROUP_SCHED is not set
CONFIG_BOOTPARAM_MAXCPUS_1=y
CONFIG_BOOTPARAM_NOSMP=y
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
# CONFIG_COMPAT_BRK is not set
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
CONFIG_MARKERS=y
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
# CONFIG_HAVE_DMA_ATTRS is not set
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_HAVE_IMMEDIATE=y
CONFIG_IMMEDIATE=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
# CONFIG_MODULE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
# CONFIG_KMOD is not set
CONFIG_BLOCK=y
CONFIG_BLK_DEV_IO_TRACE=y
# CONFIG_BLK_DEV_BSG is not set
CONFIG_BLOCK_COMPAT=y
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=m
CONFIG_IOSCHED_DEADLINE=m
# CONFIG_IOSCHED_CFQ is not set
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
# CONFIG_DEFAULT_CFQ is not set
CONFIG_DEFAULT_NOOP=y
CONFIG_DEFAULT_IOSCHED="noop"
CONFIG_CLASSIC_RCU=y
#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
# CONFIG_NO_HZ is not set
CONFIG_BOOTPARAM_NO_HZ_OFF=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
# CONFIG_BOOTPARAM_HIGHRES_OFF is not set
CONFIG_SMP=y
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_X86_RDC321X is not set
# CONFIG_X86_VSMP is not set
CONFIG_PARAVIRT_GUEST=y
# CONFIG_KVM_CLOCK is not set
# CONFIG_KVM_GUEST is not set
# CONFIG_PARAVIRT is not set
# CONFIG_MEMTEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
CONFIG_MK8=y
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_CPU=y
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_INTERNODE_CACHE_BYTES=64
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
# CONFIG_X86_DS is not set
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
# CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=8
# CONFIG_SCHED_SMT is not set
# CONFIG_SCHED_MC is not set
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_MCE is not set
CONFIG_I8K=y
CONFIG_MICROCODE=y
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=m
# CONFIG_X86_CPUID is not set
CONFIG_NUMA=y
CONFIG_K8_NUMA=y
CONFIG_X86_64_ACPI_NUMA=y
CONFIG_NODES_SPAN_OTHER_NODES=y
# CONFIG_NUMA_EMU is not set
CONFIG_NODES_SHIFT=6
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ILLEGAL_POINTER_VALUE=0xffffc10000000000
CONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not set
# CONFIG_DISCONTIGMEM_MANUAL is not set
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_NEED_MULTIPLE_NODES=y
CONFIG_HAVE_MEMORY_PRESENT=y
# CONFIG_SPARSEMEM_STATIC is not set
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y
# CONFIG_MEMORY_HOTPLUG is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_MIGRATION=y
CONFIG_RESOURCES_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MTRR=y
# CONFIG_MTRR_SANITIZER is not set
CONFIG_X86_PAT=y
CONFIG_EFI=y
# CONFIG_SECCOMP is not set
CONFIG_CC_STACKPROTECTOR_ALL=y
CONFIG_CC_STACKPROTECTOR=y
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PHYSICAL_START=0x200000
CONFIG_RELOCATABLE=y
CONFIG_PHYSICAL_ALIGN=0x200000
# CONFIG_HOTPLUG_CPU is not set
CONFIG_COMPAT_VDSO=y
CONFIG_BOOTPARAM_NOAPIC=y
# CONFIG_BOOTPARAM_NOLAPIC is not set
CONFIG_BOOTPARAM_NOLAPIC_TIMER=y
# CONFIG_BOOTPARAM_LAPIC is not set
# CONFIG_BOOTPARAM_HPET_DISABLE is not set
# CONFIG_BOOTPARAM_IDLE_MWAIT is not set
# CONFIG_BOOTPARAM_IDLE_POLL is not set
CONFIG_BOOTPARAM_HIGHMEM_512M=y
CONFIG_BOOTPARAM_NOPAT=y
CONFIG_BOOTPARAM_NOTSC=y
# CONFIG_BOOTPARAM_PCI_NOACPI is not set
# CONFIG_BOOTPARAM_PCI_NOMSI is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_HAVE_ARCH_EARLY_PFN_TO_NID=y
#
# Power management options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
CONFIG_ACPI=y
# CONFIG_ACPI_PROCFS is not set
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_SYSFS_POWER=y
# CONFIG_ACPI_PROC_EVENT is not set
# CONFIG_ACPI_AC is not set
CONFIG_ACPI_BATTERY=m
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=m
# CONFIG_ACPI_FAN is not set
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_BAY=y
# CONFIG_ACPI_PROCESSOR is not set
CONFIG_ACPI_NUMA=y
CONFIG_ACPI_WMI=y
CONFIG_ACPI_ASUS=y
# CONFIG_ACPI_TOSHIBA is not set
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_DEBUG=y
CONFIG_ACPI_DEBUG_FUNC_TRACE=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=m
# CONFIG_ACPI_SBS is not set
#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCI_MMCONFIG is not set
CONFIG_PCI_DOMAINS=y
CONFIG_DMAR=y
CONFIG_DMAR_GFX_WA=y
CONFIG_DMAR_FLOPPY_WA=y
CONFIG_PCIEPORTBUS=y
CONFIG_HOTPLUG_PCI_PCIE=m
# CONFIG_PCIEAER is not set
CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEBUG=y
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
CONFIG_PCI_LEGACY=y
# CONFIG_HT_IRQ is not set
CONFIG_ISA_DMA_API=y
CONFIG_K8_NB=y
CONFIG_PCCARD=y
# CONFIG_PCMCIA_DEBUG is not set
# CONFIG_PCMCIA is not set
CONFIG_CARDBUS=y
#
# PC-card bridges
#
CONFIG_YENTA=y
CONFIG_YENTA_O2=y
CONFIG_YENTA_RICOH=y
CONFIG_YENTA_TI=y
CONFIG_YENTA_ENE_TUNE=y
CONFIG_YENTA_TOSHIBA=y
CONFIG_PCCARD_NONSTATIC=y
CONFIG_HOTPLUG_PCI=m
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_ACPI=m
CONFIG_HOTPLUG_PCI_ACPI_IBM=m
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=m
# CONFIG_HOTPLUG_PCI_CPCI_GENERIC is not set
CONFIG_HOTPLUG_PCI_SHPC=m
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
#
# Networking
#
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_MMAP is not set
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=y
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
CONFIG_INET_AH=m
# CONFIG_INET_ESP is not set
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=y
CONFIG_INET_XFRM_MODE_TRANSPORT=y
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_LRO=y
# CONFIG_INET_DIAG is not set
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=y
CONFIG_TCP_CONG_CUBIC=m
# CONFIG_TCP_CONG_WESTWOOD is not set
# CONFIG_TCP_CONG_HTCP is not set
CONFIG_TCP_CONG_HSTCP=y
CONFIG_TCP_CONG_HYBLA=y
CONFIG_TCP_CONG_VEGAS=y
# CONFIG_TCP_CONG_SCALABLE is not set
# CONFIG_TCP_CONG_LP is not set
# CONFIG_TCP_CONG_VENO is not set
CONFIG_TCP_CONG_YEAH=m
CONFIG_TCP_CONG_ILLINOIS=m
# CONFIG_DEFAULT_BIC is not set
# CONFIG_DEFAULT_CUBIC is not set
# CONFIG_DEFAULT_HTCP is not set
CONFIG_DEFAULT_VEGAS=y
# CONFIG_DEFAULT_WESTWOOD is not set
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="vegas"
CONFIG_TCP_MD5SIG=y
# CONFIG_IP_VS is not set
CONFIG_IPV6=y
CONFIG_IPV6_PRIVACY=y
# CONFIG_IPV6_ROUTER_PREF is not set
CONFIG_IPV6_OPTIMISTIC_DAD=y
CONFIG_INET6_AH=m
CONFIG_INET6_ESP=m
# CONFIG_INET6_IPCOMP is not set
CONFIG_IPV6_MIP6=m
# CONFIG_INET6_XFRM_TUNNEL is not set
CONFIG_INET6_TUNNEL=m
CONFIG_INET6_XFRM_MODE_TRANSPORT=y
CONFIG_INET6_XFRM_MODE_TUNNEL=m
# CONFIG_INET6_XFRM_MODE_BEET is not set
CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION=y
CONFIG_IPV6_SIT=y
CONFIG_IPV6_NDISC_NODETYPE=y
CONFIG_IPV6_TUNNEL=m
CONFIG_IPV6_MULTIPLE_TABLES=y
# CONFIG_IPV6_SUBTREES is not set
CONFIG_IPV6_MROUTE=y
# CONFIG_IPV6_PIMSM_V2 is not set
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
# CONFIG_NETFILTER_ADVANCED is not set
#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_LOG=y
# CONFIG_NF_CONNTRACK is not set
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=y
CONFIG_NETFILTER_XT_TARGET_SECMARK=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_POLICY=y
#
# IP: Netfilter Configuration
#
CONFIG_IP_NF_IPTABLES=m
# CONFIG_IP_NF_FILTER is not set
# CONFIG_IP_NF_TARGET_LOG is not set
# CONFIG_IP_NF_TARGET_ULOG is not set
# CONFIG_IP_NF_MANGLE is not set
#
# IPv6: Netfilter Configuration
#
CONFIG_IP6_NF_IPTABLES=y
CONFIG_IP6_NF_MATCH_IPV6HEADER=y
# CONFIG_IP6_NF_FILTER is not set
CONFIG_IP6_NF_MANGLE=m
# CONFIG_IP_DCCP is not set
CONFIG_IP_SCTP=y
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
CONFIG_SCTP_HMAC_SHA1=y
# CONFIG_SCTP_HMAC_MD5 is not set
CONFIG_TIPC=m
# CONFIG_TIPC_ADVANCED is not set
# CONFIG_TIPC_DEBUG is not set
CONFIG_ATM=m
# CONFIG_ATM_CLIP is not set
CONFIG_ATM_LANE=m
CONFIG_ATM_MPOA=m
CONFIG_ATM_BR2684=m
# CONFIG_ATM_BR2684_IPFILTER is not set
# CONFIG_BRIDGE is not set
CONFIG_VLAN_8021Q=y
CONFIG_DECNET=y
# CONFIG_DECNET_ROUTER is not set
CONFIG_LLC=y
CONFIG_LLC2=y
CONFIG_IPX=y
CONFIG_IPX_INTERN=y
CONFIG_ATALK=m
# CONFIG_DEV_APPLETALK is not set
CONFIG_X25=y
CONFIG_LAPB=y
CONFIG_ECONET=m
CONFIG_ECONET_AUNUDP=y
# CONFIG_ECONET_NATIVE is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_SCHED is not set
CONFIG_NET_SCH_FIFO=y
#
# Network testing
#
CONFIG_NET_PKTGEN=y
CONFIG_HAMRADIO=y
#
# Packet Radio protocols
#
CONFIG_AX25=y
# CONFIG_AX25_DAMA_SLAVE is not set
# CONFIG_NETROM is not set
CONFIG_ROSE=m
#
# AX.25 network device drivers
#
CONFIG_MKISS=y
# CONFIG_6PACK is not set
# CONFIG_BPQETHER is not set
CONFIG_BAYCOM_SER_FDX=m
CONFIG_BAYCOM_SER_HDX=m
# CONFIG_YAM is not set
CONFIG_CAN=m
CONFIG_CAN_RAW=m
# CONFIG_CAN_BCM is not set
#
# CAN Device Drivers
#
# CONFIG_CAN_VCAN is not set
CONFIG_CAN_DEBUG_DEVICES=y
CONFIG_IRDA=y
#
# IrDA protocols
#
# CONFIG_IRLAN is not set
# CONFIG_IRNET is not set
CONFIG_IRCOMM=y
# CONFIG_IRDA_ULTRA is not set
#
# IrDA options
#
CONFIG_IRDA_CACHE_LAST_LSAP=y
# CONFIG_IRDA_FAST_RR is not set
CONFIG_IRDA_DEBUG=y
#
# Infrared-port device drivers
#
#
# SIR device drivers
#
# CONFIG_IRTTY_SIR is not set
#
# Dongle support
#
CONFIG_KINGSUN_DONGLE=y
CONFIG_KSDAZZLE_DONGLE=y
CONFIG_KS959_DONGLE=y
#
# FIR device drivers
#
# CONFIG_USB_IRDA is not set
# CONFIG_SIGMATEL_FIR is not set
CONFIG_NSC_FIR=m
CONFIG_WINBOND_FIR=y
# CONFIG_SMC_IRCC_FIR is not set
CONFIG_ALI_FIR=m
# CONFIG_VLSI_FIR is not set
CONFIG_VIA_FIR=m
CONFIG_MCS_FIR=y
CONFIG_BT=y
CONFIG_BT_L2CAP=m
# CONFIG_BT_SCO is not set
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
# CONFIG_BT_BNEP_MC_FILTER is not set
CONFIG_BT_BNEP_PROTO_FILTER=y
# CONFIG_BT_HIDP is not set
#
# Bluetooth device drivers
#
# CONFIG_BT_HCIUSB is not set
CONFIG_BT_HCIBTUSB=y
# CONFIG_BT_HCIBTSDIO is not set
# CONFIG_BT_HCIUART is not set
# CONFIG_BT_HCIBCM203X is not set
CONFIG_BT_HCIBPA10X=m
# CONFIG_BT_HCIBFUSB is not set
CONFIG_BT_HCIVHCI=y
CONFIG_AF_RXRPC=m
# CONFIG_AF_RXRPC_DEBUG is not set
CONFIG_RXKAD=m
CONFIG_FIB_RULES=y
#
# Wireless
#
CONFIG_CFG80211=y
CONFIG_NL80211=y
CONFIG_WIRELESS_EXT=y
CONFIG_MAC80211=y
#
# Rate control algorithm selection
#
CONFIG_MAC80211_RC_DEFAULT_PID=y
# CONFIG_MAC80211_RC_DEFAULT_NONE is not set
#
# Selecting 'y' for an algorithm will
#
#
# build the algorithm into mac80211.
#
CONFIG_MAC80211_RC_DEFAULT="pid"
CONFIG_MAC80211_RC_PID=y
CONFIG_MAC80211_MESH=y
CONFIG_MAC80211_LEDS=y
# CONFIG_MAC80211_DEBUGFS is not set
# CONFIG_MAC80211_DEBUG_PACKET_ALIGNMENT is not set
CONFIG_MAC80211_DEBUG=y
# CONFIG_MAC80211_HT_DEBUG is not set
CONFIG_MAC80211_VERBOSE_DEBUG=y
# CONFIG_MAC80211_LOWTX_FRAME_DUMP is not set
# CONFIG_TKIP_DEBUG is not set
CONFIG_MAC80211_DEBUG_COUNTERS=y
# CONFIG_MAC80211_IBSS_DEBUG is not set
# CONFIG_MAC80211_VERBOSE_PS_DEBUG is not set
# CONFIG_MAC80211_VERBOSE_MPL_DEBUG is not set
CONFIG_IEEE80211=y
CONFIG_IEEE80211_DEBUG=y
CONFIG_IEEE80211_CRYPT_WEP=y
CONFIG_IEEE80211_CRYPT_CCMP=m
CONFIG_IEEE80211_CRYPT_TKIP=m
# CONFIG_RFKILL is not set
CONFIG_NET_9P=m
# CONFIG_NET_9P_DEBUG is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=y
# CONFIG_PROC_EVENTS is not set
# CONFIG_PARPORT is not set
CONFIG_PNP=y
CONFIG_PNP_DEBUG=y
#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
CONFIG_BLK_DEV_FD=m
CONFIG_BLK_CPQ_DA=y
CONFIG_BLK_CPQ_CISS_DA=y
CONFIG_CISS_SCSI_TAPE=y
CONFIG_BLK_DEV_DAC960=y
CONFIG_BLK_DEV_UMEM=y
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_NBD is not set
CONFIG_BLK_DEV_SX8=m
CONFIG_BLK_DEV_UB=y
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=y
CONFIG_CDROM_PKTCDVD_BUFFERS=8
CONFIG_CDROM_PKTCDVD_WCACHE=y
# CONFIG_ATA_OVER_ETH is not set
# CONFIG_MISC_DEVICES is not set
CONFIG_TIFM_CORE=y
CONFIG_HAVE_IDE=y
#
# SCSI device support
#
CONFIG_RAID_ATTRS=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=m
CONFIG_SCSI_NETLINK=y
CONFIG_SCSI_PROC_FS=y
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
CONFIG_CHR_DEV_OSST=y
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
# CONFIG_CHR_DEV_SG is not set
# CONFIG_CHR_DEV_SCH is not set
#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
CONFIG_SCSI_CONSTANTS=y
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m
#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_FC_ATTRS=m
# CONFIG_SCSI_FC_TGT_ATTRS is not set
CONFIG_SCSI_ISCSI_ATTRS=y
CONFIG_SCSI_SRP_ATTRS=y
# CONFIG_SCSI_LOWLEVEL is not set
CONFIG_SCSI_AIC7XXX=y
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
# CONFIG_SATA_PMP is not set
CONFIG_SATA_AHCI=y
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
CONFIG_SATA_SVW=y
CONFIG_ATA_PIIX=y
# CONFIG_SATA_MV is not set
CONFIG_SATA_NV=y
CONFIG_PDC_ADMA=y
CONFIG_SATA_QSTOR=y
# CONFIG_SATA_PROMISE is not set
CONFIG_SATA_SX4=y
CONFIG_SATA_SIL=m
CONFIG_SATA_SIS=m
CONFIG_SATA_ULI=y
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
CONFIG_SATA_INIC162X=y
CONFIG_PATA_ACPI=m
# CONFIG_PATA_ALI is not set
CONFIG_PATA_AMD=y
CONFIG_PATA_ARTOP=y
CONFIG_PATA_ATIIXP=y
CONFIG_PATA_CMD640_PCI=m
# CONFIG_PATA_CMD64X is not set
CONFIG_PATA_CS5520=y
CONFIG_PATA_CS5530=m
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
CONFIG_ATA_GENERIC=m
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
CONFIG_PATA_IT821X=y
# CONFIG_PATA_IT8213 is not set
CONFIG_PATA_JMICRON=m
CONFIG_PATA_TRIFLEX=m
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
CONFIG_PATA_OLDPIIX=y
CONFIG_PATA_NETCELL=m
CONFIG_PATA_NINJA32=y
# CONFIG_PATA_NS87410 is not set
CONFIG_PATA_NS87415=y
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
CONFIG_PATA_PDC_OLD=m
CONFIG_PATA_RADISYS=m
CONFIG_PATA_RZ1000=m
# CONFIG_PATA_SC1200 is not set
CONFIG_PATA_SERVERWORKS=y
CONFIG_PATA_PDC2027X=m
# CONFIG_PATA_SIL680 is not set
CONFIG_PATA_SIS=y
CONFIG_PATA_VIA=y
# CONFIG_PATA_WINBOND is not set
CONFIG_PATA_SCH=m
# CONFIG_MD is not set
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
CONFIG_IEEE1394=y
#
# Subsystem Options
#
# CONFIG_IEEE1394_VERBOSEDEBUG is not set
#
# Controllers
#
CONFIG_IEEE1394_PCILYNX=m
# CONFIG_IEEE1394_OHCI1394 is not set
#
# Protocols
#
CONFIG_IEEE1394_SBP2=m
# CONFIG_IEEE1394_SBP2_PHYS_DMA is not set
# CONFIG_IEEE1394_ETH1394_ROM_ENTRY is not set
# CONFIG_IEEE1394_ETH1394 is not set
# CONFIG_IEEE1394_RAWIO is not set
CONFIG_I2O=m
CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
# CONFIG_I2O_EXT_ADAPTEC is not set
# CONFIG_I2O_CONFIG is not set
CONFIG_I2O_BUS=m
# CONFIG_I2O_BLOCK is not set
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m
CONFIG_MACINTOSH_DRIVERS=y
CONFIG_MAC_EMUMOUSEBTN=y
CONFIG_NETDEVICES=y
CONFIG_NETDEVICES_MULTIQUEUE=y
# CONFIG_DUMMY is not set
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=y
# CONFIG_VETH is not set
CONFIG_NET_SB1000=m
CONFIG_ARCNET=m
# CONFIG_ARCNET_1201 is not set
# CONFIG_ARCNET_1051 is not set
# CONFIG_ARCNET_RAW is not set
CONFIG_ARCNET_CAP=m
CONFIG_ARCNET_COM90xx=m
# CONFIG_ARCNET_COM90xxIO is not set
# CONFIG_ARCNET_RIM_I is not set
# CONFIG_ARCNET_COM20020 is not set
CONFIG_PHYLIB=y
#
# MII PHY device drivers
#
CONFIG_MARVELL_PHY=y
# CONFIG_DAVICOM_PHY is not set
CONFIG_QSEMI_PHY=m
# CONFIG_LXT_PHY is not set
CONFIG_CICADA_PHY=y
# CONFIG_VITESSE_PHY is not set
CONFIG_SMSC_PHY=m
# CONFIG_BROADCOM_PHY is not set
# CONFIG_ICPLUS_PHY is not set
CONFIG_REALTEK_PHY=m
CONFIG_FIXED_PHY=y
CONFIG_MDIO_BITBANG=m
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
CONFIG_SUNGEM=m
# CONFIG_CASSINI is not set
CONFIG_NET_VENDOR_3COM=y
CONFIG_VORTEX=y
CONFIG_TYPHOON=y
CONFIG_NET_TULIP=y
CONFIG_DE2104X=m
CONFIG_TULIP=m
CONFIG_TULIP_MWI=y
# CONFIG_TULIP_MMIO is not set
# CONFIG_TULIP_NAPI is not set
CONFIG_DE4X5=y
# CONFIG_WINBOND_840 is not set
# CONFIG_DM9102 is not set
CONFIG_ULI526X=y
CONFIG_PCMCIA_XIRCOM=m
CONFIG_HP100=m
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
CONFIG_NET_PCI=y
CONFIG_PCNET32=m
# CONFIG_AMD8111_ETH is not set
CONFIG_ADAPTEC_STARFIRE=m
# CONFIG_ADAPTEC_STARFIRE_NAPI is not set
CONFIG_B44=m
CONFIG_B44_PCI_AUTOSELECT=y
CONFIG_B44_PCICORE_AUTOSELECT=y
CONFIG_B44_PCI=y
CONFIG_FORCEDETH=y
# CONFIG_FORCEDETH_NAPI is not set
# CONFIG_EEPRO100 is not set
CONFIG_E100=y
CONFIG_FEALNX=m
CONFIG_NATSEMI=m
CONFIG_NE2K_PCI=y
CONFIG_8139CP=m
CONFIG_8139TOO=y
CONFIG_8139TOO_PIO=y
CONFIG_8139TOO_TUNE_TWISTER=y
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_R6040 is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
CONFIG_SUNDANCE=y
CONFIG_SUNDANCE_MMIO=y
CONFIG_VIA_RHINE=m
CONFIG_VIA_RHINE_MMIO=y
# CONFIG_VIA_RHINE_NAPI is not set
CONFIG_SC92031=m
CONFIG_NETDEV_1000=y
CONFIG_ACENIC=y
# CONFIG_ACENIC_OMIT_TIGON_I is not set
# CONFIG_DL2K is not set
CONFIG_E1000=y
# CONFIG_E1000_NAPI is not set
CONFIG_E1000_DISABLE_PACKET_SPLIT=y
# CONFIG_E1000E_ENABLED is not set
CONFIG_IP1000=m
CONFIG_IGB=y
CONFIG_NS83820=m
CONFIG_HAMACHI=y
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
CONFIG_SIS190=m
# CONFIG_SKGE is not set
CONFIG_SKY2=m
# CONFIG_SKY2_DEBUG is not set
# CONFIG_VIA_VELOCITY is not set
CONFIG_TIGON3=y
CONFIG_BNX2=y
# CONFIG_QLA3XXX is not set
# CONFIG_ATL1 is not set
# CONFIG_NETDEV_10000 is not set
CONFIG_TR=y
# CONFIG_IBMOL is not set
CONFIG_3C359=y
# CONFIG_TMS380TR is not set
#
# Wireless LAN
#
CONFIG_WLAN_PRE80211=y
# CONFIG_STRIP is not set
# CONFIG_WLAN_80211 is not set
# CONFIG_IWLWIFI_LEDS is not set
#
# USB Network Adapters
#
CONFIG_USB_CATC=y
CONFIG_USB_KAWETH=y
CONFIG_USB_PEGASUS=m
CONFIG_USB_RTL8150=y
# CONFIG_USB_USBNET is not set
CONFIG_WAN=y
CONFIG_LANMEDIA=y
CONFIG_HDLC=m
CONFIG_HDLC_RAW=m
# CONFIG_HDLC_RAW_ETH is not set
# CONFIG_HDLC_CISCO is not set
CONFIG_HDLC_FR=m
CONFIG_HDLC_PPP=m
CONFIG_HDLC_X25=m
CONFIG_PCI200SYN=m
CONFIG_WANXL=m
CONFIG_PC300=m
#
# Cyclades-PC300 MLPPP support is disabled.
#
#
# Refer to the file README.mlppp, provided by PC300 package.
#
# CONFIG_PC300TOO is not set
# CONFIG_FARSYNC is not set
CONFIG_DSCC4=m
CONFIG_DSCC4_PCISYNC=y
# CONFIG_DSCC4_PCI_RST is not set
CONFIG_DLCI=y
CONFIG_DLCI_MAX=8
# CONFIG_LAPBETHER is not set
# CONFIG_X25_ASY is not set
# CONFIG_SBNI is not set
# CONFIG_ATM_DRIVERS is not set
CONFIG_FDDI=y
CONFIG_DEFXX=m
CONFIG_DEFXX_MMIO=y
# CONFIG_SKFP is not set
# CONFIG_HIPPI is not set
CONFIG_PPP=y
# CONFIG_PPP_MULTILINK is not set
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
CONFIG_PPP_SYNC_TTY=y
# CONFIG_PPP_DEFLATE is not set
# CONFIG_PPP_BSDCOMP is not set
CONFIG_PPP_MPPE=m
CONFIG_PPPOE=m
# CONFIG_PPPOATM is not set
CONFIG_PPPOL2TP=m
# CONFIG_SLIP is not set
CONFIG_SLHC=y
CONFIG_NET_FC=y
CONFIG_NETCONSOLE=y
# CONFIG_NETCONSOLE_DYNAMIC is not set
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_ISDN=m
# CONFIG_ISDN_I4L is not set
# CONFIG_ISDN_CAPI is not set
CONFIG_PHONE=y
CONFIG_PHONE_IXJ=y
#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=m
CONFIG_INPUT_POLLDEV=y
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_KEYBOARD_SUNKBD=m
CONFIG_KEYBOARD_LKKBD=y
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_KEYBOARD_NEWTON=m
CONFIG_KEYBOARD_STOWAWAY=y
# CONFIG_INPUT_MOUSE is not set
CONFIG_INPUT_JOYSTICK=y
# CONFIG_JOYSTICK_ANALOG is not set
CONFIG_JOYSTICK_A3D=m
# CONFIG_JOYSTICK_ADI is not set
CONFIG_JOYSTICK_COBRA=m
CONFIG_JOYSTICK_GF2K=y
# CONFIG_JOYSTICK_GRIP is not set
CONFIG_JOYSTICK_GRIP_MP=m
# CONFIG_JOYSTICK_GUILLEMOT is not set
CONFIG_JOYSTICK_INTERACT=m
CONFIG_JOYSTICK_SIDEWINDER=y
CONFIG_JOYSTICK_TMDC=y
CONFIG_JOYSTICK_IFORCE=m
# CONFIG_JOYSTICK_IFORCE_USB is not set
# CONFIG_JOYSTICK_IFORCE_232 is not set
# CONFIG_JOYSTICK_WARRIOR is not set
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=y
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
# CONFIG_JOYSTICK_TWIDJOY is not set
# CONFIG_JOYSTICK_ZHENHUA is not set
CONFIG_JOYSTICK_JOYDUMP=y
CONFIG_JOYSTICK_XPAD=m
CONFIG_JOYSTICK_XPAD_FF=y
CONFIG_INPUT_TABLET=y
# CONFIG_TABLET_USB_ACECAD is not set
CONFIG_TABLET_USB_AIPTEK=y
CONFIG_TABLET_USB_GTCO=m
CONFIG_TABLET_USB_KBTAB=m
CONFIG_TABLET_USB_WACOM=y
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
CONFIG_SERIO_CT82C710=y
CONFIG_SERIO_PCIPS2=m
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=y
CONFIG_GAMEPORT=y
CONFIG_GAMEPORT_NS558=m
CONFIG_GAMEPORT_L4=y
CONFIG_GAMEPORT_EMU10K1=m
CONFIG_GAMEPORT_FM801=m
#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_DEVKMEM=y
CONFIG_SERIAL_NONSTANDARD=y
CONFIG_COMPUTONE=m
CONFIG_ROCKETPORT=m
# CONFIG_CYCLADES is not set
# CONFIG_DIGIEPCA is not set
# CONFIG_MOXA_INTELLIO is not set
CONFIG_MOXA_SMARTIO=m
# CONFIG_ISI is not set
CONFIG_SYNCLINK=y
CONFIG_SYNCLINKMP=y
CONFIG_SYNCLINK_GT=y
CONFIG_N_HDLC=m
# CONFIG_RISCOM8 is not set
CONFIG_SPECIALIX=y
CONFIG_SPECIALIX_RTSCTS=y
CONFIG_SX=y
CONFIG_RIO=y
CONFIG_RIO_OLDPCI=y
CONFIG_STALDRV=y
# CONFIG_NOZOMI is not set
#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
# CONFIG_SERIAL_8250_RSA is not set
#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_IPMI_HANDLER=m
CONFIG_IPMI_PANIC_EVENT=y
# CONFIG_IPMI_PANIC_STRING is not set
# CONFIG_IPMI_DEVICE_INTERFACE is not set
# CONFIG_IPMI_SI is not set
# CONFIG_IPMI_WATCHDOG is not set
CONFIG_IPMI_POWEROFF=m
# CONFIG_HW_RANDOM is not set
CONFIG_NVRAM=m
CONFIG_R3964=m
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
CONFIG_PC8736x_GPIO=m
CONFIG_NSC_GPIO=m
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
# CONFIG_HPET_RTC_IRQ is not set
# CONFIG_HPET_MMAP is not set
CONFIG_HANGCHECK_TIMER=m
CONFIG_TCG_TPM=m
CONFIG_TCG_TIS=m
# CONFIG_TCG_NSC is not set
# CONFIG_TCG_ATMEL is not set
# CONFIG_TCG_INFINEON is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
CONFIG_I2C=m
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_ALGOPCA=m
#
# I2C Hardware Bus support
#
CONFIG_I2C_ALI1535=m
# CONFIG_I2C_ALI1563 is not set
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_I810 is not set
CONFIG_I2C_PIIX4=m
CONFIG_I2C_NFORCE2=m
# CONFIG_I2C_OCORES is not set
CONFIG_I2C_PARPORT_LIGHT=m
# CONFIG_I2C_PROSAVAGE is not set
CONFIG_I2C_SAVAGE4=m
CONFIG_I2C_SIMTEC=m
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
CONFIG_I2C_SIS96X=m
CONFIG_I2C_TAOS_EVM=m
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_TINY_USB is not set
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
# CONFIG_I2C_VOODOO3 is not set
CONFIG_I2C_PCA_PLATFORM=m
#
# Miscellaneous I2C Chip support
#
CONFIG_DS1682=m
CONFIG_SENSORS_EEPROM=m
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_PCF8575 is not set
# CONFIG_SENSORS_PCF8591 is not set
CONFIG_SENSORS_MAX6875=m
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set
# CONFIG_SPI is not set
CONFIG_W1=y
# CONFIG_W1_CON is not set
#
# 1-wire Bus Masters
#
CONFIG_W1_MASTER_MATROX=y
# CONFIG_W1_MASTER_DS2490 is not set
CONFIG_W1_MASTER_DS2482=m
#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=m
# CONFIG_W1_SLAVE_SMEM is not set
CONFIG_W1_SLAVE_DS2433=y
# CONFIG_W1_SLAVE_DS2433_CRC is not set
CONFIG_W1_SLAVE_DS2760=m
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
CONFIG_BATTERY_DS2760=m
# CONFIG_HWMON is not set
CONFIG_THERMAL=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_NOWAYOUT=y
#
# Watchdog Device Drivers
#
# CONFIG_SOFT_WATCHDOG is not set
CONFIG_ACQUIRE_WDT=y
CONFIG_ADVANTECH_WDT=m
# CONFIG_ALIM1535_WDT is not set
# CONFIG_ALIM7101_WDT is not set
CONFIG_SC520_WDT=m
CONFIG_IB700_WDT=m
# CONFIG_IBMASR is not set
CONFIG_WAFER_WDT=m
CONFIG_I6300ESB_WDT=m
CONFIG_ITCO_WDT=m
CONFIG_ITCO_VENDOR_SUPPORT=y
# CONFIG_IT8712F_WDT is not set
CONFIG_HP_WATCHDOG=m
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
CONFIG_60XX_WDT=m
# CONFIG_SBC8360_WDT is not set
CONFIG_CPU5_WDT=y
CONFIG_SMSC37B787_WDT=y
# CONFIG_W83627HF_WDT is not set
CONFIG_W83697HF_WDT=m
CONFIG_W83877F_WDT=y
CONFIG_W83977F_WDT=y
CONFIG_MACHZ_WDT=m
CONFIG_SBC_EPX_C3_WATCHDOG=y
#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
# CONFIG_WDTPCI is not set
#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=y
#
# Sonics Silicon Backplane
#
CONFIG_SSB_POSSIBLE=y
CONFIG_SSB=m
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
# CONFIG_SSB_B43_PCI_BRIDGE is not set
# CONFIG_SSB_DEBUG is not set
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
#
# Multifunction device drivers
#
CONFIG_MFD_SM501=m
CONFIG_HTC_PASIC3=y
#
# Multimedia devices
#
#
# Multimedia core support
#
CONFIG_VIDEO_DEV=y
CONFIG_VIDEO_V4L2_COMMON=m
# CONFIG_VIDEO_ALLOW_V4L1 is not set
# CONFIG_VIDEO_V4L1_COMPAT is not set
# CONFIG_DVB_CORE is not set
CONFIG_VIDEO_MEDIA=y
#
# Multimedia drivers
#
# CONFIG_MEDIA_ATTACH is not set
CONFIG_MEDIA_TUNER=m
# CONFIG_MEDIA_TUNER_CUSTOMIZE is not set
CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA8290=m
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_TEA5761=m
CONFIG_MEDIA_TUNER_TEA5767=m
CONFIG_MEDIA_TUNER_MT20XX=m
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=m
CONFIG_VIDEO_V4L2=m
# CONFIG_VIDEO_CAPTURE_DRIVERS is not set
CONFIG_RADIO_ADAPTERS=y
CONFIG_RADIO_GEMTEK_PCI=m
# CONFIG_RADIO_MAXIRADIO is not set
CONFIG_RADIO_MAESTRO=m
CONFIG_USB_DSBR=m
CONFIG_USB_SI470X=m
# CONFIG_DAB is not set
#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
CONFIG_AGP_SIS=m
CONFIG_AGP_VIA=y
CONFIG_DRM=m
# CONFIG_DRM_TDFX is not set
CONFIG_DRM_R128=m
# CONFIG_DRM_RADEON is not set
CONFIG_DRM_I810=m
# CONFIG_DRM_I830 is not set
CONFIG_DRM_I915=m
# CONFIG_DRM_MGA is not set
CONFIG_DRM_SIS=m
CONFIG_DRM_VIA=m
CONFIG_DRM_SAVAGE=m
CONFIG_VGASTATE=y
CONFIG_VIDEO_OUTPUT_CONTROL=y
CONFIG_FB=y
# CONFIG_FIRMWARE_EDID is not set
CONFIG_FB_DDC=m
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
CONFIG_FB_SYS_FILLRECT=m
CONFIG_FB_SYS_COPYAREA=m
CONFIG_FB_SYS_IMAGEBLIT=m
# CONFIG_FB_FOREIGN_ENDIAN is not set
CONFIG_FB_SYS_FOPS=m
CONFIG_FB_DEFERRED_IO=y
CONFIG_FB_HECUBA=m
CONFIG_FB_SVGALIB=y
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y
#
# Frame buffer hardware drivers
#
CONFIG_FB_PM2=m
# CONFIG_FB_PM2_FIFO_DISCONNECT is not set
CONFIG_FB_CYBER2000=m
# CONFIG_FB_ARC is not set
# CONFIG_FB_IMSTT is not set
CONFIG_FB_UVESA=y
CONFIG_FB_EFI=y
# CONFIG_FB_IMAC is not set
CONFIG_FB_N411=m
CONFIG_FB_HGA=y
# CONFIG_FB_HGA_ACCEL is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
CONFIG_FB_RIVA=y
# CONFIG_FB_RIVA_I2C is not set
# CONFIG_FB_RIVA_DEBUG is not set
# CONFIG_FB_RIVA_BACKLIGHT is not set
CONFIG_FB_LE80578=y
CONFIG_FB_CARILLO_RANCH=y
# CONFIG_FB_INTEL is not set
CONFIG_FB_MATROX=m
# CONFIG_FB_MATROX_MILLENIUM is not set
CONFIG_FB_MATROX_MYSTIQUE=y
# CONFIG_FB_MATROX_G is not set
# CONFIG_FB_MATROX_I2C is not set
# CONFIG_FB_MATROX_MULTIHEAD is not set
CONFIG_FB_ATY128=m
# CONFIG_FB_ATY128_BACKLIGHT is not set
# CONFIG_FB_ATY is not set
CONFIG_FB_S3=y
CONFIG_FB_SAVAGE=m
CONFIG_FB_SAVAGE_I2C=y
# CONFIG_FB_SAVAGE_ACCEL is not set
CONFIG_FB_SIS=m
# CONFIG_FB_SIS_300 is not set
# CONFIG_FB_SIS_315 is not set
CONFIG_FB_NEOMAGIC=m
CONFIG_FB_KYRO=m
CONFIG_FB_3DFX=m
# CONFIG_FB_3DFX_ACCEL is not set
CONFIG_FB_VOODOO1=y
CONFIG_FB_VT8623=y
CONFIG_FB_TRIDENT=m
CONFIG_FB_TRIDENT_ACCEL=y
CONFIG_FB_ARK=m
CONFIG_FB_PM3=m
# CONFIG_FB_GEODE is not set
# CONFIG_FB_SM501 is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
# CONFIG_LCD_CLASS_DEVICE is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
# CONFIG_BACKLIGHT_CORGI is not set
CONFIG_BACKLIGHT_PROGEAR=m
#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
CONFIG_VIDEO_SELECT=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FONT_8x16=y
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
#
# Sound
#
CONFIG_SOUND=m
#
# Advanced Linux Sound Architecture
#
# CONFIG_SND is not set
#
# Open Sound System
#
CONFIG_SOUND_PRIME=m
CONFIG_SOUND_TRIDENT=m
# CONFIG_HID_SUPPORT is not set
CONFIG_USB_MOUSE=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set
#
# Miscellaneous USB options
#
# CONFIG_USB_DEVICEFS is not set
CONFIG_USB_DEVICE_CLASS=y
CONFIG_USB_DYNAMIC_MINORS=y
CONFIG_USB_SUSPEND=y
# CONFIG_USB_OTG is not set
#
# USB Host Controller Drivers
#
CONFIG_USB_C67X00_HCD=m
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
# CONFIG_USB_ISP116X_HCD is not set
CONFIG_USB_ISP1760_HCD=y
CONFIG_USB_ISP1760_PCI=y
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_U132_HCD=m
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
#
# USB Device Class drivers
#
CONFIG_USB_ACM=y
CONFIG_USB_PRINTER=y
# CONFIG_USB_WDM is not set
#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#
#
# may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=y
CONFIG_USB_STORAGE_DEBUG=y
# CONFIG_USB_STORAGE_DATAFAB is not set
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
CONFIG_USB_STORAGE_CYPRESS_ATACB=y
# CONFIG_USB_LIBUSUAL is not set
#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
CONFIG_USB_MICROTEK=m
CONFIG_USB_MON=y
#
# USB port drivers
#
CONFIG_USB_SERIAL=m
CONFIG_USB_EZUSB=y
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_AIRCABLE is not set
# CONFIG_USB_SERIAL_AIRPRIME is not set
CONFIG_USB_SERIAL_ARK3116=m
CONFIG_USB_SERIAL_BELKIN=m
CONFIG_USB_SERIAL_CH341=m
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_CP2101 is not set
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
# CONFIG_USB_SERIAL_FUNSOFT is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
# CONFIG_USB_SERIAL_GARMIN is not set
CONFIG_USB_SERIAL_IPW=m
# CONFIG_USB_SERIAL_IUU is not set
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
# CONFIG_USB_SERIAL_KEYSPAN_MPR is not set
CONFIG_USB_SERIAL_KEYSPAN_USA28=y
CONFIG_USB_SERIAL_KEYSPAN_USA28X=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XB=y
# CONFIG_USB_SERIAL_KEYSPAN_USA19 is not set
# CONFIG_USB_SERIAL_KEYSPAN_USA18X is not set
CONFIG_USB_SERIAL_KEYSPAN_USA19W=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y
# CONFIG_USB_SERIAL_KEYSPAN_USA19QI is not set
CONFIG_USB_SERIAL_KEYSPAN_USA49W=y
CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
CONFIG_USB_SERIAL_MOS7720=m
# CONFIG_USB_SERIAL_MOS7840 is not set
CONFIG_USB_SERIAL_MOTOROLA=m
# CONFIG_USB_SERIAL_NAVMAN is not set
CONFIG_USB_SERIAL_PL2303=m
CONFIG_USB_SERIAL_OTI6858=m
CONFIG_USB_SERIAL_SPCP8X5=m
# CONFIG_USB_SERIAL_HP4X is not set
CONFIG_USB_SERIAL_SAFE=m
# CONFIG_USB_SERIAL_SAFE_PADDED is not set
# CONFIG_USB_SERIAL_SIERRAWIRELESS is not set
# CONFIG_USB_SERIAL_TI is not set
CONFIG_USB_SERIAL_CYBERJACK=m
# CONFIG_USB_SERIAL_XIRCOM is not set
CONFIG_USB_SERIAL_OPTION=m
# CONFIG_USB_SERIAL_OMNINET is not set
CONFIG_USB_SERIAL_DEBUG=m
#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=m
CONFIG_USB_EMI26=m
CONFIG_USB_ADUTUX=m
# CONFIG_USB_AUERSWALD is not set
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
CONFIG_USB_LCD=y
CONFIG_USB_BERRY_CHARGE=y
CONFIG_USB_LED=m
CONFIG_USB_CYPRESS_CY7C63=m
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
CONFIG_USB_IDMOUSE=m
CONFIG_USB_FTDI_ELAN=m
CONFIG_USB_APPLEDISPLAY=y
CONFIG_USB_SISUSBVGA=m
CONFIG_USB_SISUSBVGA_CON=y
CONFIG_USB_LD=m
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
CONFIG_USB_ATM=m
# CONFIG_USB_SPEEDTOUCH is not set
# CONFIG_USB_CXACRU is not set
CONFIG_USB_UEAGLEATM=m
# CONFIG_USB_XUSBATM is not set
CONFIG_USB_GADGET=y
# CONFIG_USB_GADGET_DEBUG_FILES is not set
# CONFIG_USB_GADGET_DEBUG_FS is not set
CONFIG_USB_GADGET_SELECTED=y
# CONFIG_USB_GADGET_AMD5536UDC is not set
# CONFIG_USB_GADGET_ATMEL_USBA is not set
# CONFIG_USB_GADGET_FSL_USB2 is not set
# CONFIG_USB_GADGET_NET2280 is not set
# CONFIG_USB_GADGET_PXA2XX is not set
CONFIG_USB_GADGET_M66592=y
CONFIG_USB_M66592=y
# CONFIG_USB_GADGET_PXA27X is not set
# CONFIG_USB_GADGET_GOKU is not set
# CONFIG_USB_GADGET_LH7A40X is not set
# CONFIG_USB_GADGET_OMAP is not set
# CONFIG_USB_GADGET_S3C2410 is not set
# CONFIG_USB_GADGET_AT91 is not set
# CONFIG_USB_GADGET_DUMMY_HCD is not set
CONFIG_USB_GADGET_DUALSPEED=y
# CONFIG_USB_ZERO is not set
CONFIG_USB_ETH=y
# CONFIG_USB_ETH_RNDIS is not set
# CONFIG_USB_GADGETFS is not set
# CONFIG_USB_FILE_STORAGE is not set
# CONFIG_USB_G_SERIAL is not set
# CONFIG_USB_MIDI_GADGET is not set
# CONFIG_USB_G_PRINTER is not set
CONFIG_MMC=y
CONFIG_MMC_DEBUG=y
CONFIG_MMC_UNSAFE_RESUME=y
#
# MMC/SD Card Drivers
#
CONFIG_MMC_BLOCK=m
# CONFIG_MMC_BLOCK_BOUNCE is not set
# CONFIG_SDIO_UART is not set
CONFIG_MMC_TEST=y
#
# MMC/SD Host Controller Drivers
#
CONFIG_MMC_SDHCI=y
CONFIG_MMC_RICOH_MMC=m
CONFIG_MMC_WBSD=m
CONFIG_MMC_TIFM_SD=y
CONFIG_MEMSTICK=m
CONFIG_MEMSTICK_DEBUG=y
#
# MemoryStick drivers
#
# CONFIG_MEMSTICK_UNSAFE_RESUME is not set
CONFIG_MSPRO_BLOCK=m
#
# MemoryStick Host Controller Drivers
#
CONFIG_MEMSTICK_TIFM_MS=m
CONFIG_MEMSTICK_JMICRON_38X=m
CONFIG_NEW_LEDS=y
# CONFIG_LEDS_CLASS is not set
#
# LED drivers
#
#
# LED Triggers
#
CONFIG_LEDS_TRIGGERS=y
CONFIG_LEDS_TRIGGER_TIMER=m
CONFIG_LEDS_TRIGGER_HEARTBEAT=y
CONFIG_LEDS_TRIGGER_DEFAULT_ON=y
CONFIG_ACCESSIBILITY=y
CONFIG_A11Y_BRAILLE_CONSOLE=y
CONFIG_INFINIBAND=y
CONFIG_INFINIBAND_USER_MAD=y
# CONFIG_INFINIBAND_USER_ACCESS is not set
CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_MTHCA=m
CONFIG_INFINIBAND_MTHCA_DEBUG=y
# CONFIG_INFINIBAND_IPATH is not set
# CONFIG_INFINIBAND_AMSO1100 is not set
# CONFIG_MLX4_INFINIBAND is not set
CONFIG_INFINIBAND_NES=y
# CONFIG_INFINIBAND_NES_DEBUG is not set
# CONFIG_INFINIBAND_IPOIB is not set
# CONFIG_INFINIBAND_SRP is not set
# CONFIG_INFINIBAND_ISER is not set
# CONFIG_EDAC is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
# CONFIG_RTC_HCTOSYS is not set
CONFIG_RTC_DEBUG=y
#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
# CONFIG_RTC_INTF_DEV is not set
CONFIG_RTC_DRV_TEST=m
#
# I2C RTC drivers
#
CONFIG_RTC_DRV_DS1307=m
# CONFIG_RTC_DRV_DS1374 is not set
# CONFIG_RTC_DRV_DS1672 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=m
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_PCF8563=m
CONFIG_RTC_DRV_PCF8583=m
# CONFIG_RTC_DRV_M41T80 is not set
CONFIG_RTC_DRV_S35390A=m
#
# SPI RTC drivers
#
#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=m
CONFIG_RTC_DRV_DS1511=m
CONFIG_RTC_DRV_DS1553=m
# CONFIG_RTC_DRV_DS1742 is not set
CONFIG_RTC_DRV_STK17TA8=m
CONFIG_RTC_DRV_M48T86=y
# CONFIG_RTC_DRV_M48T59 is not set
CONFIG_RTC_DRV_V3020=m
#
# on-CPU RTC drivers
#
CONFIG_DMADEVICES=y
#
# DMA Devices
#
# CONFIG_INTEL_IOATDMA is not set
CONFIG_UIO=y
# CONFIG_UIO_CIF is not set
CONFIG_UIO_SMX=y
#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_EFI_VARS=m
CONFIG_DELL_RBU=y
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
# CONFIG_ISCSI_IBFT_FIND is not set
#
# File systems
#
# CONFIG_EXT2_FS is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4DEV_FS=y
CONFIG_EXT4DEV_FS_XATTR=y
# CONFIG_EXT4DEV_FS_POSIX_ACL is not set
# CONFIG_EXT4DEV_FS_SECURITY is not set
CONFIG_JBD=y
CONFIG_JBD_DEBUG=y
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
CONFIG_REISERFS_CHECK=y
# CONFIG_REISERFS_PROC_INFO is not set
# CONFIG_REISERFS_FS_XATTR is not set
CONFIG_JFS_FS=m
# CONFIG_JFS_POSIX_ACL is not set
CONFIG_JFS_SECURITY=y
CONFIG_JFS_DEBUG=y
# CONFIG_JFS_STATISTICS is not set
CONFIG_FS_POSIX_ACL=y
# CONFIG_XFS_FS is not set
CONFIG_GFS2_FS=m
CONFIG_GFS2_FS_LOCKING_NOLOCK=m
# CONFIG_GFS2_FS_LOCKING_DLM is not set
CONFIG_OCFS2_FS=y
CONFIG_OCFS2_FS_O2CB=y
# CONFIG_OCFS2_FS_USERSPACE_CLUSTER is not set
CONFIG_OCFS2_DEBUG_MASKLOG=y
# CONFIG_OCFS2_DEBUG_FS is not set
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
CONFIG_AUTOFS_FS=m
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=m
#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
# CONFIG_JOLIET is not set
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
# CONFIG_MSDOS_FS is not set
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=y
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=y
#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
CONFIG_AFFS_FS=m
CONFIG_HFS_FS=m
CONFIG_HFSPLUS_FS=y
CONFIG_BEFS_FS=m
CONFIG_BEFS_DEBUG=y
CONFIG_BFS_FS=y
# CONFIG_EFS_FS is not set
CONFIG_CRAMFS=m
CONFIG_VXFS_FS=m
# CONFIG_MINIX_FS is not set
# CONFIG_HPFS_FS is not set
CONFIG_QNX4FS_FS=m
CONFIG_ROMFS_FS=y
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_NETWORK_FILESYSTEMS is not set
#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
CONFIG_OSF_PARTITION=y
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
CONFIG_MINIX_SUBPARTITION=y
CONFIG_SOLARIS_X86_PARTITION=y
# CONFIG_UNIXWARE_DISKLABEL is not set
CONFIG_LDM_PARTITION=y
CONFIG_LDM_DEBUG=y
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
CONFIG_SUN_PARTITION=y
# CONFIG_KARMA_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=m
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=m
CONFIG_NLS_CODEPAGE_852=m
# CONFIG_NLS_CODEPAGE_855 is not set
CONFIG_NLS_CODEPAGE_857=y
CONFIG_NLS_CODEPAGE_860=m
CONFIG_NLS_CODEPAGE_861=m
CONFIG_NLS_CODEPAGE_862=y
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=y
CONFIG_NLS_CODEPAGE_865=y
CONFIG_NLS_CODEPAGE_866=m
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=y
CONFIG_NLS_CODEPAGE_950=m
# CONFIG_NLS_CODEPAGE_932 is not set
CONFIG_NLS_CODEPAGE_949=m
CONFIG_NLS_CODEPAGE_874=y
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
CONFIG_NLS_CODEPAGE_1251=y
CONFIG_NLS_ASCII=y
# CONFIG_NLS_ISO8859_1 is not set
# CONFIG_NLS_ISO8859_2 is not set
CONFIG_NLS_ISO8859_3=y
# CONFIG_NLS_ISO8859_4 is not set
CONFIG_NLS_ISO8859_5=y
CONFIG_NLS_ISO8859_6=m
CONFIG_NLS_ISO8859_7=m
CONFIG_NLS_ISO8859_9=m
CONFIG_NLS_ISO8859_13=y
CONFIG_NLS_ISO8859_14=m
CONFIG_NLS_ISO8859_15=y
CONFIG_NLS_KOI8_R=m
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=y
CONFIG_DLM=m
# CONFIG_DLM_DEBUG is not set
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
CONFIG_HEADERS_CHECK=y
# CONFIG_DEBUG_KERNEL is not set
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_TRACE_IRQFLAGS=y
CONFIG_STACKTRACE=y
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_FRAME_POINTER=y
CONFIG_LATENCYTOP=y
CONFIG_HAVE_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACING=y
CONFIG_FTRACE=y
CONFIG_IRQSOFF_TRACER=y
# CONFIG_SYSPROF_TRACER is not set
# CONFIG_SCHED_TRACER is not set
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_NONPROMISC_DEVMEM is not set
CONFIG_EARLY_PRINTK=y
CONFIG_X86_MPPARSE=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
# CONFIG_IO_DELAY_0X80 is not set
CONFIG_IO_DELAY_0XED=y
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=1
#
# Security options
#
CONFIG_KEYS=y
CONFIG_KEYS_DEBUG_PROC_KEYS=y
CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
# CONFIG_SECURITY_NETWORK_XFRM is not set
CONFIG_SECURITY_CAPABILITIES=y
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
CONFIG_SECURITY_DEFAULT_MMAP_MIN_ADDR=0
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_GF128MUL=y
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=m
# CONFIG_CRYPTO_TEST is not set
#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=y
CONFIG_CRYPTO_GCM=y
CONFIG_CRYPTO_SEQIV=y
#
# Block modes
#
CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_CTR=y
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=y
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=y
#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set
#
# Digest
#
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_MD4 is not set
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_SHA512=y
CONFIG_CRYPTO_TGR192=y
# CONFIG_CRYPTO_WP512 is not set
#
# Ciphers
#
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_X86_64=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_ARC4=y
# CONFIG_CRYPTO_BLOWFISH is not set
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
CONFIG_CRYPTO_SALSA20_X86_64=m
# CONFIG_CRYPTO_SEED is not set
CONFIG_CRYPTO_SERPENT=m
CONFIG_CRYPTO_TEA=m
# CONFIG_CRYPTO_TWOFISH is not set
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_X86_64=m
#
# Compression
#
CONFIG_CRYPTO_DEFLATE=y
# CONFIG_CRYPTO_LZO is not set
# CONFIG_CRYPTO_HW is not set
CONFIG_HAVE_KVM=y
# CONFIG_VIRTUALIZATION is not set
#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_FORCE_SUCCESSFUL_BUILD=y
CONFIG_FORCE_MINIMAL_CONFIG=y
CONFIG_FORCE_MINIMAL_CONFIG_64=y
CONFIG_FORCE_MINIMAL_CONFIG_PHYS=y
[42949372.960000] Linux version 2.6.26-rc3-sched-devel.git (mingo@titan) (gcc version 4.2.3) #115 SMP Sat May 24 04:25:10 CEST 2008
[42949372.960000] Command line: root=/dev/sda1 earlyprintk=vga console=ttyS0,115200 console=tty 3 profile=0 debug initcall_debug apic=debug apic=verbose ignore_loglevel sysrq_always_enabled pci=nomsi
[42949372.960000] BIOS-provided physical RAM map:
[42949372.960000] BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
[42949372.960000] BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
[42949372.960000] BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
[42949372.960000] BIOS-e820: 0000000000100000 - 000000003ed93000 (usable)
[42949372.960000] BIOS-e820: 000000003ed93000 - 000000003ee4d000 (ACPI NVS)
[42949372.960000] BIOS-e820: 000000003ee4d000 - 000000003fea2000 (usable)
[42949372.960000] BIOS-e820: 000000003fea2000 - 000000003fee9000 (ACPI NVS)
[42949372.960000] BIOS-e820: 000000003fee9000 - 000000003feed000 (usable)
[42949372.960000] BIOS-e820: 000000003feed000 - 000000003feff000 (ACPI data)
[42949372.960000] BIOS-e820: 000000003feff000 - 000000003ff00000 (usable)
[42949372.960000] console [earlyvga0] enabled
[42949372.960000] debug: ignoring loglevel setting.
[42949372.960000] Entering add_active_range(0, 0, 159) 0 entries of 25600 used
[42949372.960000] Entering add_active_range(0, 256, 257427) 1 entries of 25600 used
[42949372.960000] Entering add_active_range(0, 257613, 261794) 2 entries of 25600 used
[42949372.960000] Entering add_active_range(0, 261865, 261869) 3 entries of 25600 used
[42949372.960000] Entering add_active_range(0, 261887, 261888) 4 entries of 25600 used
[42949372.960000] max_pfn_mapped = 261888
[42949372.960000] init_memory_mapping
[42949372.960000] DMI 2.3 present.
[42949372.960000] ACPI: RSDP 000FE020, 0014 (r0 INTEL )
[42949372.960000] ACPI: RSDT 3FEFDE48, 003C (r1 INTEL D975XBX 4B9 MSFT 1000013)
[42949372.960000] ACPI: FACP 3FEFCF10, 0074 (r1 INTEL D975XBX 4B9 MSFT 1000013)
[42949372.960000] ACPI: DSDT 3FEF8010, 3E70 (r1 INTEL D975XBX 4B9 MSFT 1000013)
[42949372.960000] ACPI: FACS 3FEDFC40, 0040
[42949372.960000] ACPI: APIC 3FEFCE10, 0078 (r1 INTEL D975XBX 4B9 MSFT 1000013)
[42949372.960000] ACPI: WDDT 3FEF7F90, 0040 (r1 INTEL D975XBX 4B9 MSFT 1000013)
[42949372.960000] ACPI: MCFG 3FEF7F10, 003C (r1 INTEL D975XBX 4B9 MSFT 1000013)
[42949372.960000] ACPI: ASF! 3FEFCD10, 00A6 (r32 INTEL D975XBX 4B9 MSFT 1000013)
[42949372.960000] ACPI: HPET 3FEF7E90, 0038 (r1 INTEL D975XBX 4B9 MSFT 1000013)
[42949372.960000] No NUMA configuration found
[42949372.960000] Faking a node at 0000000000000000-000000003ff00000
[42949372.960000] Entering add_active_range(0, 0, 159) 0 entries of 25600 used
[42949372.960000] Entering add_active_range(0, 256, 257427) 1 entries of 25600 used
[42949372.960000] Entering add_active_range(0, 257613, 261794) 2 entries of 25600 used
[42949372.960000] Entering add_active_range(0, 261865, 261869) 3 entries of 25600 used
[42949372.960000] Entering add_active_range(0, 261887, 261888) 4 entries of 25600 used
[42949372.960000] Bootmem setup node 0 0000000000000000-000000003ff00000
[42949372.960000] NODE_DATA [000000000000a000 - 000000000003dfff]
[42949372.960000] bootmap [000000000003e000 - 0000000000045fdf] pages 8
[42949372.960000] early res: 0 [0-fff] BIOS data page
[42949372.960000] early res: 1 [6000-7fff] TRAMPOLINE
[42949372.960000] early res: 2 [200000-132a92f] TEXT DATA BSS
[42949372.960000] early res: 3 [9fc00-fffff] BIOS reserved
[42949372.960000] early res: 4 [8000-9fff] PGTABLE
[42949372.960000] Scan SMP from ffff810000000000 for 1024 bytes.
[42949372.960000] Scan SMP from ffff81000009fc00 for 1024 bytes.
[42949372.960000] Scan SMP from ffff8100000f0000 for 65536 bytes.
[42949372.960000] Zone PFN ranges:
[42949372.960000] DMA 0 -> 4096
[42949372.960000] DMA32 4096 -> 1048576
[42949372.960000] Normal 1048576 -> 1048576
[42949372.960000] Movable zone start PFN for each node
[42949372.960000] early_node_map[5] active PFN ranges
[42949372.960000] 0: 0 -> 159
[42949372.960000] 0: 256 -> 257427
[42949372.960000] 0: 257613 -> 261794
[42949372.960000] 0: 261865 -> 261869
[42949372.960000] 0: 261887 -> 261888
[42949372.960000] On node 0 totalpages: 261516
[42949372.960000] DMA zone: 56 pages used for memmap
[42949372.960000] DMA zone: 103 pages reserved
[42949372.960000] DMA zone: 3840 pages, LIFO batch:0
[42949372.960000] DMA32 zone: 3524 pages used for memmap
[42949372.960000] DMA32 zone: 253993 pages, LIFO batch:31
[42949372.960000] Normal zone: 0 pages used for memmap
[42949372.960000] Movable zone: 0 pages used for memmap
[42949372.960000] ACPI: PM-Timer IO Port: 0x408
[42949372.960000] ACPI: Local APIC address 0xfee00000
[42949372.960000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[42949372.960000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[42949372.960000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x82] disabled)
[42949372.960000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x83] disabled)
[42949372.960000] ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
[42949372.960000] ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
[42949372.960000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
[42949372.960000] IOAPIC[0]: apic_id 2, version 0, address 0xfec00000, GSI 0-23
[42949372.960000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[42949372.960000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[42949372.960000] ACPI: IRQ0 used by override.
[42949372.960000] ACPI: IRQ2 used by override.
[42949372.960000] ACPI: IRQ9 used by override.
[42949372.960000] Setting APIC routing to flat
[42949372.960000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[42949372.960000] Using ACPI (MADT) for SMP configuration information
[42949372.960000] mapped APIC to ffffffffff5fb000 ( fee00000)
[42949372.960000] mapped IOAPIC to ffffffffff5fa000 (00000000fec00000)
[42949372.960000] PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
[42949372.960000] PM: Registered nosave memory: 00000000000a0000 - 00000000000e0000
[42949372.960000] PM: Registered nosave memory: 00000000000e0000 - 0000000000100000
[42949372.960000] PM: Registered nosave memory: 000000003ed93000 - 000000003ee4d000
[42949372.960000] PM: Registered nosave memory: 000000003fea2000 - 000000003fee9000
[42949372.960000] PM: Registered nosave memory: 000000003feed000 - 000000003feff000
[42949372.960000] Allocating PCI resources starting at 40000000 (gap: 3ff00000:c0100000)
[42949372.960000] SMP: Allowing 4 CPUs, 2 hotplug CPUs
[42949372.960000] PERCPU: Allocating 40496 bytes of per cpu data
[42949372.960000] NR_CPUS: 4096, nr_cpu_ids: 4, nr_node_ids 1
[42949372.960000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 257833
[42949372.960000] Policy zone: DMA32
[42949372.960000] Kernel command line: root=/dev/sda1 earlyprintk=vga console=ttyS0,115200 console=tty 3 profile=0 debug initcall_debug apic=debug apic=verbose ignore_loglevel sysrq_always_enabled pci=nomsi
[42949372.960000] kernel profiling enabled (shift: 0)
[42949372.960000] debug: sysrq always enabled.
[42949372.960000] Initializing CPU#0
[42949372.960000] PID hash table entries: 4096 (order: 12, 32768 bytes)
[42949372.960000] TSC calibrated against PM_TIMER
[42949372.960000] time.c: Detected 2933.313 MHz processor.
[ 0.010000] Console: colour VGA+ 80x25
[ 0.010000] console handover: boot [earlyvga0] -> real [tty0]
[ 0.010000] console [ttyS0] enabled
[ 0.010000] Checking aperture...
[ 0.010000] No AGP bridge found
[ 0.010000] Calgary: detecting Calgary via BIOS EBDA area
[ 0.010000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[ 0.010000] Memory: 980576k/1047552k available (8303k kernel code, 65488k reserved, 4495k data, 1332k init)
[ 0.010000] CPA: page pool initialized 1 of 1 pages preallocated
[ 0.010000] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[ 0.010000] hpet clockevent registered
[ 0.159953] Calibrating delay using timer specific routine.. 5870.22 BogoMIPS (lpj=29351132)
[ 0.160000] Security Framework initialized
[ 0.164119] Capability LSM initialized
[ 0.167996] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[ 0.175843] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
[ 0.183045] Mount-cache hash table entries: 256
[ 0.187761] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.192514] CPU: L2 cache: 4096K
[ 0.195755] CPU 0/0 -> Node 0
[ 0.200000] CPU: Physical Processor ID: 0
[ 0.204024] CPU: Processor Core ID: 0
[ 0.207701] using mwait in idle threads.
[ 0.212614] ACPI: Core revision 20080321
[ 0.222450] enabled ExtINT on CPU#0
[ 0.226090] ENABLING IO-APIC IRQs
[ 0.229418] init IO_APIC IRQs
[ 0.232407] IO-APIC (apicid-pin) 2-0 not connected.
[ 0.237422] IOAPIC[0]: Set routing entry (2-1 -> 0x31 -> IRQ 1 Mode:0 Active:0)
[ 0.244752] IOAPIC[0]: Set routing entry (2-2 -> 0x30 -> IRQ 0 Mode:0 Active:0)
[ 0.252081] IOAPIC[0]: Set routing entry (2-3 -> 0x33 -> IRQ 3 Mode:0 Active:0)
[ 0.259402] IOAPIC[0]: Set routing entry (2-4 -> 0x34 -> IRQ 4 Mode:0 Active:0)
[ 0.266726] IOAPIC[0]: Set routing entry (2-5 -> 0x35 -> IRQ 5 Mode:0 Active:0)
[ 0.274050] IOAPIC[0]: Set routing entry (2-6 -> 0x36 -> IRQ 6 Mode:0 Active:0)
[ 0.281374] IOAPIC[0]: Set routing entry (2-7 -> 0x37 -> IRQ 7 Mode:0 Active:0)
[ 0.288696] IOAPIC[0]: Set routing entry (2-8 -> 0x38 -> IRQ 8 Mode:0 Active:0)
[ 0.296019] IOAPIC[0]: Set routing entry (2-9 -> 0x39 -> IRQ 9 Mode:1 Active:0)
[ 0.303342] IOAPIC[0]: Set routing entry (2-10 -> 0x3a -> IRQ 10 Mode:0 Active:0)
[ 0.310837] IOAPIC[0]: Set routing entry (2-11 -> 0x3b -> IRQ 11 Mode:0 Active:0)
[ 0.318332] IOAPIC[0]: Set routing entry (2-12 -> 0x3c -> IRQ 12 Mode:0 Active:0)
[ 0.325828] IOAPIC[0]: Set routing entry (2-13 -> 0x3d -> IRQ 13 Mode:0 Active:0)
[ 0.333326] IOAPIC[0]: Set routing entry (2-14 -> 0x3e -> IRQ 14 Mode:0 Active:0)
[ 0.340823] IOAPIC[0]: Set routing entry (2-15 -> 0x3f -> IRQ 15 Mode:0 Active:0)
[ 0.348688] IO-APIC (apicid-pin) 2-16, 2-17, 2-18, 2-19, 2-20, 2-21, 2-22, 2-23 not connected.
[ 0.357825] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.463886] CPU0: Intel(R) Core(TM)2 CPU E6800 @ 2.93GHz stepping 05
[ 0.471098] Using local APIC timer interrupts.
[ 0.490000] APIC timer calibration result 16666563
[ 0.494803] Detected 16.666 MHz APIC timer.
[ 0.499105] Booting processor 1/1 ip 6000
[ 0.510000] Initializing CPU#1
[ 0.510000] masked ExtINT on CPU#1
[ 0.510000] Calibrating delay using timer specific routine.. 5866.68 BogoMIPS (lpj=29333414)
[ 0.510000] CPU: L1 I cache: 32K, L1 D cache: 32K
[ 0.510000] CPU: L2 cache: 4096K
[ 0.510000] CPU 1/1 -> Node 0
[ 0.510000] CPU: Physical Processor ID: 0
[ 0.510000] CPU: Processor Core ID: 1
[ 0.659504] CPU1: Intel(R) Core(TM)2 CPU E6800 @ 2.93GHz stepping 05
[ 0.675373] checking TSC synchronization [CPU#0 -> CPU#1]: passed.
[ 0.690000] Brought up 2 CPUs
[ 0.692988] Total of 2 processors activated (11736.90 BogoMIPS).
[ 0.699097] CPU0 attaching sched-domain:
[ 0.703037] domain 0: span 0-1
[ 0.706227] groups: 0 1
[ 0.708966] domain 1: span 0-1
[ 0.712244] groups: 0-1
[ 0.715037] CPU1 attaching sched-domain:
[ 0.718971] domain 0: span 0-1
[ 0.722161] groups: 1 0
[ 0.724900] domain 1: span 0-1
[ 0.728173] groups: 0-1
[ 0.740000] calling net_ns_init+0x0/0x1a0
[ 0.740000] net_namespace: 936 bytes
[ 0.740000] initcall net_ns_init+0x0/0x1a0 returned 0 after 0 msecs
[ 0.746312] calling init_smp_flush+0x0/0x60
[ 0.750000] initcall init_smp_flush+0x0/0x60 returned 0 after 0 msecs
[ 0.756484] calling print_banner+0x0/0x20
[ 0.760596] Booting paravirtualized kernel on bare hardware
[ 0.766177] initcall print_banner+0x0/0x20 returned 0 after 0 msecs
[ 0.772487] calling sysctl_init+0x0/0x40
[ 0.776575] initcall sysctl_init+0x0/0x40 returned 0 after 0 msecs
[ 0.782800] calling ksysfs_init+0x0/0xc0
[ 0.786825] initcall ksysfs_init+0x0/0xc0 returned 0 after 0 msecs
[ 0.793049] calling init_jiffies_clocksource+0x0/0x20
[ 0.798199] initcall init_jiffies_clocksource+0x0/0x20 returned 0 after 0 msecs
[ 0.805555] calling pm_init+0x0/0x40
[ 0.809233] initcall pm_init+0x0/0x40 returned 0 after 0 msecs
[ 0.815111] calling pm_disk_init+0x0/0x20
[ 0.819222] initcall pm_disk_init+0x0/0x20 returned 0 after 0 msecs
[ 0.825533] calling swsusp_header_init+0x0/0x40
[ 0.830000] initcall swsusp_header_init+0x0/0x40 returned 0 after 0 msecs
[ 0.836831] calling filelock_init+0x0/0x40
[ 0.841031] initcall filelock_init+0x0/0x40 returned 0 after 0 msecs
[ 0.847427] calling init_misc_binfmt+0x0/0x60
[ 0.851886] initcall init_misc_binfmt+0x0/0x60 returned 0 after 0 msecs
[ 0.858541] calling init_script_binfmt+0x0/0x20
[ 0.863171] initcall init_script_binfmt+0x0/0x20 returned 0 after 0 msecs
[ 0.870000] calling init_elf_binfmt+0x0/0x20
[ 0.874371] initcall init_elf_binfmt+0x0/0x20 returned 0 after 0 msecs
[ 0.880942] calling init_compat_elf_binfmt+0x0/0x20
[ 0.885919] initcall init_compat_elf_binfmt+0x0/0x20 returned 0 after 0 msecs
[ 0.893098] calling debugfs_init+0x0/0x70
[ 0.897208] initcall debugfs_init+0x0/0x70 returned 0 after 0 msecs
[ 0.903519] calling securityfs_init+0x0/0x70
[ 0.907889] initcall securityfs_init+0x0/0x70 returned 0 after 0 msecs
[ 0.914461] calling random32_init+0x0/0x70
[ 0.918657] initcall random32_init+0x0/0x70 returned 0 after 0 msecs
[ 0.925057] calling sock_init+0x0/0x70
[ 0.928924] initcall sock_init+0x0/0x70 returned 0 after 0 msecs
[ 0.934977] calling netpoll_init+0x0/0x40
[ 0.939087] initcall netpoll_init+0x0/0x40 returned 0 after 0 msecs
[ 0.945396] calling netlink_proto_init+0x0/0x180
[ 0.950000] NET: Registered protocol family 16
[ 0.954463] initcall netlink_proto_init+0x0/0x180 returned 0 after 9 msecs
[ 0.961379] calling bdi_class_init+0x0/0x50
[ 0.970000] initcall bdi_class_init+0x0/0x50 returned 0 after 0 msecs
[ 0.976484] calling kobject_uevent_init+0x0/0x50
[ 0.980000] initcall kobject_uevent_init+0x0/0x50 returned 0 after 0 msecs
[ 0.986916] calling pcibus_class_init+0x0/0x20
[ 0.990000] initcall pcibus_class_init+0x0/0x20 returned 0 after 0 msecs
[ 0.996744] calling pci_driver_init+0x0/0x20
[ 1.000000] initcall pci_driver_init+0x0/0x20 returned 0 after 0 msecs
[ 1.006573] calling lcd_class_init+0x0/0x50
[ 1.020000] initcall lcd_class_init+0x0/0x50 returned 0 after 0 msecs
[ 1.020000] calling backlight_class_init+0x0/0x50
[ 1.030000] initcall backlight_class_init+0x0/0x50 returned 0 after 0 msecs
[ 1.030000] calling dock_init+0x0/0x56
[ 1.033895] No dock devices found.
[ 1.037310] initcall dock_init+0x0/0x56 returned 0 after 0 msecs
[ 1.043359] calling bay_init+0x0/0x54
[ 1.047302] initcall bay_init+0x0/0x54 returned -19 after 0 msecs
[ 1.053439] calling tty_class_init+0x0/0x30
[ 1.060000] initcall tty_class_init+0x0/0x30 returned 0 after 0 msecs
[ 1.066485] calling vtconsole_class_init+0x0/0xe0
[ 1.070000] initcall vtconsole_class_init+0x0/0xe0 returned 0 after 0 msecs
[ 1.077005] calling register_node_type+0x0/0x80
[ 1.080000] initcall register_node_type+0x0/0x80 returned 0 after 0 msecs
[ 1.086831] calling early_fill_mp_bus_info+0x0/0x880
[ 1.091897] initcall early_fill_mp_bus_info+0x0/0x880 returned 0 after 0 msecs
[ 1.099165] calling arch_kdebugfs_init+0x0/0x10
[ 1.103795] initcall arch_kdebugfs_init+0x0/0x10 returned 0 after 0 msecs
[ 1.110000] calling mtrr_if_init+0x0/0x90
[ 1.114111] initcall mtrr_if_init+0x0/0x90 returned 0 after 0 msecs
[ 1.120423] calling acpi_pci_init+0x0/0x60
[ 1.124618] ACPI: bus type pci registered
[ 1.128641] initcall acpi_pci_init+0x0/0x60 returned 0 after 0 msecs
[ 1.135039] calling init_acpi_device_notify+0x0/0x50
[ 1.140104] initcall init_acpi_device_notify+0x0/0x50 returned 0 after 0 msecs
[ 1.147371] calling dmi_id_init+0x0/0x310
[ 1.160000] initcall dmi_id_init+0x0/0x310 returned 0 after 0 msecs
[ 1.160000] calling pci_access_init+0x0/0x50
[ 1.164372] PCI: Using configuration type 1 for base access
[ 1.172253] initcall pci_access_init+0x0/0x50 returned 0 after 9 msecs
[ 1.178823] calling topology_init+0x0/0x90
[ 1.190000] initcall topology_init+0x0/0x90 returned 0 after 0 msecs
[ 1.190000] calling mtrr_init_finialize+0x0/0x50
[ 1.194719] initcall mtrr_init_finialize+0x0/0x50 returned 0 after 0 msecs
[ 1.200000] calling param_sysfs_init+0x0/0x250
[ 1.210000] initcall param_sysfs_init+0x0/0x250 returned 0 after 0 msecs
[ 1.216752] calling pm_sysrq_init+0x0/0x20
[ 1.220000] initcall pm_sysrq_init+0x0/0x20 returned 0 after 0 msecs
[ 1.226399] calling readahead_init+0x0/0x40
[ 1.230000] initcall readahead_init+0x0/0x40 returned 0 after 0 msecs
[ 1.236485] calling init_bio+0x0/0x100
[ 1.240000] initcall init_bio+0x0/0x100 returned 0 after 0 msecs
[ 1.246052] calling blk_settings_init+0x0/0x40
[ 1.250595] initcall blk_settings_init+0x0/0x40 returned 0 after 0 msecs
[ 1.257337] calling blk_ioc_init+0x0/0x30
[ 1.261449] initcall blk_ioc_init+0x0/0x30 returned 0 after 0 msecs
[ 1.267757] calling genhd_device_init+0x0/0x50
[ 1.280000] initcall genhd_device_init+0x0/0x50 returned 0 after 0 msecs
[ 1.280000] calling acpi_init+0x0/0x221
[ 1.290000] ACPI: EC: Look up EC in DSDT
[ 1.290000] ACPI: Interpreter enabled
[ 1.293678] ACPI: (supports S0 S4 S5)
[ 1.297458] ACPI: Using IOAPIC for interrupt routing
[ 1.302461] initcall acpi_init+0x0/0x221 returned 0 after 19 msecs
[ 1.308684] calling acpi_scan_init+0x0/0x115
[ 1.320000] initcall acpi_scan_init+0x0/0x115 returned 0 after 0 msecs
[ 1.326573] calling acpi_ec_init+0x0/0x66
[ 1.330000] initcall acpi_ec_init+0x0/0x66 returned 0 after 0 msecs
[ 1.336312] calling acpi_pci_root_init+0x0/0x2d
[ 1.340000] ACPI: PCI Root Bridge [PCI0] (0000:00)
[ 1.350000] PCI: Transparent bridge - 0000:00:1e.0
[ 1.350000] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[ 1.356256] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P32_._PRT]
[ 1.362791] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX0._PRT]
[ 1.369254] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX4._PRT]
[ 1.375717] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PEX5._PRT]
[ 1.390000] initcall acpi_pci_root_init+0x0/0x2d returned 0 after 47 msecs
[ 1.396921] calling acpi_pci_link_init+0x0/0x4d
[ 1.400000] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 7 9 10 *11 12)
[ 1.406956] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 7 9 *10 11 12)
[ 1.410000] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 7 9 *10 11 12)
[ 1.416956] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 7 *9 10 11 12)
[ 1.423907] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 7 9 10 11 12) *0, disabled.
[ 1.432058] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 7 9 10 11 12) *0, disabled.
[ 1.440000] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 7 *9 10 11 12)
[ 1.446953] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 7 9 10 *11 12)
[ 1.460000] initcall acpi_pci_link_init+0x0/0x4d returned 0 after 57 msecs
[ 1.466919] calling acpi_power_init+0x0/0x7c
[ 1.470000] initcall acpi_power_init+0x0/0x7c returned 0 after 0 msecs
[ 1.476571] calling acpi_system_init+0x0/0x1b5
[ 1.480000] initcall acpi_system_init+0x0/0x1b5 returned 0 after 0 msecs
[ 1.486744] calling acpi_wmi_init+0x0/0x54
[ 1.490000] ACPI: WMI: Mapper loaded
[ 1.493591] initcall acpi_wmi_init+0x0/0x54 returned 0 after 9 msecs
[ 1.499987] calling pnp_init+0x0/0x30
[ 1.503750] Linux Plug and Play Support v0.97 (c) Adam Belay
[ 1.510000] initcall pnp_init+0x0/0x30 returned 0 after 0 msecs
[ 1.515965] calling pnpacpi_init+0x0/0xa0
[ 1.520000] pnp: PnP ACPI init
[ 1.520000] ACPI: bus type pnp registered
[ 1.524033] pnp 00:00: parse allocated resources
[ 1.529481] pnp 00:00: add io 0xcf8-0xcff flags 0x1
[ 1.540000] pnp 00:00: Plug and Play ACPI device, IDs PNP0a03 (active)
[ 1.546610] pnp 00:01: parse allocated resources
[ 1.550000] pnp 00:01: add mem 0xf0000000-0xf3ffffff flags 0x0
[ 1.556017] pnp 00:01: add mem 0xfed13000-0xfed13fff flags 0x0
[ 1.560000] pnp 00:01: add mem 0xfed14000-0xfed17fff flags 0x0
[ 1.566017] pnp 00:01: add mem 0xfed18000-0xfed18fff flags 0x0
[ 1.570000] pnp 00:01: add mem 0xfed19000-0xfed19fff flags 0x0
[ 1.576016] pnp 00:01: add mem 0xfed1c000-0xfed1ffff flags 0x0
[ 1.582033] pnp 00:01: add mem 0xfed20000-0xfed3ffff flags 0x0
[ 1.588048] pnp 00:01: add mem 0xfed45000-0xfed99fff flags 0x0
[ 1.594065] pnp 00:01: add mem 0xc0000-0xdffff flags 0x0
[ 1.599561] pnp 00:01: add mem 0xe0000-0xfffff flags 0x0
[ 1.605059] pnp 00:01: PNP0c02: calling quirk_system_pci_resources+0x0/0x190
[ 1.620000] pnp 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 1.620000] pnp 00:02: parse allocated resources
[ 1.624634] pnp 00:02: add io 0x0-0xf flags 0x1
[ 1.629436] pnp 00:02: add io 0x81-0x83 flags 0x1
[ 1.634414] pnp 00:02: add io 0x87-0x87 flags 0x1
[ 1.639388] pnp 00:02: add io 0x89-0x8b flags 0x1
[ 1.644365] pnp 00:02: add io 0x8f-0x8f flags 0x1
[ 1.649340] pnp 00:02: add io 0xc0-0xdf flags 0x1
[ 1.654318] pnp 00:02: add dma 4 flags 0x0
[ 1.660000] pnp 00:02: Plug and Play ACPI device, IDs PNP0200 (active)
[ 1.666609] pnp 00:03: parse allocated resources
[ 1.670000] pnp 00:03: add io 0x70-0x71 flags 0x1
[ 1.674976] pnp 00:03: add io 0x74-0x77 flags 0x1
[ 1.682225] IOAPIC[0]: Set routing entry (2-8 -> 0x38 -> IRQ 8 Mode:0 Active:0)
[ 1.689547] pnp 00:03: add irq 8 flags 0x1
[ 1.700000] pnp 00:03: Plug and Play ACPI device, IDs PNP0b00 (active)
[ 1.700000] pnp 00:04: parse allocated resources
[ 1.704632] pnp 00:04: add io 0xf0-0xf0 flags 0x1
[ 1.709608] IOAPIC[0]: Set routing entry (2-13 -> 0x3d -> IRQ 13 Mode:0 Active:0)
[ 1.717106] pnp 00:04: add irq 13 flags 0x1
[ 1.730000] pnp 00:04: Plug and Play ACPI device, IDs PNP0c04 (active)
[ 1.730000] pnp 00:05: parse allocated resources
[ 1.734633] pnp 00:05: add io 0x61-0x61 flags 0x1
[ 1.740000] pnp 00:05: Plug and Play ACPI device, IDs PNP0800 (active)
[ 1.746609] pnp 00:06: parse allocated resources
[ 1.750000] pnp 00:06: add io 0x500-0x53f flags 0x1
[ 1.755150] pnp 00:06: add io 0x400-0x47f flags 0x1
[ 1.760000] pnp 00:06: add io 0x92-0x92 flags 0x1
[ 1.764976] pnp 00:06: add io 0x680-0x6ff flags 0x1
[ 1.770000] pnp 00:06: add io 0x10-0x1f flags 0x1
[ 1.774975] pnp 00:06: add io 0x72-0x73 flags 0x1
[ 1.779951] pnp 00:06: add io 0x80-0x80 flags 0x1
[ 1.784927] pnp 00:06: add io 0x84-0x86 flags 0x1
[ 1.789903] pnp 00:06: add io 0x88-0x88 flags 0x1
[ 1.794880] pnp 00:06: add io 0x8c-0x8e flags 0x1
[ 1.799855] pnp 00:06: add io 0x90-0x9f flags 0x1
[ 1.804832] pnp 00:06: PNP0c02: calling quirk_system_pci_resources+0x0/0x190
[ 1.820000] pnp 00:06: Plug and Play ACPI device, IDs PNP0c02 (active)
[ 1.820000] pnp 00:07: parse allocated resources
[ 1.824823] pnp 00:07: add io 0x3f0-0x3f5 flags 0x1
[ 1.829972] pnp 00:07: add io 0x3f0-0x3f0 flags 0x1
[ 1.835126] IOAPIC[0]: Set routing entry (2-6 -> 0x36 -> IRQ 6 Mode:0 Active:0)
[ 1.840000] pnp 00:07: add irq 6 flags 0x1
[ 1.844282] pnp 00:07: add dma 2 flags 0x0
[ 1.848565] pnp 00:07: parse resource options
[ 1.852935] pnp 00:07: new independent option
[ 1.857330] pnp 00:07: new dependent option (priority 0x0)
[ 1.862829] pnp 00:07: io min 0x3f0 max 0x3f0 align 1 size 6 flags 0x1
[ 1.869624] pnp 00:07: io min 0x3f7 max 0x3f7 align 1 size 1 flags 0x1
[ 1.876422] pnp 00:07: irq bitmask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000040 flags 0x1
[ 1.887120] pnp 00:07: dma bitmask 0x4 flags 0x0
[ 1.900000] pnp 00:07: Plug and Play ACPI device, IDs PNP0700 (active)
[ 1.900000] pnp 00:08: parse allocated resources
[ 1.904835] pnp 00:08: add io 0x378-0x37f flags 0x1
[ 1.909984] pnp 00:08: add io 0x778-0x77f flags 0x1
[ 1.915135] IOAPIC[0]: Set routing entry (2-7 -> 0x37 -> IRQ 7 Mode:0 Active:0)
[ 1.922458] pnp 00:08: add irq 7 flags 0x1
[ 1.926739] pnp 00:08: parse resource options
[ 1.930000] pnp 00:08: new independent option
[ 1.934402] pnp 00:08: new dependent option (priority 0x0)
[ 1.939898] pnp 00:08: io min 0x378 max 0x378 align 1 size 8 flags 0x1
[ 1.946696] pnp 00:08: io min 0x778 max 0x778 align 1 size 8 flags 0x1
[ 1.953493] pnp 00:08: irq bitmask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000a0 flags 0x1
[ 1.964192] pnp 00:08: dma bitmask 0xa flags 0x0
[ 1.968993] pnp 00:08: new dependent option (priority 0x0)
[ 1.974489] pnp 00:08: io min 0x278 max 0x278 align 1 size 8 flags 0x1
[ 1.981290] pnp 00:08: io min 0x678 max 0x678 align 1 size 8 flags 0x1
[ 1.988086] pnp 00:08: irq bitmask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000a0 flags 0x1
[ 1.998785] pnp 00:08: dma bitmask 0xa flags 0x0
[ 2.003591] pnp 00:08: new dependent option (priority 0x0)
[ 2.009086] pnp 00:08: io min 0x3bc max 0x3bc align 1 size 4 flags 0x1
[ 2.015884] pnp 00:08: io min 0x7bc max 0x7bc align 1 size 4 flags 0x1
[ 2.022682] pnp 00:08: irq bitmask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,000000a0 flags 0x1
[ 2.033380] pnp 00:08: dma bitmask 0xa flags 0x0
[ 2.038181] pnp 00:08: end dependent options
[ 2.050000] pnp 00:08: Plug and Play ACPI device, IDs PNP0401 (active)
[ 2.050000] pnp 00:09: parse allocated resources
[ 2.054659] IOAPIC[0]: Set routing entry (2-12 -> 0x3c -> IRQ 12 Mode:0 Active:0)
[ 2.062156] pnp 00:09: add irq 12 flags 0x1
[ 2.070000] pnp 00:09: Plug and Play ACPI device, IDs PNP0f03 PNP0f13 (active)
[ 2.077364] pnp 00:0a: parse allocated resources
[ 2.080000] pnp 00:0a: add io 0x60-0x60 flags 0x1
[ 2.084978] pnp 00:0a: add io 0x64-0x64 flags 0x1
[ 2.089955] IOAPIC[0]: Set routing entry (2-1 -> 0x31 -> IRQ 1 Mode:0 Active:0)
[ 2.097278] pnp 00:0a: add irq 1 flags 0x1
[ 2.100000] pnp 00:0a: Plug and Play ACPI device, IDs PNP0303 PNP030b (active)
[ 2.107381] pnp 00:0b: parse allocated resources
[ 2.112131] pnp 00:0b: add io 0x3f8-0x3ff flags 0x1
[ 2.117281] IOAPIC[0]: Set routing entry (2-4 -> 0x34 -> IRQ 4 Mode:0 Active:0)
[ 2.124603] pnp 00:0b: add irq 4 flags 0x1
[ 2.128885] pnp 00:0b: parse resource options
[ 2.133256] pnp 00:0b: new independent option
[ 2.137656] pnp 00:0b: new dependent option (priority 0x0)
[ 2.143156] pnp 00:0b: io min 0x3f8 max 0x3f8 align 1 size 8 flags 0x1
[ 2.149953] pnp 00:0b: irq bitmask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000010 flags 0x1
[ 2.160000] pnp 00:0b: new dependent option (priority 0x0)
[ 2.165496] pnp 00:0b: io min 0x3e8 max 0x3e8 align 1 size 8 flags 0x1
[ 2.172294] pnp 00:0b: irq bitmask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000010 flags 0x1
[ 2.182993] pnp 00:0b: new dependent option (priority 0x0)
[ 2.190826] pnp 00:0b: io min 0x2f8 max 0x2f8 align 1 size 8 flags 0x1
[ 2.197622] pnp 00:0b: irq bitmask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000008 flags 0x1
[ 2.208320] pnp 00:0b: new dependent option (priority 0x0)
[ 2.213816] pnp 00:0b: io min 0x2e8 max 0x2e8 align 1 size 8 flags 0x1
[ 2.220613] pnp 00:0b: irq bitmask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000008 flags 0x1
[ 2.231312] pnp 00:0b: new dependent option (priority 0x2)
[ 2.236807] pnp 00:0b: io min 0x100 max 0x3f8 align 8 size 8 flags 0x1
[ 2.243605] pnp 00:0b: irq bitmask 00000000,00000000,00000000,00000000,00000000,00000000,00000000,0000fdfa flags 0x1
[ 2.260000] pnp 00:0b: Plug and Play ACPI device, IDs PNP0501 (active)
[ 2.266644] pnp 00:0c: parse allocated resources
[ 2.270000] pnp 00:0c: add mem 0xfec00000-0xfec000ff flags 0x0
[ 2.280000] pnp 00:0c: Plug and Play ACPI device, IDs PNP0003 (active)
[ 2.280000] pnp: PnP ACPI: found 13 devices
[ 2.284197] ACPI: ACPI bus type pnp unregistered
[ 2.288827] initcall pnpacpi_init+0x0/0xa0 returned 0 after 734 msecs
[ 2.295312] calling misc_init+0x0/0xb0
[ 2.300000] initcall misc_init+0x0/0xb0 returned 0 after 0 msecs
[ 2.306054] calling cn_init+0x0/0xe0
[ 2.310000] initcall cn_init+0x0/0xe0 returned 0 after 0 msecs
[ 2.315879] calling tifm_init+0x0/0x90
[ 2.320000] initcall tifm_init+0x0/0x90 returned 0 after 0 msecs
[ 2.326050] calling phy_init+0x0/0x50
[ 2.330000] initcall phy_init+0x0/0x50 returned 0 after 0 msecs
[ 2.335965] calling init_scsi+0x0/0xa0
[ 2.340000] SCSI subsystem initialized
[ 2.343764] initcall init_scsi+0x0/0xa0 returned 0 after 9 msecs
[ 2.349812] calling ata_init+0x0/0x3b0
[ 2.360000] libata version 3.00 loaded.
[ 2.360000] initcall ata_init+0x0/0x3b0 returned 0 after 0 msecs
[ 2.366051] calling spi_init+0x0/0x90
[ 2.370000] initcall spi_init+0x0/0x90 returned 0 after 0 msecs
[ 2.375966] calling init_pcmcia_cs+0x0/0x30
[ 2.380000] initcall init_pcmcia_cs+0x0/0x30 returned 0 after 0 msecs
[ 2.386487] calling usb_init+0x0/0x120
[ 2.390000] usbcore: registered new interface driver usbfs
[ 2.400000] usbcore: registered new interface driver hub
[ 2.400000] usbcore: registered new device driver usb
[ 2.405065] initcall usb_init+0x0/0x120 returned 0 after 19 msecs
[ 2.410000] calling serio_init+0x0/0xb0
[ 2.420000] initcall serio_init+0x0/0xb0 returned 0 after 0 msecs
[ 2.420000] calling gameport_init+0x0/0xb0
[ 2.430000] initcall gameport_init+0x0/0xb0 returned 0 after 0 msecs
[ 2.430000] calling input_init+0x0/0x120
[ 2.440000] initcall input_init+0x0/0x120 returned 0 after 0 msecs
[ 2.440000] calling rtc_init+0x0/0x70
[ 2.450000] initcall rtc_init+0x0/0x70 returned 0 after 0 msecs
[ 2.450000] calling hwmon_init+0x0/0x50
[ 2.460000] initcall hwmon_init+0x0/0x50 returned 0 after 0 msecs
[ 2.460000] calling thermal_init+0x0/0x50
[ 2.470000] initcall thermal_init+0x0/0x50 returned 0 after 0 msecs
[ 2.470000] calling mmc_init+0x0/0x80
[ 2.480000] initcall mmc_init+0x0/0x80 returned 0 after 0 msecs
[ 2.480000] calling leds_init+0x0/0x30
[ 2.490000] initcall leds_init+0x0/0x30 returned 0 after 0 msecs
[ 2.490000] calling ac97_bus_init+0x0/0x20
[ 2.500000] initcall ac97_bus_init+0x0/0x20 returned 0 after 0 msecs
[ 2.500000] calling pci_acpi_init+0x0/0xc0
[ 2.504197] PCI: Using ACPI for IRQ routing
[ 2.508392] number of MP IRQ sources: 15.
[ 2.512416] number of IO-APIC #2 registers: 24.
[ 2.516958] testing the IO APIC.......................
[ 2.522110]
[ 2.523621] IO APIC #2......
[ 2.526518] .... register #00: 00000000
[ 2.530000] ....... : physical APIC id: 00
[ 2.534369] .... register #01: 00170020
[ 2.538218] ....... : max redirection entries: 0017
[ 2.543455] ....... : PRQ implemented: 0
[ 2.547738] ....... : IO APIC version: 0020
[ 2.552280] .... register #02: 00170020
[ 2.556128] ....... : arbitration: 00
[ 2.560000] .... IRQ redirection table:
[ 2.563850] NR Dst Mask Trig IRR Pol Stat Dmod Deli Vect:
[ 2.569607] 00 000 1 0 0 0 0 0 0 00
[ 2.574880] 01 003 0 0 0 0 0 1 1 31
[ 2.580000] 02 003 0 0 0 0 0 1 1 30
[ 2.585273] 03 003 0 0 0 0 0 1 1 33
[ 2.590546] 04 003 0 0 0 0 0 1 1 34
[ 2.595818] 05 003 0 0 0 0 0 1 1 35
[ 2.601090] 06 003 0 0 0 0 0 1 1 36
[ 2.606361] 07 003 0 0 0 0 0 1 1 37
[ 2.611633] 08 003 0 0 0 0 0 1 1 38
[ 2.616905] 09 003 0 1 0 0 0 1 1 39
[ 2.622180] 0a 003 0 0 0 0 0 1 1 3A
[ 2.627452] 0b 003 0 0 0 0 0 1 1 3B
[ 2.632725] 0c 003 0 0 0 0 0 1 1 3C
[ 2.637994] 0d 003 0 0 0 0 0 1 1 3D
[ 2.643265] 0e 003 0 0 0 0 0 1 1 3E
[ 2.648537] 0f 003 0 0 0 0 0 1 1 3F
[ 2.653810] 10 000 1 0 0 0 0 0 0 00
[ 2.659081] 11 000 1 0 0 0 0 0 0 00
[ 2.664354] 12 000 1 0 0 0 0 0 0 00
[ 2.669625] 13 000 1 0 0 0 0 0 0 00
[ 2.674898] 14 000 1 0 0 0 0 0 0 00
[ 2.680000] 15 000 1 0 0 0 0 0 0 00
[ 2.685270] 16 000 1 0 0 0 0 0 0 00
[ 2.690542] 17 000 1 0 0 0 0 0 0 00
[ 2.695811] IRQ to pin mappings:
[ 2.701305] IRQ0 -> 0:2
[ 2.703835] IRQ1 -> 0:1
[ 2.706365] IRQ3 -> 0:3
[ 2.708895] IRQ4 -> 0:4
[ 2.711427] IRQ5 -> 0:5
[ 2.713957] IRQ6 -> 0:6
[ 2.716487] IRQ7 -> 0:7
[ 2.719018] IRQ8 -> 0:8
[ 2.721549] IRQ9 -> 0:9
[ 2.724080] IRQ10 -> 0:10
[ 2.726784] IRQ11 -> 0:11
[ 2.729487] IRQ12 -> 0:12
[ 2.732191] IRQ13 -> 0:13
[ 2.734893] IRQ14 -> 0:14
[ 2.737597] IRQ15 -> 0:15
[ 2.740301] .................................... done.
[ 2.745450] initcall pci_acpi_init+0x0/0xc0 returned 0 after 228 msecs
[ 2.752020] calling pci_legacy_init+0x0/0x130
[ 2.756475] initcall pci_legacy_init+0x0/0x130 returned 0 after 0 msecs
[ 2.763131] calling pcibios_irq_init+0x0/0x540
[ 2.767672] initcall pcibios_irq_init+0x0/0x540 returned 0 after 0 msecs
[ 2.774418] calling pcibios_init+0x0/0x80
[ 2.778597] initcall pcibios_init+0x0/0x80 returned 0 after 0 msecs
[ 2.784909] calling proto_init+0x0/0x40
[ 2.788845] initcall proto_init+0x0/0x40 returned 0 after 0 msecs
[ 2.794984] calling net_dev_init+0x0/0x160
[ 2.800000] initcall net_dev_init+0x0/0x160 returned 0 after 0 msecs
[ 2.806398] calling neigh_init+0x0/0x80
[ 2.810000] initcall neigh_init+0x0/0x80 returned 0 after 0 msecs
[ 2.816136] calling fib_rules_init+0x0/0xc0
[ 2.820000] initcall fib_rules_init+0x0/0xc0 returned 0 after 0 msecs
[ 2.826483] calling genl_init+0x0/0xe0
[ 2.870261] initcall genl_init+0x0/0xe0 returned 0 after 38 msecs
[ 2.876404] calling irda_init+0x0/0xb0
[ 2.880262] NET: Registered protocol family 23
[ 2.884734] initcall irda_init+0x0/0xb0 returned 0 after 9 msecs
[ 2.890784] calling bt_init+0x0/0x70
[ 2.894460] Bluetooth: Core ver 2.11
[ 2.900000] NET: Registered protocol family 31
[ 2.904458] Bluetooth: HCI device and connection manager initialized
[ 2.910000] Bluetooth: HCI socket layer initialized
[ 2.914891] initcall bt_init+0x0/0x70 returned 0 after 19 msecs
[ 2.920000] calling atm_init+0x0/0xd0
[ 2.923764] NET: Registered protocol family 8
[ 2.928132] NET: Registered protocol family 20
[ 2.940000] initcall atm_init+0x0/0xd0 returned 0 after 9 msecs
[ 2.940000] calling wireless_nlevent_init+0x0/0x40
[ 2.944892] initcall wireless_nlevent_init+0x0/0x40 returned 0 after 0 msecs
[ 2.951985] calling cfg80211_init+0x0/0x60
[ 2.960000] initcall cfg80211_init+0x0/0x60 returned 0 after 0 msecs
[ 2.966398] calling rfkill_init+0x0/0x60
[ 2.970000] initcall rfkill_init+0x0/0x60 returned 0 after 0 msecs
[ 2.976224] calling sysctl_init+0x0/0x40
[ 2.980000] initcall sysctl_init+0x0/0x40 returned 0 after 0 msecs
[ 2.986224] calling pci_iommu_init+0x0/0x20
[ 2.990000] PCI-GART: No AMD northbridge found.
[ 2.994544] initcall pci_iommu_init+0x0/0x20 returned 0 after 9 msecs
[ 3.000000] calling hpet_late_init+0x0/0x140
[ 3.004373] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[ 3.009396] hpet0: 3 64-bit timers, 14318180 Hz
[ 3.014941] initcall hpet_late_init+0x0/0x140 returned 0 after 9 msecs
[ 3.020000] calling clocksource_done_booting+0x0/0x20
[ 3.025150] initcall clocksource_done_booting+0x0/0x20 returned 0 after 0 msecs
[ 3.032507] calling tracer_alloc_buffers+0x0/0x720
[ 3.040000] tracer: 1286 pages allocated for 65536<6> entries of 80 bytes
[ 3.046831] actual entries 65586
[ 3.050384] initcall tracer_alloc_buffers+0x0/0x720 returned 0 after 13 msecs
[ 3.057563] calling init_pipe_fs+0x0/0x60
[ 3.063931] initcall init_pipe_fs+0x0/0x60 returned 0 after 0 msecs
[ 3.070241] calling init_mnt_writers+0x0/0x70
[ 3.074698] initcall init_mnt_writers+0x0/0x70 returned 0 after 0 msecs
[ 3.081276] calling eventpoll_init+0x0/0x90
[ 3.085563] initcall eventpoll_init+0x0/0x90 returned 0 after 0 msecs
[ 3.092047] calling anon_inode_init+0x0/0x130
[ 3.096505] initcall anon_inode_init+0x0/0x130 returned 0 after 0 msecs
[ 3.103168] calling pcie_aspm_init+0x0/0x30
[ 3.108046] initcall pcie_aspm_init+0x0/0x30 returned 0 after 0 msecs
[ 3.114531] calling acpi_rtc_init+0x0/0xd6
[ 3.118736] ACPI: RTC can wake from S4
[ 3.121193] initcall acpi_rtc_init+0x0/0xd6 returned 0 after 3 msecs
[ 3.127592] calling acpi_event_init+0x0/0x57
[ 3.172670] initcall acpi_event_init+0x0/0x57 returned 0 after 37 msecs
[ 3.179334] calling pnp_system_init+0x0/0x20
[ 3.183703] pnp: the driver 'system' has been registered
[ 3.189033] system 00:01: iomem range 0xf0000000-0xf3ffffff has been reserved
[ 3.196177] system 00:01: iomem range 0xfed13000-0xfed13fff has been reserved
[ 3.203322] system 00:01: iomem range 0xfed14000-0xfed17fff has been reserved
[ 3.210089] system 00:01: iomem range 0xfed18000-0xfed18fff has been reserved
[ 3.219476] system 00:01: iomem range 0xfed19000-0xfed19fff has been reserved
[ 3.226623] system 00:01: iomem range 0xfed1c000-0xfed1ffff has been reserved
[ 3.233770] system 00:01: iomem range 0xfed20000-0xfed3ffff has been reserved
[ 3.240913] system 00:01: iomem range 0xfed45000-0xfed99fff has been reserved
[ 3.248057] system 00:01: iomem range 0xc0000-0xdffff has been reserved
[ 3.255207] system 00:01: iomem range 0xe0000-0xfffff could not be reserved
[ 3.261832] system 00:01: driver attached
[ 3.265861] system 00:06: ioport range 0x500-0x53f has been reserved
[ 3.272223] system 00:06: ioport range 0x400-0x47f has been reserved
[ 3.278586] system 00:06: ioport range 0x680-0x6ff has been reserved
[ 3.285563] system 00:06: driver attached
[ 3.291926] initcall pnp_system_init+0x0/0x20 returned 0 after 100 msecs
[ 3.298673] calling chr_dev_init+0x0/0xd0
[ 3.300174] initcall chr_dev_init+0x0/0xd0 returned 0 after 0 msecs
[ 3.306487] calling firmware_class_init+0x0/0x90
[ 3.311018] initcall firmware_class_init+0x0/0x90 returned 0 after 0 msecs
[ 3.317936] calling loopback_init+0x0/0x20
[ 3.322681] initcall loopback_init+0x0/0x20 returned 0 after 0 msecs
[ 3.329080] calling init_pcmcia_bus+0x0/0x90
[ 3.334798] initcall init_pcmcia_bus+0x0/0x90 returned 0 after 0 msecs
[ 3.340080] calling init_acpi_pm_clocksource+0x0/0xc0
[ 3.345233] initcall init_acpi_pm_clocksource+0x0/0xc0 returned 0 after 0 msecs
[ 3.351830] calling pcibios_assign_resources+0x0/0xa0
[ 3.357005] PCI: Bridge: 0000:00:01.0
[ 3.360680] IO window: 3000-3fff
[ 3.364342] MEM window: 0x50300000-0x503fffff
[ 3.368885] PREFETCH window: 0x0000000040000000-0x000000004fffffff
[ 3.375251] PCI: Bridge: 0000:00:1c.0
[ 3.378927] IO window: disabled.
[ 3.382374] MEM window: 0x50200000-0x502fffff
[ 3.386917] PREFETCH window: disabled.
[ 3.389466] PCI: Bridge: 0000:00:1c.4
[ 3.393143] IO window: disabled.
[ 3.396560] MEM window: disabled.
[ 3.400063] PREFETCH window: disabled.
[ 3.404004] PCI: Bridge: 0000:00:1c.5
[ 3.407681] IO window: 2000-2fff
[ 3.412493] MEM window: 0x50100000-0x501fffff
[ 3.417037] PREFETCH window: disabled.
[ 3.419589] PCI: Bridge: 0000:00:1e.0
[ 3.423267] IO window: 1000-1fff
[ 3.426685] MEM window: 0x50000000-0x500fffff
[ 3.431757] PREFETCH window: disabled.
[ 3.435704] IOAPIC[0]: Set routing entry (2-16 -> 0x49 -> IRQ 16 Mode:1 Active:1)
[ 3.443198] ACPI: PCI Interrupt 0000:00:01.0[A] -> GSI 16 (level, low) -> IRQ 16
[ 3.450642] PCI: Setting latency timer of device 0000:00:01.0 to 64
[ 3.456933] IOAPIC[0]: Set routing entry (2-17 -> 0x51 -> IRQ 17 Mode:1 Active:1)
[ 3.465262] ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 17 (level, low) -> IRQ 17
[ 3.472775] PCI: Setting latency timer of device 0000:00:1c.0 to 64
[ 3.479063] ACPI: PCI Interrupt 0000:00:1c.4[A] -> GSI 17 (level, low) -> IRQ 17
[ 3.486508] PCI: Setting latency timer of device 0000:00:1c.4 to 64
[ 3.492799] ACPI: PCI Interrupt 0000:00:1c.5[B] -> GSI 16 (level, low) -> IRQ 16
[ 3.500233] PCI: Setting latency timer of device 0000:00:1c.5 to 64
[ 3.506516] PCI: Setting latency timer of device 0000:00:1e.0 to 64
[ 3.513970] initcall pcibios_assign_resources+0x0/0xa0 returned 0 after 148 msecs
[ 3.520252] calling inet_init+0x0/0x370
[ 3.524196] NET: Registered protocol family 2
[ 3.622161] IP route cache hash table entries: 32768 (order: 6, 262144 bytes)
[ 3.629623] TCP established hash table entries: 131072 (order: 9, 2097152 bytes)
[ 3.637971] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[ 3.645258] TCP: Hash tables configured (established 131072 bind 65536)
[ 3.651880] TCP reno registered
[ 3.682780] initcall inet_init+0x0/0x370 returned 0 after 150 msecs
[ 3.689098] calling af_unix_init+0x0/0x70
[ 3.693216] NET: Registered protocol family 1
[ 3.697591] initcall af_unix_init+0x0/0x70 returned 0 after 4 msecs
[ 3.703904] calling populate_rootfs+0x0/0x100
[ 3.708437] initcall populate_rootfs+0x0/0x100 returned 0 after 0 msecs
[ 3.715098] calling calgary_fixup_tce_spaces+0x0/0x120
[ 3.719734] initcall calgary_fixup_tce_spaces+0x0/0x120 returned -19 after 0 msecs
[ 3.729676] calling i8259A_init_sysfs+0x0/0x30
[ 3.741842] initcall i8259A_init_sysfs+0x0/0x30 returned 0 after 0 msecs
[ 3.741842] calling vsyscall_init+0x0/0x80
[ 3.746055] initcall vsyscall_init+0x0/0x80 returned 0 after 0 msecs
[ 3.752452] calling sbf_init+0x0/0xf0
[ 3.756218] initcall sbf_init+0x0/0xf0 returned 0 after 0 msecs
[ 3.762183] calling i8237A_init_sysfs+0x0/0x30
[ 3.773562] initcall i8237A_init_sysfs+0x0/0x30 returned 0 after 0 msecs
[ 3.780308] calling cache_sysfs_init+0x0/0x70
[ 3.779681] initcall cache_sysfs_init+0x0/0x70 returned 0 after 0 msecs
[ 3.786344] calling mce_init_device+0x0/0x90
[ 3.791701] initcall mce_init_device+0x0/0x90 returned 0 after 0 msecs
[ 3.798272] calling periodic_mcheck_init+0x0/0x50
[ 3.800394] initcall periodic_mcheck_init+0x0/0x50 returned 0 after 0 msecs
[ 3.807406] calling thermal_throttle_init_device+0x0/0xa0
[ 3.812903] initcall thermal_throttle_init_device+0x0/0xa0 returned 0 after 0 msecs
[ 3.820090] calling cpuid_init+0x0/0x100
[ 3.831815] initcall cpuid_init+0x0/0x100 returned 0 after 0 msecs
[ 3.831815] calling init_lapic_sysfs+0x0/0x40
[ 3.842581] initcall init_lapic_sysfs+0x0/0x40 returned 0 after 0 msecs
[ 3.842581] calling ioapic_init_sysfs+0x0/0xe0
[ 3.853830] initcall ioapic_init_sysfs+0x0/0xe0 returned 0 after 0 msecs
[ 3.860574] calling add_pcspkr+0x0/0x60
[ 3.860620] initcall add_pcspkr+0x0/0x60 returned 0 after 0 msecs
[ 3.866759] calling init+0x0/0x20
[ 3.870177] initcall init+0x0/0x20 returned 0 after 0 msecs
[ 3.875795] calling init_vdso_vars+0x0/0x240
[ 3.879769] initcall init_vdso_vars+0x0/0x240 returned 0 after 0 msecs
[ 3.886339] calling ia32_binfmt_init+0x0/0x20
[ 3.890722] initcall ia32_binfmt_init+0x0/0x20 returned 0 after 0 msecs
[ 3.897379] calling sysenter_setup+0x0/0x380
[ 3.901752] initcall sysenter_setup+0x0/0x380 returned 0 after 0 msecs
[ 3.908323] calling init_aout_binfmt+0x0/0x20
[ 3.910410] initcall init_aout_binfmt+0x0/0x20 returned 0 after 0 msecs
[ 3.917069] calling init_sched_debug_procfs+0x0/0x30
[ 3.921527] initcall init_sched_debug_procfs+0x0/0x30 returned 0 after 0 msecs
[ 3.928797] calling create_proc_profile+0x0/0x3e0
[ 3.933873] initcall create_proc_profile+0x0/0x3e0 returned 0 after 0 msecs
[ 3.940876] calling ioresources_init+0x0/0x50
[ 3.945335] initcall ioresources_init+0x0/0x50 returned 0 after 0 msecs
[ 3.951991] calling uid_cache_init+0x0/0x90
[ 3.956276] initcall uid_cache_init+0x0/0x90 returned 0 after 0 msecs
[ 3.963468] calling init_posix_timers+0x0/0xd0
[ 3.968013] initcall init_posix_timers+0x0/0xd0 returned 0 after 0 msecs
[ 3.974757] calling init_posix_cpu_timers+0x0/0xe0
[ 3.979647] initcall init_posix_cpu_timers+0x0/0xe0 returned 0 after 0 msecs
[ 3.986737] calling nsproxy_cache_init+0x0/0x40
[ 3.990850] initcall nsproxy_cache_init+0x0/0x40 returned 0 after 0 msecs
[ 3.997682] calling timekeeping_init_device+0x0/0x30
[ 4.003061] initcall timekeeping_init_device+0x0/0x30 returned 0 after 0 msecs
[ 4.009760] calling init_clocksource_sysfs+0x0/0x60
[ 4.022056] initcall init_clocksource_sysfs+0x0/0x60 returned 0 after 0 msecs
[ 4.022056] calling init_timer_list_procfs+0x0/0x40
[ 4.027036] initcall init_timer_list_procfs+0x0/0x40 returned 0 after 0 msecs
[ 4.034267] calling futex_init+0x0/0xc0
[ 4.038212] initcall futex_init+0x0/0xc0 returned 0 after 0 msecs
[ 4.044348] calling proc_dma_init+0x0/0x30
[ 4.048545] initcall proc_dma_init+0x0/0x30 returned 0 after 0 msecs
[ 4.054943] calling kallsyms_init+0x0/0x30
[ 4.059139] initcall kallsyms_init+0x0/0x30 returned 0 after 0 msecs
[ 4.065536] calling snapshot_device_init+0x0/0x20
[ 4.070345] initcall snapshot_device_init+0x0/0x20 returned 0 after 0 msecs
[ 4.077352] calling crash_save_vmcoreinfo_init+0x0/0x430
[ 4.082771] initcall crash_save_vmcoreinfo_init+0x0/0x430 returned 0 after 0 msecs
[ 4.089977] calling crash_notes_memory_init+0x0/0x50
[ 4.095042] initcall crash_notes_memory_init+0x0/0x50 returned 0 after 0 msecs
[ 4.102671] calling utsname_sysctl_init+0x0/0x20
[ 4.107396] initcall utsname_sysctl_init+0x0/0x20 returned 0 after 0 msecs
[ 4.114312] calling init_lstats_procfs+0x0/0x30
[ 4.118941] initcall init_lstats_procfs+0x0/0x30 returned 0 after 0 msecs
[ 4.125771] calling init_sched_switch_trace+0x0/0x50
[ 4.130670] Testing tracer sched_switch: PASSED
[ 4.536573] initcall init_sched_switch_trace+0x0/0x50 returned 0 after 385 msecs
[ 4.540510] calling init_stack_trace+0x0/0x20
[ 4.545018] Testing tracer sysprof: PASSED
[ 4.670741] initcall init_stack_trace+0x0/0x20 returned 0 after 112 msecs
[ 4.670741] calling init_function_trace+0x0/0x20
[ 4.675508] Testing tracer ftrace: PASSED
[ 4.831073] initcall init_function_trace+0x0/0x20 returned 0 after 141 msecs
[ 4.831073] calling init_irqsoff_tracer+0x0/0x20
[ 4.835840] Testing tracer irqsoff: <6>( swapper-1 |#1): new 112 us maximum-latency critical section.
[ 4.847464] PASSED
[ 4.849496] initcall init_irqsoff_tracer+0x0/0x20 returned 0 after 13 msecs
[ 4.857275] calling init_per_zone_pages_min+0x0/0x60
[ 4.862414] initcall init_per_zone_pages_min+0x0/0x60 returned 0 after 0 msecs
[ 4.869680] calling pdflush_init+0x0/0x20
[ 4.877258] initcall pdflush_init+0x0/0x20 returned 0 after 0 msecs
[ 4.881609] calling kswapd_init+0x0/0x70
[ 4.887466] initcall kswapd_init+0x0/0x70 returned 0 after 0 msecs
[ 4.891992] calling setup_vmstat+0x0/0x50
[ 4.896105] initcall setup_vmstat+0x0/0x50 returned 0 after 0 msecs
[ 4.902349] calling procswaps_init+0x0/0x30
[ 4.906633] initcall procswaps_init+0x0/0x30 returned 0 after 0 msecs
[ 4.912946] calling hugetlb_init+0x0/0xc0
[ 4.917061] Total HugeTLB memory allocated, 0
[ 4.921430] initcall hugetlb_init+0x0/0xc0 returned 0 after 4 msecs
[ 4.927742] calling init_tmpfs+0x0/0xf0
[ 4.931698] initcall init_tmpfs+0x0/0xf0 returned 0 after 0 msecs
[ 4.937836] calling slab_sysfs_init+0x0/0x100
[ 4.941952] initcall slab_sysfs_init+0x0/0x100 returned 0 after 2 msecs
[ 4.948610] calling fasync_init+0x0/0x30
[ 4.955257] initcall fasync_init+0x0/0x30 returned 0 after 0 msecs
[ 4.961480] calling aio_setup+0x0/0x80
[ 4.967958] initcall aio_setup+0x0/0x80 returned 0 after 0 msecs
[ 4.971732] calling init_sys32_ioctl+0x0/0x90
[ 4.976201] initcall init_sys32_ioctl+0x0/0x90 returned 0 after 0 msecs
[ 4.982285] calling init_mbcache+0x0/0x20
[ 4.983068] initcall init_mbcache+0x0/0x20 returned 0 after 0 msecs
[ 4.989377] calling dquot_init+0x0/0x100
[ 4.993404] VFS: Disk quotas dquot_6.5.1
[ 5.002042] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[ 5.008495] initcall dquot_init+0x0/0x100 returned 0 after 9 msecs
[ 5.013862] calling init_v2_quota_format+0x0/0x20
[ 5.018666] initcall init_v2_quota_format+0x0/0x20 returned 0 after 0 msecs
[ 5.025671] calling vmcore_init+0x0/0xa50
[ 5.029781] initcall vmcore_init+0x0/0xa50 returned 0 after 0 msecs
[ 5.036092] calling configfs_init+0x0/0xf0
[ 5.042324] initcall configfs_init+0x0/0xf0 returned 0 after 0 msecs
[ 5.048722] calling init_devpts_fs+0x0/0x50
[ 5.052979] initcall init_devpts_fs+0x0/0x50 returned 0 after 0 msecs
[ 5.059463] calling init_dlm+0x0/0xa0
[ 5.061383] DLM (built May 24 2008 04:24:43) installed
[ 5.066536] initcall init_dlm+0x0/0xa0 returned 0 after 5 msecs
[ 5.072500] calling init_ext3_fs+0x0/0x80
[ 5.082884] initcall init_ext3_fs+0x0/0x80 returned 0 after 0 msecs
[ 5.082884] calling journal_init+0x0/0xf0
[ 5.093408] initcall journal_init+0x0/0xf0 returned 0 after 0 msecs
[ 5.093408] calling init_ext2_fs+0x0/0x80
[ 5.103493] initcall init_ext2_fs+0x0/0x80 returned 0 after 0 msecs
[ 5.104045] calling init_ramfs_fs+0x0/0x20
[ 5.108244] initcall init_ramfs_fs+0x0/0x20 returned 0 after 0 msecs
[ 5.114611] calling init_hugetlbfs_fs+0x0/0xc0
[ 5.125557] initcall init_hugetlbfs_fs+0x0/0xc0 returned 0 after 0 msecs
[ 5.132301] calling init_coda+0x0/0x190
[ 5.132362] initcall init_coda+0x0/0x190 returned 0 after 0 msecs
[ 5.138873] calling init_minix_fs+0x0/0x70
[ 5.143020] initcall init_minix_fs+0x0/0x70 returned 0 after 0 msecs
[ 5.149419] calling init_bfs_fs+0x0/0x70
[ 5.151289] initcall init_bfs_fs+0x0/0x70 returned 0 after 0 msecs
[ 5.157513] calling init_iso9660_fs+0x0/0x90
[ 5.161590] initcall init_iso9660_fs+0x0/0x90 returned 0 after 0 msecs
[ 5.168161] calling init_hfsplus_fs+0x0/0x80
[ 5.172584] initcall init_hfsplus_fs+0x0/0x80 returned 0 after 0 msecs
[ 5.179157] calling init_hfs_fs+0x0/0x80
[ 5.183577] initcall init_hfs_fs+0x0/0x80 returned 0 after 0 msecs
[ 5.189804] calling init_nls_cp437+0x0/0x20
[ 5.193879] initcall init_nls_cp437+0x0/0x20 returned 0 after 0 msecs
[ 5.200364] calling init_nls_cp850+0x0/0x20
[ 5.204652] initcall init_nls_cp850+0x0/0x20 returned 0 after 0 msecs
[ 5.211135] calling init_nls_cp852+0x0/0x20
[ 5.215418] initcall init_nls_cp852+0x0/0x20 returned 0 after 0 msecs
[ 5.221901] calling init_nls_cp857+0x0/0x20
[ 5.226185] initcall init_nls_cp857+0x0/0x20 returned 0 after 0 msecs
[ 5.232667] calling init_nls_cp862+0x0/0x20
[ 5.236951] initcall init_nls_cp862+0x0/0x20 returned 0 after 0 msecs
[ 5.243435] calling init_nls_cp863+0x0/0x20
[ 5.247718] initcall init_nls_cp863+0x0/0x20 returned 0 after 0 msecs
[ 5.254201] calling init_nls_cp864+0x0/0x20
[ 5.258485] initcall init_nls_cp864+0x0/0x20 returned 0 after 0 msecs
[ 5.267540] calling init_nls_cp865+0x0/0x20
[ 5.271515] initcall init_nls_cp865+0x0/0x20 returned 0 after 0 msecs
[ 5.277999] calling init_nls_cp866+0x0/0x20
[ 5.282282] initcall init_nls_cp866+0x0/0x20 returned 0 after 0 msecs
[ 5.288766] calling init_nls_cp936+0x0/0x20
[ 5.293049] initcall init_nls_cp936+0x0/0x20 returned 0 after 0 msecs
[ 5.299533] calling init_nls_cp949+0x0/0x20
[ 5.303816] initcall init_nls_cp949+0x0/0x20 returned 0 after 0 msecs
[ 5.310299] calling init_nls_cp950+0x0/0x20
[ 5.314595] initcall init_nls_cp950+0x0/0x20 returned 0 after 0 msecs
[ 5.321078] calling init_nls_cp1250+0x0/0x20
[ 5.325447] initcall init_nls_cp1250+0x0/0x20 returned 0 after 0 msecs
[ 5.332017] calling init_nls_iso8859_1+0x0/0x20
[ 5.336648] initcall init_nls_iso8859_1+0x0/0x20 returned 0 after 0 msecs
[ 5.343314] calling init_nls_iso8859_3+0x0/0x20
[ 5.347945] initcall init_nls_iso8859_3+0x0/0x20 returned 0 after 0 msecs
[ 5.354777] calling init_nls_iso8859_6+0x0/0x20
[ 5.359407] initcall init_nls_iso8859_6+0x0/0x20 returned 0 after 0 msecs
[ 5.366240] calling init_nls_cp1255+0x0/0x20
[ 5.370609] initcall init_nls_cp1255+0x0/0x20 returned 0 after 0 msecs
[ 5.377182] calling init_nls_iso8859_14+0x0/0x20
[ 5.381640] initcall init_nls_iso8859_14+0x0/0x20 returned 0 after 0 msecs
[ 5.388558] calling init_nls_koi8_r+0x0/0x20
[ 5.392928] initcall init_nls_koi8_r+0x0/0x20 returned 0 after 0 msecs
[ 5.399500] calling init_nls_koi8_u+0x0/0x20
[ 5.403870] initcall init_nls_koi8_u+0x0/0x20 returned 0 after 0 msecs
[ 5.410442] calling init_nls_koi8_ru+0x0/0x50
[ 5.415170] initcall init_nls_koi8_ru+0x0/0x50 returned 0 after 0 msecs
[ 5.421826] calling init_nls_utf8+0x0/0x40
[ 5.426024] initcall init_nls_utf8+0x0/0x40 returned 0 after 0 msecs
[ 5.432419] calling init_ufs_fs+0x0/0x70
[ 5.440890] initcall init_ufs_fs+0x0/0x70 returned 0 after 0 msecs
[ 5.443283] calling init_romfs_fs+0x0/0x70
[ 5.452875] initcall init_romfs_fs+0x0/0x70 returned 0 after 0 msecs
[ 5.453757] calling init_autofs4_fs+0x0/0x20
[ 5.458128] initcall init_autofs4_fs+0x0/0x20 returned 0 after 0 msecs
[ 5.464699] calling init_adfs_fs+0x0/0x70
[ 5.468988] initcall init_adfs_fs+0x0/0x70 returned 0 after 0 msecs
[ 5.475301] calling fuse_init+0x0/0x140
[ 5.479235] fuse init (API version 7.9)
[ 5.481622] initcall fuse_init+0x0/0x140 returned 0 after 3 msecs
[ 5.487763] calling init_xfs_fs+0x0/0x70
[ 5.491783] SGI XFS with ACLs, security attributes, large block/inode numbers, debug enabled
[ 5.508151] initcall init_xfs_fs+0x0/0x70 returned 0 after 8 msecs
[ 5.512560] calling init_befs_fs+0x0/0xa0
[ 5.512560] BeFS version: 0.9.3
[ 5.518011] initcall init_befs_fs+0x0/0xa0 returned 0 after 3 msecs
[ 5.522207] calling ipc_init+0x0/0x30
[ 5.525973] msgmni has been set to 1915 for ipc namespace ffffffff80d5fdc0
[ 5.532333] initcall ipc_init+0x0/0x30 returned 0 after 6 msecs
[ 5.538298] calling ipc_sysctl_init+0x0/0x20
[ 5.542683] initcall ipc_sysctl_init+0x0/0x20 returned 0 after 0 msecs
[ 5.549254] calling key_proc_init+0x0/0x40
[ 5.553451] initcall key_proc_init+0x0/0x40 returned 0 after 0 msecs
[ 5.559850] calling crypto_algapi_init+0x0/0x20
[ 5.564482] initcall crypto_algapi_init+0x0/0x20 returned 0 after 0 msecs
[ 5.571312] calling blkcipher_module_init+0x0/0x40
[ 5.576204] initcall blkcipher_module_init+0x0/0x40 returned 0 after 0 msecs
[ 5.583105] calling seqiv_module_init+0x0/0x20
[ 5.587648] initcall seqiv_module_init+0x0/0x20 returned 0 after 0 msecs
[ 5.594743] calling cryptomgr_init+0x0/0x20
[ 5.599025] initcall cryptomgr_init+0x0/0x20 returned 0 after 0 msecs
[ 5.605508] calling hmac_module_init+0x0/0x20
[ 5.609966] initcall hmac_module_init+0x0/0x20 returned 0 after 0 msecs
[ 5.616622] calling md4_mod_init+0x0/0x20
[ 5.620734] initcall md4_mod_init+0x0/0x20 returned 0 after 0 msecs
[ 5.627045] calling md5_mod_init+0x0/0x20
[ 5.631155] initcall md5_mod_init+0x0/0x20 returned 0 after 0 msecs
[ 5.637467] calling sha1_generic_mod_init+0x0/0x20
[ 5.641845] initcall sha1_generic_mod_init+0x0/0x20 returned 0 after 0 msecs
[ 5.648936] calling sha256_generic_mod_init+0x0/0x60
[ 5.654002] initcall sha256_generic_mod_init+0x0/0x60 returned 0 after 0 msecs
[ 5.661268] calling sha512_generic_mod_init+0x0/0x60
[ 5.666334] initcall sha512_generic_mod_init+0x0/0x60 returned 0 after 0 msecs
[ 5.673602] calling crypto_cbc_module_init+0x0/0x20
[ 5.678579] initcall crypto_cbc_module_init+0x0/0x20 returned 0 after 0 msecs
[ 5.685859] calling crypto_pcbc_module_init+0x0/0x20
[ 5.690921] initcall crypto_pcbc_module_init+0x0/0x20 returned 0 after 0 msecs
[ 5.698191] calling crypto_module_init+0x0/0x20
[ 5.702819] initcall crypto_module_init+0x0/0x20 returned 0 after 0 msecs
[ 5.709649] calling crypto_module_init+0x0/0x20
[ 5.714460] initcall crypto_module_init+0x0/0x20 returned 0 after 0 msecs
[ 5.721289] calling crypto_ctr_module_init+0x0/0x60
[ 5.726268] initcall crypto_ctr_module_init+0x0/0x60 returned 0 after 0 msecs
[ 5.733278] calling des_generic_mod_init+0x0/0x60
[ 5.738084] initcall des_generic_mod_init+0x0/0x60 returned 0 after 0 msecs
[ 5.745265] calling cast5_mod_init+0x0/0x20
[ 5.749548] initcall cast5_mod_init+0x0/0x20 returned 0 after 0 msecs
[ 5.756031] calling cast6_mod_init+0x0/0x20
[ 5.760315] initcall cast6_mod_init+0x0/0x20 returned 0 after 0 msecs
[ 5.766800] calling arc4_init+0x0/0x20
[ 5.770650] initcall arc4_init+0x0/0x20 returned 0 after 0 msecs
[ 5.778966] calling tea_mod_init+0x0/0x90
[ 5.783077] initcall tea_mod_init+0x0/0x90 returned 0 after 0 msecs
[ 5.789388] calling salsa20_generic_mod_init+0x0/0x20
[ 5.794540] initcall salsa20_generic_mod_init+0x0/0x20 returned 0 after 0 msecs
[ 5.801557] calling deflate_mod_init+0x0/0x20
[ 5.806014] initcall deflate_mod_init+0x0/0x20 returned 0 after 0 msecs
[ 5.812670] calling crc32c_mod_init+0x0/0x20
[ 5.817040] initcall crc32c_mod_init+0x0/0x20 returned 0 after 0 msecs
[ 5.823608] calling crypto_authenc_module_init+0x0/0x20
[ 5.828932] initcall crypto_authenc_module_init+0x0/0x20 returned 0 after 0 msecs
[ 5.838677] calling noop_init+0x0/0x20
[ 5.841443] io scheduler noop registered (default)
[ 5.846248] initcall noop_init+0x0/0x20 returned 0 after 4 msecs
[ 5.852296] calling deadline_init+0x0/0x20
[ 5.856493] io scheduler deadline registered
[ 5.860776] initcall deadline_init+0x0/0x20 returned 0 after 4 msecs
[ 5.867172] calling init_kmp+0x0/0x20
[ 5.870936] initcall init_kmp+0x0/0x20 returned 0 after 0 msecs
[ 5.876900] calling init_bm+0x0/0x20
[ 5.880575] initcall init_bm+0x0/0x20 returned 0 after 0 msecs
[ 5.886451] calling init_fsm+0x0/0x20
[ 5.890213] initcall init_fsm+0x0/0x20 returned 0 after 0 msecs
[ 5.896177] calling percpu_counter_startup+0x0/0x20
[ 5.901153] initcall percpu_counter_startup+0x0/0x20 returned 0 after 0 msecs
[ 5.908335] calling pci_init+0x0/0x40
[ 5.912112] pci 0000:00:1d.0: uhci_check_and_reset_hc: legsup = 0x0f10
[ 5.918649] pci 0000:00:1d.0: Performing full reset
[ 5.923550] pci 0000:00:1d.1: uhci_check_and_reset_hc: legsup = 0x0010
[ 5.926343] pci 0000:00:1d.1: Performing full reset
[ 5.936356] pci 0000:00:1d.2: uhci_check_and_reset_hc: legsup = 0x0010
[ 5.941233] pci 0000:00:1d.2: Performing full reset
[ 5.942689] pci 0000:00:1d.3: uhci_check_and_reset_hc: legsup = 0x0010
[ 5.949224] pci 0000:00:1d.3: Performing full reset
[ 5.954249] pci 0000:01:00.0: Boot video device
[ 5.958797] initcall pci_init+0x0/0x40 returned 0 after 48 msecs
[ 5.966582] calling pci_proc_init+0x0/0x80
[ 5.970811] initcall pci_proc_init+0x0/0x80 returned 0 after 0 msecs
[ 5.975483] calling pcie_portdrv_init+0x0/0x60
[ 5.983102] PCI: Setting latency timer of device 0000:00:01.0 to 64
[ 5.989401] assign_interrupt_mode Found MSI capability
[ 5.996355] Allocate Port Service[0000:00:01.0:pcie00]
[ 6.001528] PCI: Setting latency timer of device 0000:00:1c.0 to 64
[ 6.007829] assign_interrupt_mode Found MSI capability
[ 6.012978] Allocate Port Service[0000:00:1c.0:pcie00]
[ 6.022677] Allocate Port Service[0000:00:1c.0:pcie02]
[ 6.023343] PCI: Setting latency timer of device 0000:00:1c.4 to 64
[ 6.029645] assign_interrupt_mode Found MSI capability
[ 6.034794] Allocate Port Service[0000:00:1c.4:pcie00]
[ 6.045192] Allocate Port Service[0000:00:1c.4:pcie02]
[ 6.045192] PCI: Setting latency timer of device 0000:00:1c.5 to 64
[ 6.051494] assign_interrupt_mode Found MSI capability
[ 6.056714] Allocate Port Service[0000:00:1c.5:pcie00]
[ 6.061888] Allocate Port Service[0000:00:1c.5:pcie02]
[ 6.071671] initcall pcie_portdrv_init+0x0/0x60 returned 0 after 78 msecs
[ 6.072217] calling aer_service_init+0x0/0x30
[ 6.081734] initcall aer_service_init+0x0/0x30 returned 0 after 0 msecs
[ 6.083599] calling ltv350qv_init+0x0/0x20
[ 6.094484] initcall ltv350qv_init+0x0/0x20 returned 0 after 0 msecs
[ 6.101255] calling progearbl_init+0x0/0x80
[ 6.101284] ALI M7101 PMU not found.
[ 6.101284] initcall progearbl_init+0x0/0x80 returned 0 after 3 msecs
[ 6.107769] calling display_class_init+0x0/0x85
[ 6.113322] initcall display_class_init+0x0/0x85 returned 0 after 0 msecs
[ 6.120154] calling acpi_reserve_resources+0x0/0xf0
[ 6.125134] initcall acpi_reserve_resources+0x0/0xf0 returned 0 after 0 msecs
[ 6.132168] calling acpi_ac_init+0x0/0x2d
[ 6.141087] initcall acpi_ac_init+0x0/0x2d returned 0 after 0 msecs
[ 6.143459] calling acpi_battery_init+0x0/0x2d
[ 6.154370] initcall acpi_battery_init+0x0/0x2d returned 0 after 0 msecs
[ 6.161114] calling acpi_fan_init+0x0/0x37
[ 6.161158] initcall acpi_fan_init+0x0/0x37 returned 0 after 0 msecs
[ 6.167557] calling irqrouter_init_sysfs+0x0/0x3d
[ 6.171795] initcall irqrouter_init_sysfs+0x0/0x3d returned 0 after 0 msecs
[ 6.178800] calling acpi_smb_hc_init+0x0/0x1d
[ 6.181268] initcall acpi_smb_hc_init+0x0/0x1d returned 0 after 0 msecs
[ 6.187926] calling acpi_sbs_init+0x0/0x35
[ 6.192425] initcall acpi_sbs_init+0x0/0x35 returned 0 after 0 msecs
[ 6.198826] calling rand_initialize+0x0/0x40
[ 6.203065] initcall rand_initialize+0x0/0x40 returned 0 after 0 msecs
[ 6.209637] calling tty_init+0x0/0x1f0
[ 6.221079] initcall tty_init+0x0/0x1f0 returned 0 after 1 msecs
[ 6.221079] calling pty_init+0x0/0x550
[ 6.241087] initcall pty_init+0x0/0x550 returned 0 after 15 msecs
[ 6.247234] calling cy_init+0x0/0x170
[ 6.250997] Cyclades driver 2.5 (built May 24 2008 04:24:00)
[ 6.262255] initcall cy_init+0x0/0x170 returned 0 after 5 msecs
[ 6.263846] calling epca_module_init+0x0/0x590
[ 6.268391] DIGI epca driver version 1.3.0.1-LK2.6 loaded.
[ 6.291072] initcall epca_module_init+0x0/0x590 returned 0 after 23 msecs
[ 6.297913] calling specialix_init_module+0x0/0x540
[ 6.302888] sx: Specialix IO8+ driver v1.11, (c) R.E.Wolff 1997/1998.
[ 6.309338] sx: derived from work (c) D.Gorodchanin 1994-1996.
[ 6.316891] sx: DTR/RTS pin is RTS when CRTSCTS is on.
[ 6.322736] sx0: specialix IO8+ Board at 0x100 not found.
[ 6.328159] sx1: specialix IO8+ Board at 0x180 not found.
[ 6.335842] sx2: specialix IO8+ Board at 0x250 not found.
[ 6.341216] sx3: specialix IO8+ Board at 0x260 not found.
[ 6.352095] sx: No specialix IO8+ boards detected.
[ 6.356907] initcall specialix_init_module+0x0/0x540 returned -5 after 50 msecs
[ 6.363353] initcall specialix_init_module+0x0/0x540 returned with error code -5
[ 6.370880] calling mxser_module_init+0x0/0x790
[ 6.375515] MOXA Smartio/Industio family driver version 2.0.3
[ 6.382874] initcall mxser_module_init+0x0/0x790 returned 0 after 5 msecs
[ 6.389707] calling isicom_init+0x0/0x1f0
[ 6.393147] initcall isicom_init+0x0/0x1f0 returned 0 after 0 msecs
[ 6.399460] calling synclink_init+0x0/0x2d0
[ 6.403611] SyncLink serial driver $Revision: 4.38 $
[ 6.415252] SyncLink serial driver $Revision: 4.38 $, tty major#254
[ 6.421535] initcall synclink_init+0x0/0x2d0 returned 0 after 15 msecs
[ 6.428106] calling synclinkmp_init+0x0/0x1f0
[ 6.432560] SyncLink MultiPort driver $Revision: 4.38 $
[ 6.442763] SyncLink MultiPort driver $Revision: 4.38 $, tty major#253
[ 6.449309] initcall synclinkmp_init+0x0/0x1f0 returned 0 after 16 msecs
[ 6.456055] calling n_hdlc_init+0x0/0xb0
[ 6.460077] HDLC line discipline: version $Revision: 4.8 $, maxframe=4096
[ 6.466875] N_HDLC line discipline registered.
[ 6.471331] initcall n_hdlc_init+0x0/0xb0 returned 0 after 10 msecs
[ 6.477643] calling sx_init+0x0/0x120
[ 6.482940] initcall sx_init+0x0/0x120 returned 0 after 0 msecs
[ 6.488907] calling r3964_init+0x0/0x50
[ 6.492736] r3964: Philips r3964 Driver $Revision: 1.10 $
[ 6.498146] initcall r3964_init+0x0/0x50 returned 0 after 5 msecs
[ 6.502085] calling applicom_init+0x0/0x560
[ 6.506371] Applicom driver: $Id: ac.c,v 1.30 2000/03/22 16:03:57 dwmw2 Exp $
[ 6.512509] ac.o: No PCI boards found.
[ 6.516272] ac.o: For an ISA board you must supply memory and irq parameters.
[ 6.523415] initcall applicom_init+0x0/0x560 returned -6 after 17 msecs
[ 6.530073] initcall applicom_init+0x0/0x560 returned with error code -6
[ 6.537400] calling hpet_init+0x0/0x80
[ 6.541334] initcall hpet_init+0x0/0x80 returned 0 after 0 msecs
[ 6.547387] calling nvram_init+0x0/0xa0
[ 6.551682] Non-volatile memory driver v1.2
[ 6.555881] initcall nvram_init+0x0/0xa0 returned 0 after 4 msecs
[ 6.562017] calling mod_init+0x0/0xd0
[ 6.565792] initcall mod_init+0x0/0xd0 returned -19 after 0 msecs
[ 6.571928] calling pc8736x_gpio_init+0x0/0x410
[ 6.580246] platform pc8736x_gpio.0: NatSemi pc8736x GPIO Driver Initializing
[ 6.582909] platform pc8736x_gpio.0: no device found
[ 6.595071] initcall pc8736x_gpio_init+0x0/0x410 returned -19 after 11 msecs
[ 6.602164] calling nsc_gpio_init+0x0/0x20
[ 6.602193] nsc_gpio initializing
[ 6.602193] initcall nsc_gpio_init+0x0/0x20 returned 0 after 3 msecs
[ 6.608591] calling agp_init+0x0/0x30
[ 6.612353] Linux agpgart interface v0.103
[ 6.616462] initcall agp_init+0x0/0x30 returned 0 after 3 msecs
[ 6.622426] calling agp_intel_init+0x0/0x40
[ 6.633071] initcall agp_intel_init+0x0/0x40 returned 0 after 0 msecs
[ 6.634244] calling agp_sis_init+0x0/0x40
[ 6.644876] initcall agp_sis_init+0x0/0x40 returned 0 after 0 msecs
[ 6.651188] calling drm_core_init+0x0/0x110
[ 6.651221] [drm] Initialized drm 1.1.0 20060810
[ 6.651221] initcall drm_core_init+0x0/0x110 returned 0 after 4 msecs
[ 6.657706] calling tdfx_init+0x0/0x20
[ 6.661568] initcall tdfx_init+0x0/0x20 returned 0 after 0 msecs
[ 6.667619] calling i830_init+0x0/0x30
[ 6.671478] initcall i830_init+0x0/0x30 returned 0 after 0 msecs
[ 6.677527] calling via_init+0x0/0x30
[ 6.681309] initcall via_init+0x0/0x30 returned 0 after 0 msecs
[ 6.687275] calling synclink_cs_init+0x0/0x1b0
[ 6.691816] SyncLink PC Card driver $Revision: 4.34 $
[ 6.705838] SyncLink PC Card driver $Revision: 4.34 $, tty major#252
[ 6.712204] initcall synclink_cs_init+0x0/0x1b0 returned 0 after 11 msecs
[ 6.712401] calling cn_proc_init+0x0/0x50
[ 6.716512] initcall cn_proc_init+0x0/0x50 returned 0 after 0 msecs
[ 6.722823] calling serial8250_init+0x0/0x160
[ 6.727280] Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing enabled
[ 6.736719] serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 6.741871] initcall serial8250_init+0x0/0x160 returned 0 after 13 msecs
[ 6.748619] calling serial8250_pnp_init+0x0/0x20
[ 6.752501] pnp: the driver 'serial' has been registered
[ 6.765010] 00:0b: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[ 6.770663] serial 00:0b: driver attached
[ 6.771090] initcall serial8250_pnp_init+0x0/0x20 returned 0 after 14 msecs
[ 6.778094] calling serial8250_pci_init+0x0/0x20
[ 6.782199] initcall serial8250_pci_init+0x0/0x20 returned 0 after 0 msecs
[ 6.789118] calling init_serial_cs+0x0/0x20
[ 6.791499] initcall init_serial_cs+0x0/0x20 returned 0 after 0 msecs
[ 6.797983] calling parport_default_proc_register+0x0/0x20
[ 6.802306] initcall parport_default_proc_register+0x0/0x20 returned 0 after 0 msecs
[ 6.810095] calling parport_pc_init+0x0/0x1e0
[ 6.818622] pnp: the driver 'parport_pc' has been registered
[ 6.822527] parport_pc 00:08: reported by Plug and Play ACPI
[ 6.828245] parport0: PC-style at 0x378 (0x778), irq 7 [PCSPP,TRISTATE]
[ 6.941259] parport_pc 00:08: driver attached
[ 6.941259] initcall parport_pc_init+0x0/0x1e0 returned 0 after 117 msecs
[ 6.948093] calling parport_serial_init+0x0/0x20
[ 6.952807] initcall parport_serial_init+0x0/0x20 returned 0 after 0 msecs
[ 6.959726] calling init_parport_cs+0x0/0x20
[ 6.962101] initcall init_parport_cs+0x0/0x20 returned 0 after 0 msecs
[ 6.968682] calling parport_ax88796_init+0x0/0x20
[ 6.973069] initcall parport_ax88796_init+0x0/0x20 returned 0 after 0 msecs
[ 6.980078] calling topology_sysfs_init+0x0/0x60
[ 6.984925] initcall topology_sysfs_init+0x0/0x60 returned 0 after 0 msecs
[ 6.991841] calling brd_init+0x0/0x1f0
[ 7.002795] brd: module loaded
[ 7.002795] initcall brd_init+0x0/0x1f0 returned 0 after 3 msecs
[ 7.008846] calling loop_init+0x0/0x200
[ 7.012909] loop: module loaded
[ 7.012909] initcall loop_init+0x0/0x200 returned 0 after 3 msecs
[ 7.019049] calling cpqarray_init+0x0/0x2a0
[ 7.023332] Compaq SMART2 Driver (v 2.6.0)
[ 7.034993] initcall cpqarray_init+0x0/0x2a0 returned 0 after 3 msecs
[ 7.041483] calling DAC960_init_module+0x0/0x60
[ 7.041563] initcall DAC960_init_module+0x0/0x60 returned 0 after 0 msecs
[ 7.048396] calling pkt_init+0x0/0x1d0
[ 7.053093] initcall pkt_init+0x0/0x1d0 returned 0 after 0 msecs
[ 7.059145] calling mm_init+0x0/0x1b0
[ 7.063060] MM: desc_per_page = 128
[ 7.063060] initcall mm_init+0x0/0x1b0 returned 0 after 3 msecs
[ 7.069025] calling init_cryptoloop+0x0/0x40
[ 7.073394] initcall init_cryptoloop+0x0/0x40 returned 0 after 0 msecs
[ 7.079966] calling ibmasm_init+0x0/0x90
[ 7.084959] ibmasm: IBM ASM Service Processor Driver version 1.0 loaded
[ 7.091295] initcall ibmasm_init+0x0/0x90 returned 0 after 6 msecs
[ 7.097519] calling acer_wmi_init+0x0/0x550
[ 7.101802] acer-wmi: Acer Laptop ACPI-WMI Extras version 0.1
[ 7.107564] acer-wmi: No or unsupported WMI interface, unable to load
[ 7.114015] initcall acer_wmi_init+0x0/0x550 returned -19 after 11 msecs
[ 7.120761] calling asus_laptop_init+0x0/0x310
[ 7.128155] initcall asus_laptop_init+0x0/0x310 returned -19 after 0 msecs
[ 7.131976] calling phantom_init+0x0/0x120
[ 7.143162] Phantom Linux Driver, version n0.9.8, init OK
[ 7.143162] initcall phantom_init+0x0/0x120 returned 0 after 5 msecs
[ 7.149561] calling thinkpad_acpi_module_init+0x0/0xa20
[ 7.152648] initcall thinkpad_acpi_module_init+0x0/0xa20 returned -19 after 0 msecs
[ 7.160350] calling enclosure_init+0x0/0x20
[ 7.167708] initcall enclosure_init+0x0/0x20 returned 0 after 0 msecs
[ 7.172362] calling e1000_init_module+0x0/0xa0
[ 7.176906] Intel(R) PRO/1000 Network Driver - version 7.3.20-k2
[ 7.182922] Copyright (c) 1999-2006 Intel Corporation.
[ 7.188110] ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 17 (level, low) -> IRQ 17
[ 7.195561] PCI: Setting latency timer of device 0000:04:00.0 to 64
[ 7.301460] e1000: 0000:04:00.0: e1000_probe: (PCI Express:2.5Gb/s:Width x1) 00:16:76:ab:6e:84
[ 7.310115] e1000: 0000:04:00.0: e1000_probe: This device (id 8086:109a) will no longer be supported by this driver in the future.
[ 7.321855] e1000: 0000:04:00.0: e1000_probe: please use the "e1000e" driver instead.
[ 7.431662] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
[ 7.439010] initcall e1000_init_module+0x0/0xa0 returned 0 after 246 msecs
[ 7.441903] calling igb_init_module+0x0/0x60
[ 7.446275] Intel(R) Gigabit Ethernet Network Driver - version 1.0.8-k2
[ 7.452896] Copyright (c) 2007 Intel Corporation.
[ 7.463396] initcall igb_init_module+0x0/0x60 returned 0 after 10 msecs
[ 7.464559] calling ipg_init_module+0x0/0x20
[ 7.475625] initcall ipg_init_module+0x0/0x20 returned 0 after 0 msecs
[ 7.482197] calling plip_init+0x0/0x80
[ 7.482231] NET3 PLIP version 2.4-parport gniibe@mri.co.jp
[ 7.487730] plip0: Parallel port at 0x378, using IRQ 7.
[ 7.491728] initcall plip_init+0x0/0x80 returned 0 after 10 msecs
[ 7.497866] calling gem_init+0x0/0x20
[ 7.503105] initcall gem_init+0x0/0x20 returned 0 after 0 msecs
[ 7.509071] calling cas_init+0x0/0x50
[ 7.512865] initcall cas_init+0x0/0x50 returned 0 after 0 msecs
[ 7.518830] calling typhoon_init+0x0/0x20
[ 7.522627] initcall typhoon_init+0x0/0x20 returned 0 after 0 msecs
[ 7.528941] calling pcnet32_init_module+0x0/0x150
[ 7.533085] pcnet32.c:v1.35 21.Apr.2008 tsbogend@alpha.franken.de
[ 7.543840] initcall pcnet32_init_module+0x0/0x150 returned 0 after 5 msecs
[ 7.543994] calling e100_init_module+0x0/0x70
[ 7.548452] e100: Intel(R) PRO/100 Network Driver, 3.5.23-k4-NAPI
[ 7.554554] e100: Copyright(c) 1999-2006 Intel Corporation
[ 7.567091] initcall e100_init_module+0x0/0x70 returned 0 after 11 msecs
[ 7.573835] calling sis190_init_module+0x0/0x20
[ 7.573869] initcall sis190_init_module+0x0/0x20 returned 0 after 0 msecs
[ 7.580701] calling ns83820_init+0x0/0x30
[ 7.582987] ns83820.c: National Semiconductor DP83820 10/100/1000 driver.
[ 7.596275] initcall ns83820_init+0x0/0x30 returned 0 after 6 msecs
[ 7.602589] calling tg3_init+0x0/0x20
[ 7.602621] initcall tg3_init+0x0/0x20 returned 0 after 0 msecs
[ 7.608586] calling bnx2_init+0x0/0x20
[ 7.612384] initcall bnx2_init+0x0/0x20 returned 0 after 0 msecs
[ 7.618436] calling skge_init_module+0x0/0x60
[ 7.622320] initcall skge_init_module+0x0/0x60 returned 0 after 0 msecs
[ 7.628981] calling skfd_init+0x0/0x20
[ 7.633474] initcall skfd_init+0x0/0x20 returned 0 after 0 msecs
[ 7.639526] calling rhine_init+0x0/0x80
[ 7.643409] initcall rhine_init+0x0/0x80 returned 0 after 0 msecs
[ 7.649547] calling velocity_init_module+0x0/0x50
[ 7.653518] initcall velocity_init_module+0x0/0x50 returned 0 after 0 msecs
[ 7.660525] calling cicada_init+0x0/0x60
[ 7.662985] initcall cicada_init+0x0/0x60 returned 0 after 0 msecs
[ 7.669211] calling lxt_init+0x0/0x60
[ 7.673281] initcall lxt_init+0x0/0x60 returned 0 after 0 msecs
[ 7.679247] calling qs6612_init+0x0/0x20
[ 7.683055] initcall qs6612_init+0x0/0x20 returned 0 after 0 msecs
[ 7.689283] calling vsc8244_init+0x0/0x20
[ 7.693330] initcall vsc8244_init+0x0/0x20 returned 0 after 0 msecs
[ 7.699643] calling ip175c_init+0x0/0x20
[ 7.703778] initcall ip175c_init+0x0/0x20 returned 0 after 0 msecs
[ 7.710005] calling fixed_mdio_bus_init+0x0/0xc0
[ 7.714051] Fixed MDIO Bus: probed
[ 7.717470] initcall fixed_mdio_bus_init+0x0/0xc0 returned 0 after 3 msecs
[ 7.724387] calling sundance_init+0x0/0x20
[ 7.735715] initcall sundance_init+0x0/0x20 returned 0 after 0 msecs
[ 7.742115] calling hamachi_init+0x0/0x20
[ 7.742148] initcall hamachi_init+0x0/0x20 returned 0 after 0 msecs
[ 7.748461] calling net_olddevs_init+0x0/0xb0
[ 7.752605] initcall net_olddevs_init+0x0/0xb0 returned 0 after 0 msecs
[ 7.759263] calling hp100_module_init+0x0/0x20
[ 7.763723] initcall hp100_module_init+0x0/0x20 returned 0 after 0 msecs
[ 7.770470] calling init_nic+0x0/0x20
[ 7.772669] initcall init_nic+0x0/0x20 returned 0 after 0 msecs
[ 7.778636] calling slip_init+0x0/0xc0
[ 7.782433] SLIP: version 0.8.4-NET3.019-NEWTTY (dynamic channels, max=256).
[ 7.789489] CSLIP: code copyright 1989 Regents of the University of California.
[ 7.796811] initcall slip_init+0x0/0xc0 returned 0 after 13 msecs
[ 7.802948] calling dfx_init+0x0/0x20
[ 7.809271] initcall dfx_init+0x0/0x20 returned 0 after 0 msecs
[ 7.812947] calling cp_init+0x0/0x20
[ 7.818533] initcall cp_init+0x0/0x20 returned 0 after 0 msecs
[ 7.822622] calling rtl8139_init_module+0x0/0x20
[ 7.830504] initcall rtl8139_init_module+0x0/0x20 returned 0 after 0 msecs
[ 7.833250] calling tun_init+0x0/0xb0
[ 7.837013] tun: Universal TUN/TAP device driver, 1.6
[ 7.842074] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[ 7.855220] initcall tun_init+0x0/0xb0 returned 0 after 10 msecs
[ 7.855220] calling veth_init+0x0/0x30
[ 7.859073] initcall veth_init+0x0/0x30 returned 0 after 0 msecs
[ 7.865121] calling rio_init+0x0/0x20
[ 7.874967] initcall rio_init+0x0/0x20 returned 0 after 0 msecs
[ 7.874967] calling rtl8169_init_module+0x0/0x20
[ 7.885683] initcall rtl8169_init_module+0x0/0x20 returned 0 after 0 msecs
[ 7.892602] calling amd8111e_init+0x0/0x20
[ 7.892635] initcall amd8111e_init+0x0/0x20 returned 0 after 0 msecs
[ 7.899033] calling mlx4_init+0x0/0x40
[ 7.903263] initcall mlx4_init+0x0/0x40 returned 0 after 0 msecs
[ 7.909314] calling init_tc589+0x0/0x20
[ 7.913222] initcall init_tc589+0x0/0x20 returned 0 after 0 msecs
[ 7.919360] calling init_tc574+0x0/0x20
[ 7.923326] initcall init_tc574+0x0/0x20 returned 0 after 0 msecs
[ 7.929464] calling init_fmvj18x_cs+0x0/0x20
[ 7.933426] initcall init_fmvj18x_cs+0x0/0x20 returned 0 after 0 msecs
[ 7.940000] calling init_pcnet_cs+0x0/0x20
[ 7.942018] initcall init_pcnet_cs+0x0/0x20 returned 0 after 0 msecs
[ 7.948416] calling init_xirc2ps_cs+0x0/0x20
[ 7.952665] initcall init_xirc2ps_cs+0x0/0x20 returned 0 after 0 msecs
[ 7.959237] calling init_axnet_cs+0x0/0x20
[ 7.963634] initcall init_axnet_cs+0x0/0x20 returned 0 after 0 msecs
[ 7.970031] calling kaweth_init+0x0/0x40
[ 7.974051] drivers/net/usb/kaweth.c: Driver loading
[ 7.985631] usbcore: registered new interface driver kaweth
[ 7.985631] initcall kaweth_init+0x0/0x40 returned 0 after 10 msecs
[ 7.991943] calling pegasus_init+0x0/0x1e0
[ 7.996141] pegasus: v0.6.14 (2006/09/27), Pegasus/Pegasus II USB Ethernet driver
[ 8.001750] usbcore: registered new interface driver pegasus
[ 8.007422] initcall pegasus_init+0x0/0x1e0 returned 0 after 12 msecs
[ 8.013903] calling usb_rtl8150_init+0x0/0x30
[ 8.018360] rtl8150: rtl8150 based usb-ethernet driver v0.6.2 (2004/08/27)
[ 8.028122] usbcore: registered new interface driver rtl8150
[ 8.032783] initcall usb_rtl8150_init+0x0/0x30 returned 0 after 11 msecs
[ 8.039528] calling asix_init+0x0/0x20
[ 8.042759] usbcore: registered new interface driver asix
[ 8.048171] initcall asix_init+0x0/0x20 returned 0 after 5 msecs
[ 8.054220] calling cdc_init+0x0/0x20
[ 8.063289] usbcore: registered new interface driver cdc_ether
[ 8.064324] initcall cdc_init+0x0/0x20 returned 0 after 5 msecs
[ 8.070290] calling net1080_init+0x0/0x20
[ 8.073782] usbcore: registered new interface driver net1080
[ 8.079453] initcall net1080_init+0x0/0x20 returned 0 after 5 msecs
[ 8.085966] calling plusb_init+0x0/0x20
[ 8.091119] usbcore: registered new interface driver plusb
[ 8.096619] initcall plusb_init+0x0/0x20 returned 0 after 5 msecs
[ 8.101740] calling cdc_subset_init+0x0/0x20
[ 8.107275] usbcore: registered new interface driver cdc_subset
[ 8.112253] initcall cdc_subset_init+0x0/0x20 returned 0 after 5 msecs
[ 8.118825] calling zaurus_init+0x0/0x20
[ 8.122401] usbcore: registered new interface driver zaurus
[ 8.127987] initcall zaurus_init+0x0/0x20 returned 0 after 5 msecs
[ 8.134210] calling mcs7830_init+0x0/0x20
[ 8.144480] usbcore: registered new interface driver MOSCHIP usb-ethernet driver
[ 8.151890] initcall mcs7830_init+0x0/0x20 returned 0 after 7 msecs
[ 8.152185] calling usbnet_init+0x0/0x40
[ 8.156213] initcall usbnet_init+0x0/0x40 returned 0 after 0 msecs
[ 8.162436] calling init_orinoco+0x0/0x30
[ 8.166546] orinoco 0.15 (David Gibson <hermes@gibson.dropbear.id.au>, Pavel Roskin <proski@gnu.org>, et al)
[ 8.176380] initcall init_orinoco+0x0/0x30 returned 0 after 9 msecs
[ 8.182690] calling init_hermes+0x0/0x10
[ 8.186715] initcall init_hermes+0x0/0x10 returned 0 after 0 msecs
[ 8.192937] calling init_orinoco_cs+0x0/0x30
[ 8.197308] orinoco_cs 0.15 (David Gibson <hermes@gibson.dropbear.id.au>, Pavel Roskin <proski@gnu.org>, et al)
[ 8.211528] initcall init_orinoco_cs+0x0/0x30 returned 0 after 9 msecs
[ 8.213729] calling orinoco_pci_init+0x0/0x40
[ 8.218187] orinoco_pci 0.15 (Pavel Roskin <proski@gnu.org>, David Gibson <hermes@gibson.dropbear.id.au> & Jean Tourrilhes <jt@hpl.hp.com>)
[ 8.237316] initcall orinoco_pci_init+0x0/0x40 returned 0 after 11 msecs
[ 8.244060] calling orinoco_tmd_init+0x0/0x40
[ 8.244093] orinoco_tmd 0.15 (Joerg Dorchain <joerg@dorchain.net>)
[ 8.254742] initcall orinoco_tmd_init+0x0/0x40 returned 0 after 5 msecs
[ 8.254742] calling orinoco_nortel_init+0x0/0x40
[ 8.259460] orinoco_nortel 0.15 (Tobias Hoffmann & Christoph Jungegger <disdos@traum404.de>)
[ 8.273778] initcall orinoco_nortel_init+0x0/0x40 returned 0 after 8 msecs
[ 8.274601] calling airo_init_module+0x0/0x100
[ 8.279147] airo(): Probing for PCI adapters
[ 8.286098] airo(): Finished probing for PCI adapters
[ 8.291165] initcall airo_init_module+0x0/0x100 returned 0 after 8 msecs
[ 8.297911] calling atmel_cs_init+0x0/0x20
[ 8.302230] initcall atmel_cs_init+0x0/0x20 returned 0 after 0 msecs
[ 8.308628] calling prism54_module_init+0x0/0x40
[ 8.312857] Loaded prism54 driver, version 1.2
[ 8.318052] initcall prism54_module_init+0x0/0x40 returned 0 after 4 msecs
[ 8.322034] calling init_ray_cs+0x0/0xa0
[ 8.328737] initcall init_ray_cs+0x0/0xa0 returned 0 after 0 msecs
[ 8.333007] calling wl3501_init_module+0x0/0x20
[ 8.342082] initcall wl3501_init_module+0x0/0x20 returned 0 after 0 msecs
[ 8.343892] calling xircom_init+0x0/0x20
[ 8.355141] initcall xircom_init+0x0/0x20 returned 0 after 0 msecs
[ 8.355141] calling de_init+0x0/0x20
[ 8.365077] initcall de_init+0x0/0x20 returned 0 after 0 msecs
[ 8.365077] calling de4x5_module_init+0x0/0x20
[ 8.375531] initcall de4x5_module_init+0x0/0x20 returned 0 after 0 msecs
[ 8.382276] calling uli526x_init_module+0x0/0xb0
[ 8.382309] uli526x: ULi M5261/M5263 net driver, version 0.9.3 (2005-7-29)
[ 8.393690] initcall uli526x_init_module+0x0/0xb0 returned 0 after 6 msecs
[ 8.393913] calling usb_irda_init+0x0/0x50
[ 8.404892] usbcore: registered new interface driver irda-usb
[ 8.405062] USB IrDA support registered
[ 8.408912] initcall usb_irda_init+0x0/0x50 returned 0 after 9 msecs
[ 8.414694] calling stir_init+0x0/0x20
[ 8.424944] usbcore: registered new interface driver stir4200
[ 8.424944] initcall stir_init+0x0/0x20 returned 0 after 5 msecs
[ 8.430996] calling smsc_ircc_init+0x0/0x670
[ 8.434398] initcall smsc_ircc_init+0x0/0x670 returned -19 after 0 msecs
[ 8.441143] calling vlsi_mod_init+0x0/0x180
[ 8.446785] initcall vlsi_mod_init+0x0/0x180 returned 0 after 0 msecs
[ 8.452316] calling mcs_init+0x0/0x60
[ 8.457938] usbcore: registered new interface driver mcs7780
[ 8.462599] initcall mcs_init+0x0/0x60 returned 0 after 5 msecs
[ 8.468565] calling irtty_sir_init+0x0/0x60
[ 8.471880] initcall irtty_sir_init+0x0/0x60 returned 0 after 0 msecs
[ 8.478363] calling sir_wq_init+0x0/0x40
[ 8.482650] initcall sir_wq_init+0x0/0x40 returned 0 after 0 msecs
[ 8.488876] calling esi_sir_init+0x0/0x20
[ 8.492914] initcall esi_sir_init+0x0/0x20 returned 0 after 0 msecs
[ 8.499226] calling litelink_sir_init+0x0/0x20
[ 8.503337] initcall litelink_sir_init+0x0/0x20 returned 0 after 0 msecs
[ 8.510082] calling girbil_sir_init+0x0/0x20
[ 8.514453] initcall girbil_sir_init+0x0/0x20 returned 0 after 0 msecs
[ 8.521024] calling old_belkin_sir_init+0x0/0x20
[ 8.525740] initcall old_belkin_sir_init+0x0/0x20 returned 0 after 0 msecs
[ 8.532656] calling mcp2120_sir_init+0x0/0x20
[ 8.537113] initcall mcp2120_sir_init+0x0/0x20 returned 0 after 0 msecs
[ 8.543770] calling act200l_sir_init+0x0/0x20
[ 8.548227] initcall act200l_sir_init+0x0/0x20 returned 0 after 0 msecs
[ 8.555162] calling toim3232_sir_init+0x0/0x40
[ 8.559703] initcall toim3232_sir_init+0x0/0x40 returned 0 after 0 msecs
[ 8.566447] calling init_netconsole+0x0/0x280
[ 8.570908] console [netcon0] enabled
[ 8.573109] netconsole: network logging started
[ 8.577653] initcall init_netconsole+0x0/0x280 returned 0 after 7 msecs
[ 8.585795] calling dabusb_init+0x0/0xd0
[ 8.591386] usbcore: registered new interface driver dabusb
[ 8.596972] dabusb: v1.54:DAB-USB Interface Driver for Linux (c)1999
[ 8.602087] initcall dabusb_init+0x0/0xd0 returned 0 after 11 msecs
[ 8.608397] calling raid_init+0x0/0x20
[ 8.612298] initcall raid_init+0x0/0x20 returned 0 after 0 msecs
[ 8.618351] calling spi_transport_init+0x0/0x40
[ 8.622311] initcall spi_transport_init+0x0/0x40 returned 0 after 0 msecs
[ 8.629142] calling fc_transport_init+0x0/0x60
[ 8.633817] initcall fc_transport_init+0x0/0x60 returned 0 after 0 msecs
[ 8.640562] calling iscsi_transport_init+0x0/0x130
[ 8.645454] Loading iSCSI transport class v2.0-869.
[ 8.652461] initcall iscsi_transport_init+0x0/0x130 returned 0 after 4 msecs
[ 8.659553] calling sas_transport_init+0x0/0xd0
[ 8.662175] initcall sas_transport_init+0x0/0xd0 returned 0 after 0 msecs
[ 8.669007] calling srp_transport_init+0x0/0x60
[ 8.673761] initcall srp_transport_init+0x0/0x60 returned 0 after 0 msecs
[ 8.680593] calling ahc_linux_init+0x0/0x70
[ 8.685802] initcall ahc_linux_init+0x0/0x70 returned 0 after 0 msecs
[ 8.691929] calling ahd_linux_init+0x0/0xa0
[ 8.698364] initcall ahd_linux_init+0x0/0xa0 returned 0 after 0 msecs
[ 8.702737] calling aac_init+0x0/0x80
[ 8.706501] Adaptec aacraid driver 1.1-5[2456]-ms
[ 8.713025] initcall aac_init+0x0/0x80 returned 0 after 4 msecs
[ 8.718990] calling ips_module_init+0x0/0x230
[ 8.723743] initcall ips_module_init+0x0/0x230 returned -19 after 0 msecs
[ 8.730576] calling qla1280_init+0x0/0x20
[ 8.732538] initcall qla1280_init+0x0/0x20 returned 0 after 0 msecs
[ 8.738851] calling qla2x00_module_init+0x0/0x1b0
[ 8.743167] QLogic Fibre Channel HBA Driver
[ 8.753204] initcall qla2x00_module_init+0x0/0x1b0 returned 0 after 4 msecs
[ 8.754583] calling dc395x_module_init+0x0/0x20
[ 8.766255] initcall dc395x_module_init+0x0/0x20 returned 0 after 0 msecs
[ 8.773087] calling dc390_module_init+0x0/0xb0
[ 8.773120] DC390: clustering now enabled by default. If you get problems load
[ 8.773120] with "disable_clustering=1" and report to maintainers
[ 8.791092] initcall dc390_module_init+0x0/0xb0 returned 0 after 12 msecs
[ 8.795492] calling megaraid_init+0x0/0xd0
[ 8.800676] initcall megaraid_init+0x0/0xd0 returned 0 after 0 msecs
[ 8.802156] calling megasas_init+0x0/0x180
[ 8.806354] megasas: 00.00.03.20-rc1 Mon. March 10 11:02:31 PDT 2008
[ 8.812788] initcall megasas_init+0x0/0x180 returned 0 after 6 msecs
[ 8.819186] calling atp870u_init+0x0/0x20
[ 8.823210] initcall atp870u_init+0x0/0x20 returned 0 after 0 msecs
[ 8.829522] calling inia100_init+0x0/0x20
[ 8.833665] initcall inia100_init+0x0/0x20 returned 0 after 0 msecs
[ 8.839975] calling ipr_init+0x0/0x40
[ 8.843736] ipr: IBM Power RAID SCSI Device Driver version: 2.4.1 (April 24, 2007)
[ 8.857841] initcall ipr_init+0x0/0x40 returned 0 after 7 msecs
[ 8.863807] calling stex_init+0x0/0x40
[ 8.863841] stex: Promise SuperTrak EX Driver version: 3.6.0000.1
[ 8.873796] initcall stex_init+0x0/0x40 returned 0 after 5 msecs
[ 8.873796] calling init_sd+0x0/0xd0
[ 8.881255] Driver 'sd' needs updating - please use bus_type methods
[ 8.883559] initcall init_sd+0x0/0xd0 returned 0 after 6 msecs
[ 8.889439] calling init_sr+0x0/0x60
[ 8.893114] Driver 'sr' needs updating - please use bus_type methods
[ 8.905893] initcall init_sr+0x0/0x60 returned 0 after 6 msecs
[ 8.911772] calling init_sg+0x0/0xe0
[ 8.911795] initcall init_sg+0x0/0xe0 returned 0 after 0 msecs
[ 8.911795] calling ahci_init+0x0/0x20
[ 8.922772] initcall ahci_init+0x0/0x20 returned 0 after 0 msecs
[ 8.925225] calling piix_init+0x0/0x30
[ 8.929089] ata_piix 0000:00:1f.1: version 2.12
[ 8.933642] IOAPIC[0]: Set routing entry (2-18 -> 0x59 -> IRQ 18 Mode:1 Active:1)
[ 8.941139] ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 18
[ 8.949986] PCI: Setting latency timer of device 0000:00:1f.1 to 64
[ 8.951947] scsi0 : ata_piix
[ 8.963871] scsi1 : ata_piix
[ 8.963921] ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x40b0 irq 14
[ 8.970892] ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x40b8 irq 15
[ 9.136770] ata2: port disabled. ignoring.
[ 9.141989] IOAPIC[0]: Set routing entry (2-19 -> 0x61 -> IRQ 19 Mode:1 Active:1)
[ 9.149487] ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 19
[ 9.156931] ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
[ 9.162263] PCI: Setting latency timer of device 0000:00:1f.2 to 64
[ 9.172701] scsi2 : ata_piix
[ 9.174067] scsi3 : ata_piix
[ 9.174067] ata3: SATA max UDMA/133 cmd 0x40c8 ctl 0x40e4 bmdma 0x40a0 irq 19
[ 9.181211] ata4: SATA max UDMA/133 cmd 0x40c0 ctl 0x40e0 bmdma 0x40a8 irq 19
[ 9.366945] ata3.00: ATA-7: HDT722516DLA380, V43OA96A, max UDMA/133
[ 9.372355] ata3.00: 321672960 sectors, multi 16: LBA48 NCQ (depth 0/32)
[ 9.416158] ata3.00: configured for UDMA/133
[ 9.616546] ata4.00: ATA-7: HDT722516DLA380, V43OA96A, max UDMA/133
[ 9.622832] ata4.00: 321672960 sectors, multi 16: LBA48 NCQ (depth 0/32)
[ 9.629548] ata4.01: ATAPI: PLEXTOR DVDR PX-755A, 1.03, max UDMA/66
[ 9.666619] ata4.00: configured for UDMA/133
[ 9.686511] ata4.01: configured for UDMA/66
[ 9.693010] isa bounce pool size: 16 pages
[ 9.693010] scsi 2:0:0:0: Direct-Access ATA HDT722516DLA380 V43O PQ: 0 ANSI: 5
[ 9.709702] sd 2:0:0:0: [sda] 321672960 512-byte hardware sectors (164697 MB)
[ 9.716857] sd 2:0:0:0: [sda] Write Protect is off
[ 9.716941] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 9.722025] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 9.731126] sd 2:0:0:0: [sda] 321672960 512-byte hardware sectors (164697 MB)
[ 9.738280] sd 2:0:0:0: [sda] Write Protect is off
[ 9.743083] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[ 9.743092] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 9.752148] sda: sda1 sda2
[ 9.773042] sd 2:0:0:0: [sda] Attached SCSI disk
[ 9.785767] sd 2:0:0:0: Attached scsi generic sg0 type 0
[ 9.785767] scsi 3:0:0:0: Direct-Access ATA HDT722516DLA380 V43O PQ: 0 ANSI: 5
[ 9.799308] sd 3:0:0:0: [sdb] 321672960 512-byte hardware sectors (164697 MB)
[ 9.806462] sd 3:0:0:0: [sdb] Write Protect is off
[ 9.806547] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 9.811630] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 9.820715] sd 3:0:0:0: [sdb] 321672960 512-byte hardware sectors (164697 MB)
[ 9.825500] sd 3:0:0:0: [sdb] Write Protect is off
[ 9.830305] sd 3:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[ 9.835385] sd 3:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 9.844439] sdb: sdb1
[ 9.863083] sd 3:0:0:0: [sdb] Attached SCSI disk
[ 9.873464] sd 3:0:0:0: Attached scsi generic sg1 type 0
[ 9.874522] scsi 3:0:1:0: CD-ROM PLEXTOR DVDR PX-755A 1.03 PQ: 0 ANSI: 5
[ 9.888613] sr0: scsi3-mmc drive: 40x/40x writer cd/rw xa/form2 cdda tray
[ 9.895422] Uniform CD-ROM driver Revision: 3.20
[ 9.897369] sr 3:0:1:0: Attached scsi CD-ROM sr0
[ 9.906723] sr 3:0:1:0: Attached scsi generic sg2 type 5
[ 9.906723] initcall piix_init+0x0/0x30 returned 0 after 931 msecs
[ 9.912949] calling pdc_ata_init+0x0/0x20
[ 9.915815] initcall pdc_ata_init+0x0/0x20 returned 0 after 0 msecs
[ 9.922129] calling sil24_init+0x0/0x20
[ 9.926451] initcall sil24_init+0x0/0x20 returned 0 after 0 msecs
[ 9.932961] calling sis_init+0x0/0x20
[ 9.936565] initcall sis_init+0x0/0x20 returned 0 after 0 msecs
[ 9.942531] calling nv_init+0x0/0x20
[ 9.946701] initcall nv_init+0x0/0x20 returned 0 after 0 msecs
[ 9.952581] calling inic_init+0x0/0x20
[ 9.956295] initcall inic_init+0x0/0x20 returned 0 after 0 msecs
[ 9.962348] calling adma_ata_init+0x0/0x20
[ 9.966234] initcall adma_ata_init+0x0/0x20 returned 0 after 0 msecs
[ 9.972633] calling amd_init+0x0/0x20
[ 9.976867] initcall amd_init+0x0/0x20 returned 0 after 0 msecs
[ 9.982833] calling atiixp_init+0x0/0x20
[ 9.986632] initcall atiixp_init+0x0/0x20 returned 0 after 0 msecs
[ 9.992857] calling cmd640_init+0x0/0x20
[ 9.997235] initcall cmd640_init+0x0/0x20 returned 0 after 0 msecs
[ 10.003322] calling cmd64x_init+0x0/0x20
[ 10.008230] initcall cmd64x_init+0x0/0x20 returned 0 after 0 msecs
[ 10.013605] calling cs5520_init+0x0/0x20
[ 10.019085] initcall cs5520_init+0x0/0x20 returned 0 after 0 msecs
[ 10.023891] calling cs5530_init+0x0/0x20
[ 10.029936] initcall cs5530_init+0x0/0x20 returned 0 after 0 msecs
[ 10.034174] calling it821x_init+0x0/0x20
[ 10.040793] initcall it821x_init+0x0/0x20 returned 0 after 0 msecs
[ 10.044461] calling it8213_init+0x0/0x20
[ 10.051652] initcall it8213_init+0x0/0x20 returned 0 after 0 msecs
[ 10.054747] calling netcell_init+0x0/0x20
[ 10.062770] initcall netcell_init+0x0/0x20 returned 0 after 0 msecs
[ 10.065118] calling ns87410_init+0x0/0x20
[ 10.074060] initcall ns87410_init+0x0/0x20 returned 0 after 0 msecs
[ 10.075579] calling opti_init+0x0/0x20
[ 10.084658] initcall opti_init+0x0/0x20 returned 0 after 0 msecs
[ 10.085778] calling oldpiix_init+0x0/0x20
[ 10.095510] initcall oldpiix_init+0x0/0x20 returned 0 after 0 msecs
[ 10.095975] calling pcmcia_init+0x0/0x20
[ 10.106348] initcall pcmcia_init+0x0/0x20 returned 0 after 0 msecs
[ 10.106348] calling radisys_init+0x0/0x20
[ 10.116717] initcall radisys_init+0x0/0x20 returned 0 after 0 msecs
[ 10.116717] calling serverworks_init+0x0/0x20
[ 10.127526] initcall serverworks_init+0x0/0x20 returned 0 after 0 msecs
[ 10.134185] calling sl82c105_init+0x0/0x20
[ 10.134221] initcall sl82c105_init+0x0/0x20 returned 0 after 0 msecs
[ 10.140621] calling sis_init+0x0/0x20
[ 10.144854] initcall sis_init+0x0/0x20 returned 0 after 0 msecs
[ 10.150820] calling triflex_init+0x0/0x20
[ 10.154621] initcall triflex_init+0x0/0x20 returned 0 after 0 msecs
[ 10.160934] calling i2o_iop_init+0x0/0x60
[ 10.165042] I2O subsystem v1.325
[ 10.165081] i2o: max drivers = 8
[ 10.175679] initcall i2o_iop_init+0x0/0x60 returned 0 after 6 msecs
[ 10.175679] calling i2o_bus_init+0x0/0x50
[ 10.179789] I2O Bus Adapter OSM v1.317
[ 10.186226] initcall i2o_bus_init+0x0/0x50 returned 0 after 3 msecs
[ 10.192540] calling i2o_scsi_init+0x0/0x50
[ 10.193950] I2O SCSI Peripheral OSM v1.316
[ 10.201008] initcall i2o_scsi_init+0x0/0x50 returned 0 after 3 msecs
[ 10.204640] calling i2o_proc_init+0x0/0x1b0
[ 10.208925] I2O ProcFS OSM v1.316
[ 10.215353] initcall i2o_proc_init+0x0/0x1b0 returned 0 after 3 msecs
[ 10.221839] calling fusion_init+0x0/0x130
[ 10.225196] Fusion MPT base driver 3.04.06
[ 10.229304] Copyright (c) 1999-2007 LSI Corporation
[ 10.233417] initcall fusion_init+0x0/0x130 returned 0 after 8 msecs
[ 10.239728] calling mptspi_init+0x0/0x100
[ 10.243835] Fusion MPT SPI Host driver 3.04.06
[ 10.248351] initcall mptspi_init+0x0/0x100 returned 0 after 4 msecs
[ 10.253192] calling mptfc_init+0x0/0x110
[ 10.257216] Fusion MPT FC Host driver 3.04.06
[ 10.262388] initcall mptfc_init+0x0/0x110 returned 0 after 4 msecs
[ 10.268616] calling mptsas_init+0x0/0x110
[ 10.272725] Fusion MPT SAS Host driver 3.04.06
[ 10.274200] initcall mptsas_init+0x0/0x110 returned 0 after 4 msecs
[ 10.280513] calling cdrom_init+0x0/0x20
[ 10.284449] initcall cdrom_init+0x0/0x20 returned 0 after 0 msecs
[ 10.290587] calling at25_init+0x0/0x20
[ 10.296816] initcall at25_init+0x0/0x20 returned 0 after 0 msecs
[ 10.302869] calling spidev_init+0x0/0x90
[ 10.306780] initcall spidev_init+0x0/0x90 returned 0 after 0 msecs
[ 10.313004] calling tle62x0_init+0x0/0x20
[ 10.317851] initcall tle62x0_init+0x0/0x20 returned 0 after 0 msecs
[ 10.323535] calling nonstatic_sysfs_init+0x0/0x20
[ 10.328341] initcall nonstatic_sysfs_init+0x0/0x20 returned 0 after 0 msecs
[ 10.334678] calling yenta_socket_init+0x0/0x20
[ 10.345355] initcall yenta_socket_init+0x0/0x20 returned 0 after 0 msecs
[ 10.346228] calling pd6729_module_init+0x0/0x20
[ 10.357644] initcall pd6729_module_init+0x0/0x20 returned 0 after 0 msecs
[ 10.364478] calling aoe_init+0x0/0xc0
[ 10.364516] aoe: AoE v47 initialised.
[ 10.364516] initcall aoe_init+0x0/0xc0 returned 0 after 3 msecs
[ 10.370481] calling ehci_hcd_init+0x0/0xa0
[ 10.374677] ehci_hcd: block sizes: qh 160 qtd 96 itd 192 sitd 96
[ 10.380720] IOAPIC[0]: Set routing entry (2-23 -> 0x69 -> IRQ 23 Mode:1 Active:1)
[ 10.388302] ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 23
[ 10.395830] PCI: Setting latency timer of device 0000:00:1d.7 to 64
[ 10.402108] ehci_hcd 0000:00:1d.7: EHCI Host Controller
[ 10.410605] ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
[ 10.414791] ehci_hcd 0000:00:1d.7: reset hcs_params 0x104208 dbg=1 cc=4 pcc=2 ordered !ppc ports=8
[ 10.423759] ehci_hcd 0000:00:1d.7: reset hcc_params 6871 thresh 7 uframes 1024 64 bit addr
[ 10.432046] ehci_hcd 0000:00:1d.7: reset command 080002 (park)=0 ithresh=8 period=1024 Reset HALT
[ 10.444828] ehci_hcd 0000:00:1d.7: debug port 1
[ 10.449375] PCI: cache line size of 32 is not supported by device 0000:00:1d.7
[ 10.456837] ehci_hcd 0000:00:1d.7: irq 23, io mem 0x50404400
[ 10.462508] ehci_hcd 0000:00:1d.7: reset command 080002 (park)=0 ithresh=8 period=1024 Reset HALT
[ 10.475277] ehci_hcd 0000:00:1d.7: init command 010001 (park)=0 ithresh=1 period=1024 RUN
[ 10.500735] ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[ 10.503222] usb usb1: default language 0x0409
[ 10.507616] usb usb1: uevent
[ 10.514341] usb usb1: usb_probe_device
[ 10.515121] usb usb1: configuration #1 chosen from 1 choice
[ 10.520708] usb usb1: adding 1-0:1.0 (config #1, interface 0)
[ 10.527497] usb 1-0:1.0: uevent
[ 10.530734] hub 1-0:1.0: usb_probe_interface
[ 10.533248] hub 1-0:1.0: usb_probe_interface - got id
[ 10.538313] hub 1-0:1.0: USB hub found
[ 10.542080] hub 1-0:1.0: 8 ports detected
[ 10.546104] hub 1-0:1.0: standalone hub
[ 10.549952] hub 1-0:1.0: no power switching (usb 1.0)
[ 10.553582] hub 1-0:1.0: individual port over-current protection
[ 10.559600] hub 1-0:1.0: power on to power good time: 20ms
[ 10.568518] hub 1-0:1.0: local power source is good
[ 10.573406] hub 1-0:1.0: trying to enable port power on non-switchable hub
[ 10.693431] hub 1-0:1.0: state 7 ports 8 chg 0000 evt 0000
[ 10.688039] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[ 10.688042] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 10.688044] usb usb1: Product: EHCI Host Controller
[ 10.688046] usb usb1: Manufacturer: Linux 2.6.26-rc3-sched-devel.git ehci_hcd
[ 10.688048] usb usb1: SerialNumber: 0000:00:1d.7
[ 10.688102] initcall ehci_hcd_init+0x0/0xa0 returned 0 after 296 msecs
[ 10.688106] calling isp116x_init+0x0/0x50
[ 10.688107] 116x: driver isp116x-hcd, 03 Nov 2005
[ 10.688159] initcall isp116x_init+0x0/0x50 returned 0 after 0 msecs
[ 10.688161] calling ohci_hcd_mod_init+0x0/0xb0
[ 10.688163] ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver
[ 10.688164] ohci_hcd: block sizes: ed 80 td 96
[ 10.688199] initcall ohci_hcd_mod_init+0x0/0xb0 returned 0 after 0 msecs
[ 10.688201] calling uhci_hcd_init+0x0/0x100
[ 10.688202] USB Universal Host Controller Interface driver v3.0
[ 10.688221] ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 23
[ 10.688226] PCI: Setting latency timer of device 0000:00:1d.0 to 64
[ 10.688228] uhci_hcd 0000:00:1d.0: UHCI Host Controller
[ 10.688256] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
[ 10.688262] uhci_hcd 0000:00:1d.0: detected 2 ports
[ 10.688265] uhci_hcd 0000:00:1d.0: uhci_check_and_reset_hc: cmd = 0x0000
[ 10.688266] uhci_hcd 0000:00:1d.0: Performing full reset
[ 10.688284] uhci_hcd 0000:00:1d.0: irq 23, io base 0x00004080
[ 10.688314] usb usb2: default language 0x0409
[ 10.688339] usb usb2: uevent
[ 10.688363] usb usb2: usb_probe_device
[ 10.688395] usb usb2: configuration #1 chosen from 1 choice
[ 10.688399] usb usb2: adding 2-0:1.0 (config #1, interface 0)
[ 10.688408] usb 2-0:1.0: uevent
[ 10.688431] hub 2-0:1.0: usb_probe_interface
[ 10.688432] hub 2-0:1.0: usb_probe_interface - got id
[ 10.688433] hub 2-0:1.0: USB hub found
[ 10.688437] hub 2-0:1.0: 2 ports detected
[ 10.688438] hub 2-0:1.0: standalone hub
[ 10.688439] hub 2-0:1.0: no power switching (usb 1.0)
[ 10.688440] hub 2-0:1.0: individual port over-current protection
[ 10.688441] hub 2-0:1.0: power on to power good time: 2ms
[ 10.688444] hub 2-0:1.0: local power source is good
[ 10.688445] hub 2-0:1.0: trying to enable port power on non-switchable hub
[ 10.797961] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
[ 10.797962] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 10.797963] usb usb2: Product: UHCI Host Controller
[ 10.797964] usb usb2: Manufacturer: Linux 2.6.26-rc3-sched-devel.git uhci_hcd
[ 10.797966] usb usb2: SerialNumber: 0000:00:1d.0
[ 10.797985] ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 19
[ 10.797990] PCI: Setting latency timer of device 0000:00:1d.1 to 64
[ 10.797992] uhci_hcd 0000:00:1d.1: UHCI Host Controller
[ 10.798021] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
[ 10.798027] uhci_hcd 0000:00:1d.1: detected 2 ports
[ 10.798031] uhci_hcd 0000:00:1d.1: uhci_check_and_reset_hc: cmd = 0x0000
[ 10.798032] uhci_hcd 0000:00:1d.1: Performing full reset
[ 10.798048] uhci_hcd 0000:00:1d.1: irq 19, io base 0x00004060
[ 10.798075] usb usb3: default language 0x0409
[ 10.798099] usb usb3: uevent
[ 10.798122] usb usb3: usb_probe_device
[ 10.798156] usb usb3: configuration #1 chosen from 1 choice
[ 10.798159] usb usb3: adding 3-0:1.0 (config #1, interface 0)
[ 10.798173] usb 3-0:1.0: uevent
[ 10.798196] hub 3-0:1.0: usb_probe_interface
[ 10.798197] hub 3-0:1.0: usb_probe_interface - got id
[ 10.798198] hub 3-0:1.0: USB hub found
[ 10.798204] hub 3-0:1.0: 2 ports detected
[ 10.798204] hub 3-0:1.0: standalone hub
[ 10.798205] hub 3-0:1.0: no power switching (usb 1.0)
[ 10.798206] hub 3-0:1.0: individual port over-current protection
[ 10.798207] hub 3-0:1.0: power on to power good time: 2ms
[ 10.798210] hub 3-0:1.0: local power source is good
[ 10.798211] hub 3-0:1.0: trying to enable port power on non-switchable hub
[ 10.907928] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001
[ 10.907930] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 10.907931] usb usb3: Product: UHCI Host Controller
[ 10.907932] usb usb3: Manufacturer: Linux 2.6.26-rc3-sched-devel.git uhci_hcd
[ 10.907933] usb usb3: SerialNumber: 0000:00:1d.1
[ 10.907955] ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 18
[ 10.907961] PCI: Setting latency timer of device 0000:00:1d.2 to 64
[ 10.907963] uhci_hcd 0000:00:1d.2: UHCI Host Controller
[ 10.907993] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
[ 10.907999] uhci_hcd 0000:00:1d.2: detected 2 ports
[ 10.908003] uhci_hcd 0000:00:1d.2: uhci_check_and_reset_hc: cmd = 0x0000
[ 10.908004] uhci_hcd 0000:00:1d.2: Performing full reset
[ 10.908026] uhci_hcd 0000:00:1d.2: irq 18, io base 0x00004040
[ 10.908055] usb usb4: default language 0x0409
[ 10.908079] usb usb4: uevent
[ 10.908102] usb usb4: usb_probe_device
[ 10.908136] usb usb4: configuration #1 chosen from 1 choice
[ 10.908139] usb usb4: adding 4-0:1.0 (config #1, interface 0)
[ 10.908149] usb 4-0:1.0: uevent
[ 10.908171] hub 4-0:1.0: usb_probe_interface
[ 10.908172] hub 4-0:1.0: usb_probe_interface - got id
[ 10.908173] hub 4-0:1.0: USB hub found
[ 10.908178] hub 4-0:1.0: 2 ports detected
[ 10.908178] hub 4-0:1.0: standalone hub
[ 10.908179] hub 4-0:1.0: no power switching (usb 1.0)
[ 10.908180] hub 4-0:1.0: individual port over-current protection
[ 10.908181] hub 4-0:1.0: power on to power good time: 2ms
[ 10.908184] hub 4-0:1.0: local power source is good
[ 10.908185] hub 4-0:1.0: trying to enable port power on non-switchable hub
[ 11.017899] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[ 11.017900] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 11.017902] usb usb4: Product: UHCI Host Controller
[ 11.017903] usb usb4: Manufacturer: Linux 2.6.26-rc3-sched-devel.git uhci_hcd
[ 11.017904] usb usb4: SerialNumber: 0000:00:1d.2
[ 11.017921] ACPI: PCI Interrupt 0000:00:1d.3[D] -> GSI 16 (level, low) -> IRQ 16
[ 11.017926] PCI: Setting latency timer of device 0000:00:1d.3 to 64
[ 11.017928] uhci_hcd 0000:00:1d.3: UHCI Host Controller
[ 11.017957] uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5
[ 11.017963] uhci_hcd 0000:00:1d.3: detected 2 ports
[ 11.017966] uhci_hcd 0000:00:1d.3: uhci_check_and_reset_hc: cmd = 0x0000
[ 11.017967] uhci_hcd 0000:00:1d.3: Performing full reset
[ 11.017989] uhci_hcd 0000:00:1d.3: irq 16, io base 0x00004020
[ 11.018020] usb usb5: default language 0x0409
[ 11.018044] usb usb5: uevent
[ 11.018068] usb usb5: usb_probe_device
[ 11.018105] usb usb5: configuration #1 chosen from 1 choice
[ 11.018108] usb usb5: adding 5-0:1.0 (config #1, interface 0)
[ 11.018118] usb 5-0:1.0: uevent
[ 11.018141] hub 5-0:1.0: usb_probe_interface
[ 11.018142] hub 5-0:1.0: usb_probe_interface - got id
[ 11.018143] hub 5-0:1.0: USB hub found
[ 11.018147] hub 5-0:1.0: 2 ports detected
[ 11.018148] hub 5-0:1.0: standalone hub
[ 11.018149] hub 5-0:1.0: no power switching (usb 1.0)
[ 11.018150] hub 5-0:1.0: individual port over-current protection
[ 11.018151] hub 5-0:1.0: power on to power good time: 2ms
[ 11.018154] hub 5-0:1.0: local power source is good
[ 11.018155] hub 5-0:1.0: trying to enable port power on non-switchable hub
[ 11.127870] usb usb5: New USB device found, idVendor=1d6b, idProduct=0001
[ 11.127871] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 11.127873] usb usb5: Product: UHCI Host Controller
[ 11.127874] usb usb5: Manufacturer: Linux 2.6.26-rc3-sched-devel.git uhci_hcd
[ 11.127875] usb usb5: SerialNumber: 0000:00:1d.3
[ 11.127917] initcall uhci_hcd_init+0x0/0x100 returned 0 after 419 msecs
[ 11.127919] calling c67x00_init+0x0/0x20
[ 11.127947] initcall c67x00_init+0x0/0x20 returned 0 after 0 msecs
[ 11.127949] calling acm_init+0x0/0x160
[ 11.127978] usbcore: registered new interface driver cdc_acm
[ 11.127980] cdc_acm: v0.26:USB Abstract Control Model driver for USB modems and ISDN adapters
[ 11.127982] initcall acm_init+0x0/0x160 returned 0 after 0 msecs
[ 11.127984] calling usblp_init+0x0/0x20
[ 11.128010] usbcore: registered new interface driver usblp
[ 11.128013] initcall usblp_init+0x0/0x20 returned 0 after 0 msecs
[ 11.128015] calling wdm_init+0x0/0x20
[ 11.128040] usbcore: registered new interface driver cdc_wdm
[ 11.128043] initcall wdm_init+0x0/0x20 returned 0 after 0 msecs
[ 11.128045] calling usb_stor_init+0x0/0x60
[ 11.128045] Initializing USB Mass Storage driver...
[ 11.128073] usbcore: registered new interface driver usb-storage
[ 11.128074] USB Mass Storage support registered.
[ 11.128076] initcall usb_stor_init+0x0/0x60 returned 0 after 0 msecs
[ 11.128078] calling usb_usual_init+0x0/0x50
[ 11.128108] usbcore: registered new interface driver libusual
[ 11.128111] initcall usb_usual_init+0x0/0x50 returned 0 after 0 msecs
[ 11.128112] calling adu_init+0x0/0xb0
[ 11.128113] drivers/usb/misc/adutux.c : adu_init : enter
[ 11.128139] usbcore: registered new interface driver adutux
[ 11.128141] adutux: adutux adutux (see www.ontrak.net) v0.0.13
[ 11.128142] adutux: adutux is an experimental driver. Use at your own risk
[ 11.128143] drivers/usb/misc/adutux.c : adu_init : leave, return value 0
[ 11.128145] initcall adu_init+0x0/0xb0 returned 0 after 0 msecs
[ 11.128146] calling appledisplay_init+0x0/0x60
[ 11.128191] usbcore: registered new interface driver appledisplay
[ 11.128193] initcall appledisplay_init+0x0/0x60 returned 0 after 0 msecs
[ 11.128195] calling cypress_init+0x0/0x50
[ 11.128221] usbcore: registered new interface driver cypress_cy7c63
[ 11.128224] initcall cypress_init+0x0/0x50 returned 0 after 0 msecs
[ 11.128226] calling ftdi_elan_init+0x0/0x180
[ 11.128227] driver ftdi-elan built at 04:24:37 on May 24 2008
[ 11.128288] usbcore: registered new interface driver ftdi-elan
[ 11.128290] initcall ftdi_elan_init+0x0/0x180 returned 0 after 0 msecs
[ 11.128292] calling usb_idmouse_init+0x0/0x50
[ 11.128293] idmouse: Siemens ID Mouse FingerTIP Sensor Driver 0.6
[ 11.128320] usbcore: registered new interface driver idmouse
[ 11.128322] initcall usb_idmouse_init+0x0/0x50 returned 0 after 0 msecs
[ 11.128324] calling lego_usb_tower_init+0x0/0xa0
[ 11.128325] drivers/usb/misc/legousbtower.c: lego_usb_tower_init: enter
[ 11.128351] usbcore: registered new interface driver legousbtower
[ 11.128353] legousbtower: LEGO USB Tower Driver v0.96
[ 11.128354] drivers/usb/misc/legousbtower.c: lego_usb_tower_init: leave, return value 0
[ 11.128356] initcall lego_usb_tower_init+0x0/0xa0 returned 0 after 0 msecs
[ 11.128357] calling usb_rio_init+0x0/0x40
[ 11.128383] usbcore: registered new interface driver rio500
[ 11.128385] rio500: v1.1:USB Rio 500 driver
[ 11.128387] initcall usb_rio_init+0x0/0x40 returned 0 after 0 msecs
[ 11.128388] calling uss720_init+0x0/0x70
[ 11.128416] usbcore: registered new interface driver uss720
[ 11.128417] uss720: v0.6:USB Parport Cable driver for Cables using the Lucent Technologies USS720 Chip
[ 11.128418] uss720: NOTE: this is a special purpose driver to allow nonstandard
[ 11.128419] uss720: protocols (eg. bitbang) over USS720 usb to parallel cables
[ 11.128420] uss720: If you just want to connect to a printer, use usblp instead
[ 11.128422] initcall uss720_init+0x0/0x70 returned 0 after 0 msecs
[ 11.128424] calling cxacru_init+0x0/0x20
[ 11.128450] usbcore: registered new interface driver cxacru
[ 11.128452] initcall cxacru_init+0x0/0x20 returned 0 after 0 msecs
[ 11.128454] calling usbatm_usb_init+0x0/0x70
[ 11.128455] drivers/usb/atm/usbatm.c: usbatm_usb_init: driver version 1.10
[ 11.128457] initcall usbatm_usb_init+0x0/0x70 returned 0 after 0 msecs
[ 11.128459] calling xusbatm_init+0x0/0x140
[ 11.128460] drivers/usb/atm/xusbatm.c: xusbatm_init
[ 11.128460] xusbatm: malformed module parameters
[ 11.128462] initcall xusbatm_init+0x0/0x140 returned -22 after 0 msecs
[ 11.128464] initcall xusbatm_init+0x0/0x140 returned with error code -22
[ 11.128465] calling i8042_init+0x0/0x410
[ 11.128467] pnp: the driver 'i8042 kbd' has been registered
[ 11.128485] i8042 kbd 00:0a: driver attached
[ 11.128510] pnp: the driver 'i8042 aux' has been registered
[ 11.128522] i8042 aux 00:09: driver attached
[ 11.128545] PNP: PS/2 Controller [PNP0303:PS2K,PNP0f03:PS2M] at 0x60,0x64 irq 1,12
[ 11.131379] serio: i8042 KBD port at 0x60,0x64 irq 1
[ 11.131383] serio: i8042 AUX port at 0x60,0x64 irq 12
[ 11.131386] initcall i8042_init+0x0/0x410 returned 0 after 2 msecs
[ 11.131388] calling parkbd_init+0x0/0x1c0
[ 11.131391] parport0: cannot grant exclusive access for device parkbd
[ 11.131392] initcall parkbd_init+0x0/0x1c0 returned -19 after 0 msecs
[ 11.131394] calling serio_raw_init+0x0/0x20
[ 11.131420] initcall serio_raw_init+0x0/0x20 returned 0 after 0 msecs
[ 11.131422] calling l4_init+0x0/0x2b0
[ 11.131433] initcall l4_init+0x0/0x2b0 returned -19 after 0 msecs
[ 11.131435] calling ns558_init+0x0/0x360
[ 11.131436] pnp: the driver 'ns558' has been registered
[ 11.152342] initcall ns558_init+0x0/0x360 returned 0 after 19 msecs
[ 11.152344] calling mousedev_init+0x0/0x80
[ 11.152372] mice: PS/2 mouse device common for all mice
[ 11.152374] initcall mousedev_init+0x0/0x80 returned 0 after 0 msecs
[ 11.152376] calling evdev_init+0x0/0x20
[ 11.152378] initcall evdev_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152379] calling evbug_init+0x0/0x20
[ 11.152381] initcall evbug_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152383] calling atkbd_init+0x0/0x30
[ 11.152407] initcall atkbd_init+0x0/0x30 returned 0 after 0 msecs
[ 11.152409] calling sunkbd_init+0x0/0x20
[ 11.152438] initcall sunkbd_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152440] calling xtkbd_init+0x0/0x20
[ 11.152464] initcall xtkbd_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152466] calling nkbd_init+0x0/0x20
[ 11.152491] initcall nkbd_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152493] calling gc_init+0x0/0x730
[ 11.152494] initcall gc_init+0x0/0x730 returned -19 after 0 msecs
[ 11.152496] calling iforce_init+0x0/0x20
[ 11.152523] initcall iforce_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152524] calling magellan_init+0x0/0x20
[ 11.152549] initcall magellan_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152551] calling sw_init+0x0/0x20
[ 11.152554] initcall sw_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152556] calling tmdc_init+0x0/0x20
[ 11.152557] initcall tmdc_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152559] calling usb_xpad_init+0x0/0x40
[ 11.152586] usbcore: registered new interface driver xpad
[ 11.152588] xpad: X-Box pad driver
[ 11.152589] initcall usb_xpad_init+0x0/0x40 returned 0 after 0 msecs
[ 11.152591] calling usb_acecad_init+0x0/0x40
[ 11.152618] usbcore: registered new interface driver usb_acecad
[ 11.152619] acecad: v3.2:USB Acecad Flair tablet driver
[ 11.152621] initcall usb_acecad_init+0x0/0x40 returned 0 after 0 msecs
[ 11.152623] calling aiptek_init+0x0/0x50
[ 11.152650] usbcore: registered new interface driver aiptek
[ 11.152652] aiptek: v2.3 (May 2, 2007): Bryan W. Headley/Chris Atenasio/Cedric Brun/Rene van Paassen
[ 11.152653] aiptek: Aiptek HyperPen USB Tablet Driver (Linux 2.6.x)
[ 11.152655] initcall aiptek_init+0x0/0x50 returned 0 after 0 msecs
[ 11.152656] calling gtco_init+0x0/0x60
[ 11.152682] usbcore: registered new interface driver gtco
[ 11.152684] GTCO usb driver version: 2.00.0006initcall gtco_init+0x0/0x60 returned 0 after 0 msecs
[ 11.152687] calling kbtab_init+0x0/0x40
[ 11.152713] usbcore: registered new interface driver kbtab
[ 11.152715] kbtab: v0.0.2:USB KB Gear JamStudio Tablet driver
[ 11.152717] initcall kbtab_init+0x0/0x40 returned 0 after 0 msecs
[ 11.152718] calling wacom_init+0x0/0x50
[ 11.152748] usbcore: registered new interface driver wacom
[ 11.152750] wacom: v1.48:USB Wacom Graphire and Wacom Intuos tablet driver
[ 11.152752] initcall wacom_init+0x0/0x50 returned 0 after 0 msecs
[ 11.152754] calling cmos_init+0x0/0x20
[ 11.152755] pnp: the driver 'rtc_cmos' has been registered
[ 11.152796] rtc_cmos 00:03: rtc core: registered rtc_cmos as rtc0
[ 11.152815] rtc0: alarms up to one month
[ 11.152815] rtc_cmos 00:03: driver attached
[ 11.152843] initcall cmos_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152845] calling ds1511_rtc_init+0x0/0x20
[ 11.152873] initcall ds1511_rtc_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152875] calling ds1742_init+0x0/0x20
[ 11.152900] initcall ds1742_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152902] calling m48t59_rtc_init+0x0/0x20
[ 11.152930] initcall m48t59_rtc_init+0x0/0x20 returned 0 after 0 msecs
[ 11.152932] calling max6902_init+0x0/0x30
[ 11.152933] max6902 spi driver
[ 11.152957] initcall max6902_init+0x0/0x30 returned 0 after 0 msecs
[ 11.152959] calling test_init+0x0/0xe0
[ 11.153044] rtc-test rtc-test.0: rtc core: registered test as rtc1
[ 11.153102] rtc-test rtc-test.1: rtc core: registered test as rtc2
[ 11.153105] initcall test_init+0x0/0xe0 returned 0 after 0 msecs
[ 11.153107] calling v3020_init+0x0/0x20
[ 11.153135] initcall v3020_init+0x0/0x20 returned 0 after 0 msecs
[ 11.153137] calling abituguru_init+0x0/0x1c0
[ 11.153138] initcall abituguru_init+0x0/0x1c0 returned -19 after 0 msecs
[ 11.153140] calling abituguru3_init+0x0/0x1a0
[ 11.153146] abituguru3: no Abit uGuru3 found, data = 0xFF, cmd = 0xFF
[ 11.153148] initcall abituguru3_init+0x0/0x1a0 returned -19 after 0 msecs
[ 11.153150] calling applesmc_init+0x0/0x620
[ 11.153152] applesmc: supported laptop not found!
[ 11.153153] applesmc: driver init failed (ret=-19)!
[ 11.153155] initcall applesmc_init+0x0/0x620 returned -19 after 0 msecs
[ 11.153157] calling f71805f_init+0x0/0x190
[ 11.153185] initcall f71805f_init+0x0/0x190 returned -19 after 0 msecs
[ 11.153187] calling hdaps_init+0x0/0x200
[ 11.153193] hdaps: supported laptop not found!
[ 11.153194] hdaps: driver init failed (ret=-19)!
[ 11.153196] initcall hdaps_init+0x0/0x200 returned -19 after 0 msecs
[ 11.153197] calling i5k_amb_init+0x0/0x80
[ 11.153260] initcall i5k_amb_init+0x0/0x80 returned 0 after 0 msecs
[ 11.153262] calling sm_sis5595_init+0x0/0x20
[ 11.153298] initcall sm_sis5595_init+0x0/0x20 returned 0 after 0 msecs
[ 11.153300] calling sm_smsc47m1_init+0x0/0x250
[ 11.153311] initcall sm_smsc47m1_init+0x0/0x250 returned -19 after 0 msecs
[ 11.153313] calling vhci_init+0x0/0x70
[ 11.153314] Bluetooth: Virtual HCI driver ver 1.2
[ 11.153343] initcall vhci_init+0x0/0x70 returned 0 after 0 msecs
[ 11.153345] calling bcm203x_init+0x0/0x70
[ 11.153346] Bluetooth: Broadcom Blutonium firmware driver ver 1.1
[ 11.153374] usbcore: registered new interface driver bcm203x
[ 11.153377] initcall bcm203x_init+0x0/0x70 returned 0 after 0 msecs
[ 11.153379] calling bpa10x_init+0x0/0x40
[ 11.153380] Bluetooth: Digianswer Bluetooth USB driver ver 0.9
[ 11.153406] usbcore: registered new interface driver bpa10x
[ 11.153408] initcall bpa10x_init+0x0/0x40 returned 0 after 0 msecs
[ 11.153410] calling bfusb_init+0x0/0x70
[ 11.153411] Bluetooth: BlueFRITZ! USB driver ver 1.1
[ 11.153437] usbcore: registered new interface driver bfusb
[ 11.153439] initcall bfusb_init+0x0/0x70 returned 0 after 0 msecs
[ 11.153441] calling init_dtl1_cs+0x0/0x20
[ 11.153452] initcall init_dtl1_cs+0x0/0x20 returned 0 after 0 msecs
[ 11.153452] calling init_bluecard_cs+0x0/0x20
[ 11.153452] initcall init_bluecard_cs+0x0/0x20 returned 0 after 0 msecs
[ 11.153452] calling btsdio_init+0x0/0x30
[ 11.153452] Bluetooth: Generic Bluetooth SDIO driver ver 0.1
[ 11.153452] initcall btsdio_init+0x0/0x30 returned 0 after 0 msecs
[ 11.153452] calling sdio_uart_init+0x0/0x130
[ 11.153452] initcall sdio_uart_init+0x0/0x130 returned 0 after 0 msecs
[ 11.153452] calling sdhci_drv_init+0x0/0x40
[ 11.153452] sdhci: Secure Digital Host Controller Interface driver
[ 11.153452] sdhci: Copyright(c) Pierre Ossman
[ 11.153452] initcall sdhci_drv_init+0x0/0x40 returned 0 after 0 msecs
[ 11.153452] calling tifm_sd_init+0x0/0x20
[ 11.153452] initcall tifm_sd_init+0x0/0x20 returned 0 after 0 msecs
[ 11.153452] calling mmc_spi_init+0x0/0x20
[ 11.153452] initcall mmc_spi_init+0x0/0x20 returned 0 after 0 msecs
[ 11.153452] calling ib_core_init+0x0/0x50
[ 11.153452] initcall ib_core_init+0x0/0x50 returned 0 after 0 msecs
[ 11.153452] calling ib_mad_init_module+0x0/0xb0
[ 11.153452] initcall ib_mad_init_module+0x0/0xb0 returned 0 after 0 msecs
[ 11.153452] calling ib_sa_init+0x0/0x90
[ 11.153452] initcall ib_sa_init+0x0/0x90 returned 0 after 0 msecs
[ 11.153452] calling ib_cm_init+0x0/0x150
[ 11.153452] initcall ib_cm_init+0x0/0x150 returned 0 after 0 msecs
[ 11.153452] calling iw_cm_init+0x0/0x40
[ 11.153452] initcall iw_cm_init+0x0/0x40 returned 0 after 0 msecs
[ 11.153452] calling addr_init+0x0/0x50
[ 11.153452] initcall addr_init+0x0/0x50 returned 0 after 0 msecs
[ 11.153452] calling cma_init+0x0/0xe0
[ 11.153452] initcall cma_init+0x0/0xe0 returned 0 after 0 msecs
[ 11.153452] calling mthca_init+0x0/0x160
[ 11.153452] initcall mthca_init+0x0/0x160 returned 0 after 0 msecs
[ 11.153452] calling c2_init_module+0x0/0x20
[ 11.153452] initcall c2_init_module+0x0/0x20 returned 0 after 0 msecs
[ 11.153452] calling mlx4_ib_init+0x0/0x20
[ 11.153452] initcall mlx4_ib_init+0x0/0x20 returned 0 after 0 msecs
[ 11.153452] calling iser_init+0x0/0x130
[ 11.153452] iscsi: registered transport (iser)
[ 11.153452] initcall iser_init+0x0/0x130 returned 0 after 0 msecs
[ 11.153452] calling efivars_init+0x0/0x210
[ 11.153452] initcall efivars_init+0x0/0x210 returned -19 after 0 msecs
[ 11.153452] calling dcdbas_init+0x0/0x90
[ 11.153452] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.2)
[ 11.153452] initcall dcdbas_init+0x0/0x90 returned 0 after 0 msecs
[ 11.153452] calling init_soundcore+0x0/0x60
[ 11.153452] initcall init_soundcore+0x0/0x60 returned 0 after 0 msecs
[ 11.153452] calling trident_init_module+0x0/0x30
[ 11.153452] Trident 4DWave/SiS 7018/ALi 5451,Tvia CyberPro 5050 PCI Audio, version 0.14.10j-2.6, 04:23:42 May 24 2008
[ 11.153452] initcall trident_init_module+0x0/0x30 returned 0 after 0 msecs
[ 11.153452] calling alsa_sound_init+0x0/0x90
[ 11.153452] Advanced Linux Sound Architecture Driver Version 1.0.16.
[ 11.153452] initcall alsa_sound_init+0x0/0x90 returned 0 after 0 msecs
[ 11.153452] calling alsa_hwdep_init+0x0/0x70
[ 11.153452] initcall alsa_hwdep_init+0x0/0x70 returned 0 after 0 msecs
[ 11.153452] calling alsa_timer_init+0x0/0x220
[ 11.153452] initcall alsa_timer_init+0x0/0x220 returned 0 after 0 msecs
[ 11.153452] calling alsa_pcm_init+0x0/0x90
[ 11.153452] initcall alsa_pcm_init+0x0/0x90 returned 0 after 0 msecs
[ 11.153452] calling snd_mem_init+0x0/0x40
[ 11.153452] initcall snd_mem_init+0x0/0x40 returned 0 after 0 msecs
[ 11.153452] calling alsa_rawmidi_init+0x0/0x30
[ 11.153452] initcall alsa_rawmidi_init+0x0/0x30 returned 0 after 0 msecs
[ 11.153452] calling snd_mts64_module_init+0x0/0x60
[ 11.702034] initcall snd_mts64_module_init+0x0/0x60 returned -19 after 523 msecs
[ 11.702036] calling alsa_opl3_init+0x0/0x10
[ 11.702038] initcall alsa_opl3_init+0x0/0x10 returned 0 after 0 msecs
[ 11.702040] calling alsa_mpu401_uart_init+0x0/0x10
[ 11.702042] initcall alsa_mpu401_uart_init+0x0/0x10 returned 0 after 0 msecs
[ 11.702044] calling alsa_card_mpu401_init+0x0/0xf0
[ 11.702071] pnp: the driver 'mpu401' has been registered
[ 11.702121] pnp: the driver 'mpu401' has been unregistered
[ 11.702144] initcall alsa_card_mpu401_init+0x0/0xf0 returned -19 after 0 msecs
[ 11.702146] calling alsa_sb_common_init+0x0/0x10
[ 11.702148] initcall alsa_sb_common_init+0x0/0x10 returned 0 after 0 msecs
[ 11.702150] calling alsa_card_als4000_init+0x0/0x20
[ 11.702185] initcall alsa_card_als4000_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702188] calling alsa_card_cmipci_init+0x0/0x20
[ 11.702222] initcall alsa_card_cmipci_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702224] calling alsa_card_cs4281_init+0x0/0x20
[ 11.702259] initcall alsa_card_cs4281_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702261] calling alsa_card_es1968_init+0x0/0x20
[ 11.702295] initcall alsa_card_es1968_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702297] calling alsa_card_fm801_init+0x0/0x20
[ 11.702331] initcall alsa_card_fm801_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702333] calling alsa_card_rme32_init+0x0/0x20
[ 11.702368] initcall alsa_card_rme32_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702370] calling alsa_card_via82xx_init+0x0/0x20
[ 11.702405] initcall alsa_card_via82xx_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702408] calling alsa_card_via82xx_init+0x0/0x20
[ 11.702444] initcall alsa_card_via82xx_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702446] calling alsa_ac97_init+0x0/0x10
[ 11.702447] initcall alsa_ac97_init+0x0/0x10 returned 0 after 0 msecs
[ 11.702449] calling alsa_card_vortex_init+0x0/0x20
[ 11.702483] initcall alsa_card_vortex_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702485] calling alsa_card_vortex_init+0x0/0x20
[ 11.702519] initcall alsa_card_vortex_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702521] calling alsa_card_vortex_init+0x0/0x20
[ 11.702556] initcall alsa_card_vortex_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702559] calling alsa_card_ca0106_init+0x0/0x20
[ 11.702592] initcall alsa_card_ca0106_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702594] calling alsa_card_cs46xx_init+0x0/0x20
[ 11.702629] initcall alsa_card_cs46xx_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702631] calling alsa_card_echo_init+0x0/0x20
[ 11.702664] initcall alsa_card_echo_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702666] calling alsa_card_echo_init+0x0/0x20
[ 11.702699] initcall alsa_card_echo_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702701] calling alsa_card_echo_init+0x0/0x20
[ 11.702737] initcall alsa_card_echo_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702738] calling alsa_card_echo_init+0x0/0x20
[ 11.702773] initcall alsa_card_echo_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702775] calling alsa_card_echo_init+0x0/0x20
[ 11.702810] initcall alsa_card_echo_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702812] calling alsa_card_echo_init+0x0/0x20
[ 11.702846] initcall alsa_card_echo_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702847] calling alsa_card_echo_init+0x0/0x20
[ 11.702881] initcall alsa_card_echo_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702883] calling alsa_card_emu10k1_init+0x0/0x20
[ 11.702918] initcall alsa_card_emu10k1_init+0x0/0x20 returned 0 after 0 msecs
[ 11.702920] calling alsa_card_azx_init+0x0/0x20
[ 11.702950] IOAPIC[0]: Set routing entry (2-22 -> 0x71 -> IRQ 22 Mode:1 Active:1)
[ 11.702952] ACPI: PCI Interrupt 0000:00:1b.0[A] -> GSI 22 (level, low) -> IRQ 22
[ 11.702974] PCI: Setting latency timer of device 0000:00:1b.0 to 64
[ 11.778272] hda-codec: No codec parser is available
[ 11.778273] hda-intel: no codecs initialized
[ 11.796885] usb usb2: suspend_rh (auto-stop)
[ 12.046816] usb usb3: suspend_rh (auto-stop)
[ 12.046839] usb usb4: suspend_rh (auto-stop)
[ 12.296748] usb usb5: suspend_rh (auto-stop)
[ 13.190075] ACPI: PCI interrupt for device 0000:00:1b.0 disabled
[ 13.187047] hub 2-0:1.0: state 7 ports 2 chg 0000 evt 0000
[ 13.187056] hub 3-0:1.0: state 7 ports 2 chg 0000 evt 0000
[ 13.187066] hub 4-0:1.0: state 7 ports 2 chg 0000 evt 0000
[ 13.187077] hub 5-0:1.0: state 7 ports 2 chg 0000 evt 0000
[ 13.221608] initcall alsa_card_azx_init+0x0/0x20 returned 0 after 1445 msecs
[ 13.225048] calling alsa_card_nm256_init+0x0/0x20
[ 13.237058] initcall alsa_card_nm256_init+0x0/0x20 returned 0 after 0 msecs
[ 13.244098] calling alsa_card_oxygen_init+0x0/0x20
[ 13.244119] initcall alsa_card_oxygen_init+0x0/0x20 returned 0 after 0 msecs
[ 13.251219] calling alsa_card_xonar_init+0x0/0x20
[ 13.253904] initcall alsa_card_xonar_init+0x0/0x20 returned 0 after 0 msecs
[ 13.263236] calling alsa_util_mem_init+0x0/0x10
[ 13.265764] initcall alsa_util_mem_init+0x0/0x10 returned 0 after 0 msecs
[ 13.272597] calling snd_module_init+0x0/0x20
[ 13.283433] input: AT Translated Set 2 keyboard as /class/input/input0
[ 13.278390] usbcore: registered new interface driver snd-usb-caiaq
[ 13.278392] initcall snd_module_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278394] calling init_pdacf+0x0/0x20
[ 13.278419] initcall init_pdacf+0x0/0x20 returned 0 after 0 msecs
[ 13.278423] calling snd_soc_init+0x0/0x30
[ 13.278424] ASoC version 0.13.2
[ 13.278452] initcall snd_soc_init+0x0/0x30 returned 0 after 0 msecs
[ 13.278454] calling alsa_sound_last_init+0x0/0x70
[ 13.278455] ALSA device list:
[ 13.278455] No soundcards found.
[ 13.278457] initcall alsa_sound_last_init+0x0/0x70 returned 0 after 0 msecs
[ 13.278458] calling sysctl_core_init+0x0/0x20
[ 13.278495] initcall sysctl_core_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278497] calling flow_cache_init+0x0/0x1f0
[ 13.278522] initcall flow_cache_init+0x0/0x1f0 returned 0 after 0 msecs
[ 13.278524] calling llc_init+0x0/0x30
[ 13.278525] initcall llc_init+0x0/0x30 returned 0 after 0 msecs
[ 13.278527] calling llc2_init+0x0/0xe0
[ 13.278562] NET: Registered protocol family 26
[ 13.278563] initcall llc2_init+0x0/0xe0 returned 0 after 0 msecs
[ 13.278565] calling snap_init+0x0/0x40
[ 13.278567] initcall snap_init+0x0/0x40 returned 0 after 0 msecs
[ 13.278568] calling xt_init+0x0/0xe0
[ 13.278571] initcall xt_init+0x0/0xe0 returned 0 after 0 msecs
[ 13.278572] calling tcpudp_mt_init+0x0/0x20
[ 13.278575] initcall tcpudp_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278577] calling nflog_tg_init+0x0/0x20
[ 13.278578] initcall nflog_tg_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278580] calling xt_rateest_tg_init+0x0/0x60
[ 13.278584] initcall xt_rateest_tg_init+0x0/0x60 returned 0 after 0 msecs
[ 13.278586] calling comment_mt_init+0x0/0x20
[ 13.278588] initcall comment_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278589] calling dccp_mt_init+0x0/0x60
[ 13.278591] initcall dccp_mt_init+0x0/0x60 returned 0 after 0 msecs
[ 13.278592] calling dscp_mt_init+0x0/0x20
[ 13.278594] initcall dscp_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278596] calling hashlimit_mt_init+0x0/0xb0
[ 13.278603] initcall hashlimit_mt_init+0x0/0xb0 returned 0 after 0 msecs
[ 13.278605] calling iprange_mt_init+0x0/0x20
[ 13.278607] initcall iprange_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278608] calling length_mt_init+0x0/0x20
[ 13.278610] initcall length_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278612] calling mark_mt_init+0x0/0x20
[ 13.278614] initcall mark_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278615] calling multiport_mt_init+0x0/0x20
[ 13.278617] initcall multiport_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278619] calling owner_mt_init+0x0/0x20
[ 13.278621] initcall owner_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278622] calling sctp_mt_init+0x0/0x20
[ 13.278624] initcall sctp_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278625] calling string_mt_init+0x0/0x20
[ 13.278627] initcall string_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278629] calling time_mt_init+0x0/0x20
[ 13.278630] initcall time_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278632] calling u32_mt_init+0x0/0x20
[ 13.278634] initcall u32_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.278635] calling sysctl_ipv4_init+0x0/0x60
[ 13.278850] initcall sysctl_ipv4_init+0x0/0x60 returned 0 after 0 msecs
[ 13.278852] calling ipip_init+0x0/0x80
[ 13.278852] IPv4 over IPv4 tunneling driver
[ 13.279027] initcall ipip_init+0x0/0x80 returned 0 after 0 msecs
[ 13.279029] calling ipgre_init+0x0/0x90
[ 13.279030] GRE over IPv4 tunneling driver
[ 13.279198] initcall ipgre_init+0x0/0x90 returned 0 after 0 msecs
[ 13.279200] calling ah4_init+0x0/0x80
[ 13.279203] initcall ah4_init+0x0/0x80 returned 0 after 0 msecs
[ 13.279205] calling esp4_init+0x0/0x80
[ 13.279207] initcall esp4_init+0x0/0x80 returned 0 after 0 msecs
[ 13.279208] calling ipcomp4_init+0x0/0x80
[ 13.279210] initcall ipcomp4_init+0x0/0x80 returned 0 after 0 msecs
[ 13.279212] calling ipip_init+0x0/0x80
[ 13.279213] initcall ipip_init+0x0/0x80 returned 0 after 0 msecs
[ 13.279215] calling xfrm4_beet_init+0x0/0x20
[ 13.279217] initcall xfrm4_beet_init+0x0/0x20 returned 0 after 0 msecs
[ 13.279218] calling tunnel4_init+0x0/0x40
[ 13.279220] initcall tunnel4_init+0x0/0x40 returned 0 after 0 msecs
[ 13.279222] calling xfrm4_transport_init+0x0/0x20
[ 13.279223] initcall xfrm4_transport_init+0x0/0x20 returned 0 after 0 msecs
[ 13.279227] calling ipv4_netfilter_init+0x0/0x20
[ 13.279229] initcall ipv4_netfilter_init+0x0/0x20 returned 0 after 0 msecs
[ 13.279230] calling ip_tables_init+0x0/0xc0
[ 13.279235] ip_tables: (C) 2000-2006 Netfilter Core Team
[ 13.279237] initcall ip_tables_init+0x0/0xc0 returned 0 after 0 msecs
[ 13.279239] calling iptable_filter_init+0x0/0x80
[ 13.279247] initcall iptable_filter_init+0x0/0x80 returned 0 after 0 msecs
[ 13.279248] calling iptable_mangle_init+0x0/0x60
[ 13.279255] initcall iptable_mangle_init+0x0/0x60 returned 0 after 0 msecs
[ 13.279256] calling addrtype_mt_init+0x0/0x20
[ 13.279258] initcall addrtype_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.279260] calling ah_mt_init+0x0/0x20
[ 13.279261] initcall ah_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.279263] calling ecn_mt_init+0x0/0x20
[ 13.279264] initcall ecn_mt_init+0x0/0x20 returned 0 after 0 msecs
[ 13.279267] calling ecn_tg_init+0x0/0x20
[ 13.279268] initcall ecn_tg_init+0x0/0x20 returned 0 after 0 msecs
[ 13.279270] calling log_tg_init+0x0/0x30
[ 13.279272] initcall log_tg_init+0x0/0x30 returned 0 after 0 msecs
[ 13.279274] calling ttl_tg_init+0x0/0x20
[ 13.279275] initcall ttl_tg_init+0x0/0x20 returned 0 after 0 msecs
[ 13.279277] calling ulog_tg_init+0x0/0x120
[ 13.279285] initcall ulog_tg_init+0x0/0x120 returned 0 after 0 msecs
[ 13.279286] calling ip_vs_init+0x0/0xd0
[ 13.279330] IPVS: Registered protocols ()
[ 13.279345] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
[ 13.279364] IPVS: ipvs loaded.
[ 13.279366] initcall ip_vs_init+0x0/0xd0 returned 0 after 0 msecs
[ 13.279368] calling ip_vs_rr_init+0x0/0x30
[ 13.279369] IPVS: [rr] scheduler registered.
[ 13.279371] initcall ip_vs_rr_init+0x0/0x30 returned 0 after 0 msecs
[ 13.279372] calling ip_vs_lc_init+0x0/0x30
[ 13.279373] IPVS: [lc] scheduler registered.
[ 13.279375] initcall ip_vs_lc_init+0x0/0x30 returned 0 after 0 msecs
[ 13.279376] calling ip_vs_wlc_init+0x0/0x30
[ 13.279378] IPVS: [wlc] scheduler registered.
[ 13.279379] initcall ip_vs_wlc_init+0x0/0x30 returned 0 after 0 msecs
[ 13.279381] calling ip_vs_lblc_init+0x0/0x70
[ 13.279390] IPVS: [lblc] scheduler registered.
[ 13.279391] initcall ip_vs_lblc_init+0x0/0x70 returned 0 after 0 msecs
[ 13.279393] calling ip_vs_dh_init+0x0/0x30
[ 13.279394] IPVS: [dh] scheduler registered.
[ 13.279396] initcall ip_vs_dh_init+0x0/0x30 returned 0 after 0 msecs
[ 13.279397] calling ip_vs_sh_init+0x0/0x30
[ 13.279398] IPVS: [sh] scheduler registered.
[ 13.279400] initcall ip_vs_sh_init+0x0/0x30 returned 0 after 0 msecs
[ 13.279402] calling ip_vs_sed_init+0x0/0x30
[ 13.279403] IPVS: [sed] scheduler registered.
[ 13.279404] initcall ip_vs_sed_init+0x0/0x30 returned 0 after 0 msecs
[ 13.279406] calling ip_vs_nq_init+0x0/0x30
[ 13.279407] IPVS: [nq] scheduler registered.
[ 13.279408] initcall ip_vs_nq_init+0x0/0x30 returned 0 after 0 msecs
[ 13.279410] calling cubictcp_register+0x0/0x80
[ 13.279412] TCP cubic registered
[ 13.279413] initcall cubictcp_register+0x0/0x80 returned 0 after 0 msecs
[ 13.279415] calling packet_init+0x0/0x50
[ 13.279416] NET: Registered protocol family 17
[ 13.279420] initcall packet_init+0x0/0x50 returned 0 after 0 msecs
[ 13.279422] calling ipsec_pfkey_init+0x0/0xb0
[ 13.279423] NET: Registered protocol family 15
[ 13.279425] initcall ipsec_pfkey_init+0x0/0xb0 returned 0 after 0 msecs
[ 13.279427] calling br_init+0x0/0xe0
[ 13.279437] initcall br_init+0x0/0xe0 returned 0 after 0 msecs
[ 13.279439] calling ipx_init+0x0/0x120
[ 13.279494] NET: Registered protocol family 4
[ 14.047678] initcall ipx_init+0x0/0x120 returned 0 after 732 msecs
[ 14.053493] calling atalk_init+0x0/0xb0
[ 14.057432] NET: Registered protocol family 5
[ 14.127434] initcall atalk_init+0x0/0xb0 returned 0 after 66 msecs
[ 14.133635] calling x25_init+0x0/0x70
[ 14.137398] NET: Registered protocol family 9
[ 14.141767] X.25 for Linux Version 0.2
[ 14.143664] initcall x25_init+0x0/0x70 returned 0 after 7 msecs
[ 14.149628] calling ircomm_init+0x0/0xb0
[ 14.153652] IrCOMM protocol (Dag Brattli)
[ 14.153636] evbug.c: Connected device: "AT Translated Set 2 keyboard", isa0060/serio0/input0
[ 14.165409] initcall ircomm_init+0x0/0xb0 returned 0 after 11 msecs
[ 14.171720] calling ircomm_tty_init+0x0/0x190
[ 14.185053] initcall ircomm_tty_init+0x0/0x190 returned 0 after 1 msecs
[ 14.185053] calling af_rxrpc_init+0x0/0x1c0
[ 14.197432] NET: Registered protocol family 33
[ 14.197432] initcall af_rxrpc_init+0x0/0x1c0 returned 0 after 4 msecs
[ 14.203917] calling decnet_init+0x0/0xa0
[ 14.206063] NET4: DECnet for Linux: V.2.5.68s (C) 1995-2003 Linux DECnet Project Team
[ 14.220310] DECnet: Routing cache hash table of 1024 buckets, 16Kbytes
[ 14.226860] NET: Registered protocol family 12
[ 14.227000] initcall decnet_init+0x0/0xa0 returned 0 after 18 msecs
[ 14.233317] calling dn_rtmsg_init+0x0/0x80
[ 14.235361] initcall dn_rtmsg_init+0x0/0x80 returned 0 after 0 msecs
[ 14.241759] calling econet_proto_init+0x0/0x50
[ 14.246051] NET: Registered protocol family 19
[ 14.250509] initcall econet_proto_init+0x0/0x50 returned 0 after 4 msecs
[ 14.263453] calling vlan_proto_init+0x0/0xd0
[ 14.264178] 802.1Q VLAN Support v1.8 Ben Greear <greearb@candelatech.com>
[ 14.274179] All bugs added by David S. Miller <davem@redhat.com>
[ 14.275347] initcall vlan_proto_init+0x0/0xd0 returned 0 after 12 msecs
[ 14.285562] calling sctp_init+0x0/0x700
[ 14.285562] SCTP: Hash tables configured (established 65536 bind 65536)
[ 14.359861] sctp_init_sock(sk: ffff81003c1b8000)
[ 14.363775] initcall sctp_init+0x0/0x700 returned 0 after 72 msecs
[ 14.370002] calling rfkill_handler_init+0x0/0x20
[ 14.374675] initcall rfkill_handler_init+0x0/0x20 returned 0 after 0 msecs
[ 14.381594] calling hpet_insert_resource+0x0/0x30
[ 14.383935] initcall hpet_insert_resource+0x0/0x30 returned 0 after 0 msecs
[ 14.390939] calling lapic_insert_resource+0x0/0x50
[ 14.395750] initcall lapic_insert_resource+0x0/0x50 returned 0 after 0 msecs
[ 14.402841] calling init_lapic_nmi_sysfs+0x0/0x40
[ 14.407737] initcall init_lapic_nmi_sysfs+0x0/0x40 returned 0 after 0 msecs
[ 14.414740] calling ioapic_insert_resources+0x0/0x70
[ 14.419805] initcall ioapic_insert_resources+0x0/0x70 returned 0 after 0 msecs
[ 14.427074] calling sched_init_debug+0x0/0x30
[ 14.431537] initcall sched_init_debug+0x0/0x30 returned 0 after 0 msecs
[ 14.438197] calling init_oops_id+0x0/0x30
[ 14.442308] initcall init_oops_id+0x0/0x30 returned 0 after 0 msecs
[ 14.448620] calling disable_boot_consoles+0x0/0x40
[ 14.453508] initcall disable_boot_consoles+0x0/0x40 returned 0 after 0 msecs
[ 14.460600] calling pm_qos_power_init+0x0/0xa0
[ 14.465414] initcall pm_qos_power_init+0x0/0xa0 returned 0 after 0 msecs
[ 14.472159] calling software_resume+0x0/0x160
[ 14.476615] initcall software_resume+0x0/0x160 returned -2 after 0 msecs
[ 14.483360] initcall software_resume+0x0/0x160 returned with error code -2
[ 14.490676] calling afs_init+0x0/0x1e0
[ 14.494525] kAFS: Red Hat AFS client v0.1 registering.
[ 14.506425] initcall afs_init+0x0/0x1e0 returned 0 after 5 msecs
[ 14.506425] calling random32_reseed+0x0/0x80
[ 14.510804] initcall random32_reseed+0x0/0x80 returned 0 after 0 msecs
[ 14.516994] calling pci_sysfs_init+0x0/0x60
[ 14.521378] initcall pci_sysfs_init+0x0/0x60 returned 0 after 0 msecs
[ 14.527865] calling acpi_wakeup_device_init+0x0/0xac
[ 14.532934] initcall acpi_wakeup_device_init+0x0/0xac returned 0 after 0 msecs
[ 14.540202] calling acpi_sleep_proc_init+0x0/0x38
[ 14.544405] initcall acpi_sleep_proc_init+0x0/0x38 returned 0 after 0 msecs
[ 14.551410] calling seqgen_init+0x0/0x20
[ 14.555439] initcall seqgen_init+0x0/0x20 returned 0 after 0 msecs
[ 14.561668] calling scsi_complete_async_scans+0x0/0x130
[ 14.566991] initcall scsi_complete_async_scans+0x0/0x130 returned 0 after 0 msecs
[ 14.574184] calling edd_init+0x0/0x2c0
[ 14.578036] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[ 14.584051] EDD information not available.
[ 14.588163] initcall edd_init+0x0/0x2c0 returned 1 after 9 msecs
[ 14.594213] initcall edd_init+0x0/0x2c0 returned with error code 1
[ 14.600526] calling tcp_congestion_default+0x0/0x20
[ 14.605424] initcall tcp_congestion_default+0x0/0x20 returned 0 after 0 msecs
[ 14.612602] calling ip_auto_config+0x0/0xf90
[ 14.617851] initcall ip_auto_config+0x0/0xf90 returned 0 after 0 msecs
[ 14.643884] kjournald starting. Commit interval 5 seconds
[ 14.640379] EXT3-fs: mounted filesystem with ordered data mode.
[ 14.640390] VFS: Mounted root (ext3 filesystem) readonly.
[ 14.640408] Freeing unused kernel memory: 1332k freed
[ 16.641668] usb usb1: uevent
[ 16.644601] usb 1-0:1.0: uevent
[ 16.647855] usb usb2: uevent
[ 16.651506] usb 2-0:1.0: uevent
[ 16.651521] usb usb3: uevent
[ 16.651535] usb 3-0:1.0: uevent
[ 16.651549] usb usb4: uevent
[ 16.651563] usb 4-0:1.0: uevent
[ 16.651579] usb usb5: uevent
[ 16.651592] usb 5-0:1.0: uevent
[ 109.151140] EXT3 FS on sda1, internal journal
[ 109.251429] kjournald starting. Commit interval 5 seconds
[ 109.245820] EXT3 FS on sdb1, internal journal
[ 109.245825] EXT3-fs: mounted filesystem with ordered data mode.
[ 109.601916] Adding 8193140k swap on /dev/sda2. Priority:-1 extents:1 across:8193140k
[ 110.436787] e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
[ 115.287552] warning: `dbus-daemon' uses 32-bit capabilities (legacy support in use)
00:00.0 Host bridge: Intel Corporation 82975X Memory Controller Hub
Subsystem: Intel Corporation Unknown device 5842
Flags: bus master, fast devsel, latency 0
Capabilities: <access denied>
00:01.0 PCI bridge: Intel Corporation 82975X PCI Express Root Port (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00003000-00003fff
Memory behind bridge: 50300000-503fffff
Prefetchable memory behind bridge: 0000000040000000-000000004ff00000
Capabilities: <access denied>
00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller (rev 01)
Subsystem: Intel Corporation Unknown device 0417
Flags: fast devsel, IRQ 22
Memory at 50400000 (64-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port 1 (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
Memory behind bridge: 50200000-502fffff
Capabilities: <access denied>
00:1c.4 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 5 (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
Capabilities: <access denied>
00:1c.5 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI Express Port 6 (rev 01) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: 50100000-501fffff
Capabilities: <access denied>
00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI #1 (rev 01) (prog-if 00 [UHCI])
Subsystem: Intel Corporation Unknown device 5842
Flags: bus master, medium devsel, latency 0, IRQ 23
I/O ports at 4080 [size=32]
00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI #2 (rev 01) (prog-if 00 [UHCI])
Subsystem: Intel Corporation Unknown device 5842
Flags: bus master, medium devsel, latency 0, IRQ 19
I/O ports at 4060 [size=32]
00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI #3 (rev 01) (prog-if 00 [UHCI])
Subsystem: Intel Corporation Unknown device 5842
Flags: bus master, medium devsel, latency 0, IRQ 18
I/O ports at 4040 [size=32]
00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI #4 (rev 01) (prog-if 00 [UHCI])
Subsystem: Intel Corporation Unknown device 5842
Flags: bus master, medium devsel, latency 0, IRQ 16
I/O ports at 4020 [size=32]
00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller (rev 01) (prog-if 20 [EHCI])
Subsystem: Intel Corporation Unknown device 5842
Flags: bus master, medium devsel, latency 0, IRQ 23
Memory at 50404400 (32-bit, non-prefetchable) [size=1K]
Capabilities: <access denied>
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1) (prog-if 01 [Subtractive decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=05, subordinate=05, sec-latency=32
I/O behind bridge: 00001000-00001fff
Memory behind bridge: 50000000-500fffff
Capabilities: <access denied>
00:1f.0 ISA bridge: Intel Corporation 82801GH (ICH7DH) LPC Interface Bridge (rev 01)
Subsystem: Intel Corporation Unknown device 5842
Flags: bus master, medium devsel, latency 0
Capabilities: <access denied>
00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller (rev 01) (prog-if 8a [Master SecP PriP])
Subsystem: Intel Corporation Unknown device 5842
Flags: bus master, medium devsel, latency 0, IRQ 18
I/O ports at 01f0 [size=8]
I/O ports at 03f4 [size=1]
I/O ports at 0170 [size=8]
I/O ports at 0374 [size=1]
I/O ports at 40b0 [size=16]
00:1f.2 IDE interface: Intel Corporation 82801GB/GR/GH (ICH7 Family) Serial ATA Storage Controller IDE (rev 01) (prog-if 8f [Master SecP SecO PriP PriO])
Subsystem: Intel Corporation Unknown device 5842
Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 19
I/O ports at 40c8 [size=8]
I/O ports at 40e4 [size=4]
I/O ports at 40c0 [size=8]
I/O ports at 40e0 [size=4]
I/O ports at 40a0 [size=16]
Memory at 50404000 (32-bit, non-prefetchable) [size=1K]
Capabilities: <access denied>
00:1f.3 SMBus: Intel Corporation 82801G (ICH7 Family) SMBus Controller (rev 01)
Subsystem: Intel Corporation Unknown device 5842
Flags: medium devsel, IRQ 9
I/O ports at 4000 [size=32]
01:00.0 VGA compatible controller: ATI Technologies Inc R580 [Radeon X1900 XT] Primary (prog-if 00 [VGA])
Subsystem: ATI Technologies Inc Unknown device 0b12
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at 40000000 (64-bit, prefetchable) [size=256M]
Memory at 50300000 (64-bit, non-prefetchable) [size=64K]
I/O ports at 3000 [size=256]
Expansion ROM at 50320000 [disabled] [size=128K]
Capabilities: <access denied>
01:00.1 Display controller: ATI Technologies Inc R580 [Radeon X1900 XT] Secondary
Subsystem: ATI Technologies Inc Unknown device 0b13
Flags: bus master, fast devsel, latency 0
Memory at 50310000 (64-bit, non-prefetchable) [size=64K]
Capabilities: <access denied>
02:00.0 Multimedia controller: Philips Semiconductors Unknown device 7162
Subsystem: Animation Technologies Inc. Unknown device 0820
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at 50200000 (64-bit, non-prefetchable) [size=1M]
Capabilities: <access denied>
04:00.0 Ethernet controller: Intel Corporation 82573L Gigabit Ethernet Controller
Subsystem: Intel Corporation Unknown device 30a5
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at 50100000 (32-bit, non-prefetchable) [size=128K]
I/O ports at 2000 [size=32]
Capabilities: <access denied>
05:02.0 Network controller: RaLink RT2561/RT61 802.11g PCI
Subsystem: Linksys WMP54G ver 4.1
Flags: bus master, slow devsel, latency 32, IRQ 10
Memory at 50000000 (32-bit, non-prefetchable) [size=32K]
Capabilities: <access denied>
05:04.0 FireWire (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller (PHY/Link) (prog-if 10 [OHCI])
Subsystem: Intel Corporation Unknown device 5842
Flags: bus master, medium devsel, latency 32, IRQ 10
Memory at 5000c000 (32-bit, non-prefetchable) [size=2K]
Memory at 50008000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
05:05.0 RAID bus controller: Silicon Image, Inc. SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
Subsystem: Intel Corporation Unknown device 7114
Flags: bus master, 66MHz, medium devsel, latency 32, IRQ 10
I/O ports at 1018 [size=8]
I/O ports at 1024 [size=4]
I/O ports at 1010 [size=8]
I/O ports at 1020 [size=4]
I/O ports at 1000 [size=16]
Memory at 5000c800 (32-bit, non-prefetchable) [size=1K]
Expansion ROM at fff80000 [disabled] [size=512K]
Capabilities: <access denied>
IcmpMsg:
InType3: 32
InType8: 1
OutType0: 1
OutType3: 32
Tcp:
1541 active connections openings
152 passive connection openings
2 failed connection attempts
0 connection resets received
5 connections established
350291 segments received
160513 segments send out
0 segments retransmited
0 bad segments received.
2 resets sent
UdpLite:
----- End forwarded message -----
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 11:56 [bug] stuck localhost TCP connections, v2.6.26-rc3+ Ingo Molnar
@ 2008-05-26 13:28 ` Ilpo Järvinen
2008-05-26 13:59 ` Ingo Molnar
2008-05-30 16:23 ` Ray Lee
2008-05-26 16:24 ` Arjan van de Ven
` (2 subsequent siblings)
3 siblings, 2 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-26 13:28 UTC (permalink / raw)
To: Ingo Molnar
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
On Mon, 26 May 2008, Ingo Molnar wrote:
> in an overnight -tip testruns that is based on recent -git i got two
> stuck TCP connections:
>
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED
> tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015 ESTABLISHED
>
> on a previously reliable machine. That connection has been stuck for 9
> hours so it does not time out, etc. - and the distcc run that goes over
> that connection is stuck as well.
>
> kernel config is attached.
>
> in terms of debugging there's not much i can do i'm afraid. It's not
> possible to get a tcpdump of this incident, given the extreme amount of
> load these testboxes handle.
...but you can still tcpdump that particular flow once the situation is
discovered to see if TCP still tries to do something, no? One needs to
tcpdump couple of minutes at minimum. Also please get /proc/net/tcp for
that flow around the same time.
> This problem started sometime around rc3
> and it occured on two boxes (on a laptop and on a desktop), both are SMP
> Core2Duo based systems. I never saw this problem before on thousands of
> similar bootups, so i'm 99.9% sure the bug is either new or became
> easier to trigger.
>
> It's not possible to bisect it as it needs up to 12 hours of heavy
> workload to trigger. The incident happened about 5 times since the first
> incident a couple of days ago - 4 times on one box and once on another
> box. The first failing head i became aware of was 78b58e549a3098. (-tip
> has other changes beyond -git but changes nothing in networking.)
>
> One clue (which might or might not matter) is that distcc is one of the
> very few applications that makes use of sendfile().
Can you please try with /proc/sys/net/ipv4/tcp_frto set to zero though
recv-q symptom seems weird would it be related to that (but there were
some recent fixes to FRTO and retrans_stamp change could have some
significance here)?
Other than that, nothing since -rc1 seems suspicious to me (though
I hardly understand every part of networking).
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 13:28 ` Ilpo Järvinen
@ 2008-05-26 13:59 ` Ingo Molnar
2008-05-26 14:12 ` Ingo Molnar
2008-05-30 16:23 ` Ray Lee
1 sibling, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-05-26 13:59 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
* Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> > in terms of debugging there's not much i can do i'm afraid. It's not
> > possible to get a tcpdump of this incident, given the extreme amount
> > of load these testboxes handle.
>
> ...but you can still tcpdump that particular flow once the situation
> is discovered to see if TCP still tries to do something, no? One needs
> to tcpdump couple of minutes at minimum. Also please get /proc/net/tcp
> for that flow around the same time.
ok, will try those.
> > One clue (which might or might not matter) is that distcc is one of
> > the very few applications that makes use of sendfile().
>
> Can you please try with /proc/sys/net/ipv4/tcp_frto set to zero though
> recv-q symptom seems weird would it be related to that (but there were
> some recent fixes to FRTO and retrans_stamp change could have some
> significance here)?
>
> Other than that, nothing since -rc1 seems suspicious to me (though I
> hardly understand every part of networking).
ok, i will first wait for it to trigger on a box and will do the tcpdump
session (and /proc/net/tcp output), then i'll continue the tests with
this done in the rc.local:
echo 0 > /proc/sys/net/ipv4/tcp_frto
and will see whether the hung connections still occur. The cycle of
testing will be very slow i suspect.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 13:59 ` Ingo Molnar
@ 2008-05-26 14:12 ` Ingo Molnar
2008-05-26 14:17 ` Ingo Molnar
2008-05-26 14:58 ` Ilpo Järvinen
0 siblings, 2 replies; 96+ messages in thread
From: Ingo Molnar @ 2008-05-26 14:12 UTC (permalink / raw)
To: Ilpo J?rvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
[-- Attachment #1: Type: text/plain, Size: 3206 bytes --]
* Ingo Molnar <mingo@elte.hu> wrote:
> > ...but you can still tcpdump that particular flow once the situation
> > is discovered to see if TCP still tries to do something, no? One
> > needs to tcpdump couple of minutes at minimum. Also please get
> > /proc/net/tcp for that flow around the same time.
>
> ok, will try those.
i got lucky ... the bug just reproduced again - find the netstat and
/proc/net/tcp output below. New config attached.
i also ran this for 15 minutes:
[root@europe ~]# tcpdump src and dst 10.0.1.15 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
but there was no output at all - i.e. TCP has not attempted to do
anything to these connections.
there's a hung distcc task on the system, waiting for socket action
forever:
[root@europe ~]# strace -fp 19578
Process 19578 attached - interrupt to quit
select(5, NULL, [4], [4], {82, 90000} <unfinished ...>
disturbing that task via strace did not change the state of the socket -
and that's not unexpected as it's a select(). [TCP state might be
affected if strace impacted a recvmsg or a sendmsg wait directly.]
i'll keep this system in its hung state for an hour, just in case you
can think of anything else to extract out of it.
Ingo
------------->
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 10.0.1.15:22 10.0.1.16:42843 ESTABLISHED
tcp 0 207232 10.0.1.15:37198 10.0.1.15:3632 ESTABLISHED
tcp 0 0 10.0.1.15:22 10.0.1.16:36284 ESTABLISHED
tcp 72283 0 10.0.1.15:3632 10.0.1.15:37198 ESTABLISHED
Tcp:
1931 active connections openings
149 passive connection openings
0 failed connection attempts
0 connection resets received
4 connections established
436004 segments received
98819 segments send out
4 segments retransmited
0 bad segments received.
0 resets sent
UdpLite:
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:0E30 00000000:0000 0A 00000000:00000001 00:00000000 00000000 100 0 7526 1 f6110640 300 0 0 2 -1
1: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7132 1 f61c8000 300 0 0 2 -1
2: 0F01000A:0016 1001000A:A75B 01 00000000:00000000 02:000AF5CA 00000000 0 0 39283 4 e6826400 21 6 15 4 -1
3: 0F01000A:914E 0F01000A:0E30 01 00032980:00000000 04:00001AAA 00000000 500 0 31436 2 f6113200 22 0 0 4 -1
4: 0F01000A:0016 1001000A:8DBC 01 00000000:00000000 02:0009141B 00000000 0 0 10784 2 f6110000 21 4 0 4 -1
5: 0F01000A:0E30 0F01000A:914E 01 00000000:00011A5B 00:00000000 00000000 0 0 0 1 f6115dc0 22 8 0 2 -1
[-- Attachment #2: config --]
[-- Type: text/plain, Size: 47961 bytes --]
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.26-rc3
# Mon May 26 15:38:43 2008
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
# CONFIG_X86_64 is not set
CONFIG_X86=y
CONFIG_DEFCONFIG_LIST="arch/x86/configs/i386_defconfig"
# CONFIG_GENERIC_LOCKBREAK is not set
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
# CONFIG_GENERIC_GPIO is not set
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
# CONFIG_ARCH_HAS_ILOG2_U32 is not set
# CONFIG_ARCH_HAS_ILOG2_U64 is not set
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
# CONFIG_GENERIC_TIME_VSYSCALL is not set
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ZONE_DMA32 is not set
CONFIG_ARCH_POPULATES_NODE_MAP=y
# CONFIG_AUDIT_ARCH is not set
CONFIG_ARCH_SUPPORTS_AOUT=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_X86_32_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_KTIME_SCALAR=y
CONFIG_BOOTPARAM_SUPPORT=y
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_BROKEN_BOOT_ALLOWED3=y
CONFIG_BROKEN_BOOT_ALLOWED2=y
# CONFIG_BROKEN_BOOT_ALLOWED is not set
# CONFIG_BROKEN_BOOT_EUROPE is not set
# CONFIG_BROKEN_BOOT_TITAN is not set
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
# CONFIG_SWAP is not set
CONFIG_SYSVIPC=y
# CONFIG_POSIX_MQUEUE is not set
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
# CONFIG_TASK_XACCT is not set
# CONFIG_AUDIT is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=20
CONFIG_CGROUPS=y
CONFIG_CGROUP_DEBUG=y
CONFIG_CGROUP_NS=y
# CONFIG_CGROUP_DEVICE is not set
CONFIG_CPUSETS=y
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
# CONFIG_FAIR_GROUP_SCHED is not set
# CONFIG_RT_GROUP_SCHED is not set
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
# CONFIG_CGROUP_MEM_RES_CTLR is not set
CONFIG_SYSFS_DEPRECATED=y
# CONFIG_SYSFS_DEPRECATED_V2 is not set
# CONFIG_PROC_PID_CPUSET is not set
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
CONFIG_IPC_NS=y
# CONFIG_USER_NS is not set
CONFIG_PID_NS=y
# CONFIG_BLK_DEV_INITRD is not set
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_EMBEDDED=y
# CONFIG_UID16 is not set
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_COMPAT_BRK=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
# CONFIG_EPOLL is not set
# CONFIG_SIGNALFD is not set
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
# CONFIG_VM_EVENT_COUNTERS is not set
# CONFIG_SLUB_DEBUG is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
CONFIG_PROFILING=y
CONFIG_MARKERS=y
CONFIG_OPROFILE=y
CONFIG_HAVE_OPROFILE=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
# CONFIG_HAVE_DMA_ATTRS is not set
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_HAVE_IMMEDIATE=y
CONFIG_IMMEDIATE=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
# CONFIG_MODULES is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
# CONFIG_LBD is not set
# CONFIG_BLK_DEV_IO_TRACE is not set
CONFIG_LSF=y
CONFIG_BLK_DEV_BSG=y
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_AS is not set
# CONFIG_IOSCHED_DEADLINE is not set
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
# CONFIG_DEFAULT_CFQ is not set
CONFIG_DEFAULT_NOOP=y
CONFIG_DEFAULT_IOSCHED="noop"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_CLASSIC_RCU=y
#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
# CONFIG_BOOTPARAM_NO_HZ_OFF is not set
# CONFIG_HIGH_RES_TIMERS is not set
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
# CONFIG_BOOTPARAM_HIGHRES_OFF is not set
# CONFIG_SMP_SUPPORT is not set
CONFIG_UP_WANTED_1=y
CONFIG_UP_WANTED_2=y
# CONFIG_UP_WANTED is not set
CONFIG_SMP=y
# CONFIG_X86_PC is not set
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
CONFIG_X86_BIGSMP=y
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_X86_RDC321X is not set
# CONFIG_X86_VSMP is not set
# CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER is not set
# CONFIG_PARAVIRT_GUEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
CONFIG_MVIAC7=y
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_GENERIC=y
CONFIG_X86_CPU=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_X86_XADD=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_TSC=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=4
CONFIG_X86_DEBUGCTLMSR=y
# CONFIG_X86_DS is not set
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
# CONFIG_IOMMU_HELPER is not set
CONFIG_NR_CPUS=32
# CONFIG_SCHED_SMT is not set
# CONFIG_SCHED_MC is not set
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
# CONFIG_X86_MCE_P4THERMAL is not set
# CONFIG_VM86 is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
CONFIG_X86_REBOOTFIXUPS=y
CONFIG_MICROCODE=y
CONFIG_MICROCODE_OLD_INTERFACE=y
# CONFIG_X86_MSR is not set
CONFIG_X86_CPUID=y
CONFIG_NOHIGHMEM=y
# CONFIG_HIGHMEM4G is not set
# CONFIG_HIGHMEM64G is not set
# CONFIG_VMSPLIT_3G is not set
# CONFIG_VMSPLIT_3G_OPT is not set
# CONFIG_VMSPLIT_2G is not set
CONFIG_VMSPLIT_2G_OPT=y
# CONFIG_VMSPLIT_1G is not set
CONFIG_PAGE_OFFSET=0x78000000
# CONFIG_X86_PAE is not set
CONFIG_ILLEGAL_POINTER_VALUE=0
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
# CONFIG_SPARSEMEM_STATIC is not set
# CONFIG_SPARSEMEM_VMEMMAP_ENABLE is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
# CONFIG_RESOURCES_64BIT is not set
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MATH_EMULATION=y
# CONFIG_MTRR is not set
CONFIG_EFI=y
# CONFIG_IRQBALANCE is not set
# CONFIG_SECCOMP is not set
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
# CONFIG_SCHED_HRTICK is not set
# CONFIG_KEXEC is not set
CONFIG_PHYSICAL_START=0x100000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x100000
CONFIG_HOTPLUG_CPU=y
# CONFIG_COMPAT_VDSO is not set
CONFIG_BOOTPARAM_NOLAPIC_TIMER=y
# CONFIG_BOOTPARAM_LAPIC is not set
CONFIG_BOOTPARAM_HPET_DISABLE=y
CONFIG_BOOTPARAM_IDLE_MWAIT=y
# CONFIG_BOOTPARAM_IDLE_POLL is not set
# CONFIG_BOOTPARAM_HIGHMEM_512M is not set
# CONFIG_BOOTPARAM_NOPAT is not set
# CONFIG_BOOTPARAM_NOTSC is not set
CONFIG_BOOTPARAM_PCI_NOMSI=y
#
# Power management options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_SLEEP=y
CONFIG_SUSPEND=y
CONFIG_SUSPEND_FREEZER=y
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
CONFIG_ACPI_AC=y
CONFIG_ACPI_BATTERY=y
# CONFIG_ACPI_BUTTON is not set
# CONFIG_ACPI_VIDEO is not set
# CONFIG_ACPI_FAN is not set
CONFIG_ACPI_DOCK=y
# CONFIG_ACPI_BAY is not set
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
# CONFIG_ACPI_THERMAL is not set
CONFIG_ACPI_WMI=y
CONFIG_ACPI_ASUS=y
CONFIG_ACPI_TOSHIBA=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_DEBUG=y
# CONFIG_ACPI_DEBUG_FUNC_TRACE is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
# CONFIG_X86_PM_TIMER is not set
CONFIG_ACPI_CONTAINER=y
# CONFIG_ACPI_SBS is not set
# CONFIG_APM is not set
#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
# CONFIG_CPU_IDLE is not set
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
CONFIG_PCI_GODIRECT=y
# CONFIG_PCI_GOANY is not set
# CONFIG_PCI_GOOLPC is not set
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_DOMAINS=y
CONFIG_PCIEPORTBUS=y
# CONFIG_HOTPLUG_PCI_PCIE is not set
# CONFIG_PCIEAER is not set
# CONFIG_PCIEASPM is not set
CONFIG_ARCH_SUPPORTS_MSI=y
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY=y
CONFIG_PCI_DEBUG=y
# CONFIG_HT_IRQ is not set
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
CONFIG_EISA=y
CONFIG_EISA_PCI_EISA=y
# CONFIG_EISA_NAMES is not set
CONFIG_MCA=y
# CONFIG_MCA_LEGACY is not set
CONFIG_SCx200=y
# CONFIG_SCx200HR_TIMER is not set
# CONFIG_OLPC is not set
# CONFIG_PCCARD is not set
CONFIG_HOTPLUG_PCI=y
# CONFIG_HOTPLUG_PCI_FAKE is not set
CONFIG_HOTPLUG_PCI_COMPAQ=y
# CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM is not set
CONFIG_HOTPLUG_PCI_IBM=y
CONFIG_HOTPLUG_PCI_ACPI=y
CONFIG_HOTPLUG_PCI_ACPI_IBM=y
CONFIG_HOTPLUG_PCI_CPCI=y
CONFIG_HOTPLUG_PCI_CPCI_ZT5550=y
CONFIG_HOTPLUG_PCI_CPCI_GENERIC=y
CONFIG_HOTPLUG_PCI_SHPC=y
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_AOUT is not set
# CONFIG_BINFMT_MISC is not set
#
# Networking
#
CONFIG_NET=y
#
# Networking options
#
CONFIG_PACKET=y
# CONFIG_PACKET_MMAP is not set
CONFIG_UNIX=y
CONFIG_XFRM=y
# CONFIG_XFRM_USER is not set
CONFIG_XFRM_SUB_POLICY=y
CONFIG_XFRM_MIGRATE=y
CONFIG_XFRM_STATISTICS=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
CONFIG_IP_ADVANCED_ROUTER=y
# CONFIG_ASK_IP_FIB_HASH is not set
CONFIG_IP_FIB_TRIE=y
# CONFIG_IP_FIB_HASH is not set
CONFIG_IP_FIB_TRIE_STATS=y
# CONFIG_IP_MULTIPLE_TABLES is not set
CONFIG_IP_ROUTE_MULTIPATH=y
# CONFIG_IP_ROUTE_VERBOSE is not set
# CONFIG_IP_PNP is not set
CONFIG_NET_IPIP=y
# CONFIG_NET_IPGRE is not set
CONFIG_ARPD=y
# CONFIG_SYN_COOKIES is not set
CONFIG_INET_AH=y
# CONFIG_INET_ESP is not set
CONFIG_INET_IPCOMP=y
CONFIG_INET_XFRM_TUNNEL=y
CONFIG_INET_TUNNEL=y
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
CONFIG_INET_XFRM_MODE_TUNNEL=y
CONFIG_INET_XFRM_MODE_BEET=y
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6=y
# CONFIG_IPV6_PRIVACY is not set
CONFIG_IPV6_ROUTER_PREF=y
# CONFIG_IPV6_ROUTE_INFO is not set
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
# CONFIG_INET6_AH is not set
CONFIG_INET6_ESP=y
CONFIG_INET6_IPCOMP=y
# CONFIG_IPV6_MIP6 is not set
CONFIG_INET6_XFRM_TUNNEL=y
CONFIG_INET6_TUNNEL=y
# CONFIG_INET6_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET6_XFRM_MODE_TUNNEL is not set
# CONFIG_INET6_XFRM_MODE_BEET is not set
# CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set
# CONFIG_IPV6_SIT is not set
CONFIG_IPV6_TUNNEL=y
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
CONFIG_NETLABEL=y
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NETFILTER is not set
CONFIG_IP_DCCP=y
CONFIG_INET_DCCP_DIAG=y
CONFIG_IP_DCCP_ACKVEC=y
#
# DCCP CCIDs Configuration (EXPERIMENTAL)
#
CONFIG_IP_DCCP_CCID2=y
CONFIG_IP_DCCP_CCID2_DEBUG=y
CONFIG_IP_DCCP_CCID3=y
# CONFIG_IP_DCCP_CCID3_DEBUG is not set
CONFIG_IP_DCCP_CCID3_RTO=100
CONFIG_IP_DCCP_TFRC_LIB=y
#
# DCCP Kernel Hacking
#
CONFIG_IP_DCCP_DEBUG=y
# CONFIG_IP_SCTP is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
CONFIG_BRIDGE=y
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
CONFIG_LLC=y
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
CONFIG_LAPB=y
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_SCHED is not set
CONFIG_NET_SCH_FIFO=y
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
#
# Wireless
#
CONFIG_CFG80211=y
CONFIG_NL80211=y
CONFIG_WIRELESS_EXT=y
CONFIG_MAC80211=y
#
# Rate control algorithm selection
#
# CONFIG_MAC80211_RC_DEFAULT_PID is not set
CONFIG_MAC80211_RC_DEFAULT_NONE=y
#
# Selecting 'y' for an algorithm will
#
#
# build the algorithm into mac80211.
#
CONFIG_MAC80211_RC_DEFAULT=""
CONFIG_MAC80211_RC_PID=y
# CONFIG_MAC80211_MESH is not set
CONFIG_MAC80211_LEDS=y
CONFIG_MAC80211_DEBUGFS=y
CONFIG_MAC80211_DEBUG_PACKET_ALIGNMENT=y
# CONFIG_MAC80211_DEBUG is not set
CONFIG_IEEE80211=y
# CONFIG_IEEE80211_DEBUG is not set
CONFIG_IEEE80211_CRYPT_WEP=y
# CONFIG_IEEE80211_CRYPT_CCMP is not set
# CONFIG_IEEE80211_CRYPT_TKIP is not set
# CONFIG_RFKILL is not set
CONFIG_NET_9P=y
# CONFIG_NET_9P_DEBUG is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_DEBUG_DRIVER=y
CONFIG_DEBUG_DEVRES=y
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
# CONFIG_PARPORT_PC_FIFO is not set
CONFIG_PARPORT_PC_SUPERIO=y
# CONFIG_PARPORT_GSC is not set
# CONFIG_PARPORT_AX88796 is not set
CONFIG_PARPORT_1284=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set
#
# Protocols
#
# CONFIG_ISAPNP is not set
CONFIG_PNPBIOS=y
CONFIG_PNPBIOS_PROC_FS=y
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_DEV_XD is not set
CONFIG_BLK_CPQ_DA=y
# CONFIG_BLK_CPQ_CISS_DA is not set
CONFIG_BLK_DEV_DAC960=y
CONFIG_BLK_DEV_UMEM=y
# CONFIG_BLK_DEV_COW_COMMON is not set
# CONFIG_BLK_DEV_LOOP is not set
# CONFIG_BLK_DEV_NBD is not set
CONFIG_BLK_DEV_SX8=y
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
# CONFIG_BLK_DEV_XIP is not set
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set
CONFIG_MISC_DEVICES=y
CONFIG_IBM_ASM=y
CONFIG_PHANTOM=y
CONFIG_EEPROM_93CX6=y
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set
CONFIG_ACER_WMI=y
# CONFIG_FUJITSU_LAPTOP is not set
# CONFIG_TC1100_WMI is not set
CONFIG_MSI_LAPTOP=y
# CONFIG_SONY_LAPTOP is not set
CONFIG_THINKPAD_ACPI=y
CONFIG_THINKPAD_ACPI_DEBUG=y
CONFIG_THINKPAD_ACPI_BAY=y
CONFIG_THINKPAD_ACPI_VIDEO=y
CONFIG_THINKPAD_ACPI_HOTKEY_POLL=y
# CONFIG_EEEPC_LAPTOP is not set
CONFIG_ENCLOSURE_SERVICES=y
CONFIG_HAVE_IDE=y
#
# SCSI device support
#
CONFIG_RAID_ATTRS=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
CONFIG_SCSI_NETLINK=y
# CONFIG_SCSI_PROC_FS is not set
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=y
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not set
# CONFIG_CHR_DEV_SG is not set
CONFIG_CHR_DEV_SCH=y
CONFIG_SCSI_ENCLOSURE=y
#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
CONFIG_SCSI_MULTI_LUN=y
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
CONFIG_SCSI_FC_ATTRS=y
CONFIG_SCSI_ISCSI_ATTRS=y
CONFIG_SCSI_SAS_ATTRS=y
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
CONFIG_BLK_DEV_3W_XXXX_RAID=y
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_7000FASST is not set
CONFIG_SCSI_ACARD=y
# CONFIG_SCSI_AHA152X is not set
CONFIG_SCSI_AHA1542=y
# CONFIG_SCSI_AHA1740 is not set
CONFIG_SCSI_AACRAID=y
CONFIG_SCSI_AIC7XXX=y
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=5000
# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
CONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
CONFIG_SCSI_AIC79XX=y
CONFIG_AIC79XX_CMDS_PER_DEVICE=32
CONFIG_AIC79XX_RESET_DELAY_MS=5000
# CONFIG_AIC79XX_DEBUG_ENABLE is not set
CONFIG_AIC79XX_DEBUG_MASK=0
# CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
CONFIG_SCSI_IN2000=y
CONFIG_SCSI_ARCMSR=y
CONFIG_MEGARAID_NEWGEN=y
CONFIG_MEGARAID_MM=y
CONFIG_MEGARAID_MAILBOX=y
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
CONFIG_SCSI_HPTIOP=y
CONFIG_SCSI_BUSLOGIC=y
# CONFIG_SCSI_FLASHPOINT is not set
CONFIG_SCSI_DMX3191D=y
# CONFIG_SCSI_DTC3280 is not set
CONFIG_SCSI_EATA=y
CONFIG_SCSI_EATA_TAGGED_QUEUE=y
CONFIG_SCSI_EATA_LINKED_COMMANDS=y
CONFIG_SCSI_EATA_MAX_TAGS=16
CONFIG_SCSI_FUTURE_DOMAIN=y
CONFIG_SCSI_GDTH=y
CONFIG_SCSI_GENERIC_NCR5380=y
# CONFIG_SCSI_GENERIC_NCR5380_MMIO is not set
# CONFIG_SCSI_GENERIC_NCR53C400 is not set
CONFIG_SCSI_IBMMCA=y
CONFIG_IBMMCA_SCSI_ORDER_STANDARD=y
CONFIG_IBMMCA_SCSI_DEV_RESET=y
CONFIG_SCSI_IPS=y
CONFIG_SCSI_INITIO=y
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_PPA is not set
CONFIG_SCSI_IMM=y
CONFIG_SCSI_IZIP_EPP16=y
# CONFIG_SCSI_IZIP_SLOW_CTR is not set
# CONFIG_SCSI_NCR53C406A is not set
CONFIG_SCSI_NCR_D700=y
CONFIG_SCSI_STEX=y
CONFIG_SCSI_SYM53C8XX_2=y
CONFIG_SCSI_SYM53C8XX_DMA_ADDRESSING_MODE=1
CONFIG_SCSI_SYM53C8XX_DEFAULT_TAGS=16
CONFIG_SCSI_SYM53C8XX_MAX_TAGS=64
# CONFIG_SCSI_SYM53C8XX_MMIO is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_NCR_Q720 is not set
# CONFIG_SCSI_PAS16 is not set
CONFIG_SCSI_QLOGIC_FAS=y
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA_FC=y
CONFIG_SCSI_QLA_ISCSI=y
# CONFIG_SCSI_LPFC is not set
CONFIG_SCSI_SIM710=y
# CONFIG_SCSI_SYM53C416 is not set
CONFIG_SCSI_DC395x=y
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
CONFIG_SCSI_NSP32=y
# CONFIG_SCSI_SRP is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
# CONFIG_SATA_PMP is not set
CONFIG_SATA_AHCI=y
CONFIG_SATA_SIL24=y
CONFIG_ATA_SFF=y
CONFIG_SATA_SVW=y
CONFIG_ATA_PIIX=y
CONFIG_SATA_MV=y
# CONFIG_SATA_NV is not set
# CONFIG_PDC_ADMA is not set
CONFIG_SATA_QSTOR=y
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
CONFIG_SATA_VITESSE=y
# CONFIG_SATA_INIC162X is not set
# CONFIG_PATA_ACPI is not set
# CONFIG_PATA_ALI is not set
CONFIG_PATA_AMD=y
CONFIG_PATA_ARTOP=y
CONFIG_PATA_ATIIXP=y
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
CONFIG_PATA_CS5520=y
# CONFIG_PATA_CS5530 is not set
CONFIG_PATA_CS5535=y
CONFIG_PATA_CS5536=y
# CONFIG_PATA_CYPRESS is not set
CONFIG_PATA_EFAR=y
CONFIG_ATA_GENERIC=y
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
CONFIG_PATA_HPT3X3=y
# CONFIG_PATA_HPT3X3_DMA is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
CONFIG_PATA_JMICRON=y
# CONFIG_PATA_LEGACY is not set
# CONFIG_PATA_TRIFLEX is not set
CONFIG_PATA_MARVELL=y
CONFIG_PATA_MPIIX=y
CONFIG_PATA_OLDPIIX=y
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
CONFIG_PATA_NS87410=y
CONFIG_PATA_NS87415=y
CONFIG_PATA_OPTI=y
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC_OLD is not set
CONFIG_PATA_QDI=y
CONFIG_PATA_RADISYS=y
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
CONFIG_PATA_VIA=y
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_WINBOND_VLB is not set
CONFIG_PATA_PLATFORM=y
CONFIG_PATA_SCH=y
CONFIG_MD=y
# CONFIG_BLK_DEV_MD is not set
# CONFIG_BLK_DEV_DM is not set
CONFIG_FUSION=y
# CONFIG_FUSION_SPI is not set
# CONFIG_FUSION_FC is not set
CONFIG_FUSION_SAS=y
CONFIG_FUSION_MAX_SGE=128
# CONFIG_FUSION_CTL is not set
CONFIG_FUSION_LOGGING=y
#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
CONFIG_IEEE1394=y
#
# Subsystem Options
#
CONFIG_IEEE1394_VERBOSEDEBUG=y
#
# Controllers
#
#
# Texas Instruments PCILynx requires I2C
#
# CONFIG_IEEE1394_OHCI1394 is not set
#
# Protocols
#
# CONFIG_IEEE1394_SBP2 is not set
CONFIG_IEEE1394_ETH1394_ROM_ENTRY=y
CONFIG_IEEE1394_ETH1394=y
CONFIG_IEEE1394_RAWIO=y
CONFIG_I2O=y
# CONFIG_I2O_LCT_NOTIFY_ON_CHANGES is not set
# CONFIG_I2O_EXT_ADAPTEC is not set
CONFIG_I2O_CONFIG=y
CONFIG_I2O_CONFIG_OLD_IOCTL=y
CONFIG_I2O_BUS=y
# CONFIG_I2O_BLOCK is not set
CONFIG_I2O_SCSI=y
CONFIG_I2O_PROC=y
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
# CONFIG_NETDEVICES_MULTIQUEUE is not set
CONFIG_DUMMY=y
CONFIG_BONDING=y
# CONFIG_MACVLAN is not set
CONFIG_EQUALIZER=y
# CONFIG_TUN is not set
CONFIG_VETH=y
CONFIG_NET_SB1000=y
CONFIG_ARCNET=y
CONFIG_ARCNET_1201=y
# CONFIG_ARCNET_1051 is not set
CONFIG_ARCNET_RAW=y
# CONFIG_ARCNET_CAP is not set
CONFIG_ARCNET_COM90xx=y
CONFIG_ARCNET_COM90xxIO=y
# CONFIG_ARCNET_RIM_I is not set
# CONFIG_ARCNET_COM20020 is not set
# CONFIG_PHYLIB is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
CONFIG_NET_VENDOR_RACAL=y
CONFIG_NI52=y
CONFIG_NI65=y
CONFIG_NET_TULIP=y
# CONFIG_DE2104X is not set
CONFIG_TULIP=y
# CONFIG_TULIP_MWI is not set
CONFIG_TULIP_MMIO=y
# CONFIG_TULIP_NAPI is not set
CONFIG_DE4X5=y
CONFIG_WINBOND_840=y
# CONFIG_DM9102 is not set
CONFIG_ULI526X=y
CONFIG_AT1700=y
# CONFIG_DEPCA is not set
# CONFIG_HP100 is not set
CONFIG_NET_ISA=y
CONFIG_E2100=y
CONFIG_EWRK3=y
# CONFIG_EEXPRESS is not set
CONFIG_EEXPRESS_PRO=y
# CONFIG_HPLAN_PLUS is not set
# CONFIG_HPLAN is not set
# CONFIG_LP486E is not set
CONFIG_ETH16I=y
CONFIG_NE2000=y
# CONFIG_ZNET is not set
# CONFIG_SEEQ8005 is not set
CONFIG_IBMLANA=y
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
CONFIG_NET_PCI=y
CONFIG_PCNET32=y
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
CONFIG_AC3200=y
CONFIG_APRICOT=y
# CONFIG_B44 is not set
CONFIG_FORCEDETH=y
# CONFIG_FORCEDETH_NAPI is not set
# CONFIG_CS89x0 is not set
# CONFIG_EEPRO100 is not set
CONFIG_E100=y
CONFIG_LNE390=y
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
CONFIG_NE2K_PCI=y
CONFIG_NE3210=y
# CONFIG_ES3210 is not set
# CONFIG_8139CP is not set
CONFIG_8139TOO=y
# CONFIG_8139TOO_PIO is not set
CONFIG_8139TOO_TUNE_TWISTER=y
CONFIG_8139TOO_8129=y
CONFIG_8139_OLD_RX_RESET=y
# CONFIG_R6040 is not set
CONFIG_SIS900=y
# CONFIG_EPIC100 is not set
CONFIG_SUNDANCE=y
CONFIG_SUNDANCE_MMIO=y
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
CONFIG_SC92031=y
# CONFIG_NET_POCKET is not set
CONFIG_NETDEV_1000=y
CONFIG_ACENIC=y
CONFIG_ACENIC_OMIT_TIGON_I=y
CONFIG_DL2K=y
CONFIG_E1000=y
CONFIG_E1000_NAPI=y
# CONFIG_E1000_DISABLE_PACKET_SPLIT is not set
# CONFIG_E1000E_ENABLED is not set
CONFIG_IP1000=y
# CONFIG_IGB is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
CONFIG_R8169=y
# CONFIG_R8169_NAPI is not set
# CONFIG_SIS190 is not set
CONFIG_SKGE=y
# CONFIG_SKGE_DEBUG is not set
# CONFIG_SKY2 is not set
# CONFIG_VIA_VELOCITY is not set
CONFIG_TIGON3=y
CONFIG_BNX2=y
CONFIG_QLA3XXX=y
CONFIG_ATL1=y
CONFIG_NETDEV_10000=y
# CONFIG_CHELSIO_T1 is not set
CONFIG_CHELSIO_T3=y
CONFIG_IXGBE=y
CONFIG_IXGB=y
CONFIG_IXGB_NAPI=y
# CONFIG_S2IO is not set
CONFIG_MYRI10GE=y
# CONFIG_NETXEN_NIC is not set
# CONFIG_NIU is not set
# CONFIG_MLX4_CORE is not set
# CONFIG_TEHUTI is not set
# CONFIG_BNX2X is not set
# CONFIG_SFC is not set
CONFIG_TR=y
CONFIG_IBMTR=y
# CONFIG_IBMOL is not set
CONFIG_IBMLS=y
CONFIG_3C359=y
# CONFIG_TMS380TR is not set
# CONFIG_SMCTR is not set
#
# Wireless LAN
#
CONFIG_WLAN_PRE80211=y
CONFIG_STRIP=y
CONFIG_ARLAN=y
# CONFIG_WAVELAN is not set
CONFIG_WLAN_80211=y
CONFIG_IPW2100=y
# CONFIG_IPW2100_MONITOR is not set
CONFIG_IPW2100_DEBUG=y
# CONFIG_IPW2200 is not set
# CONFIG_LIBERTAS is not set
# CONFIG_AIRO is not set
# CONFIG_HERMES is not set
# CONFIG_ATMEL is not set
CONFIG_PRISM54=y
CONFIG_USB_ZD1201=y
# CONFIG_USB_NET_RNDIS_WLAN is not set
CONFIG_RTL8180=y
CONFIG_RTL8187=y
# CONFIG_ADM8211 is not set
CONFIG_P54_COMMON=y
# CONFIG_P54_USB is not set
# CONFIG_P54_PCI is not set
CONFIG_ATH5K=y
# CONFIG_ATH5K_DEBUG is not set
CONFIG_IWLWIFI=y
CONFIG_IWLCORE=y
CONFIG_IWLWIFI_LEDS=y
# CONFIG_IWLWIFI_RFKILL is not set
CONFIG_IWL4965=y
# CONFIG_IWL4965_HT is not set
CONFIG_IWL4965_LEDS=y
CONFIG_IWL4965_SPECTRUM_MEASUREMENT=y
# CONFIG_IWL4965_SENSITIVITY is not set
CONFIG_IWLWIFI_DEBUG=y
# CONFIG_IWLWIFI_DEBUGFS is not set
# CONFIG_IWL3945 is not set
CONFIG_HOSTAP=y
# CONFIG_HOSTAP_FIRMWARE is not set
# CONFIG_HOSTAP_PLX is not set
# CONFIG_HOSTAP_PCI is not set
CONFIG_B43=y
CONFIG_B43_PCI_AUTOSELECT=y
CONFIG_B43_PCICORE_AUTOSELECT=y
CONFIG_B43_LEDS=y
# CONFIG_B43_DEBUG is not set
CONFIG_B43LEGACY=y
CONFIG_B43LEGACY_PCI_AUTOSELECT=y
CONFIG_B43LEGACY_PCICORE_AUTOSELECT=y
CONFIG_B43LEGACY_LEDS=y
# CONFIG_B43LEGACY_DEBUG is not set
CONFIG_B43LEGACY_PIO=y
# CONFIG_B43LEGACY_DMA_AND_PIO_MODE is not set
# CONFIG_B43LEGACY_DMA_MODE is not set
CONFIG_B43LEGACY_PIO_MODE=y
CONFIG_ZD1211RW=y
CONFIG_ZD1211RW_DEBUG=y
#
# USB Network Adapters
#
CONFIG_USB_CATC=y
# CONFIG_USB_KAWETH is not set
CONFIG_USB_PEGASUS=y
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
CONFIG_WAN=y
CONFIG_LANMEDIA=y
# CONFIG_HDLC is not set
CONFIG_DLCI=y
CONFIG_DLCI_MAX=8
# CONFIG_SDLA is not set
CONFIG_SBNI=y
CONFIG_SBNI_MULTILINE=y
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
CONFIG_PPP=y
# CONFIG_PPP_MULTILINK is not set
# CONFIG_PPP_FILTER is not set
# CONFIG_PPP_ASYNC is not set
CONFIG_PPP_SYNC_TTY=y
# CONFIG_PPP_DEFLATE is not set
CONFIG_PPP_BSDCOMP=y
# CONFIG_PPP_MPPE is not set
CONFIG_PPPOE=y
CONFIG_PPPOL2TP=y
CONFIG_SLIP=y
CONFIG_SLIP_COMPRESSED=y
CONFIG_SLHC=y
# CONFIG_SLIP_SMART is not set
CONFIG_SLIP_MODE_SLIP6=y
# CONFIG_NET_FC is not set
CONFIG_NETCONSOLE=y
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_ISDN=y
CONFIG_ISDN_I4L=y
# CONFIG_ISDN_PPP is not set
# CONFIG_ISDN_AUDIO is not set
#
# ISDN feature submodules
#
# CONFIG_ISDN_DIVERSION is not set
#
# ISDN4Linux hardware drivers
#
#
# Passive cards
#
CONFIG_ISDN_DRV_HISAX=y
#
# D-channel protocol features
#
# CONFIG_HISAX_EURO is not set
CONFIG_HISAX_1TR6=y
CONFIG_HISAX_NI1=y
CONFIG_HISAX_MAX_CARDS=8
#
# HiSax supported cards
#
CONFIG_HISAX_16_0=y
CONFIG_HISAX_16_3=y
# CONFIG_HISAX_TELESPCI is not set
CONFIG_HISAX_S0BOX=y
CONFIG_HISAX_AVM_A1=y
# CONFIG_HISAX_FRITZPCI is not set
CONFIG_HISAX_AVM_A1_PCMCIA=y
# CONFIG_HISAX_ELSA is not set
CONFIG_HISAX_IX1MICROR2=y
CONFIG_HISAX_DIEHLDIVA=y
# CONFIG_HISAX_ASUSCOM is not set
# CONFIG_HISAX_TELEINT is not set
CONFIG_HISAX_HFCS=y
CONFIG_HISAX_SEDLBAUER=y
# CONFIG_HISAX_SPORTSTER is not set
CONFIG_HISAX_MIC=y
# CONFIG_HISAX_NETJET is not set
# CONFIG_HISAX_NETJET_U is not set
# CONFIG_HISAX_NICCY is not set
# CONFIG_HISAX_ISURF is not set
CONFIG_HISAX_HSTSAPHIR=y
CONFIG_HISAX_BKM_A4T=y
# CONFIG_HISAX_SCT_QUADRO is not set
# CONFIG_HISAX_GAZEL is not set
# CONFIG_HISAX_HFC_PCI is not set
CONFIG_HISAX_W6692=y
# CONFIG_HISAX_HFC_SX is not set
# CONFIG_HISAX_DEBUG is not set
#
# HiSax PCMCIA card service modules
#
#
# HiSax sub driver modules
#
CONFIG_HISAX_ST5481=y
# CONFIG_HISAX_HFCUSB is not set
# CONFIG_HISAX_HFC4S8S is not set
CONFIG_HISAX_FRITZ_PCIPNP=y
CONFIG_HISAX_HDLC=y
#
# Active cards
#
CONFIG_ISDN_DRV_PCBIT=y
# CONFIG_ISDN_DRV_SC is not set
CONFIG_ISDN_DRV_ACT2000=y
CONFIG_ISDN_DRV_GIGASET=y
CONFIG_GIGASET_BASE=y
# CONFIG_GIGASET_M105 is not set
CONFIG_GIGASET_M101=y
CONFIG_GIGASET_DEBUG=y
CONFIG_GIGASET_UNDOCREQ=y
# CONFIG_ISDN_CAPI is not set
# CONFIG_PHONE is not set
#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
CONFIG_INPUT_POLLDEV=y
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
CONFIG_INPUT_EVDEV=y
CONFIG_INPUT_EVBUG=y
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_KEYBOARD_SUNKBD=y
# CONFIG_KEYBOARD_LKKBD is not set
CONFIG_KEYBOARD_XTKBD=y
# CONFIG_KEYBOARD_NEWTON is not set
CONFIG_KEYBOARD_STOWAWAY=y
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
CONFIG_INPUT_TABLET=y
# CONFIG_TABLET_USB_ACECAD is not set
CONFIG_TABLET_USB_AIPTEK=y
CONFIG_TABLET_USB_GTCO=y
# CONFIG_TABLET_USB_KBTAB is not set
# CONFIG_TABLET_USB_WACOM is not set
CONFIG_INPUT_TOUCHSCREEN=y
# CONFIG_TOUCHSCREEN_FUJITSU is not set
CONFIG_TOUCHSCREEN_GUNZE=y
# CONFIG_TOUCHSCREEN_ELO is not set
CONFIG_TOUCHSCREEN_MTOUCH=y
CONFIG_TOUCHSCREEN_MK712=y
# CONFIG_TOUCHSCREEN_PENMOUNT is not set
# CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set
# CONFIG_TOUCHSCREEN_TOUCHWIN is not set
CONFIG_TOUCHSCREEN_UCB1400=y
CONFIG_TOUCHSCREEN_WM97XX=y
CONFIG_TOUCHSCREEN_WM9705=y
# CONFIG_TOUCHSCREEN_WM9712 is not set
# CONFIG_TOUCHSCREEN_WM9713 is not set
CONFIG_TOUCHSCREEN_USB_COMPOSITE=y
CONFIG_TOUCHSCREEN_USB_EGALAX=y
CONFIG_TOUCHSCREEN_USB_PANJIT=y
CONFIG_TOUCHSCREEN_USB_3M=y
# CONFIG_TOUCHSCREEN_USB_ITM is not set
CONFIG_TOUCHSCREEN_USB_ETURBO=y
# CONFIG_TOUCHSCREEN_USB_GUNZE is not set
CONFIG_TOUCHSCREEN_USB_DMC_TSC10=y
# CONFIG_TOUCHSCREEN_USB_IRTOUCH is not set
# CONFIG_TOUCHSCREEN_USB_IDEALTEK is not set
CONFIG_TOUCHSCREEN_USB_GENERAL_TOUCH=y
# CONFIG_TOUCHSCREEN_USB_GOTOP is not set
CONFIG_INPUT_MISC=y
# CONFIG_INPUT_PCSPKR is not set
# CONFIG_INPUT_WISTRON_BTNS is not set
CONFIG_INPUT_ATLAS_BTNS=y
# CONFIG_INPUT_ATI_REMOTE is not set
CONFIG_INPUT_ATI_REMOTE2=y
CONFIG_INPUT_KEYSPAN_REMOTE=y
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_UINPUT is not set
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
CONFIG_SERIO_CT82C710=y
# CONFIG_SERIO_PARKBD is not set
CONFIG_SERIO_PCIPS2=y
CONFIG_SERIO_LIBPS2=y
CONFIG_SERIO_RAW=y
CONFIG_GAMEPORT=y
CONFIG_GAMEPORT_NS558=y
CONFIG_GAMEPORT_L4=y
# CONFIG_GAMEPORT_EMU10K1 is not set
CONFIG_GAMEPORT_FM801=y
#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_DEVKMEM=y
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set
#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
# CONFIG_SERIAL_8250_PCI is not set
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set
CONFIG_SERIAL_8250_MCA=y
#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
# CONFIG_PRINTER is not set
CONFIG_PPDEV=y
CONFIG_HVC_DRIVER=y
# CONFIG_IPMI_HANDLER is not set
CONFIG_HW_RANDOM=y
CONFIG_HW_RANDOM_INTEL=y
CONFIG_HW_RANDOM_AMD=y
CONFIG_HW_RANDOM_GEODE=y
CONFIG_HW_RANDOM_VIA=y
CONFIG_NVRAM=y
CONFIG_RTC=y
CONFIG_DTLK=y
CONFIG_R3964=y
# CONFIG_APPLICOM is not set
CONFIG_SONYPI=y
# CONFIG_MWAVE is not set
CONFIG_SCx200_GPIO=y
CONFIG_PC8736x_GPIO=y
CONFIG_NSC_GPIO=y
# CONFIG_CS5535_GPIO is not set
CONFIG_RAW_DRIVER=y
CONFIG_MAX_RAW_DEVS=256
# CONFIG_HPET is not set
CONFIG_HANGCHECK_TIMER=y
CONFIG_TCG_TPM=y
# CONFIG_TCG_TIS is not set
# CONFIG_TCG_NSC is not set
# CONFIG_TCG_ATMEL is not set
CONFIG_TCG_INFINEON=y
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
# CONFIG_I2C is not set
# CONFIG_SPI is not set
CONFIG_W1=y
CONFIG_W1_CON=y
#
# 1-wire Bus Masters
#
# CONFIG_W1_MASTER_MATROX is not set
# CONFIG_W1_MASTER_DS2490 is not set
#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=y
# CONFIG_W1_SLAVE_SMEM is not set
# CONFIG_W1_SLAVE_DS2433 is not set
CONFIG_W1_SLAVE_DS2760=y
CONFIG_POWER_SUPPLY=y
CONFIG_POWER_SUPPLY_DEBUG=y
CONFIG_PDA_POWER=y
# CONFIG_BATTERY_DS2760 is not set
CONFIG_HWMON=y
# CONFIG_HWMON_VID is not set
# CONFIG_SENSORS_ABITUGURU is not set
CONFIG_SENSORS_ABITUGURU3=y
CONFIG_SENSORS_K8TEMP=y
# CONFIG_SENSORS_I5K_AMB is not set
CONFIG_SENSORS_F71805F=y
CONFIG_SENSORS_F71882FG=y
# CONFIG_SENSORS_CORETEMP is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_PC87360 is not set
CONFIG_SENSORS_PC87427=y
# CONFIG_SENSORS_SIS5595 is not set
CONFIG_SENSORS_SMSC47M1=y
# CONFIG_SENSORS_SMSC47B397 is not set
CONFIG_SENSORS_VIA686A=y
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83627HF is not set
# CONFIG_SENSORS_W83627EHF is not set
CONFIG_SENSORS_HDAPS=y
CONFIG_SENSORS_APPLESMC=y
# CONFIG_HWMON_DEBUG_CHIP is not set
CONFIG_THERMAL=y
# CONFIG_WATCHDOG is not set
#
# Sonics Silicon Backplane
#
CONFIG_SSB_POSSIBLE=y
CONFIG_SSB=y
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
CONFIG_SSB_B43_PCI_BRIDGE=y
# CONFIG_SSB_SILENT is not set
# CONFIG_SSB_DEBUG is not set
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y
#
# Multifunction device drivers
#
CONFIG_MFD_SM501=y
# CONFIG_HTC_PASIC3 is not set
#
# Multimedia devices
#
#
# Multimedia core support
#
# CONFIG_VIDEO_DEV is not set
# CONFIG_DVB_CORE is not set
# CONFIG_VIDEO_MEDIA is not set
#
# Multimedia drivers
#
CONFIG_DAB=y
CONFIG_USB_DABUSB=y
#
# Graphics support
#
# CONFIG_AGP is not set
# CONFIG_DRM is not set
# CONFIG_VGASTATE is not set
CONFIG_VIDEO_OUTPUT_CONTROL=y
# CONFIG_FB is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
# CONFIG_LCD_CLASS_DEVICE is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=y
# CONFIG_BACKLIGHT_CORGI is not set
CONFIG_BACKLIGHT_PROGEAR=y
#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
# CONFIG_VIDEO_SELECT is not set
CONFIG_DUMMY_CONSOLE=y
CONFIG_FONT_8x16=y
#
# Sound
#
CONFIG_SOUND=y
#
# Advanced Linux Sound Architecture
#
# CONFIG_SND is not set
#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=y
# CONFIG_HID_SUPPORT is not set
CONFIG_USB_MOUSE=y
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set
#
# Miscellaneous USB options
#
# CONFIG_USB_DEVICEFS is not set
CONFIG_USB_DEVICE_CLASS=y
CONFIG_USB_DYNAMIC_MINORS=y
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
CONFIG_USB_OTG_WHITELIST=y
# CONFIG_USB_OTG_BLACKLIST_HUB is not set
#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
CONFIG_USB_ISP116X_HCD=y
# CONFIG_USB_ISP1760_HCD is not set
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_OHCI_HCD_SSB is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_U132_HCD=y
# CONFIG_USB_SL811_HCD is not set
CONFIG_USB_R8A66597_HCD=y
#
# USB Device Class drivers
#
CONFIG_USB_ACM=y
CONFIG_USB_PRINTER=y
# CONFIG_USB_WDM is not set
#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#
#
# may also be needed; see USB_STORAGE Help for more information
#
# CONFIG_USB_STORAGE is not set
# CONFIG_USB_LIBUSUAL is not set
#
# USB Imaging devices
#
CONFIG_USB_MDC800=y
# CONFIG_USB_MICROTEK is not set
CONFIG_USB_MON=y
#
# USB port drivers
#
# CONFIG_USB_USS720 is not set
# CONFIG_USB_SERIAL is not set
#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=y
CONFIG_USB_EMI26=y
# CONFIG_USB_ADUTUX is not set
CONFIG_USB_AUERSWALD=y
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
CONFIG_USB_LED=y
CONFIG_USB_CYPRESS_CY7C63=y
CONFIG_USB_CYTHERM=y
CONFIG_USB_PHIDGET=y
# CONFIG_USB_PHIDGETKIT is not set
CONFIG_USB_PHIDGETMOTORCONTROL=y
# CONFIG_USB_PHIDGETSERVO is not set
# CONFIG_USB_IDMOUSE is not set
CONFIG_USB_FTDI_ELAN=y
# CONFIG_USB_APPLEDISPLAY is not set
CONFIG_USB_SISUSBVGA=y
CONFIG_USB_SISUSBVGA_CON=y
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
CONFIG_USB_IOWARRIOR=y
CONFIG_USB_GADGET=y
# CONFIG_USB_GADGET_DEBUG is not set
CONFIG_USB_GADGET_DEBUG_FILES=y
# CONFIG_USB_GADGET_DEBUG_FS is not set
CONFIG_USB_GADGET_SELECTED=y
# CONFIG_USB_GADGET_AMD5536UDC is not set
# CONFIG_USB_GADGET_ATMEL_USBA is not set
# CONFIG_USB_GADGET_FSL_USB2 is not set
# CONFIG_USB_GADGET_NET2280 is not set
# CONFIG_USB_GADGET_PXA2XX is not set
# CONFIG_USB_GADGET_M66592 is not set
# CONFIG_USB_GADGET_PXA27X is not set
# CONFIG_USB_GADGET_GOKU is not set
# CONFIG_USB_GADGET_LH7A40X is not set
# CONFIG_USB_GADGET_OMAP is not set
# CONFIG_USB_GADGET_S3C2410 is not set
# CONFIG_USB_GADGET_AT91 is not set
CONFIG_USB_GADGET_DUMMY_HCD=y
CONFIG_USB_DUMMY_HCD=y
CONFIG_USB_GADGET_DUALSPEED=y
CONFIG_USB_ZERO=y
# CONFIG_USB_ETH is not set
# CONFIG_USB_GADGETFS is not set
# CONFIG_USB_FILE_STORAGE is not set
# CONFIG_USB_G_SERIAL is not set
# CONFIG_USB_MIDI_GADGET is not set
# CONFIG_USB_G_PRINTER is not set
# CONFIG_MMC is not set
CONFIG_MEMSTICK=y
CONFIG_MEMSTICK_DEBUG=y
#
# MemoryStick drivers
#
CONFIG_MEMSTICK_UNSAFE_RESUME=y
# CONFIG_MSPRO_BLOCK is not set
#
# MemoryStick Host Controller Drivers
#
# CONFIG_MEMSTICK_TIFM_MS is not set
# CONFIG_MEMSTICK_JMICRON_38X is not set
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y
#
# LED drivers
#
# CONFIG_LEDS_NET48XX is not set
CONFIG_LEDS_WRAP=y
# CONFIG_LEDS_CLEVO_MAIL is not set
#
# LED Triggers
#
CONFIG_LEDS_TRIGGERS=y
# CONFIG_LEDS_TRIGGER_TIMER is not set
CONFIG_LEDS_TRIGGER_HEARTBEAT=y
# CONFIG_LEDS_TRIGGER_DEFAULT_ON is not set
# CONFIG_ACCESSIBILITY is not set
CONFIG_INFINIBAND=y
CONFIG_INFINIBAND_USER_MAD=y
CONFIG_INFINIBAND_USER_ACCESS=y
CONFIG_INFINIBAND_USER_MEM=y
CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_MTHCA=y
# CONFIG_INFINIBAND_MTHCA_DEBUG is not set
CONFIG_INFINIBAND_AMSO1100=y
# CONFIG_INFINIBAND_AMSO1100_DEBUG is not set
# CONFIG_INFINIBAND_CXGB3 is not set
# CONFIG_MLX4_INFINIBAND is not set
# CONFIG_INFINIBAND_NES is not set
CONFIG_INFINIBAND_IPOIB=y
CONFIG_INFINIBAND_IPOIB_CM=y
# CONFIG_INFINIBAND_IPOIB_DEBUG is not set
# CONFIG_INFINIBAND_SRP is not set
CONFIG_INFINIBAND_ISER=y
# CONFIG_EDAC is not set
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set
CONFIG_AUXDISPLAY=y
# CONFIG_KS0108 is not set
CONFIG_UIO=y
CONFIG_UIO_CIF=y
# CONFIG_UIO_SMX is not set
#
# Firmware Drivers
#
CONFIG_EDD=y
CONFIG_EDD_OFF=y
CONFIG_EFI_VARS=y
# CONFIG_DELL_RBU is not set
CONFIG_DCDBAS=y
# CONFIG_DMIID is not set
CONFIG_ISCSI_IBFT_FIND=y
# CONFIG_ISCSI_IBFT is not set
#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
# CONFIG_EXT2_FS_POSIX_ACL is not set
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
# CONFIG_EXT4DEV_FS is not set
CONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
# CONFIG_REISERFS_FS_XATTR is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
CONFIG_XFS_FS=y
CONFIG_XFS_QUOTA=y
# CONFIG_XFS_POSIX_ACL is not set
CONFIG_XFS_RT=y
# CONFIG_XFS_DEBUG is not set
# CONFIG_OCFS2_FS is not set
# CONFIG_DNOTIFY is not set
# CONFIG_INOTIFY is not set
CONFIG_QUOTA=y
# CONFIG_QUOTA_NETLINK_INTERFACE is not set
CONFIG_PRINT_QUOTA_WARNING=y
CONFIG_QFMT_V1=y
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_AUTOFS_FS=y
CONFIG_AUTOFS4_FS=y
# CONFIG_FUSE_FS is not set
#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
# CONFIG_VFAT_FS is not set
CONFIG_FAT_DEFAULT_CODEPAGE=437
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
# CONFIG_PROC_SYSCTL is not set
CONFIG_SYSFS=y
# CONFIG_TMPFS is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_CONFIGFS_FS=y
#
# Miscellaneous filesystems
#
CONFIG_ADFS_FS=y
CONFIG_ADFS_FS_RW=y
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
CONFIG_HFSPLUS_FS=y
CONFIG_BEFS_FS=y
CONFIG_BEFS_DEBUG=y
CONFIG_BFS_FS=y
CONFIG_EFS_FS=y
CONFIG_CRAMFS=y
# CONFIG_VXFS_FS is not set
CONFIG_MINIX_FS=y
CONFIG_HPFS_FS=y
CONFIG_QNX4FS_FS=y
CONFIG_ROMFS_FS=y
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_NETWORK_FILESYSTEMS is not set
#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
CONFIG_ACORN_PARTITION=y
# CONFIG_ACORN_PARTITION_CUMANA is not set
# CONFIG_ACORN_PARTITION_EESOX is not set
# CONFIG_ACORN_PARTITION_ICS is not set
CONFIG_ACORN_PARTITION_ADFS=y
CONFIG_ACORN_PARTITION_POWERTEC=y
CONFIG_ACORN_PARTITION_RISCIX=y
CONFIG_OSF_PARTITION=y
# CONFIG_AMIGA_PARTITION is not set
CONFIG_ATARI_PARTITION=y
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
CONFIG_LDM_PARTITION=y
# CONFIG_LDM_DEBUG is not set
CONFIG_SGI_PARTITION=y
CONFIG_ULTRIX_PARTITION=y
CONFIG_SUN_PARTITION=y
# CONFIG_KARMA_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
CONFIG_SYSV68_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_CODEPAGE_737=y
CONFIG_NLS_CODEPAGE_775=y
CONFIG_NLS_CODEPAGE_850=y
CONFIG_NLS_CODEPAGE_852=y
CONFIG_NLS_CODEPAGE_855=y
# CONFIG_NLS_CODEPAGE_857 is not set
CONFIG_NLS_CODEPAGE_860=y
CONFIG_NLS_CODEPAGE_861=y
CONFIG_NLS_CODEPAGE_862=y
CONFIG_NLS_CODEPAGE_863=y
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
CONFIG_NLS_CODEPAGE_869=y
# CONFIG_NLS_CODEPAGE_936 is not set
CONFIG_NLS_CODEPAGE_950=y
CONFIG_NLS_CODEPAGE_932=y
# CONFIG_NLS_CODEPAGE_949 is not set
CONFIG_NLS_CODEPAGE_874=y
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
# CONFIG_NLS_ISO8859_1 is not set
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=y
CONFIG_NLS_KOI8_R=y
CONFIG_NLS_KOI8_U=y
CONFIG_NLS_UTF8=y
# CONFIG_DLM is not set
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
# CONFIG_DETECT_SOFTLOCKUP is not set
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
CONFIG_RT_MUTEX_TESTER=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
CONFIG_LOCKDEP=y
# CONFIG_LOCK_STAT is not set
CONFIG_DEBUG_LOCKDEP=y
CONFIG_TRACE_IRQFLAGS=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
CONFIG_DEBUG_LOCKING_API_SELFTESTS=y
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_BUGVERBOSE is not set
# CONFIG_DEBUG_VM is not set
CONFIG_DEBUG_WRITECOUNT=y
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
CONFIG_FRAME_POINTER=y
CONFIG_BOOT_PRINTK_DELAY=y
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_FAULT_INJECTION is not set
CONFIG_LATENCYTOP=y
CONFIG_HAVE_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_TRACING=y
# CONFIG_FTRACE is not set
CONFIG_IRQSOFF_TRACER=y
# CONFIG_SYSPROF_TRACER is not set
CONFIG_SCHED_TRACER=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_FTRACE_SELFTEST=y
CONFIG_FTRACE_STARTUP_TEST=y
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
# CONFIG_NONPROMISC_DEVMEM is not set
CONFIG_EARLY_PRINTK=y
# CONFIG_DEBUG_STACKOVERFLOW is not set
CONFIG_DEBUG_STACK_USAGE=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_DEBUG_PER_CPU_MAPS=y
CONFIG_X86_PTDUMP=y
CONFIG_DEBUG_RODATA=y
CONFIG_DEBUG_RODATA_TEST=y
CONFIG_4KSTACKS=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_DOUBLEFAULT=y
CONFIG_MMIOTRACE_HOOKS=y
CONFIG_MMIOTRACE=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
# CONFIG_IO_DELAY_0X80 is not set
# CONFIG_IO_DELAY_0XED is not set
CONFIG_IO_DELAY_UDELAY=y
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=2
# CONFIG_DEBUG_BOOT_PARAMS is not set
CONFIG_CPA_DEBUG=y
#
# Security options
#
CONFIG_KEYS=y
# CONFIG_KEYS_DEBUG_PROC_KEYS is not set
CONFIG_SECURITY=y
# CONFIG_SECURITY_NETWORK is not set
CONFIG_SECURITY_CAPABILITIES=y
CONFIG_SECURITY_FILE_CAPABILITIES=y
CONFIG_SECURITY_DEFAULT_MMAP_MIN_ADDR=0
CONFIG_CRYPTO=y
#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_GF128MUL=y
# CONFIG_CRYPTO_NULL is not set
CONFIG_CRYPTO_CRYPTD=y
CONFIG_CRYPTO_AUTHENC=y
#
# Authenticated Encryption with Associated Data
#
CONFIG_CRYPTO_CCM=y
CONFIG_CRYPTO_GCM=y
CONFIG_CRYPTO_SEQIV=y
#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=y
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=y
# CONFIG_CRYPTO_PCBC is not set
CONFIG_CRYPTO_XTS=y
#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set
#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
CONFIG_CRYPTO_MD4=y
CONFIG_CRYPTO_MD5=y
# CONFIG_CRYPTO_MICHAEL_MIC is not set
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=y
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
CONFIG_CRYPTO_WP512=y
#
# Ciphers
#
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_586 is not set
CONFIG_CRYPTO_ANUBIS=y
CONFIG_CRYPTO_ARC4=y
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
CONFIG_CRYPTO_CAST6=y
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_FCRYPT=y
CONFIG_CRYPTO_KHAZAD=y
CONFIG_CRYPTO_SALSA20=y
CONFIG_CRYPTO_SALSA20_586=y
# CONFIG_CRYPTO_SEED is not set
CONFIG_CRYPTO_SERPENT=y
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_TWOFISH is not set
CONFIG_CRYPTO_TWOFISH_COMMON=y
CONFIG_CRYPTO_TWOFISH_586=y
#
# Compression
#
CONFIG_CRYPTO_DEFLATE=y
# CONFIG_CRYPTO_LZO is not set
# CONFIG_CRYPTO_HW is not set
CONFIG_HAVE_KVM=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y
# CONFIG_KVM_INTEL is not set
# CONFIG_KVM_AMD is not set
CONFIG_KVM_TRACE=y
CONFIG_LGUEST=y
# CONFIG_VIRTIO_PCI is not set
# CONFIG_VIRTIO_BALLOON is not set
#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
CONFIG_CRC7=y
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_FORCE_SUCCESSFUL_BUILD=y
CONFIG_FORCE_MINIMAL_CONFIG=y
CONFIG_FORCE_MINIMAL_CONFIG_PHYS=y
CONFIG_X86_32_ALWAYS_ON=y
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 14:12 ` Ingo Molnar
@ 2008-05-26 14:17 ` Ingo Molnar
2008-05-26 14:29 ` Ingo Molnar
2008-05-26 14:43 ` Ilpo Järvinen
2008-05-26 14:58 ` Ilpo Järvinen
1 sibling, 2 replies; 96+ messages in thread
From: Ingo Molnar @ 2008-05-26 14:17 UTC (permalink / raw)
To: Ilpo J?rvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
* Ingo Molnar <mingo@elte.hu> wrote:
> i also ran this for 15 minutes:
>
> [root@europe ~]# tcpdump src and dst 10.0.1.15 -n
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
>
> but there was no output at all - i.e. TCP has not attempted to do
> anything to these connections.
... which is not surprising as it traced eth0 ;-) [10.0.1.15 is the IP
of the box and routes back to loopback]
with the proper tcpdump it gave:
[root@europe ~]# tcpdump -i lo
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes
16:15:05.260000 IP europe.37198 > europe.distcc: . ack 2022529163 win 257 <nop,nop,timestamp 209526 197526>
16:15:05.260000 IP europe.distcc > europe.37198: . ack 1 win 0 <nop,nop,timestamp 209526 7020>
hm, a window of 257 and zero, is that right?
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 14:17 ` Ingo Molnar
@ 2008-05-26 14:29 ` Ingo Molnar
2008-05-26 14:43 ` Ilpo Järvinen
1 sibling, 0 replies; 96+ messages in thread
From: Ingo Molnar @ 2008-05-26 14:29 UTC (permalink / raw)
To: Ilpo J?rvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
there's TCP ACK activity every 120 seconds:
16:19:05.260000 IP europe.37198 > europe.distcc: . ack 2022529163 win 257
<nop,nop,timestamp 233526 221526>
16:19:05.260000 IP europe.distcc > europe.37198: . ack 1 win 0
<nop,nop,timestamp 233526 7020>
16:21:05.260000 IP europe.37198 > europe.distcc: . ack 1 win 257
<nop,nop,timestamp 245526 233526>
16:21:05.260000 IP europe.distcc > europe.37198: . ack 1 win 0
<nop,nop,timestamp 245526 7020>
16:23:05.260000 IP europe.37198 > europe.distcc: . ack 1 win 257
<nop,nop,timestamp 257526 245526>
16:23:05.260000 IP europe.distcc > europe.37198: . ack 1 win 0
<nop,nop,timestamp 257526 7020>
16:25:05.260000 IP europe.37198 > europe.distcc: . ack 1 win 257
<nop,nop,timestamp 269526 257526>
16:25:05.260000 IP europe.distcc > europe.37198: . ack 1 win 0
<nop,nop,timestamp 269526 7020>
16:27:05.260000 IP europe.37198 > europe.distcc: . ack 1 win 257
<nop,nop,timestamp 281526 269526>
16:27:05.260000 IP europe.distcc > europe.37198: . ack 1 win 0
<nop,nop,timestamp 281526 7020>
The socket on the other side of the localhost connection,
10.0.1.15:3632, produces no packets.
a detail i forgot to mention: i reproduced the same TCP hang on a 32-bit
system and on a 64-bit system as well.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 14:17 ` Ingo Molnar
2008-05-26 14:29 ` Ingo Molnar
@ 2008-05-26 14:43 ` Ilpo Järvinen
1 sibling, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-26 14:43 UTC (permalink / raw)
To: Ingo Molnar
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
On Mon, 26 May 2008, Ingo Molnar wrote:
>
> * Ingo Molnar <mingo@elte.hu> wrote:
>
> > i also ran this for 15 minutes:
> >
> > [root@europe ~]# tcpdump src and dst 10.0.1.15 -n
> > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> > listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
> >
> > but there was no output at all - i.e. TCP has not attempted to do
> > anything to these connections.
>
> ... which is not surprising as it traced eth0 ;-) [10.0.1.15 is the IP
> of the box and routes back to loopback]
Thanks, it's much easier to diagnose right with symptoms that are based
on reality :-).
> with the proper tcpdump it gave:
>
> [root@europe ~]# tcpdump -i lo
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes
> 16:15:05.260000 IP europe.37198 > europe.distcc: . ack 2022529163 win 257 <nop,nop,timestamp 209526 197526>
> 16:15:05.260000 IP europe.distcc > europe.37198: . ack 1 win 0 <nop,nop,timestamp 209526 7020>
>
> hm, a window of 257 and zero, is that right?
...Yes, though that 257 is lacks the effect of window scaling (and it's
mostly irrelevant anyway since the problem occurs in opposite dir). The
sender is using just window probes (/proc/net/tcp shows that too), that's
why you see them occassionally. So the main problem seems to be that
receiver doesn't read, rest is just a consequence of that (from TCP POV).
I doubt that tcp_frto will affect to that (and I read what a stale
retrans_stamp could do, it would hard cause any troubles, and those minor
things wouldn't realize in your setup anyway).
I'll take a look into pre -rc1 changes now but it may well be beoynd
what I'm able to figure out.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 14:12 ` Ingo Molnar
2008-05-26 14:17 ` Ingo Molnar
@ 2008-05-26 14:58 ` Ilpo Järvinen
2008-05-26 16:23 ` Ingo Molnar
1 sibling, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-26 14:58 UTC (permalink / raw)
To: Ingo Molnar
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
On Mon, 26 May 2008, Ingo Molnar wrote:
> there's a hung distcc task on the system, waiting for socket action
> forever:
>
> [root@europe ~]# strace -fp 19578
> Process 19578 attached - interrupt to quit
> select(5, NULL, [4], [4], {82, 90000} <unfinished ...>
Hmm, readfds is NULL isn't it?!? Are you sure you straced the right
process?
> disturbing that task via strace did not change the state of the socket -
> and that's not unexpected as it's a select(). [TCP state might be
> affected if strace impacted a recvmsg or a sendmsg wait directly.]
I fail to understand this paragraph due to excessive negation... :-)
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 14:58 ` Ilpo Järvinen
@ 2008-05-26 16:23 ` Ingo Molnar
2008-05-26 16:32 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-05-26 16:23 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
* Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> On Mon, 26 May 2008, Ingo Molnar wrote:
>
> > there's a hung distcc task on the system, waiting for socket action
> > forever:
> >
> > [root@europe ~]# strace -fp 19578
> > Process 19578 attached - interrupt to quit
> > select(5, NULL, [4], [4], {82, 90000} <unfinished ...>
>
> Hmm, readfds is NULL isn't it?!? Are you sure you straced the right
> process?
yes, i'm stracing the task that is hung unexpectedly.
> > disturbing that task via strace did not change the state of the
> > socket - and that's not unexpected as it's a select(). [TCP state
> > might be affected if strace impacted a recvmsg or a sendmsg wait
> > directly.]
>
> I fail to understand this paragraph due to excessive negation... :-)
i mean, sometimes a TCP connection can get 'unstuck' if you strace a
task - that is because the TCP related syscall the task sits in gets
interrupted. But in this case it's select() which doesnt explicitly take
the socket, doesnt do any tcp_push_pending_frames() processing, etc. -
it just its on the socket waitqueue AFAICS. And that's expected.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 11:56 [bug] stuck localhost TCP connections, v2.6.26-rc3+ Ingo Molnar
2008-05-26 13:28 ` Ilpo Järvinen
@ 2008-05-26 16:24 ` Arjan van de Ven
2008-05-28 9:27 ` Peter Zijlstra
2008-05-29 8:45 ` Ingo Molnar
3 siblings, 0 replies; 96+ messages in thread
From: Arjan van de Ven @ 2008-05-26 16:24 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton
On Mon, 26 May 2008 13:56:28 +0200
Ingo Molnar <mingo@elte.hu> wrote:
>
> in an overnight -tip testruns that is based on recent -git i got two
> stuck TCP connections:
>
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign
> Address State tcp 0 174592
> 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED
> tcp 72134 0 10.0.1.14:3632
> 10.0.1.14:58015 ESTABLISHED
>
> on a previously reliable machine. That connection has been stuck for
> 9 hours so it does not time out, etc. - and the distcc run that goes
> over that connection is stuck as well.
>
interestingly enough, I think I'm seeing something very similar to this
with the claws email client; as of 2.6.26-rc3 (sorry, didn't run
earlier ones because they broke my wireless) claws gets "stuck" once in
a while, really stuck, when talking to the imap server.....
It seems to happen a little less frequently when running under strace,
but it still happens, so that could just be a timing thing.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 16:23 ` Ingo Molnar
@ 2008-05-26 16:32 ` Ilpo Järvinen
2008-05-26 16:54 ` Ingo Molnar
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-26 16:32 UTC (permalink / raw)
To: Ingo Molnar
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1862 bytes --]
On Mon, 26 May 2008, Ingo Molnar wrote:
>
> * Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
>
> > On Mon, 26 May 2008, Ingo Molnar wrote:
> >
> > > there's a hung distcc task on the system, waiting for socket action
> > > forever:
> > >
> > > [root@europe ~]# strace -fp 19578
> > > Process 19578 attached - interrupt to quit
> > > select(5, NULL, [4], [4], {82, 90000} <unfinished ...>
> >
> > Hmm, readfds is NULL isn't it?!? Are you sure you straced the right
> > process?
>
> yes, i'm stracing the task that is hung unexpectedly.
But that wasn't the receiving process? (I didn't quickly find into which
direction distcc ports go, so I couldn't confirm this). If you still have
that situation at hand, could you check which is the receiving process
(e.g., using netstat -p, the end which has Recv-Q is the right one) and
where it's stuck?
> > > disturbing that task via strace did not change the state of the
> > > socket - and that's not unexpected as it's a select(). [TCP state
> > > might be affected if strace impacted a recvmsg or a sendmsg wait
> > > directly.]
> >
> > I fail to understand this paragraph due to excessive negation... :-)
>
> i mean, sometimes a TCP connection can get 'unstuck' if you strace a
> task - that is because the TCP related syscall the task sits in gets
> interrupted. But in this case it's select() which doesnt explicitly take
> the socket, doesnt do any tcp_push_pending_frames() processing, etc. -
> it just its on the socket waitqueue AFAICS. And that's expected.
This is not in the sender end at all. It's correct behavior of the flow
control to stop the sender until more room is made available by the
reading end. Thus push_pending_frames couldn't send anything.
...It may still be that the receiving process is stuck due to the non-net
related changes you have there.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 16:32 ` Ilpo Järvinen
@ 2008-05-26 16:54 ` Ingo Molnar
2008-05-26 17:08 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-05-26 16:54 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
* Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> > > Hmm, readfds is NULL isn't it?!? Are you sure you straced the
> > > right process?
> >
> > yes, i'm stracing the task that is hung unexpectedly.
>
> But that wasn't the receiving process? (I didn't quickly find into
> which direction distcc ports go, so I couldn't confirm this). If you
> still have that situation at hand, could you check which is the
> receiving process (e.g., using netstat -p, the end which has Recv-Q is
> the right one) and where it's stuck?
it wasnt the receiving process. There's no receiving process - which is
weird:
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 207232 europe:37198 europe:distcc ESTABLISHED 19578/distcc
tcp 0 0 europe:ssh dione:36284 ESTABLISHED -
tcp 0 0 europe:ssh e2:45910 ESTABLISHED -
tcp 72283 0 europe:distcc europe:37198 ESTABLISHED -
> > i mean, sometimes a TCP connection can get 'unstuck' if you strace a
> > task - that is because the TCP related syscall the task sits in gets
> > interrupted. But in this case it's select() which doesnt explicitly
> > take the socket, doesnt do any tcp_push_pending_frames() processing,
> > etc. - it just its on the socket waitqueue AFAICS. And that's
> > expected.
>
> This is not in the sender end at all. It's correct behavior of the
> flow control to stop the sender until more room is made available by
> the reading end. Thus push_pending_frames couldn't send anything.
i just gave it as a general example of why sometimes stracing a task can
'disturb' the observed system and can kick the TCP state machine out of
a stall. I did not say it's occuring here.
> ...It may still be that the receiving process is stuck due to the
> non-net related changes you have there.
the socket does not seem to be owned. It should have closed down?
Refcounting issue?
find below the sysrq-t dump.
Ingo
-------------------------->
[11811.540000] SysRq : Show State
[11811.540000] task PC stack pid father
[11811.540000] init S f7c67b04 0 1 0
[11811.540000] f7c70000 00000086 00000002 f7c67b04 f7c67b0c 00000000 00000046 f7ca08b8
[11811.540000] f7c67b2c 00119272 00000002 80130c53 00000000 01ca0000 f7c67b08 f7c70164
[11811.540000] 82cbb080 0011907f 00000000 00000000 00000000 00000000 000000ff f7c67b2c
[11811.540000] Call Trace:
[11811.540000] [<80130c53>] internal_add_timer+0x103/0x110
[11811.540000] [<806ab7da>] schedule_timeout+0x4a/0xd0
[11811.540000] [<801310e0>] process_timeout+0x0/0x40
[11811.540000] [<806ab7d5>] schedule_timeout+0x45/0xd0
[11811.540000] [<8019868c>] do_select+0x3dc/0x500
[11811.540000] [<801982b0>] do_select+0x0/0x500
[11811.540000] [<80198d60>] __pollwait+0x0/0xe0
[11811.540000] [<80120070>] default_wake_function+0x0/0x10
[11811.540000] [<801852b6>] check_bytes_and_report+0x26/0xd0
[11811.540000] [<80184f6e>] slab_pad_check+0x7e/0x100
[11811.540000] [<80185695>] check_object+0xe5/0x1f0
[11811.540000] [<80185e08>] __slab_free+0x238/0x2e0
[11811.540000] [<801202e4>] __wake_up+0x24/0x60
[11811.540000] [<801e6717>] journal_stop+0x1a7/0x2a0
[11811.540000] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540000] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540000] [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540000] [<801987d1>] core_sys_select+0x21/0x350
[11811.540000] [<802b0e1c>] selinux_inode_getattr+0x5c/0x70
[11811.540000] [<80185e08>] __slab_free+0x238/0x2e0
[11811.540000] [<802e62af>] copy_to_user+0x3f/0x70
[11811.540000] [<8018e3e9>] cp_new_stat64+0xf9/0x110
[11811.540000] [<80198f27>] sys_select+0xe7/0x1b0
[11811.540000] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540000] =======================
[11811.540000] kthreadd S f7c69fac 0 2 0
[11811.540000] f7c70ed0 00000096 00000002 f7c69fac f7c69fb4 00000000 f7ca8000 00000003
[11811.540000] 00000246 00000000 00000000 808e2c50 00000000 01000246 f7c69fb0 f7c71034
[11811.540000] 82c68080 ffffcb74 00000000 00000000 00000000 00000000 000000ff 808e2c20
[11811.540000] Call Trace:
[11811.540000] [<8013b551>] kthreadd+0x151/0x160
[11811.540000] [<8013b400>] kthreadd+0x0/0x160
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] migration/0 S 82c68090 0 3 2
[11811.540000] f7c71da0 00000086 00000000 82c68090 f7c71da0 00000046 00000000 00000001
[11811.540000] 00000046 00000000 00000000 82c68090 00000000 01000046 00000046 f7c71f04
[11811.540000] 82c68080 82c68090 00000002 806b1e00 00000001 82c68080 82c6d620 82c687f8
[11811.540000] Call Trace:
[11811.540000] [<80124931>] migration_thread+0x141/0x220
[11811.540000] [<801247f0>] migration_thread+0x0/0x220
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] ksoftirqd/0 S f7c6bfa4 0 4 2
[11811.540000] f7c72c70 00000082 00000002 f7c6bfa4 f7c6bfac 00000000 f7c6b000 802e5b64
[11811.540000] 02223000 00000000 808dd400 80103f3c 00000000 01c68080 f7c6bfa8 f7c72dd4
[11811.540000] 82c68080 00119094 00000000 00000000 00000000 00000000 000000ff 00000000
[11811.540000] Call Trace:
[11811.540000] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540000] [<80103f3c>] restore_nocheck_notrace+0x0/0xe
[11811.540000] [<8012dbd5>] ksoftirqd+0xf5/0x100
[11811.540000] [<8012dae0>] ksoftirqd+0x0/0x100
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] watchdog/0 S f7c6cf8c 0 5 2
[11811.540000] f7c73b40 00000092 00000002 f7c6cf8c f7c6cf94 00000000 0011908c 00979c85
[11811.540000] 00000abe 82c68880 ffff0600 00000abd 00000000 01140dab f7c6cf90 f7c73ca4
[11811.540000] 82c68080 0011908d 00000000 00000000 00000000 00000000 000000ff eb400ed0
[11811.540000] Call Trace:
[11811.540000] [<8015d375>] watchdog+0x65/0x1d0
[11811.540000] [<8015d310>] watchdog+0x0/0x1d0
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] migration/1 S 82cbb090 0 6 2
[11811.540000] f7c74a10 00000086 00000000 82cbb090 f7c74a10 00000046 00000000 00000001
[11811.540000] 00000046 00000000 00000000 82cbb090 00000000 01000046 00000046 f7c74b74
[11811.540000] 82cbb080 82cbb090 00000002 00000000 00000001 82cbb080 82c1a620 82cbb7f8
[11811.540000] Call Trace:
[11811.540000] [<80124931>] migration_thread+0x141/0x220
[11811.540000] [<801247f0>] migration_thread+0x0/0x220
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] ksoftirqd/1 S f7c7afa4 0 7 2
[11811.540000] f7c758e0 00000082 00000002 f7c7afa4 f7c7afac 00000000 80a44e80 80a44e80
[11811.540000] f7c7a000 802e5b64 f7ca8000 00000001 00000000 01103f3c f7c7afa8 f7c75a44
[11811.540000] 82cbb080 00119094 00000000 00000000 00000000 00000000 000000ff 00000001
[11811.540000] Call Trace:
[11811.540000] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540000] [<8012dbd5>] ksoftirqd+0xf5/0x100
[11811.540000] [<8012dae0>] ksoftirqd+0x0/0x100
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] watchdog/1 S f7c7bf8c 0 8 2
[11811.540000] f7c767b0 00000092 00000002 f7c7bf8c f7c7bf94 00000000 0011908d 0097de40
[11811.540000] 00000abe 82cbb880 00979c80 00000abe 00000000 01140dab f7c7bf90 f7c76914
[11811.540000] 82cbb080 0011908e 00000000 00000000 00000000 00000000 000000ff fffffffc
[11811.540000] Call Trace:
[11811.540000] [<8015d375>] watchdog+0x65/0x1d0
[11811.540000] [<8015d310>] watchdog+0x0/0x1d0
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] events/0 S f7c7ef90 0 9 2
[11811.540000] f7ca8ed0 00000096 00000002 f7c7ef90 f7c7ef98 00000000 f7c11d48 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 f7c7ef94 f7ca9034
[11811.540000] 82c68080 00119084 00000000 00000000 00000000 00000000 000000ff f7c11d10
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] events/1 S f7c7ff90 0 10 2
[11811.540000] f7ca9da0 00000096 00000002 f7c7ff90 f7c7ff98 00000000 f7c11df8 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 f7c7ff94 f7ca9f04
[11811.540000] 82cbb080 001190a7 00000000 00000000 00000000 00000000 000000ff f7c11dc0
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] khelper S 82c1a620 0 11 2
[11811.540000] f7caac70 00000096 00000afb 82c1a620 00000002 f7c88f8c f7c11ea8 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 f7c88f94 f7caadd4
[11811.540000] 82c68080 00114d91 00000000 00000000 00000000 00000afb 000000ff f7c11e70
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] kblockd/0 S 00000046 0 183 2
[11811.540000] f7cabb40 00000096 00000046 00000046 00000000 00000000 f7c8d508 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f7cabca4
[11811.540000] 82c68080 00000002 00000001 00000001 f7c8d4f8 00000286 801388e0 f7c8d4d0
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] kblockd/1 S 00000046 0 184 2
[11811.540000] f7caca10 00000096 00000046 00000046 00000000 00000000 f7c8d5b8 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f7cacb74
[11811.540000] 82cbb080 00000002 00000001 00000001 f7c8d5a8 00000286 801388e0 f7c8d580
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] kacpid S 00000046 0 185 2
[11811.540000] f7cad8e0 00000096 00000046 00000046 00000000 00000000 f7c8d668 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f7cada44
[11811.540000] 82cbb080 00000000 00000001 00000001 f7c8d658 00000286 801388e0 f7c8d630
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] [<8063e9ec>] ip6_mc_add_src+0x7c/0x2c0
[11811.540000] =======================
[11811.540000] kacpi_notify S f7ce5f90 0 186 2
[11811.540000] f7cae7b0 00000096 00000002 f7ce5f90 f7ce5f98 00000000 f7c8d7c8 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 f7ce5f94 f7cae914
[11811.540000] 82cbb080 ffff8b1f 00000000 00000000 00000000 00000000 000000ff f7c8d790
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] ata/0 S f7d7cf90 0 276 2
[11811.540000] f7db0000 00000096 00000002 f7d7cf90 f7d7cf98 00000000 f7d47b38 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 f7d7cf94 f7db0164
[11811.540000] 82c68080 ffff8df0 00000000 00000000 00000000 00000000 000000ff f7d47b00
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] ata/1 S 00000046 0 277 2
[11811.540000] f7db0ed0 00000096 00000046 00000046 00000000 00000000 f7d47be8 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f7db1034
[11811.540000] 82cbb080 00000002 00000001 00000001 f7d47bd8 00000286 801388e0 f7d47bb0
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] ata_aux S f7d7ef90 0 278 2
[11811.540000] f7db1da0 00000096 00000002 f7d7ef90 f7d7ef98 00000000 f7d47c98 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 f7d7ef94 f7db1f04
[11811.540000] 82cbb080 ffff8db2 00000000 00000000 00000000 00000000 000000ff f7d47c60
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] ksuspend_usbd S 00000046 0 280 2
[11811.540000] f7db2c70 00000096 00000046 00000046 00000000 00000000 f7d47df8 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f7db2dd4
[11811.540000] 82cbb080 00000002 00000001 00000001 f7d47de8 00000286 801388e0 f7d47dc0
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] khubd S 00000046 0 286 2
[11811.540000] f7db3b40 00000086 00000046 00000046 00000000 00000000 80918b70 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000292 f7db3ca4
[11811.540000] 82c68080 00000002 00000001 00000001 80918b60 00000292 f6ae3810 80918b40
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<804ecdcd>] hub_thread+0x1dd/0x1060
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<804ecbf0>] hub_thread+0x0/0x1060
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] kseriod S f7d84f80 0 289 2
[11811.540000] f7db4a10 00000092 00000002 f7d84f80 f7d84f88 00000000 8094f8b0 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 f7d84f84 f7db4b74
[11811.540000] 82c68080 ffff8e59 00000000 00000000 00000000 00000000 000000ff 8094f880
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<805292b8>] serio_thread+0xa8/0x2d0
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<80529210>] serio_thread+0x0/0x2d0
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] kondemand/0 S f6dbef90 0 316 2
[11811.540000] f7db58e0 00000096 00000002 f6dbef90 f6dbef98 00000000 f7d85b38 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6dbef94 f7db5a44
[11811.540000] 82c68080 ffffcc75 00000000 00000000 00000000 00000000 000000ff f7d85b00
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] kondemand/1 S f6dbff90 0 317 2
[11811.540000] f7db67b0 00000096 00000002 f6dbff90 f6dbff98 00000000 f7d85be8 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6dbff94 f7db6914
[11811.540000] 82cbb080 ffffcc75 00000000 00000000 00000000 00000000 000000ff f7d85bb0
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] pageattr-test S f67fdef0 0 340 2
[11811.540000] f7cb0ed0 00000092 00000002 f67fdef0 f67fdef8 00000000 00000046 80b6a678
[11811.540000] f67fdf18 00119af6 00000002 80130c53 00000000 01b69d80 f67fdef4 f7cb1034
[11811.540000] 82c68080 00118f3b 00000000 00000000 00000000 00000000 000000ff f67fdf18
[11811.540000] Call Trace:
[11811.540000] [<80130c53>] internal_add_timer+0x103/0x110
[11811.540000] [<806ab7da>] schedule_timeout+0x4a/0xd0
[11811.540000] [<8011ae25>] print_split+0x85/0x1b0
[11811.540000] [<8011b7c8>] change_page_attr_set_clr+0xc8/0x190
[11811.540000] [<801310e0>] process_timeout+0x0/0x40
[11811.540000] [<806ab7d5>] schedule_timeout+0x45/0xd0
[11811.540000] [<8011b8b0>] do_pageattr_test+0x20/0x4e0
[11811.540000] [<8011b890>] do_pageattr_test+0x0/0x4e0
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] rt-test-0 S f7cb67b0 0 346 2
[11811.540000] f7cb67b0 00000096 00000000 f7cb67b0 806ab08b 00000046 00000000 00000000
[11811.540000] f6df0a94 00000046 00000046 00000046 00000000 0112c7fe f6df0a94 f7cb6914
[11811.540000] 82c68080 00000001 f6df0a84 00000000 8014cef0 00000000 806ad940 00000001
[11811.540000] Call Trace:
[11811.540000] [<806ab08b>] schedule+0x25b/0x670
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<8014cf49>] test_func+0x59/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] rt-test-1 S f7cb58e0 0 348 2
[11811.540000] f7cb58e0 00000096 00000000 f7cb58e0 806ab08b 00000046 00000000 00000000
[11811.540000] f6df1014 00000046 00000046 00000046 00000000 0112c7fe f6df1014 f7cb5a44
[11811.540000] 82c68080 00000001 f6df1004 00000000 8014cef0 00000000 806ad940 00000001
[11811.540000] Call Trace:
[11811.540000] [<806ab08b>] schedule+0x25b/0x670
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<8014cf49>] test_func+0x59/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] rt-test-2 S f7cb4a10 0 350 2
[11811.540000] f7cb4a10 00000096 00000000 f7cb4a10 806ab08b 00000046 00000000 00000000
[11811.540000] f6df1594 00000046 00000046 00000046 00000000 0112c7fe f6df1594 f7cb4b74
[11811.540000] 82c68080 00000001 f6df1584 00000000 8014cef0 00000000 806ad940 00000001
[11811.540000] Call Trace:
[11811.540000] [<806ab08b>] schedule+0x25b/0x670
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<8014cf49>] test_func+0x59/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] rt-test-3 S f7cb3b40 0 352 2
[11811.540000] f7cb3b40 00000096 00000000 f7cb3b40 806ab08b 00000046 00000000 00000000
[11811.540000] f6df1b14 00000046 00000046 00000046 00000000 0112c7fe f6df1b14 f7cb3ca4
[11811.540000] 82c68080 00000001 f6df1b04 00000000 8014cef0 00000000 806ad940 00000001
[11811.540000] Call Trace:
[11811.540000] [<806ab08b>] schedule+0x25b/0x670
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<8014cf49>] test_func+0x59/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] rt-test-4 S f7cb2c70 0 354 2
[11811.540000] f7cb2c70 00000096 00000000 f7cb2c70 806ab08b 00000046 00000000 00000000
[11811.540000] f6df2094 00000046 00000046 00000046 00000000 0112c7fe f6df2094 f7cb2dd4
[11811.540000] 82c68080 00000001 f6df2084 00000000 8014cef0 00000000 806ad940 00000001
[11811.540000] Call Trace:
[11811.540000] [<806ab08b>] schedule+0x25b/0x670
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<8014cf49>] test_func+0x59/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] rt-test-5 S f7cb1da0 0 356 2
[11811.540000] f7cb1da0 00000096 00000000 f7cb1da0 806ab08b 00000046 00000000 00000000
[11811.540000] f6df2614 00000046 00000046 00000046 00000000 0112c7fe f6df2614 f7cb1f04
[11811.540000] 82c68080 00000001 f6df2604 00000000 8014cef0 00000000 806ad940 00000001
[11811.540000] Call Trace:
[11811.540000] [<806ab08b>] schedule+0x25b/0x670
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<8014cf49>] test_func+0x59/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] rt-test-6 S f7cb0000 0 358 2
[11811.540000] f7cb0000 00000096 00000000 f7cb0000 806ab08b 00000046 00000000 00000000
[11811.540000] f6df2b94 00000046 00000046 00000046 00000000 0112c7fe f6df2b94 f7cb0164
[11811.540000] 82c68080 00000001 f6df2b84 00000000 8014cef0 00000000 806ad940 00000001
[11811.540000] Call Trace:
[11811.540000] [<806ab08b>] schedule+0x25b/0x670
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<8014cf49>] test_func+0x59/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] rt-test-7 S f6df8000 0 360 2
[11811.540000] f6df8000 00000096 00000000 f6df8000 806ab08b 00000046 00000000 00000000
[11811.540000] f6df3114 00000046 00000046 00000046 00000000 0112c7fe f6df3114 f6df8164
[11811.540000] 82c68080 00000001 f6df3104 00000000 8014cef0 00000000 806ad940 00000001
[11811.540000] Call Trace:
[11811.540000] [<806ab08b>] schedule+0x25b/0x670
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540000] [<8014cef0>] test_func+0x0/0x90
[11811.540000] [<8014cf49>] test_func+0x59/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] pdflush S f6743f90 0 362 2
[11811.540000] f6df8ed0 00000096 00000002 f6743f90 f6743f98 00000000 f6df8ed0 80124aa3
[11811.540000] 00000046 00000000 00000000 808e6db0 00000000 01000046 f6743f94 f6df9034
[11811.540000] 82cbb080 ffff8b1f 00000000 00000000 00000000 00000000 000000ff f6743fc0
[11811.540000] Call Trace:
[11811.540000] [<80124aa3>] set_cpus_allowed_ptr+0x53/0xd0
[11811.540000] [<8016f790>] pdflush+0x0/0x1a0
[11811.540000] [<8016f842>] pdflush+0xb2/0x1a0
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] pdflush S f6751f90 0 363 2
[11811.540000] f6df9da0 00000096 00000002 f6751f90 f6751f98 00000000 00000000 00000296
[11811.540000] 00000046 00000000 00000000 808e6db0 00000000 01000046 f6751f94 f6df9f04
[11811.540000] 82cbb080 00118f23 00000000 00000000 00000000 00000000 000000ff f6751fc0
[11811.540000] Call Trace:
[11811.540000] [<8016f790>] pdflush+0x0/0x1a0
[11811.540000] [<8016f842>] pdflush+0xb2/0x1a0
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] kswapd0 S f67f9f28 0 364 2
[11811.540000] f6dfac70 00000082 00000002 f67f9f28 f67f9f30 00000000 808e6c54 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 f67f9f2c f6dfadd4
[11811.540000] 82cbb080 ffff8b1f 00000000 00000000 00000000 00000000 000000ff 808e4c00
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<80172c80>] kswapd+0x0/0x400
[11811.540000] [<8017304a>] kswapd+0x3ca/0x400
[11811.540000] [<806ab08b>] schedule+0x25b/0x670
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<80172c80>] kswapd+0x0/0x400
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] aio/0 S 00000046 0 451 2
[11811.540000] f6dfbb40 00000096 00000046 00000046 00000000 00000000 f6dc59d8 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f6dfbca4
[11811.540000] 82c68080 00000002 00000001 00000001 f6dc59c8 00000286 801388e0 f6dc59a0
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] aio/1 S 00000046 0 452 2
[11811.540000] f6dfca10 00000096 00000046 00000046 00000000 00000000 f6dc5a88 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f6dfcb74
[11811.540000] 82cbb080 00000002 00000001 00000001 f6dc5a78 00000286 801388e0 f6dc5a50
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] jfsIO S 808ea3d0 0 476 2
[11811.540000] f6dfd8e0 00000092 00000000 808ea3d0 f6dfd8e0 00000002 82c68080 f7c70ed0
[11811.540000] 00000046 00000000 00000000 808ea3d0 00000000 01000046 00000046 f6dfda44
[11811.540000] 82c68080 808ea3d0 00000002 80248770 00000001 808ea3c0 00000000 00000000
[11811.540000] Call Trace:
[11811.540000] [<80248770>] jfsIOWait+0x0/0x70
[11811.540000] [<80248770>] jfsIOWait+0x0/0x70
[11811.540000] [<802487c4>] jfsIOWait+0x54/0x70
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] jfsCommit S 00000046 0 477 2
[11811.540000] f6dfe7b0 00000092 80d7d174 00000046 00000046 00000000 00000000 808ea490
[11811.540000] 00000046 00000000 00000001 00000001 00000000 01000046 808ea480 f6dfe914
[11811.540000] 82c68080 00000002 00000000 00000001 80d7d164 00000286 8024c150 fffffffc
[11811.540000] Call Trace:
[11811.540000] [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000] [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000] [<8024c304>] jfs_lazycommit+0x1b4/0x200
[11811.540000] [<80120070>] default_wake_function+0x0/0x10
[11811.540000] [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] jfsCommit S 00000046 0 478 2
[11811.540000] f6780000 00000092 80d7d174 00000046 00000046 00000000 00000000 808ea490
[11811.540000] 00000046 00000000 00000001 00000001 00000000 01000046 808ea480 f6780164
[11811.540000] 82c68080 00000002 00000000 00000001 80d7d164 00000286 8024c150 fffffffc
[11811.540000] Call Trace:
[11811.540000] [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000] [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000] [<8024c304>] jfs_lazycommit+0x1b4/0x200
[11811.540000] [<80120070>] default_wake_function+0x0/0x10
[11811.540000] [<8024c150>] jfs_lazycommit+0x0/0x200
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] jfsSync S f6776f9c 0 479 2
[11811.540000] f6780ed0 00000096 00000002 f6776f9c f6776fa4 00000000 00000000 f6780ed0
[11811.540000] 00000246 00000000 00000000 808ea470 00000000 01000246 f6776fa0 f6781034
[11811.540000] 82c68080 ffff8b4b 00000000 00000000 00000000 00000000 000000ff fffffffc
[11811.540000] Call Trace:
[11811.540000] [<8024b370>] jfs_sync+0x0/0x1c0
[11811.540000] [<8024b49a>] jfs_sync+0x12a/0x1c0
[11811.540000] [<8024b370>] jfs_sync+0x0/0x1c0
[11811.540000] [<8024b370>] jfs_sync+0x0/0x1c0
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] ocfs2_wq S 00000046 0 482 2
[11811.540000] f6781da0 00000096 00000046 00000046 00000000 00000000 f6773c98 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f6781f04
[11811.540000] 82c68080 00000002 00000001 00000001 f6773c88 00000286 801388e0 f6773c60
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] fakephp S 00000046 0 488 2
[11811.540000] f6782c70 00000096 00000046 00000046 00000000 00000000 f676b5b8 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000286 f6782dd4
[11811.540000] 82c68080 00000002 00000001 00000001 f676b5a8 00000286 801388e0 f676b580
[11811.540000] Call Trace:
[11811.540000] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013895a>] worker_thread+0x7a/0x90
[11811.540000] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540000] [<801388e0>] worker_thread+0x0/0x90
[11811.540000] [<8013b3d2>] kthread+0x42/0x70
[11811.540000] [<8013b390>] kthread+0x0/0x70
[11811.540000] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540000] =======================
[11811.540000] netxen S f6e9ff90 0 1672 2
[11811.540000] f6783b40 00000096 00000002 f6e9ff90 f6e9ff98 00000000 f6f20b38 00000046
[11811.540000] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6e9ff94 f6783ca4
[11811.540001] 82cbb080 ffff8d29 00000000 00000000 00000000 00000000 000000ff f6f20b00
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013895a>] worker_thread+0x7a/0x90
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] sfc_refill/0 S f67b1f90 0 1675 2
[11811.540001] f6784a10 00000096 00000002 f67b1f90 f67b1f98 00000000 f6f20c98 00000046
[11811.540001] 00000000 00000001 00000001 8013b87c 00000000 01000046 f67b1f94 f6784b74
[11811.540001] 82c68080 ffff8d1f 00000000 00000000 00000000 00000000 000000ff f6f20c60
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013895a>] worker_thread+0x7a/0x90
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] sfc_refill/1 S f6ea6f90 0 1676 2
[11811.540001] f67858e0 00000096 00000002 f6ea6f90 f6ea6f98 00000000 f6f20d48 00000046
[11811.540001] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6ea6f94 f6785a44
[11811.540001] 82cbb080 ffff8d29 00000000 00000000 00000000 00000000 000000ff f6f20d10
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013895a>] worker_thread+0x7a/0x90
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] scsi_eh_0 S 00000000 0 1697 2
[11811.540001] f67867b0 00000086 00000046 00000000 00000046 00000001 8048944d f6fe8884
[11811.540001] 00000002 00000286 00000001 f6fe8874 00000000 01fe8830 00000000 f6786914
[11811.540001] 82c68080 00000000 80489479 00000286 00000292 f6fe8830 8048d230 00000292
[11811.540001] Call Trace:
[11811.540001] [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001] [<80489479>] __scsi_iterate_devices+0x49/0x70
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] scsi_eh_1 S f6774f64 0 1701 2
[11811.540001] f6a88000 00000086 00000002 f6774f64 f6774f6c 00000000 8048944d f6fe90b4
[11811.540001] 00000002 00000286 00000001 f6fe90a4 00000000 01fe9060 f6774f68 f6a88164
[11811.540001] 82cbb080 ffff8db3 00000000 00000000 00000000 00000000 000000ff 00000292
[11811.540001] Call Trace:
[11811.540001] [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] scsi_eh_2 S f67b7f64 0 1703 2
[11811.540001] f6a88ed0 00000086 00000002 f67b7f64 f67b7f6c 00000000 8048944d f6fe98e4
[11811.540001] 00000002 00000286 00000001 f6fe98d4 00000000 01fe9890 f67b7f68 f6a89034
[11811.540001] 82cbb080 ffff8db3 00000000 00000000 00000000 00000000 000000ff 00000292
[11811.540001] Call Trace:
[11811.540001] [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] scsi_eh_3 S f6772f64 0 1705 2
[11811.540001] f6a89da0 00000086 00000002 f6772f64 f6772f6c 00000000 8048944d f6fea114
[11811.540001] 00000002 00000286 00000001 f6fea104 00000000 01fea0c0 f6772f68 f6a89f04
[11811.540001] 82cbb080 ffff8db3 00000000 00000000 00000000 00000000 000000ff 00000292
[11811.540001] Call Trace:
[11811.540001] [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] scsi_eh_4 S 00000000 0 1716 2
[11811.540001] f6a8ac70 00000086 00000046 00000000 00000046 00000001 8048944d f6feb174
[11811.540001] 00000002 00000286 00000001 f6feb164 00000000 01feb120 f6fec180 f6a8add4
[11811.540001] 82c68080 f6fec188 80489484 00000286 f6fec180 f6feb120 8048d230 00000292
[11811.540001] Call Trace:
[11811.540001] [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001] [<80489484>] __scsi_iterate_devices+0x54/0x70
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] scsi_eh_5 S f6f3af64 0 1718 2
[11811.540001] f6a8bb40 00000086 00000002 f6f3af64 f6f3af6c 00000000 8048944d f6feb9a4
[11811.540001] 00000002 00000286 00000001 f6feb994 00000000 01feb950 f6f3af68 f6a8bca4
[11811.540001] 82cbb080 ffff8def 00000000 00000000 00000000 00000000 000000ff 00000292
[11811.540001] Call Trace:
[11811.540001] [<8048944d>] __scsi_iterate_devices+0x1d/0x70
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8048d281>] scsi_error_handler+0x51/0x2e0
[11811.540001] [<8048d230>] scsi_error_handler+0x0/0x2e0
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] w1_control S f6a09f4c 0 1852 2
[11811.540001] f6a8ca10 00000096 00000002 f6a09f4c f6a09f54 00000000 00000046 f7ca0678
[11811.540001] f6a09f74 001190ca 00000002 80130c53 00000000 01ca0000 f6a09f50 f6a8cb74
[11811.540001] 82cbb080 00119066 00000000 00000000 00000000 00000000 000000ff f6a09f74
[11811.540001] Call Trace:
[11811.540001] [<80130c53>] internal_add_timer+0x103/0x110
[11811.540001] [<8053f990>] w1_control+0x0/0x1c0
[11811.540001] [<806ab7da>] schedule_timeout+0x4a/0xd0
[11811.540001] [<806ab08b>] schedule+0x25b/0x670
[11811.540001] [<801310e0>] process_timeout+0x0/0x40
[11811.540001] [<806ab7d5>] schedule_timeout+0x45/0xd0
[11811.540001] [<80131576>] msleep_interruptible+0x26/0x40
[11811.540001] [<8053f9be>] w1_control+0x2e/0x1c0
[11811.540001] [<8053f990>] w1_control+0x0/0x1c0
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] kmemstick S f6a02f90 0 1875 2
[11811.540001] f6a8d8e0 00000096 00000002 f6a02f90 f6a02f98 00000000 f6ac5f58 00000046
[11811.540001] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6a02f94 f6a8da44
[11811.540001] 82cbb080 ffff8e68 00000000 00000000 00000000 00000000 000000ff f6ac5f20
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013895a>] worker_thread+0x7a/0x90
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] ib_mcast S 82c6d620 0 1881 2
[11811.540001] f6a8e7b0 00000096 00000818 82c6d620 00000002 f6ae7f8c f6a08248 00000046
[11811.540001] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6ae7f94 f6a8e914
[11811.540001] 82cbb080 ffff8e68 00000000 00000000 00000000 00000818 000000ff f6a08210
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013895a>] worker_thread+0x7a/0x90
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] ib_cm/0 S f6afcf90 0 1883 2
[11811.540001] f6a20000 00000096 00000002 f6afcf90 f6afcf98 00000000 f6a083a8 00000046
[11811.540001] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6afcf94 f6a20164
[11811.540001] 82c68080 ffff8e59 00000000 00000000 00000000 00000000 000000ff f6a08370
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013895a>] worker_thread+0x7a/0x90
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] ib_cm/1 S 82c6d620 0 1884 2
[11811.540001] f6a20ed0 00000096 00000818 82c6d620 00000002 f6adef8c f6a08458 00000046
[11811.540001] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6adef94 f6a21034
[11811.540001] 82cbb080 ffff8e69 00000000 00000000 00000000 00000818 000000ff f6a08420
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013895a>] worker_thread+0x7a/0x90
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] iw_cm_wq S f6adbf90 0 1885 2
[11811.540001] f6a21da0 00000096 00000002 f6adbf90 f6adbf98 00000000 f6a08508 00000046
[11811.540001] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6adbf94 f6a21f04
[11811.540001] 82cbb080 ffff8e69 00000000 00000000 00000000 00000000 000000ff f6a084d0
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013895a>] worker_thread+0x7a/0x90
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] ib_addr S f6adaf90 0 1886 2
[11811.540001] f6a22c70 00000096 00000002 f6adaf90 f6adaf98 00000000 f6a08668 00000046
[11811.540001] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6adaf94 f6a22dd4
[11811.540001] 82cbb080 00118f4d 00000000 00000000 00000000 00000000 000000ff f6a08630
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013895a>] worker_thread+0x7a/0x90
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] rdma_cm S f6ec2f90 0 1887 2
[11811.540001] f6a23b40 00000096 00000002 f6ec2f90 f6ec2f98 00000000 f6a087c8 00000046
[11811.540001] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6ec2f94 f6a23ca4
[11811.540001] 82cbb080 ffff8e69 00000000 00000000 00000000 00000000 000000ff f6a08790
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013895a>] worker_thread+0x7a/0x90
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] mthca_catas S f6ec0f90 0 1888 2
[11811.540001] f6a24a10 00000096 00000002 f6ec0f90 f6ec0f98 00000000 f6a08928 00000046
[11811.540001] 00000000 00000001 00000001 8013b87c 00000000 01000046 f6ec0f94 f6a24b74
[11811.540001] 82cbb080 ffff8e69 00000000 00000000 00000000 00000000 000000ff f6a088f0
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013895a>] worker_thread+0x7a/0x90
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801388e0>] worker_thread+0x0/0x90
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] kirqd S f664df20 0 1909 2
[11811.540001] f6a258e0 00000096 00000002 f664df20 f664df28 00000000 00000046 f7ca08b0
[11811.540001] f664df48 00119180 00000002 80130c53 00000000 01ca0000 f664df24 f6a25a44
[11811.540001] 82cbb080 00118f8d 00000000 00000000 00000000 00000000 000000ff f664df48
[11811.540001] Call Trace:
[11811.540001] [<80130c53>] internal_add_timer+0x103/0x110
[11811.540001] [<806ab7da>] schedule_timeout+0x4a/0xd0
[11811.540001] [<801310e0>] process_timeout+0x0/0x40
[11811.540001] [<806ab7d5>] schedule_timeout+0x45/0xd0
[11811.540001] [<80114143>] balanced_irq+0x73/0x4e0
[11811.540001] [<801140d0>] balanced_irq+0x0/0x4e0
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] kjournald S 00000046 0 1913 2
[11811.540001] f6a267b0 00000082 00000046 00000046 00000000 00000000 f6afb37c 00000046
[11811.540001] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000246 f6a26914
[11811.540001] 82cbb080 00000002 00000001 00000001 00000001 f6afb254 f6afb254 f6afb240
[11811.540001] Call Trace:
[11811.540001] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540001] [<801ec9e5>] kjournald+0x2f5/0x340
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801ec6f0>] kjournald+0x0/0x340
[11811.540001] [<8013b3d2>] kthread+0x42/0x70
[11811.540001] [<8013b390>] kthread+0x0/0x70
[11811.540001] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540001] =======================
[11811.540001] udevd S f66efb04 0 1977 1
[11811.540001] f6a61da0 00000086 00000002 f66efb04 f66efb0c 00000000 f7824380 00001000
[11811.540001] 00000046 00000046 00000000 f6dd1190 00000000 01000046 f66efb08 f6a61f04
[11811.540001] 82cbb080 00114d79 00000000 00000000 00000000 00000000 000000ff 00000000
[11811.540001] Call Trace:
[11811.540001] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540001] [<80190af2>] pipe_poll+0x32/0xb0
[11811.540001] [<8019868c>] do_select+0x3dc/0x500
[11811.540001] [<801982b0>] do_select+0x0/0x500
[11811.540001] [<80198d60>] __pollwait+0x0/0xe0
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<801d5c3e>] ext3_get_blocks_handle+0x8e/0xa60
[11811.540001] [<801ad1b9>] __find_get_block_slow+0x49/0x110
[11811.540001] [<801ad1d1>] __find_get_block_slow+0x61/0x110
[11811.540001] [<801ad55d>] __find_get_block+0x9d/0x190
[11811.540001] [<801d5c3e>] ext3_get_blocks_handle+0x8e/0xa60
[11811.540001] [<801e1257>] ext3fs_dirhash+0x107/0x210
[11811.540001] [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540001] [<801987d1>] core_sys_select+0x21/0x350
[11811.540001] [<80185e08>] __slab_free+0x238/0x2e0
[11811.540001] [<801202e4>] __wake_up+0x24/0x60
[11811.540001] [<8013b8d3>] remove_wait_queue+0x13/0x40
[11811.540001] [<8012b335>] do_wait+0x1d5/0xa80
[11811.540001] [<8011a07b>] do_page_fault+0xcb/0x710
[11811.540001] [<80198e8b>] sys_select+0x4b/0x1b0
[11811.540001] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001] =======================
[11811.540001] rsyslogd R running 0 5059 1
[11811.540001] f62e4a10 00000086 00000002 808e3e00 00000002 f6e8ce20 00000046 000000df
[11811.540001] f6e8cae8 00000000 00000000 f784ad28 00000000 0184ad18 00000046 f62e4b74
[11811.540001] 82cbb080 00000000 00000000 f784ad28 00000046 00000000 00000001 00000000
[11811.540001] Call Trace:
[11811.540001] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540001] [<805b7e87>] datagram_poll+0x17/0x100
[11811.540001] [<8019868c>] do_select+0x3dc/0x500
[11811.540001] [<801982b0>] do_select+0x0/0x500
[11811.540001] [<80198d60>] __pollwait+0x0/0xe0
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<8012098e>] hrtick_set+0x2e/0x120
[11811.540001] [<801209c9>] hrtick_set+0x69/0x120
[11811.540001] [<801852b6>] check_bytes_and_report+0x26/0xd0
[11811.540001] [<80184f32>] slab_pad_check+0x42/0x100
[11811.540001] [<80185695>] check_object+0xe5/0x1f0
[11811.540001] [<80185e08>] __slab_free+0x238/0x2e0
[11811.540001] [<805b7f7a>] skb_free_datagram+0xa/0x30
[11811.540001] [<80621254>] unix_dgram_recvmsg+0x184/0x2c0
[11811.540001] [<80178642>] handle_mm_fault+0x532/0x5b0
[11811.540001] [<805aef4c>] sock_recvmsg+0x13c/0x190
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540001] [<801987d1>] core_sys_select+0x21/0x350
[11811.540001] [<8018c4c3>] fget_light+0x53/0xe0
[11811.540001] [<805ae922>] sockfd_lookup_light+0x32/0x60
[11811.540001] [<805b02d4>] sys_recvfrom+0x154/0x160
[11811.540001] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540001] [<8014a2d6>] futex_wake+0xc6/0xf0
[11811.540001] [<8015ed90>] handle_edge_irq+0x0/0x120
[11811.540001] [<80107035>] do_IRQ+0x85/0xe0
[11811.540001] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540001] [<80103f3c>] restore_nocheck_notrace+0x0/0xe
[11811.540001] [<80198e8b>] sys_select+0x4b/0x1b0
[11811.540001] [<802e62af>] copy_to_user+0x3f/0x70
[11811.540001] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001] =======================
[11811.540001] rsyslogd R running 0 5061 1
[11811.540001] f62e1da0 00000082 00000000 00000046 00000046 00000000 00000000 f6e75e30
[11811.540001] 00000046 00000000 00000001 00000001 00000000 01000046 f6e75e20 f62e1f04
[11811.540001] 82cbb080 00000002 00000000 00000001 f6e75e20 00000282 00000000 80d4c0cc
[11811.540001] Call Trace:
[11811.540001] [<8014a175>] futex_wait+0x345/0x3b0
[11811.540001] [<806ad970>] _read_unlock_irq+0x20/0x30
[11811.540001] [<80167a7d>] find_get_pages_tag+0x6d/0x80
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<8014b01c>] do_futex+0xbc/0x950
[11811.540001] [<806ad970>] _read_unlock_irq+0x20/0x30
[11811.540001] [<80167a7d>] find_get_pages_tag+0x6d/0x80
[11811.540001] [<8017009a>] pagevec_lookup_tag+0x2a/0x40
[11811.540001] [<8016844f>] wait_on_page_writeback_range+0x5f/0x110
[11811.540001] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540001] [<801a90cd>] sync_inode+0x2d/0x40
[11811.540001] [<8014b947>] sys_futex+0x97/0x120
[11811.540001] [<801ac37c>] do_fsync+0x8c/0xb0
[11811.540001] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001] =======================
[11811.540001] rklogd R running 0 5064 1
[11811.540001] 82cbb090 00000046 00000046 00000001 00000001 8011fc6f 82cbb090 00000046
[11811.540001] f62e4a10 82cbb090 806ad489 00000000 82cbb080 00000002 00000001 80186499
[11811.540001] 82cbb080 802aa1b4 00000046 00000046 00000000 00000000 f6a6c304 f6a6c000
[11811.540001] Call Trace:
[11811.540001] [<80185695>] check_object+0xe5/0x1f0
[11811.540001] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540001] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540001] [<806ad9f5>] _spin_lock_irqsave+0x75/0x90
[11811.540001] [<8013b810>] prepare_to_wait_exclusive+0x20/0x70
[11811.540001] [<8061fa91>] unix_wait_for_peer+0xb1/0xc0
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<8061ff71>] unix_dgram_sendmsg+0x331/0x4f0
[11811.540001] [<805ae7a9>] sock_aio_write+0x109/0x120
[11811.540001] [<8018aba5>] do_sync_write+0xd5/0x120
[11811.540001] [<802e62af>] copy_to_user+0x3f/0x70
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<802a53ec>] security_file_permission+0xc/0x10
[11811.540001] [<8018ad6e>] rw_verify_area+0x5e/0xd0
[11811.540001] [<8018b4a9>] vfs_write+0x119/0x120
[11811.540001] [<8018bb8a>] sys_write+0x4a/0xa0
[11811.540001] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001] =======================
[11811.540001] dbus-daemon S 801312f7 0 5071 1
[11811.540001] f7cdac70 00000082 00000001 801312f7 f7ca0010 f7ca0000 00000046 f63a5bcc
[11811.540001] 806ad9f5 00000000 f7cdbb40 f5fc8328 00000000 01ca0000 00000046 f7cdadd4
[11811.540001] 82cbb080 00000000 00000000 f5fc8328 00000046 00000000 00000001 00000000
[11811.540001] Call Trace:
[11811.540001] [<801312f7>] lock_timer_base+0x27/0x60
[11811.540001] [<806ad9f5>] _spin_lock_irqsave+0x75/0x90
[11811.540001] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540001] [<8061f0a7>] unix_poll+0x17/0xb0
[11811.540001] [<80198076>] do_sys_poll+0x246/0x390
[11811.540001] [<80198d60>] __pollwait+0x0/0xe0
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80184f6e>] slab_pad_check+0x7e/0x100
[11811.540001] [<80122852>] enqueue_task_fair+0xc2/0xe0
[11811.540001] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540001] [<8011e94b>] __wake_up_common+0x4b/0x80
[11811.540001] [<806ad2f4>] _read_unlock+0x14/0x20
[11811.540001] [<80620f76>] unix_stream_sendmsg+0x1d6/0x330
[11811.540001] [<805ae7a9>] sock_aio_write+0x109/0x120
[11811.540001] [<805ae6a0>] sock_aio_write+0x0/0x120
[11811.540001] [<8018aa8e>] do_sync_readv_writev+0xce/0x110
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<8018b28b>] do_readv_writev+0x10b/0x170
[11811.540001] [<805ae6a0>] sock_aio_write+0x0/0x120
[11811.540001] [<8019d56d>] dput+0x6d/0x120
[11811.540001] [<801a20d8>] mntput_no_expire+0x18/0xe0
[11811.540001] [<802e62af>] copy_to_user+0x3f/0x70
[11811.540001] [<801981ed>] sys_poll+0x2d/0x80
[11811.540001] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001] =======================
[11811.540001] sshd S f613cb04 0 5085 1
[11811.540001] f7cd8000 00200086 00000002 f613cb04 f613cb0c 00000000 f7824380 00001000
[11811.540001] 00200246 00000000 00000000 f784b928 00000000 01e339c0 00200046 f7cd8164
[11811.540001] 82cbb080 00000000 00000000 f784b928 00200046 00000000 00000001 00000000
[11811.540001] Call Trace:
[11811.540001] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540001] [<805e92a7>] tcp_poll+0x17/0x130
[11811.540001] [<8019868c>] do_select+0x3dc/0x500
[11811.540001] [<801982b0>] do_select+0x0/0x500
[11811.540001] [<80198d60>] __pollwait+0x0/0xe0
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<801852b6>] check_bytes_and_report+0x26/0xd0
[11811.540001] [<80184f6e>] slab_pad_check+0x7e/0x100
[11811.540001] [<80185695>] check_object+0xe5/0x1f0
[11811.540001] [<80185e08>] __slab_free+0x238/0x2e0
[11811.540001] [<801202e4>] __wake_up+0x24/0x60
[11811.540001] [<801e6717>] journal_stop+0x1a7/0x2a0
[11811.540001] [<801e6717>] journal_stop+0x1a7/0x2a0
[11811.540001] [<801e6717>] journal_stop+0x1a7/0x2a0
[11811.540001] [<801852b6>] check_bytes_and_report+0x26/0xd0
[11811.540001] [<80184f6e>] slab_pad_check+0x7e/0x100
[11811.540001] [<80185695>] check_object+0xe5/0x1f0
[11811.540001] [<801864d9>] __slab_alloc+0x3e9/0x510
[11811.540001] [<805b36ad>] sock_def_readable+0x1d/0x80
[11811.540001] [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540001] [<801987d1>] core_sys_select+0x21/0x350
[11811.540001] [<801852b6>] check_bytes_and_report+0x26/0xd0
[11811.540001] [<80184f6e>] slab_pad_check+0x7e/0x100
[11811.540001] [<80185695>] check_object+0xe5/0x1f0
[11811.540001] [<80185e08>] __slab_free+0x238/0x2e0
[11811.540001] [<8019c48d>] d_kill+0x3d/0x60
[11811.540001] [<8019c48d>] d_kill+0x3d/0x60
[11811.540001] [<8019d56d>] dput+0x6d/0x120
[11811.540001] [<80198e8b>] sys_select+0x4b/0x1b0
[11811.540001] [<8018a4b9>] sys_close+0x89/0x100
[11811.540001] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001] =======================
[11811.540001] gpm S f63a0b04 0 5099 1
[11811.540001] f6a63b40 00000086 00000002 f63a0b04 f63a0b0c 00000000 00000046 80b6a9e8
[11811.540001] f63a0b2c 00839d26 00000002 80130c53 00000000 01b69d80 f63a0b08 f6a63ca4
[11811.540001] 82c68080 ffffc71d 00000000 00000000 00000000 00000000 000000ff f63a0b2c
[11811.540001] Call Trace:
[11811.540001] [<80130c53>] internal_add_timer+0x103/0x110
[11811.540001] [<806ab7da>] schedule_timeout+0x4a/0xd0
[11811.540001] [<801310e0>] process_timeout+0x0/0x40
[11811.540001] [<806ab7d5>] schedule_timeout+0x45/0xd0
[11811.540001] [<8019868c>] do_select+0x3dc/0x500
[11811.540001] [<801982b0>] do_select+0x0/0x500
[11811.540001] [<801864d9>] __slab_alloc+0x3e9/0x510
[11811.540001] [<80198d60>] __pollwait+0x0/0xe0
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<801e7e71>] journal_dirty_metadata+0xf1/0x110
[11811.540001] [<801e7e71>] journal_dirty_metadata+0xf1/0x110
[11811.540001] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540001] [<801ea8a2>] find_revoke_record+0x42/0x80
[11811.540001] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540001] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540001] [<802aabe1>] task_has_capability+0x71/0xb0
[11811.540001] [<802a5129>] security_capable+0x9/0x10
[11811.540001] [<8012f9b8>] __capable+0x8/0x20
[11811.540001] [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540001] [<801987d1>] core_sys_select+0x21/0x350
[11811.540001] [<801852b6>] check_bytes_and_report+0x26/0xd0
[11811.540001] [<80184f32>] slab_pad_check+0x42/0x100
[11811.540001] [<8052fa59>] mousedev_read+0xe9/0x1a0
[11811.540001] [<802e62af>] copy_to_user+0x3f/0x70
[11811.540001] [<8052fac9>] mousedev_read+0x159/0x1a0
[11811.540001] [<80198f27>] sys_select+0xe7/0x1b0
[11811.540001] [<802e62af>] copy_to_user+0x3f/0x70
[11811.540001] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001] =======================
[11811.540001] console-kit-d S f7cebbc8 0 5110 1
[11811.540001] f7cdbb40 00000082 00000002 f7cebbc8 f7cebbd0 00000000 f6afb4bc 00000246
[11811.540001] 00000246 00000000 00000000 f5fc8628 00000000 01000246 00000046 f7cdbca4
[11811.540001] 82cbb080 00000000 00000000 f5fc8628 00000046 00000000 00000001 00000000
[11811.540001] Call Trace:
[11811.540001] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540001] [<8061f0a7>] unix_poll+0x17/0xb0
[11811.540001] [<80198076>] do_sys_poll+0x246/0x390
[11811.540001] [<80198d60>] __pollwait+0x0/0xe0
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80184f6e>] slab_pad_check+0x7e/0x100
[11811.540001] [<80122852>] enqueue_task_fair+0xc2/0xe0
[11811.540001] [<801852b6>] check_bytes_and_report+0x26/0xd0
[11811.540001] [<80184f32>] slab_pad_check+0x42/0x100
[11811.540001] [<805b522a>] skb_dequeue+0x1a/0x70
[11811.540001] [<80620542>] unix_stream_recvmsg+0x3a2/0x540
[11811.540001] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540001] [<805ae8e3>] sock_aio_read+0x123/0x130
[11811.540001] [<8018acc5>] do_sync_read+0xd5/0x120
[11811.540001] [<8014afda>] do_futex+0x7a/0x950
[11811.540001] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540001] [<802a53ec>] security_file_permission+0xc/0x10
[11811.540001] [<8018ad6e>] rw_verify_area+0x5e/0xd0
[11811.540001] [<8018b5c7>] vfs_read+0x117/0x120
[11811.540001] [<8018bb38>] sys_read+0x98/0xa0
[11811.540001] [<801981ed>] sys_poll+0x2d/0x80
[11811.540001] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001] =======================
[11811.540001] console-kit-d S 82c1a620 0 5114 1
[11811.540001] f7cd8ed0 00000082 00000600 82c1a620 00000002 f7ca6dc4 00000046 00000002
[11811.540001] 82cbb080 f5438000 00000002 0000000f 00000000 018e0f80 f7ca6dcc f7cd9034
[11811.540001] 82c68080 0002fda2 00000000 00000000 00000000 00000600 000000ff 00000001
[11811.540001] Call Trace:
[11811.540001] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540001] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540001] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540001] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540001] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540001] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540001] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540001] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540001] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540001] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540001] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540001] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540001] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540001] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540001] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540001] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540001] =======================
[11811.540001] console-kit-d S 82c1a620 0 5115 1
[11811.540001] f7cd9da0 00000082 00000600 82c1a620 00000002 f6270dc4 00000046 00000002
[11811.540001] 82cbb080 f5430ed0 00000002 0000000f 00000000 018e0f80 f6270dcc f7cd9f04
[11811.540001] 82c68080 0002fda2 00000000 00000000 00000000 00000600 000000ff 00000002
[11811.540001] Call Trace:
[11811.540001] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540001] [<80120070>] default_wake_function+0x0/0x10
[11811.540001] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540001] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540001] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540001] [<8016c479>] rmqueue_bulk+0x69/0x80
[11811.540001] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540001] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540001] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540001] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540001] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540001] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540001] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540001] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540001] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540001] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540001] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540001] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540001] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540001] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5116 1
[11811.540002] f7cdd8e0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82c68080 f7cde7b0 00000002 0000000f 00000000 018e0f80 00000282 f7cdda44
[11811.540002] 82c68080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000003
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5117 1
[11811.540002] f7cde7b0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82c68080 f62e2c70 00000002 0000000f 00000000 018e0f80 00000282 f7cde914
[11811.540002] 82c68080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000004
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5118 1
[11811.540002] f62e2c70 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82c68080 e689e7b0 00000002 0000000f 00000000 018e0f80 00000282 f62e2dd4
[11811.540002] 82c68080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000005
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 82c1a620 0 5119 1
[11811.540002] f62e0ed0 00000082 00000600 82c1a620 00000002 f6233dc4 00000046 00000002
[11811.540002] 82cbb080 f54367b0 00000002 0000000f 00000000 018e0f80 f6233dcc f62e1034
[11811.540002] 82c68080 0002fda2 00000000 00000000 00000000 00000600 000000ff 00000006
[11811.540002] Call Trace:
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 82c1a620 0 5120 1
[11811.540002] f62e3b40 00000082 00000600 82c1a620 00000002 f6b33dc4 00000046 00000002
[11811.540002] 82cbb080 f542e7b0 00000002 0000000f 00000000 018e0f80 f6b33dcc f62e3ca4
[11811.540002] 82c68080 0002fda2 00000000 00000000 00000000 00000600 000000ff 00000007
[11811.540002] Call Trace:
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 82c1a620 0 5121 1
[11811.540002] f62e58e0 00000082 00000600 82c1a620 00000002 f60fddc4 00000046 00000002
[11811.540002] 82cbb080 f5432c70 00000002 0000000f 00000000 018e0f80 f60fddcc f62e5a44
[11811.540002] 82c68080 0002fda2 00000000 00000000 00000000 00000600 000000ff 00000008
[11811.540002] Call Trace:
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 82c1a620 0 5122 1
[11811.540002] f5400000 00000082 00000600 82c1a620 00000002 f63dcdc4 00000046 00000002
[11811.540002] 82cbb080 f5431da0 00000002 0000000f 00000000 018e0f80 f63dcdcc f5400164
[11811.540002] 82c68080 0002fda2 00000000 00000000 00000000 00000600 000000ff 00000009
[11811.540002] Call Trace:
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<80170286>] __pagevec_lru_add_active+0x46/0xb0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 82c1a620 0 5124 1
[11811.540002] f5401da0 00000082 00000600 82c1a620 00000002 f63dedc4 00000046 00000002
[11811.540002] 82cbb080 f5430000 00000002 0000000f 00000000 018e0f80 f63dedcc f5401f04
[11811.540002] 82c68080 0002fda2 00000000 00000000 00000000 00000600 000000ff 0000000b
[11811.540002] Call Trace:
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5125 1
[11811.540002] f5402c70 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82cbb080 f5403b40 00000002 0000000f 00000000 018e0f80 00000282 f5402dd4
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000000c
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540002] [<8016c479>] rmqueue_bulk+0x69/0x80
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5126 1
[11811.540002] f5403b40 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82cbb080 f5404a10 00000002 0000000f 00000000 018e0f80 00000282 f5403ca4
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000000d
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5127 1
[11811.540002] f5404a10 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82cbb080 f542ca10 00000002 0000000f 00000000 018e0f80 00000282 f5404b74
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000000e
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5128 1
[11811.540002] f54058e0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82cbb080 f54067b0 00000002 0000000f 00000000 018e0f80 00000282 f5405a44
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000000f
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5129 1
[11811.540002] f54067b0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82cbb080 f5428000 00000002 0000000f 00000000 018e0f80 00000282 f5406914
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000010
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<80170286>] __pagevec_lru_add_active+0x46/0xb0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5130 1
[11811.540002] f5428000 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82cbb080 f5428ed0 00000002 0000000f 00000000 018e0f80 00000282 f5428164
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000011
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5131 1
[11811.540002] f5428ed0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82cbb080 f5429da0 00000002 0000000f 00000000 018e0f80 00000282 f5429034
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000012
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5132 1
[11811.540002] f5429da0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82cbb080 f542ac70 00000002 0000000f 00000000 018e0f80 00000282 f5429f04
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000013
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540002] [<8016c479>] rmqueue_bulk+0x69/0x80
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5133 1
[11811.540002] f542ac70 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82cbb080 f542bb40 00000002 0000000f 00000000 018e0f80 00000282 f542add4
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000014
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5134 1
[11811.540002] f542bb40 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82cbb080 f5402c70 00000002 0000000f 00000000 018e0f80 00000282 f542bca4
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000015
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5135 1
[11811.540002] f542ca10 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82cbb080 f542d8e0 00000002 0000000f 00000000 018e0f80 00000282 f542cb74
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000016
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5136 1
[11811.540002] f542d8e0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82c68080 f62e3b40 00000002 0000000f 00000000 018e0f80 00000282 f542da44
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000017
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<80170286>] __pagevec_lru_add_active+0x46/0xb0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5137 1
[11811.540002] f542e7b0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82c68080 f5401da0 00000002 0000000f 00000000 018e0f80 00000282 f542e914
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000018
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540002] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540002] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540002] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540002] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540002] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540002] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540002] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540002] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540002] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540002] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540002] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540002] =======================
[11811.540002] console-kit-d S 8011ff89 0 5138 1
[11811.540002] f5430000 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540002] 82c68080 f5400000 00000002 0000000f 00000000 018e0f80 00000282 f5430164
[11811.540002] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000019
[11811.540002] Call Trace:
[11811.540002] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540002] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540002] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540002] [<80120070>] default_wake_function+0x0/0x10
[11811.540002] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540002] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540002] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540002] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540002] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5139 1
[11811.540003] f5430ed0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82c68080 f7cdd8e0 00000002 0000000f 00000000 018e0f80 00000282 f5431034
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000001a
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5140 1
[11811.540003] f5431da0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82c68080 f62e58e0 00000002 0000000f 00000000 018e0f80 00000282 f5431f04
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000001b
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5141 1
[11811.540003] f5432c70 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f5433b40 00000002 0000000f 00000000 018e0f80 00000282 f5432dd4
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000001c
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5142 1
[11811.540003] f5433b40 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82c68080 e689e7b0 00000002 0000000f 00000000 018e0f80 00000282 f5433ca4
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000001d
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540003] [<8016c479>] rmqueue_bulk+0x69/0x80
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5143 1
[11811.540003] f5434a10 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f54358e0 00000002 0000000f 00000000 018e0f80 00000282 f5434b74
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000001e
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5144 1
[11811.540003] f54358e0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82c68080 f62e0ed0 00000002 0000000f 00000000 018e0f80 00000282 f5435a44
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000001f
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5145 1
[11811.540003] f54367b0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82c68080 f7cd8ed0 00000002 0000000f 00000000 018e0f80 00000282 f5436914
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000020
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5146 1
[11811.540003] f5438000 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82c68080 f7cd9da0 00000002 0000000f 00000000 018e0f80 00000282 f5438164
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000021
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5147 1
[11811.540003] f5438ed0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f54058e0 00000002 0000000f 00000000 018e0f80 00000282 f5439034
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000022
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5148 1
[11811.540003] f5439da0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f543bb40 00000002 0000000f 00000000 018e0f80 00000282 f5439f04
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000023
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5149 1
[11811.540003] f543ac70 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f543ca10 00000002 0000000f 00000000 018e0f80 00000282 f543add4
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000024
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540003] [<8016c479>] rmqueue_bulk+0x69/0x80
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5150 1
[11811.540003] f543bb40 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f5464a10 00000002 0000000f 00000000 018e0f80 00000282 f543bca4
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000025
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5151 1
[11811.540003] f543ca10 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f543d8e0 00000002 0000000f 00000000 018e0f80 00000282 f543cb74
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000026
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5152 1
[11811.540003] f543d8e0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f543e7b0 00000002 0000000f 00000000 018e0f80 00000282 f543da44
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000027
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5153 1
[11811.540003] f543e7b0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f5460000 00000002 0000000f 00000000 018e0f80 00000282 f543e914
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000028
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5154 1
[11811.540003] f5460000 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f5460ed0 00000002 0000000f 00000000 018e0f80 00000282 f5460164
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000029
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5155 1
[11811.540003] f5460ed0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f5461da0 00000002 0000000f 00000000 018e0f80 00000282 f5461034
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000002a
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5156 1
[11811.540003] f5461da0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f5462c70 00000002 0000000f 00000000 018e0f80 00000282 f5461f04
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000002b
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<80170286>] __pagevec_lru_add_active+0x46/0xb0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5157 1
[11811.540003] f5462c70 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f5439da0 00000002 0000000f 00000000 018e0f80 00000282 f5462dd4
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000002c
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5158 1
[11811.540003] f5463b40 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f5468000 00000002 0000000f 00000000 018e0f80 00000282 f5463ca4
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000002d
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540003] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540003] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540003] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540003] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540003] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540003] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540003] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540003] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540003] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540003] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540003] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540003] =======================
[11811.540003] console-kit-d S 8011ff89 0 5159 1
[11811.540003] f5464a10 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540003] 82cbb080 f54658e0 00000002 0000000f 00000000 018e0f80 00000282 f5464b74
[11811.540003] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000002e
[11811.540003] Call Trace:
[11811.540003] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540003] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540003] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540003] [<80120070>] default_wake_function+0x0/0x10
[11811.540003] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540003] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540003] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540003] [<8016c479>] rmqueue_bulk+0x69/0x80
[11811.540003] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540003] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540003] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5160 1
[11811.540004] f54658e0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f54667b0 00000002 0000000f 00000000 018e0f80 00000282 f5465a44
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000002f
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5161 1
[11811.540004] f54667b0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f5463b40 00000002 0000000f 00000000 018e0f80 00000282 f5466914
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000030
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5162 1
[11811.540004] f5468000 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f5400ed0 00000002 0000000f 00000000 018e0f80 00000282 f5468164
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000031
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5163 1
[11811.540004] f5468ed0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f5472c70 00000002 0000000f 00000000 018e0f80 00000282 f5469034
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000032
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<80170286>] __pagevec_lru_add_active+0x46/0xb0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5165 1
[11811.540004] f5469da0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f5468ed0 00000002 0000000f 00000000 018e0f80 00000282 f5469f04
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000033
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5166 1
[11811.540004] f6a667b0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f5473b40 00000002 0000000f 00000000 018e0f80 00000282 f6a66914
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000034
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540004] [<8016c479>] rmqueue_bulk+0x69/0x80
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5167 1
[11811.540004] f6a60000 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f5469da0 00000002 0000000f 00000000 018e0f80 00000282 f6a60164
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000035
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5168 1
[11811.540004] f6a64a10 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f6a667b0 00000002 0000000f 00000000 018e0f80 00000282 f6a64b74
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000036
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5169 1
[11811.540004] f5470000 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f6a60000 00000002 0000000f 00000000 018e0f80 00000282 f5470164
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000037
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5170 1
[11811.540004] f5470ed0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f6a64a10 00000002 0000000f 00000000 018e0f80 00000282 f5471034
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000038
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5171 1
[11811.540004] f5471da0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f5470ed0 00000002 0000000f 00000000 018e0f80 00000282 f5471f04
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000039
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5172 1
[11811.540004] f5472c70 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f5471da0 00000002 0000000f 00000000 018e0f80 00000282 f5472dd4
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000003a
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<80170286>] __pagevec_lru_add_active+0x46/0xb0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5173 1
[11811.540004] f5473b40 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f5474a10 00000002 0000000f 00000000 018e0f80 00000282 f5473ca4
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000003b
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540004] [<8016c479>] rmqueue_bulk+0x69/0x80
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5174 1
[11811.540004] f5474a10 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f54758e0 00000002 0000000f 00000000 018e0f80 00000282 f5474b74
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000003c
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 8011ff89 0 5175 1
[11811.540004] f54758e0 00000082 00000000 8011ff89 8098a8e8 00000000 00000046 00000002
[11811.540004] 82cbb080 f543ac70 00000002 0000000f 00000000 018e0f80 00000282 f5475a44
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 0000003d
[11811.540004] Call Trace:
[11811.540004] [<8011ff89>] try_to_wake_up+0x79/0x160
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<8011d3d1>] kunmap_atomic+0x91/0xb0
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<8016d37e>] get_page_from_freelist+0x2ae/0x4c0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<8017858d>] handle_mm_fault+0x47d/0x5b0
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] console-kit-d S 00000001 0 24872 1
[11811.540004] f557d8e0 00000082 00000000 00000001 00000001 8013fdd1 00000046 808e0fc0
[11811.540004] 00000082 00005607 00000002 00000000 00000000 018e0f80 00000282 f557da44
[11811.540004] 82cbb080 00000282 00020b19 00020b19 80129157 00000282 00000000 00000000
[11811.540004] Call Trace:
[11811.540004] [<8013fdd1>] up+0x11/0x40
[11811.540004] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540004] [<803490f9>] vt_waitactive+0x59/0xb0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80349c51>] vt_ioctl+0xb01/0x19b0
[11811.540004] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540004] [<80349150>] vt_ioctl+0x0/0x19b0
[11811.540004] [<803440f6>] tty_ioctl+0xa6/0xdf0
[11811.540004] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540004] [<802ab122>] inode_has_perm+0x52/0x90
[11811.540004] [<802ab207>] file_has_perm+0xa7/0xb0
[11811.540004] [<80344050>] tty_ioctl+0x0/0xdf0
[11811.540004] [<8019749b>] vfs_ioctl+0x2b/0x90
[11811.540004] [<80197751>] do_vfs_ioctl+0x251/0x2a0
[11811.540004] [<80197806>] sys_ioctl+0x66/0xb0
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] crond S f5454f10 0 5176 1
[11811.540004] f546ac70 00000092 00000002 f5454f10 f5454f18 00000000 00000001 82c6dae0
[11811.540004] 00000282 00000034 82c6db34 00000282 00000000 01454f48 f5454f14 f546add4
[11811.540004] 82cbb080 00117951 00000000 00000000 00000000 00000000 000000ff f5454f48
[11811.540004] Call Trace:
[11811.540004] [<806ac501>] do_nanosleep+0x71/0xa0
[11811.540004] [<8013f0ce>] hrtimer_nanosleep+0x5e/0xd0
[11811.540004] [<8013ea20>] hrtimer_wakeup+0x0/0x20
[11811.540004] [<806ac4ee>] do_nanosleep+0x5e/0xa0
[11811.540004] [<8013f198>] sys_nanosleep+0x58/0x60
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] xfs S f6a53b04 0 5207 1
[11811.540004] f546e7b0 00000086 00000002 f6a53b04 f6a53b0c 00000000 00000046 f7ca0a68
[11811.540004] f6a53b2c 00121986 00000002 80130c53 00000000 01ca0000 f6a53b08 f546e914
[11811.540004] 82cbb080 00112f27 00000000 00000000 00000000 00000000 000000ff f6a53b2c
[11811.540004] Call Trace:
[11811.540004] [<80130c53>] internal_add_timer+0x103/0x110
[11811.540004] [<806ab7da>] schedule_timeout+0x4a/0xd0
[11811.540004] [<801310e0>] process_timeout+0x0/0x40
[11811.540004] [<806ab7d5>] schedule_timeout+0x45/0xd0
[11811.540004] [<8019868c>] do_select+0x3dc/0x500
[11811.540004] [<801982b0>] do_select+0x0/0x500
[11811.540004] [<802da923>] blk_recount_segments+0x33/0x70
[11811.540004] [<80198d60>] __pollwait+0x0/0xe0
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<801ad55d>] __find_get_block+0x9d/0x190
[11811.540004] [<801ad67c>] __getblk+0x2c/0x200
[11811.540004] [<801d66a8>] ext3_getblk+0x98/0x190
[11811.540004] [<801da1f0>] ext3_find_entry+0x1f0/0x660
[11811.540004] [<80185695>] check_object+0xe5/0x1f0
[11811.540004] [<8019c2c0>] d_rehash+0x20/0x50
[11811.540004] [<8019d57f>] dput+0x7f/0x120
[11811.540004] [<801852b6>] check_bytes_and_report+0x26/0xd0
[11811.540004] [<80184f32>] slab_pad_check+0x42/0x100
[11811.540004] [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540004] [<801987d1>] core_sys_select+0x21/0x350
[11811.540004] [<80175b29>] __do_fault+0xe9/0x400
[11811.540004] [<8011d37f>] kunmap_atomic+0x3f/0xb0
[11811.540004] [<80175be5>] __do_fault+0x1a5/0x400
[11811.540004] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540004] [<80178642>] handle_mm_fault+0x532/0x5b0
[11811.540004] [<8011a07b>] do_page_fault+0xcb/0x710
[11811.540004] [<80198f27>] sys_select+0xe7/0x1b0
[11811.540004] [<802e62af>] copy_to_user+0x3f/0x70
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] atd S f545ff10 0 5213 1
[11811.540004] f6a60ed0 00000092 00000002 f545ff10 f545ff18 00000000 00000001 82c6dae0
[11811.540004] 00000282 00000034 82c6db34 00000282 00000000 0145ff48 f545ff14 f6a61034
[11811.540004] 82cbb080 001044e0 00000000 00000000 00000000 00000000 000000ff f545ff48
[11811.540004] Call Trace:
[11811.540004] [<806ac501>] do_nanosleep+0x71/0xa0
[11811.540004] [<8013f0ce>] hrtimer_nanosleep+0x5e/0xd0
[11811.540004] [<8013ea20>] hrtimer_wakeup+0x0/0x20
[11811.540004] [<806ac4ee>] do_nanosleep+0x5e/0xa0
[11811.540004] [<8013f198>] sys_nanosleep+0x58/0x60
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] kauditd S f54eef90 0 5244 2
[11811.540004] f54d9da0 00000096 00000002 f54eef90 f54eef98 00000000 00000000 808e3b30
[11811.540004] 00000046 00000000 00000001 00000001 00000000 01000046 f54eef94 f54d9f04
[11811.540004] 82c68080 ffffcb74 00000000 00000000 00000000 00000000 000000ff 00000000
[11811.540004] Call Trace:
[11811.540004] [<80153ee0>] kauditd_thread+0x0/0x170
[11811.540004] [<80154020>] kauditd_thread+0x140/0x170
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<8013b3d2>] kthread+0x42/0x70
[11811.540004] [<8013b390>] kthread+0x0/0x70
[11811.540004] [<80104b1b>] kernel_thread_helper+0x7/0x1c
[11811.540004] =======================
[11811.540004] distccd S f5502ef0 0 5297 1
[11811.540004] f54767b0 00000092 00000002 f5502ef0 f5502ef8 00000000 00000004 f5502ecc
[11811.540004] 00000000 00000000 00000001 ffffffff 00000000 01000000 f5502ef4 f5476914
[11811.540004] 82cbb080 ffffe5cc 00000000 00000000 00000000 00000000 000000ff f54767a8
[11811.540004] Call Trace:
[11811.540004] [<8012b865>] do_wait+0x705/0xa80
[11811.540004] [<80120070>] default_wake_function+0x0/0x10
[11811.540004] [<8012bc5c>] sys_wait4+0x7c/0xc0
[11811.540004] [<8012bcc7>] sys_waitpid+0x27/0x30
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] distccd S f62e67b0 0 5298 5297
[11811.540004] f62e67b0 00000096 e6b66d40 f62e67b0 805e28e5 00000000 f611067c f62e67b0
[11811.540004] 00000000 00000000 f5770ed0 00000000 00000000 01000000 f611067c f62e6914
[11811.540004] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540004] Call Trace:
[11811.540004] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540004] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540004] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540004] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540004] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540004] [<80607d08>] inet_accept+0x28/0xd0
[11811.540004] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540004] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540004] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540004] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540004] [<8019c48d>] d_kill+0x3d/0x60
[11811.540004] [<8019c48d>] d_kill+0x3d/0x60
[11811.540004] [<8019d56d>] dput+0x6d/0x120
[11811.540004] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540004] [<8018a4b9>] sys_close+0x89/0x100
[11811.540004] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] distccd S f54d0000 0 5332 5297
[11811.540004] f54d0000 00000096 ee6d9710 f54d0000 805e28e5 00000000 f611067c f54d0000
[11811.540004] 00000000 00000000 f2132c70 00000000 00000000 01000000 f611067c f54d0164
[11811.540004] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540004] Call Trace:
[11811.540004] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540004] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540004] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540004] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540004] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540004] [<80607d08>] inet_accept+0x28/0xd0
[11811.540004] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540004] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540004] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540004] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540004] [<8019c48d>] d_kill+0x3d/0x60
[11811.540004] [<8019c48d>] d_kill+0x3d/0x60
[11811.540004] [<8019d56d>] dput+0x6d/0x120
[11811.540004] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540004] [<8018a4b9>] sys_close+0x89/0x100
[11811.540004] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540004] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540004] =======================
[11811.540004] distccd S f557e7b0 0 5479 5297
[11811.540004] f557e7b0 00000096 e6be9dd0 f557e7b0 805e28e5 00000000 f611067c f557e7b0
[11811.540004] 00000000 00000000 eb661da0 00000000 00000000 01000000 f611067c f557e914
[11811.540004] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540004] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] smartd S f5514f10 0 5492 1
[11811.540005] f54d4a10 00000092 00000002 f5514f10 f5514f18 00000000 00000001 82c6dae0
[11811.540005] 00000282 00000034 82c6db34 00000282 00000000 01514f48 f5514f14 f54d4b74
[11811.540005] 82cbb080 00104886 00000000 00000000 00000000 00000000 000000ff f5514f48
[11811.540005] Call Trace:
[11811.540005] [<806ac501>] do_nanosleep+0x71/0xa0
[11811.540005] [<8013f0ce>] hrtimer_nanosleep+0x5e/0xd0
[11811.540005] [<8013ea20>] hrtimer_wakeup+0x0/0x20
[11811.540005] [<806ac4ee>] do_nanosleep+0x5e/0xa0
[11811.540005] [<8013f198>] sys_nanosleep+0x58/0x60
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] mingetty S 84388000 0 5494 1
[11811.540005] f557ac70 00000082 8034f35f 84388000 00000000 80351500 00000004 00000000
[11811.540005] f670151c f557ac70 fffffe00 0700b338 00000000 01000046 00000000 f557add4
[11811.540005] 82cbb080 f557ac70 00000000 00000002 00000046 00000046 00000000 00000000
[11811.540005] Call Trace:
[11811.540005] [<8034f35f>] notify_update+0x1f/0x30
[11811.540005] [<80351500>] do_con_write+0x1a0/0x1aa0
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<8013b966>] add_wait_queue+0x16/0x40
[11811.540005] [<8034720a>] read_chan+0x1ea/0x6b0
[11811.540005] [<80120070>] default_wake_function+0x0/0x10
[11811.540005] [<80343ec6>] tty_read+0x86/0xc0
[11811.540005] [<80347020>] read_chan+0x0/0x6b0
[11811.540005] [<80343e40>] tty_read+0x0/0xc0
[11811.540005] [<8018b549>] vfs_read+0x99/0x120
[11811.540005] [<8018baea>] sys_read+0x4a/0xa0
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] mingetty S 00000000 0 5495 1
[11811.540005] f5578ed0 00000082 0000000d 00000000 33679376 0000000d 82c680ec 00000000
[11811.540005] f6f91d4c f5578ed0 82c680d0 82c680ec 00000000 01000046 00000000 f5579034
[11811.540005] 82c68080 f5578ed0 00000000 00000002 00000046 00000046 00000000 00000000
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<8013b966>] add_wait_queue+0x16/0x40
[11811.540005] [<8034720a>] read_chan+0x1ea/0x6b0
[11811.540005] [<80120070>] default_wake_function+0x0/0x10
[11811.540005] [<80343ec6>] tty_read+0x86/0xc0
[11811.540005] [<80347020>] read_chan+0x0/0x6b0
[11811.540005] [<80343e40>] tty_read+0x0/0xc0
[11811.540005] [<8018b549>] vfs_read+0x99/0x120
[11811.540005] [<8018baea>] sys_read+0x4a/0xa0
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] mingetty S f6068d20 0 5496 1
[11811.540005] f546bb40 00000082 8034f35f f6068d20 00000000 80351500 00000046 00000000
[11811.540005] f6f5b5dc f546bb40 fffffe00 0700b338 00000000 01000046 00000000 f546bca4
[11811.540005] 82cbb080 f546bb40 00000000 00000002 00000046 00000046 00000000 00000000
[11811.540005] Call Trace:
[11811.540005] [<8034f35f>] notify_update+0x1f/0x30
[11811.540005] [<80351500>] do_con_write+0x1a0/0x1aa0
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<8013b966>] add_wait_queue+0x16/0x40
[11811.540005] [<8034720a>] read_chan+0x1ea/0x6b0
[11811.540005] [<80120070>] default_wake_function+0x0/0x10
[11811.540005] [<80343ec6>] tty_read+0x86/0xc0
[11811.540005] [<80347020>] read_chan+0x0/0x6b0
[11811.540005] [<80343e40>] tty_read+0x0/0xc0
[11811.540005] [<8018b549>] vfs_read+0x99/0x120
[11811.540005] [<8018baea>] sys_read+0x4a/0xa0
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] mingetty S f621aaf0 0 5497 1
[11811.540005] f546ca10 00000082 8034f35f f621aaf0 00000000 80351500 00000046 00000000
[11811.540005] f6f8ef2c f546ca10 fffffe00 0700b338 00000000 01000046 00000000 f546cb74
[11811.540005] 82cbb080 f546ca10 00000000 00000002 00000046 00000046 00000000 00000000
[11811.540005] Call Trace:
[11811.540005] [<8034f35f>] notify_update+0x1f/0x30
[11811.540005] [<80351500>] do_con_write+0x1a0/0x1aa0
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<8013b966>] add_wait_queue+0x16/0x40
[11811.540005] [<8034720a>] read_chan+0x1ea/0x6b0
[11811.540005] [<80120070>] default_wake_function+0x0/0x10
[11811.540005] [<80343ec6>] tty_read+0x86/0xc0
[11811.540005] [<80347020>] read_chan+0x0/0x6b0
[11811.540005] [<80343e40>] tty_read+0x0/0xc0
[11811.540005] [<8018b549>] vfs_read+0x99/0x120
[11811.540005] [<8018baea>] sys_read+0x4a/0xa0
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] mingetty S f54f9e84 0 5498 1
[11811.540005] f54dd8e0 00000082 00000002 f54f9e84 f54f9e8c 00000000 00000000 00000000
[11811.540005] f6a3ef2c f54dd8e0 82c68080 00000046 00000000 01000046 f54f9e88 f54dda44
[11811.540005] 82c68080 ffffcdc9 00000000 00000000 00000000 00000000 000000ff 00000000
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<8013b966>] add_wait_queue+0x16/0x40
[11811.540005] [<8034720a>] read_chan+0x1ea/0x6b0
[11811.540005] [<80120070>] default_wake_function+0x0/0x10
[11811.540005] [<80343ec6>] tty_read+0x86/0xc0
[11811.540005] [<80347020>] read_chan+0x0/0x6b0
[11811.540005] [<80343e40>] tty_read+0x0/0xc0
[11811.540005] [<8018b549>] vfs_read+0x99/0x120
[11811.540005] [<8018baea>] sys_read+0x4a/0xa0
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] mingetty S 00000000 0 5499 1
[11811.540005] f54dca10 00000082 0000000d 00000000 33679376 0000000d 82c680ec 00000000
[11811.540005] f6a3e6fc f54dca10 82c680d0 82c680ec 00000000 01000046 00000000 f54dcb74
[11811.540005] 82c68080 f54dca10 00000000 00000002 00000046 00000046 00000000 00000000
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<8013b966>] add_wait_queue+0x16/0x40
[11811.540005] [<8034720a>] read_chan+0x1ea/0x6b0
[11811.540005] [<80120070>] default_wake_function+0x0/0x10
[11811.540005] [<80343ec6>] tty_read+0x86/0xc0
[11811.540005] [<80347020>] read_chan+0x0/0x6b0
[11811.540005] [<80343e40>] tty_read+0x0/0xc0
[11811.540005] [<8018b549>] vfs_read+0x99/0x120
[11811.540005] [<8018baea>] sys_read+0x4a/0xa0
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] mingetty S f66c6d20 0 5500 1
[11811.540005] f54d8000 00000082 8034f35f f66c6d20 00000000 80351500 00000046 00000000
[11811.540005] f6a3decc f54d8000 fffffe00 0700b338 00000000 01000046 00000000 f54d8164
[11811.540005] 82c68080 f54d8000 00000000 00000002 00000046 00000046 00000000 00000000
[11811.540005] Call Trace:
[11811.540005] [<8034f35f>] notify_update+0x1f/0x30
[11811.540005] [<80351500>] do_con_write+0x1a0/0x1aa0
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<8013b966>] add_wait_queue+0x16/0x40
[11811.540005] [<8034720a>] read_chan+0x1ea/0x6b0
[11811.540005] [<80120070>] default_wake_function+0x0/0x10
[11811.540005] [<80343ec6>] tty_read+0x86/0xc0
[11811.540005] [<80347020>] read_chan+0x0/0x6b0
[11811.540005] [<80343e40>] tty_read+0x0/0xc0
[11811.540005] [<8018b549>] vfs_read+0x99/0x120
[11811.540005] [<8018baea>] sys_read+0x4a/0xa0
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] mingetty S f5554e84 0 5501 1
[11811.540005] f54d8ed0 00000082 00000002 f5554e84 f5554e8c 00000000 82c680ec 00000000
[11811.540005] f6f8decc f54d8ed0 82c680d0 82c680ec 00000000 01000046 f5554e88 f54d9034
[11811.540005] 82cbb080 ffffcdcd 00000000 00000000 00000000 00000000 000000ff 00000000
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<8013b966>] add_wait_queue+0x16/0x40
[11811.540005] [<8034720a>] read_chan+0x1ea/0x6b0
[11811.540005] [<80120070>] default_wake_function+0x0/0x10
[11811.540005] [<80343ec6>] tty_read+0x86/0xc0
[11811.540005] [<80347020>] read_chan+0x0/0x6b0
[11811.540005] [<80343e40>] tty_read+0x0/0xc0
[11811.540005] [<8018b549>] vfs_read+0x99/0x120
[11811.540005] [<8018baea>] sys_read+0x4a/0xa0
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] mingetty S 00000000 0 5502 1
[11811.540005] f54dbb40 00000082 0000000d 00000000 33679376 0000000d 82c680ec 00000000
[11811.540005] f6f84e6c f54dbb40 82c680d0 82c680ec 00000000 01000046 00000000 f54dbca4
[11811.540005] 82c68080 f54dbb40 00000000 00000002 00000046 00000046 00000000 00000000
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<8013b966>] add_wait_queue+0x16/0x40
[11811.540005] [<8034720a>] read_chan+0x1ea/0x6b0
[11811.540005] [<80120070>] default_wake_function+0x0/0x10
[11811.540005] [<80343ec6>] tty_read+0x86/0xc0
[11811.540005] [<80347020>] read_chan+0x0/0x6b0
[11811.540005] [<80343e40>] tty_read+0x0/0xc0
[11811.540005] [<8018b549>] vfs_read+0x99/0x120
[11811.540005] [<8018baea>] sys_read+0x4a/0xa0
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] mingetty S 00000000 0 5503 1
[11811.540005] f54de7b0 00000082 0000000d 00000000 33679376 0000000d 82c680ec 00000000
[11811.540005] f6a3f75c f54de7b0 82c680d0 82c680ec 00000000 01000046 00000000 f54de914
[11811.540005] 82c68080 f54de7b0 00000000 00000002 00000046 00000046 00000000 00000000
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<8013b966>] add_wait_queue+0x16/0x40
[11811.540005] [<8034720a>] read_chan+0x1ea/0x6b0
[11811.540005] [<80120070>] default_wake_function+0x0/0x10
[11811.540005] [<80343ec6>] tty_read+0x86/0xc0
[11811.540005] [<80347020>] read_chan+0x0/0x6b0
[11811.540005] [<80343e40>] tty_read+0x0/0xc0
[11811.540005] [<8018b549>] vfs_read+0x99/0x120
[11811.540005] [<8018baea>] sys_read+0x4a/0xa0
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f7cdca10 0 5504 5297
[11811.540005] f7cdca10 00000096 e19c7200 f7cdca10 805e28e5 00000000 f611067c f7cdca10
[11811.540005] 00000000 00000000 eb4a0ed0 00000000 00000000 01000000 f611067c f7cdcb74
[11811.540005] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f5673b40 0 5622 5297
[11811.540005] f5673b40 00000096 f21793b0 f5673b40 805e28e5 00000000 f611067c f5673b40
[11811.540005] 00000000 00000000 f21793b0 00000000 00000000 01000000 f611067c f5673ca4
[11811.540005] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f5671da0 0 5626 5297
[11811.540005] f5671da0 00000096 f21559e0 f5671da0 805e28e5 00000000 f611067c f5671da0
[11811.540005] 00000000 00000000 f21559e0 00000000 00000000 01000000 f611067c f5671f04
[11811.540005] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f5670000 0 5631 5297
[11811.540005] f5670000 00000096 f2179050 f5670000 805e28e5 00000000 f611067c f5670000
[11811.540005] 00000000 00000000 f2179050 00000000 00000000 01000000 f611067c f5670164
[11811.540005] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f5570000 0 5658 5297
[11811.540005] f5570000 00000096 f21209e0 f5570000 f7c02c54 00000000 f611067c f5570000
[11811.540005] 00000000 00000000 f21209e0 00000000 00000000 01000000 f611067c f5570164
[11811.540005] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f5571da0 0 5663 5297
[11811.540005] f5571da0 00000096 e6be25f0 f5571da0 805e28e5 00000000 f611067c f5571da0
[11811.540005] 00000000 00000000 e6be25f0 00000000 00000000 01000000 f611067c f5571f04
[11811.540005] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f5570ed0 0 5666 5297
[11811.540005] f5570ed0 00000096 e6b1ff80 f5570ed0 f7c02c54 00000000 f611067c f5570ed0
[11811.540005] 00000000 00000000 e6b1ff80 00000000 00000000 01000000 f611067c f5571034
[11811.540005] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f54d1da0 0 5678 5297
[11811.540005] f54d1da0 00000096 f2120d40 f54d1da0 00000000 00000000 f611067c f54d1da0
[11811.540005] 00000000 00000000 eb581da0 00000000 00000000 01000000 f611067c f54d1f04
[11811.540005] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f54d3b40 0 5681 5297
[11811.540005] f54d3b40 00000096 ee6285f0 f54d3b40 805e28e5 00000000 f611067c f54d3b40
[11811.540005] 00000000 00000000 ee6285f0 00000000 00000000 01000000 f611067c f54d3ca4
[11811.540005] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f54d0ed0 0 5686 5297
[11811.540005] f54d0ed0 00000096 f21f8050 f54d0ed0 805e28e5 00000000 f611067c f54d0ed0
[11811.540005] 00000000 00000000 f21f8050 00000000 00000000 01000000 f611067c f54d1034
[11811.540005] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f6a658e0 0 5709 5297
[11811.540005] f6a658e0 00000096 ee7a0320 f6a658e0 f7c02c54 00000000 f611067c f6a658e0
[11811.540005] 00000000 00000000 ee7a0320 00000000 00000000 01000000 f611067c f6a65a44
[11811.540005] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f54d2c70 0 5742 5297
[11811.540005] f54d2c70 00000096 eb697d40 f54d2c70 805e28e5 00000000 f611067c f54d2c70
[11811.540005] 00000000 00000000 eb697d40 00000000 00000000 01000000 f611067c f54d2dd4
[11811.540005] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f54d58e0 0 5745 5297
[11811.540005] f54d58e0 00000096 eb5c6050 f54d58e0 829ae510 00000000 f611067c f54d58e0
[11811.540005] 00000000 00000000 eb5c6050 00000000 00000000 01000000 f611067c f54d5a44
[11811.540005] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f5572c70 0 5748 5297
[11811.540005] f5572c70 00000096 e6be2830 f5572c70 805e28e5 00000000 f611067c f5572c70
[11811.540005] 00000000 00000000 e6be2830 00000000 00000000 01000000 f611067c f5572dd4
[11811.540005] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f5574a10 0 5751 5297
[11811.540005] f5574a10 00000096 eb4c4170 f5574a10 f7c02c54 00000000 f611067c f5574a10
[11811.540005] 00000000 00000000 eb4c4170 00000000 00000000 01000000 f611067c f5574b74
[11811.540005] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f5674a10 0 5753 5297
[11811.540005] f5674a10 00000096 eb6ff440 f5674a10 805e28e5 00000000 f611067c f5674a10
[11811.540005] 00000000 00000000 ee634a10 00000000 00000000 01000000 f611067c f5674b74
[11811.540005] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540005] [<80607d08>] inet_accept+0x28/0xd0
[11811.540005] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540005] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540005] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540005] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019c48d>] d_kill+0x3d/0x60
[11811.540005] [<8019d56d>] dput+0x6d/0x120
[11811.540005] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540005] [<8018a4b9>] sys_close+0x89/0x100
[11811.540005] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540005] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540005] =======================
[11811.540005] distccd S f56758e0 0 5755 5297
[11811.540005] f56758e0 00000096 eb4c4950 f56758e0 805e28e5 00000000 f611067c f56758e0
[11811.540005] 00000000 00000000 eb670000 00000000 00000000 01000000 f611067c f5675a44
[11811.540005] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540005] Call Trace:
[11811.540005] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540005] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540005] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540005] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540005] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f56767b0 0 5758 5297
[11811.540006] f56767b0 00000096 e18f1cb0 f56767b0 805e28e5 00000000 f611067c f56767b0
[11811.540006] 00000000 00000000 e18f1cb0 00000000 00000000 01000000 f611067c f5676914
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5670ed0 0 5761 5297
[11811.540006] f5670ed0 00000096 eb7ad200 f5670ed0 805e28e5 00000000 f611067c f5670ed0
[11811.540006] 00000000 00000000 eb7ad200 00000000 00000000 01000000 f611067c f5671034
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f54d67b0 0 5764 5297
[11811.540006] f54d67b0 00000096 ee4ab710 f54d67b0 805e28e5 00000000 f611067c f54d67b0
[11811.540006] 00000000 00000000 ee4ab710 00000000 00000000 01000000 f611067c f54d6914
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f56d0000 0 5767 5297
[11811.540006] f56d0000 00000096 ee41c200 f56d0000 805e28e5 00000000 f611067c f56d0000
[11811.540006] 00000000 00000000 f57958e0 00000000 00000000 01000000 f611067c f56d0164
[11811.540006] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f56d1da0 0 5770 5297
[11811.540006] f56d1da0 00000096 e6a04170 f56d1da0 805e28e5 00000000 f611067c f56d1da0
[11811.540006] 00000000 00000000 e6a04170 00000000 00000000 01000000 f611067c f56d1f04
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f56d2c70 0 5773 5297
[11811.540006] f56d2c70 00000096 e6b1f440 f56d2c70 829fb5f0 00000000 f611067c f56d2c70
[11811.540006] 00000000 00000000 e6b1f440 00000000 00000000 01000000 f611067c f56d2dd4
[11811.540006] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f56d3b40 0 5776 5297
[11811.540006] f56d3b40 00000096 f2362dd0 f56d3b40 f7c02c54 00000000 f611067c f56d3b40
[11811.540006] 00000000 00000000 f2362dd0 00000000 00000000 01000000 f611067c f56d3ca4
[11811.540006] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f56d4a10 0 5779 5297
[11811.540006] f56d4a10 00000096 f21f97a0 f56d4a10 8266fc38 00000000 f611067c f56d4a10
[11811.540006] 00000000 00000000 ee6c8000 00000000 00000000 01000000 f611067c f56d4b74
[11811.540006] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f56d0ed0 0 5781 5297
[11811.540006] f56d0ed0 00000096 f2155950 f56d0ed0 805e28e5 00000000 f611067c f56d0ed0
[11811.540006] 00000000 00000000 f2155950 00000000 00000000 01000000 f611067c f56d1034
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f56d58e0 0 5783 5297
[11811.540006] f56d58e0 00000096 e6b663b0 f56d58e0 805e28e5 00000000 f611067c f56d58e0
[11811.540006] 00000000 00000000 ee500000 00000000 00000000 01000000 f611067c f56d5a44
[11811.540006] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f56b8e0c 0 5786 5297
[11811.540006] f55767b0 00000096 00000002 f56b8e0c f56b8e14 00000000 f611067c f55767b0
[11811.540006] 00000000 00000000 eb401da0 00000000 00000000 01000000 f611067c f5576914
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<80175be5>] __do_fault+0x1a5/0x400
[11811.540006] [<801760df>] do_wp_page+0x29f/0x5a0
[11811.540006] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540006] [<80178642>] handle_mm_fault+0x532/0x5b0
[11811.540006] [<8011a07b>] do_page_fault+0xcb/0x710
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8012007b>] default_wake_function+0xb/0x10
[11811.540006] [<80119fb0>] do_page_fault+0x0/0x710
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5710000 0 5789 5297
[11811.540006] f5710000 00000096 eb71ac20 f5710000 f7c02c54 00000000 f611067c f5710000
[11811.540006] 00000000 00000000 ee6f58e0 00000000 00000000 01000000 f611067c f5710164
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5711da0 0 5792 5297
[11811.540006] f5711da0 00000096 ee4a17a0 f5711da0 805e28e5 00000000 f611067c f5711da0
[11811.540006] 00000000 00000000 ee4a17a0 00000000 00000000 01000000 f611067c f5711f04
[11811.540006] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5710ed0 0 5793 5297
[11811.540006] f5710ed0 00000096 f55b8950 f5710ed0 f7c02c54 00000000 f611067c f5710ed0
[11811.540006] 00000000 00000000 eb69e7b0 00000000 00000000 01000000 f611067c f5711034
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5712c70 0 5794 5297
[11811.540006] f5712c70 00000096 e18ddb90 f5712c70 805e28e5 00000000 f611067c f5712c70
[11811.540006] 00000000 00000000 e18ddb90 00000000 00000000 01000000 f611067c f5712dd4
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5713b40 0 5795 5297
[11811.540006] f5713b40 00000096 e19054d0 f5713b40 805e28e5 00000000 f611067c f5713b40
[11811.540006] 00000000 00000000 f23e0ed0 00000000 00000000 01000000 f611067c f5713ca4
[11811.540006] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5728e0c 0 5796 5297
[11811.540006] f5714a10 00000096 00000002 f5728e0c f5728e14 00000000 f611067c f5714a10
[11811.540006] 00000000 00000000 eb664a10 00000000 00000000 01000000 f611067c f5714b74
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<80175be5>] __do_fault+0x1a5/0x400
[11811.540006] [<801760df>] do_wp_page+0x29f/0x5a0
[11811.540006] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540006] [<80178642>] handle_mm_fault+0x532/0x5b0
[11811.540006] [<8011a07b>] do_page_fault+0xcb/0x710
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8012007b>] default_wake_function+0xb/0x10
[11811.540006] [<80119fb0>] do_page_fault+0x0/0x710
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S 82c6d620 0 5797 5297
[11811.540006] f57158e0 00000096 00000419 82c6d620 00000002 f5732e08 f611067c f57158e0
[11811.540006] 00000000 00000000 e18c0830 00000000 00000000 01000000 f5732e10 f5715a44
[11811.540006] 82cbb080 00002bcc 00000001 00000000 00000000 00000419 000000ff f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f57167b0 0 5798 5297
[11811.540006] f57167b0 00000096 f55b85f0 f57167b0 f7c02c54 00000000 f611067c f57167b0
[11811.540006] 00000000 00000000 f55b85f0 00000000 00000000 01000000 f611067c f5716914
[11811.540006] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<80185e08>] __slab_free+0x238/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5745e0c 0 5799 5297
[11811.540006] f56d67b0 00000096 00000002 f5745e0c f5745e14 00000000 f611067c f56d67b0
[11811.540006] 00000000 00000000 e6a867b0 00000000 00000000 01000000 f611067c f56d6914
[11811.540006] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<80175be5>] __do_fault+0x1a5/0x400
[11811.540006] [<801760df>] do_wp_page+0x29f/0x5a0
[11811.540006] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540006] [<80178642>] handle_mm_fault+0x532/0x5b0
[11811.540006] [<8011a07b>] do_page_fault+0xcb/0x710
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8012007b>] default_wake_function+0xb/0x10
[11811.540006] [<80119fb0>] do_page_fault+0x0/0x710
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f546d8e0 0 5800 5297
[11811.540006] f546d8e0 00000096 eb62c710 f546d8e0 829ae510 00000000 f611067c f546d8e0
[11811.540006] 00000000 00000000 eb62c710 00000000 00000000 01000000 f611067c f546da44
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5672c70 0 5807 5297
[11811.540006] f5672c70 00000096 ee4aa050 f5672c70 805e28e5 00000000 f611067c f5672c70
[11811.540006] 00000000 00000000 eb4a67b0 00000000 00000000 01000000 f611067c f5672dd4
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5773b40 0 5960 5297
[11811.540006] f5773b40 00000096 e69cd0e0 f5773b40 805e28e5 00000000 f611067c f5773b40
[11811.540006] 00000000 00000000 eb583b40 00000000 00000000 01000000 f611067c f5773ca4
[11811.540006] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5768ed0 0 6098 5297
[11811.540006] f5768ed0 00000096 e18c0b00 f5768ed0 805e28e5 00000000 f611067c f5768ed0
[11811.540006] 00000000 00000000 e18c0b00 00000000 00000000 01000000 f611067c f5769034
[11811.540006] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540006] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540006] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540006] [<80607d08>] inet_accept+0x28/0xd0
[11811.540006] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540006] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540006] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540006] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019c48d>] d_kill+0x3d/0x60
[11811.540006] [<8019d56d>] dput+0x6d/0x120
[11811.540006] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540006] [<8018a4b9>] sys_close+0x89/0x100
[11811.540006] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540006] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540006] =======================
[11811.540006] distccd S f5784a10 0 6241 5297
[11811.540006] f5784a10 00000096 ee4a1710 f5784a10 82a053a8 00000000 f611067c f5784a10
[11811.540006] 00000000 00000000 ee4a1710 00000000 00000000 01000000 f611067c f5784b74
[11811.540006] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540006] Call Trace:
[11811.540006] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540006] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5772c70 0 6440 5297
[11811.540007] f5772c70 00000096 e18b8290 f5772c70 f7c02c54 00000000 f611067c f5772c70
[11811.540007] 00000000 00000000 eb583b40 00000000 00000000 01000000 f611067c f5772dd4
[11811.540007] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5791da0 0 6591 5297
[11811.540007] f5791da0 00000096 eb778e60 f5791da0 f7c02c54 00000000 f611067c f5791da0
[11811.540007] 00000000 00000000 eb778e60 00000000 00000000 01000000 f611067c f5791f04
[11811.540007] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5781da0 0 6788 5297
[11811.540007] f5781da0 00000096 eb433440 f5781da0 805e28e5 00000000 f611067c f5781da0
[11811.540007] 00000000 00000000 ee500ed0 00000000 00000000 01000000 f611067c f5781f04
[11811.540007] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5780000 0 6835 5297
[11811.540007] f5780000 00000096 ee7617a0 f5780000 f7c02c54 00000000 f611067c f5780000
[11811.540007] 00000000 00000000 ee7617a0 00000000 00000000 01000000 f611067c f5780164
[11811.540007] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5782c70 0 6900 5297
[11811.540007] f5782c70 00000096 ee6288c0 f5782c70 805e28e5 00000000 f611067c f5782c70
[11811.540007] 00000000 00000000 ee6288c0 00000000 00000000 01000000 f611067c f5782dd4
[11811.540007] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f557ca10 0 6973 5297
[11811.540007] f557ca10 00000096 e6a66ef0 f557ca10 805e28e5 00000000 f611067c f557ca10
[11811.540007] 00000000 00000000 eb614a10 00000000 00000000 01000000 f611067c f557cb74
[11811.540007] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5579da0 0 7021 5297
[11811.540007] f5579da0 00000096 ee4e9830 f5579da0 f7c02c54 00000000 f611067c f5579da0
[11811.540007] 00000000 00000000 ee7267b0 00000000 00000000 01000000 f611067c f5579f04
[11811.540007] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5794a10 0 7079 5297
[11811.540007] f5794a10 00000096 e18c09e0 f5794a10 805e28e5 00000000 f611067c f5794a10
[11811.540007] 00000000 00000000 e18c09e0 00000000 00000000 01000000 f611067c f5794b74
[11811.540007] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S 82c1a620 0 7147 5297
[11811.540007] f5780ed0 00000096 00000019 82c1a620 00000002 00000000 f611067c f5780ed0
[11811.540007] 00000000 00000000 eb610000 00000000 00000000 01000000 f611067c f5781034
[11811.540007] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5573b40 0 7199 5297
[11811.540007] f5573b40 00000096 ee702f80 f5573b40 805e28e5 00000000 f611067c f5573b40
[11811.540007] 00000000 00000000 ee702f80 00000000 00000000 01000000 f611067c f5573ca4
[11811.540007] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5790000 0 7302 5297
[11811.540007] f5790000 00000096 00000000 f5790000 806ab08b 00000000 f611067c f5790000
[11811.540007] f61106a0 00000046 f5578000 00000000 00000000 01000000 f611067c f5790164
[11811.540007] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab08b>] schedule+0x25b/0x670
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5792c70 0 7378 5297
[11811.540007] f5792c70 00000096 f56e2950 f5792c70 805e28e5 00000000 f611067c f5792c70
[11811.540007] 00000000 00000000 f56e2950 00000000 00000000 01000000 f611067c f5792dd4
[11811.540007] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f57758e0 0 7460 5297
[11811.540007] f57758e0 00000096 ee6ab290 f57758e0 f7c02c54 00000000 f611067c f57758e0
[11811.540007] 00000000 00000000 e199ca10 00000000 00000000 01000000 f611067c f5775a44
[11811.540007] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f205ce0c 0 7466 5297
[11811.540007] f6a62c70 00000096 00000002 f205ce0c f205ce14 00000000 f611067c f6a62c70
[11811.540007] 00000000 00000000 f21f0ed0 00000000 00000000 01000000 f611067c f6a62dd4
[11811.540007] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<80175be5>] __do_fault+0x1a5/0x400
[11811.540007] [<801760df>] do_wp_page+0x29f/0x5a0
[11811.540007] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540007] [<80178642>] handle_mm_fault+0x532/0x5b0
[11811.540007] [<8011a07b>] do_page_fault+0xcb/0x710
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8012007b>] default_wake_function+0xb/0x10
[11811.540007] [<80119fb0>] do_page_fault+0x0/0x710
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f55758e0 0 7467 5297
[11811.540007] f55758e0 00000096 ee5105f0 f55758e0 f7c02c54 00000000 f611067c f55758e0
[11811.540007] 00000000 00000000 ee4758e0 00000000 00000000 01000000 f611067c f5575a44
[11811.540007] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5793b40 0 7468 5297
[11811.540007] f5793b40 00000096 eb470dd0 f5793b40 829e9220 00000000 f611067c f5793b40
[11811.540007] 00000000 00000000 e6ba2c70 00000000 00000000 01000000 f611067c f5793ca4
[11811.540007] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5790ed0 0 7469 5297
[11811.540007] f5790ed0 00000096 e6a04ef0 f5790ed0 f7c02c54 00000000 f611067c f5790ed0
[11811.540007] 00000000 00000000 e6a04ef0 00000000 00000000 01000000 f611067c f5791034
[11811.540007] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f57867b0 0 7471 5297
[11811.540007] f57867b0 00000096 e69f9b00 f57867b0 805e28e5 00000000 f611067c f57867b0
[11811.540007] 00000000 00000000 f54dac70 00000000 00000000 01000000 f611067c f5786914
[11811.540007] 82cbb080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] distccd S f5783b40 0 7472 5297
[11811.540007] f5783b40 00000096 eb533ef0 f5783b40 f7c02c54 00000000 f611067c f5783b40
[11811.540007] 00000000 00000000 ee630ed0 00000000 00000000 01000000 f611067c f5783ca4
[11811.540007] 82c68080 00000246 00000046 00000246 00000000 00000000 f611067c f6110640
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805e777d>] inet_csk_accept+0x11d/0x230
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80607d08>] inet_accept+0x28/0xd0
[11811.540007] [<802aff04>] selinux_socket_accept+0x24/0x60
[11811.540007] [<805b05a9>] sys_accept+0xc9/0x210
[11811.540007] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540007] [<80185d6f>] __slab_free+0x19f/0x2e0
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019c48d>] d_kill+0x3d/0x60
[11811.540007] [<8019d56d>] dput+0x6d/0x120
[11811.540007] [<805b093f>] sys_socketcall+0x24f/0x2e0
[11811.540007] [<8018a4b9>] sys_close+0x89/0x100
[11811.540007] [<802e5b64>] trace_hardirqs_on_thunk+0xc/0x10
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] sshd S f562ec68 0 7474 5085
[11811.540007] f57858e0 00000082 f562ec88 f562ec68 00000000 00000002 00000292 00000000
[11811.540007] f6b1d0ac f57858e0 f569de60 f6bd2620 00000000 01000046 00000000 f5785a44
[11811.540007] 82cbb080 00000046 00000000 00000046 00000046 00000000 00000000 f6b1ce00
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540007] [<806ad9f5>] _spin_lock_irqsave+0x75/0x90
[11811.540007] [<80620496>] unix_stream_recvmsg+0x2f6/0x540
[11811.540007] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<805ae8e3>] sock_aio_read+0x123/0x130
[11811.540007] [<8018acc5>] do_sync_read+0xd5/0x120
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<802a53ec>] security_file_permission+0xc/0x10
[11811.540007] [<8018ad6e>] rw_verify_area+0x5e/0xd0
[11811.540007] [<8018b5c7>] vfs_read+0x117/0x120
[11811.540007] [<8018baea>] sys_read+0x4a/0xa0
[11811.540007] [<80119fb0>] do_page_fault+0x0/0x710
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] sshd S f57b1b04 0 7482 7474
[11811.540007] f5774a10 00000086 00000002 f57b1b04 f57b1b0c 00000000 00000000 808e3e00
[11811.540007] f5774a10 00000000 00000000 f1f84f28 00000000 01000000 00000246 f5774b74
[11811.540007] 82c68080 00000000 00000000 f1f84f28 00000046 00000000 00000001 00000000
[11811.540007] Call Trace:
[11811.540007] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540007] [<8061f0a7>] unix_poll+0x17/0xb0
[11811.540007] [<8019868c>] do_select+0x3dc/0x500
[11811.540007] [<801982b0>] do_select+0x0/0x500
[11811.540007] [<80198d60>] __pollwait+0x0/0xe0
[11811.540007] [<80120070>] default_wake_function+0x0/0x10
[11811.540007] [<80120070>] default_wake_function+0x0/0x10
[11811.540007] [<80120070>] default_wake_function+0x0/0x10
[11811.540007] [<80120070>] default_wake_function+0x0/0x10
[11811.540007] [<805bdeae>] dev_queue_xmit+0x3e/0x3a0
[11811.540007] [<805e2d7d>] ip_finish_output+0x13d/0x2b0
[11811.540007] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540007] [<805e3462>] ip_queue_xmit+0x1a2/0x350
[11811.540007] [<801312f7>] lock_timer_base+0x27/0x60
[11811.540007] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540007] [<80130c53>] internal_add_timer+0x103/0x110
[11811.540007] [<801314c4>] __mod_timer+0x94/0xe0
[11811.540007] [<805f5da8>] __tcp_push_pending_frames+0x118/0x770
[11811.540007] [<805b1734>] release_sock+0x24/0xc0
[11811.540007] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540007] [<805ea632>] tcp_sendmsg+0x702/0xa90
[11811.540007] [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540007] [<801987d1>] core_sys_select+0x21/0x350
[11811.540007] [<8018aba5>] do_sync_write+0xd5/0x120
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80198e8b>] sys_select+0x4b/0x1b0
[11811.540007] [<8018bb8a>] sys_write+0x4a/0xa0
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] bash S f57a4ef0 0 7489 7482
[11811.540007] f576bb40 00000092 00000002 f57a4ef0 f57a4ef8 00000000 00000004 f57a4ecc
[11811.540007] 00000000 00000000 00000001 ffffffff 00000000 01000000 f57a4ef4 f576bca4
[11811.540007] 82cbb080 ffffe748 00000000 00000000 00000000 00000000 000000ff f576bb38
[11811.540007] Call Trace:
[11811.540007] [<8012b865>] do_wait+0x705/0xa80
[11811.540007] [<80120070>] default_wake_function+0x0/0x10
[11811.540007] [<8012bc5c>] sys_wait4+0x7c/0xc0
[11811.540007] [<8012bcc7>] sys_waitpid+0x27/0x30
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] m32 S 82c6d620 0 7520 7489
[11811.540007] f576ca10 00000092 00000400 82c6d620 00000002 f2030eec 00000004 f2030ecc
[11811.540007] 00000000 00000000 00000001 ffffffff 00000000 01000000 f2030ef4 f576cb74
[11811.540007] 82cbb080 ffffe764 00000000 00000000 00000000 00000400 000000ff f576ca08
[11811.540007] Call Trace:
[11811.540007] [<8012b865>] do_wait+0x705/0xa80
[11811.540007] [<8011a07b>] do_page_fault+0xcb/0x710
[11811.540007] [<80120070>] default_wake_function+0x0/0x10
[11811.540007] [<8012bc5c>] sys_wait4+0x7c/0xc0
[11811.540007] [<8012bcc7>] sys_waitpid+0x27/0x30
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] m32 S f20d9ef0 0 7528 7520
[11811.540007] f5769da0 00000092 00000002 f20d9ef0 f20d9ef8 00000000 00000004 f20d9ecc
[11811.540007] 00000000 00000000 00000001 ffffffff 00000000 01000000 f20d9ef4 f5769f04
[11811.540007] 82cbb080 ffffe764 00000000 00000000 00000000 00000000 000000ff f5769d98
[11811.540007] Call Trace:
[11811.540007] [<8012b865>] do_wait+0x705/0xa80
[11811.540007] [<80120070>] default_wake_function+0x0/0x10
[11811.540007] [<8012bc5c>] sys_wait4+0x7c/0xc0
[11811.540007] [<8012bcc7>] sys_waitpid+0x27/0x30
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] tee S 00000046 0 7529 7520
[11811.540007] f5768000 00000082 00000046 00000046 00000000 00000000 f55d35f0 00000046
[11811.540007] 00000000 00000001 00000001 8013b87c 00000000 01000046 00000292 f5768164
[11811.540007] 82c68080 00000002 00000001 f4d5a5a4 f4d5a5c8 00000246 00000000 f55d35e0
[11811.540007] Call Trace:
[11811.540007] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540007] [<80191630>] pipe_wait+0x50/0x80
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<80191bc3>] pipe_read+0xe3/0x3f0
[11811.540007] [<801d3ab0>] ext3_file_write+0x30/0xc0
[11811.540007] [<8018acc5>] do_sync_read+0xd5/0x120
[11811.540007] [<80107035>] do_IRQ+0x85/0xe0
[11811.540007] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540007] [<802a53ec>] security_file_permission+0xc/0x10
[11811.540007] [<8018ad6e>] rw_verify_area+0x5e/0xd0
[11811.540007] [<8018abf0>] do_sync_read+0x0/0x120
[11811.540007] [<8018b549>] vfs_read+0x99/0x120
[11811.540007] [<8018baea>] sys_read+0x4a/0xa0
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] time S f20e8f04 0 7531 7528
[11811.540007] f576e7b0 00000086 00000002 f20e8f04 f20e8f0c 00000000 00000004 f20e8ee0
[11811.540007] 00000000 00000000 00000001 ffffffff 00000000 01000000 f20e8f08 f576e914
[11811.540007] 82cbb080 ffffe764 00000000 00000000 00000000 00000000 000000ff f576e7a8
[11811.540007] Call Trace:
[11811.540007] [<8012b865>] do_wait+0x705/0xa80
[11811.540007] [<80132367>] do_sigaction+0xe7/0x170
[11811.540007] [<80120070>] default_wake_function+0x0/0x10
[11811.540007] [<8012bc5c>] sys_wait4+0x7c/0xc0
[11811.540007] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540007] =======================
[11811.540007] make S f20dbf04 0 7532 7531
[11811.540007] f576d8e0 00000086 00000002 f20dbf04 f20dbf0c 00000000 00000004 f20dbee0
[11811.540007] 00000000 00000000 00000001 ffffffff 00000000 01000000 f20dbf08 f576da44
[11811.540007] 82cbb080 00003742 00000000 00000000 00000000 00000000 000000ff f576d8d8
[11811.540007] Call Trace:
[11811.540008] [<8012b865>] do_wait+0x705/0xa80
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<8012bc5c>] sys_wait4+0x7c/0xc0
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] make S eb7ecf04 0 10479 7532
[11811.540008] e6898000 00000086 00000002 eb7ecf04 eb7ecf0c 00000000 00000004 eb7ecee0
[11811.540008] 00000000 00000000 00000001 ffffffff 00000000 01000000 eb7ecf08 e6898164
[11811.540008] 82c68080 00003ceb 00000000 00000000 00000000 00000000 000000ff e6897ff8
[11811.540008] Call Trace:
[11811.540008] [<8012b865>] do_wait+0x705/0xa80
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<8012bc5c>] sys_wait4+0x7c/0xc0
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] make S ee7dbee0 0 17987 10479
[11811.540008] eb69d8e0 00000086 00000000 ee7dbee0 00000001 802aa1b4 00000004 ee7dbee0
[11811.540008] 00000000 00000000 00000001 ffffffff 00000000 01000000 00000246 eb69da44
[11811.540008] 82c68080 00000003 00000004 00000000 00000001 8098c800 00000000 eb69d8d8
[11811.540008] Call Trace:
[11811.540008] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540008] [<8012b865>] do_wait+0x705/0xa80
[11811.540008] [<8018eb8e>] sys_stat64+0x1e/0x30
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<8012bc5c>] sys_wait4+0x7c/0xc0
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] sh S eb5bdecc 0 19573 17987
[11811.540008] ee631da0 00000092 00000000 eb5bdecc 00000001 802aa1b4 00000004 eb5bdecc
[11811.540008] 00000000 00000000 00000001 ffffffff 00000000 01000000 00000246 ee631f04
[11811.540008] 82cbb080 00000003 00000004 00000000 00000001 8098c800 00000000 ee631d98
[11811.540008] Call Trace:
[11811.540008] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540008] [<8012b865>] do_wait+0x705/0xa80
[11811.540008] [<8013f7f4>] up_read+0x14/0x30
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<8012bc5c>] sys_wait4+0x7c/0xc0
[11811.540008] [<8012bcc7>] sys_waitpid+0x27/0x30
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] distcc S e6945b04 0 19578 19573
[11811.540008] eb4058e0 00000086 00000002 e6945b04 e6945b0c 00000000 00000046 80b6a7e0
[11811.540008] e6945b2c 0011f5a7 00000002 80130c53 00000000 01b69d80 e6945b08 eb405a44
[11811.540008] 82c68080 00118078 00000000 00000000 00000000 00000000 000000ff e6945b2c
[11811.540008] Call Trace:
[11811.540008] [<80130c53>] internal_add_timer+0x103/0x110
[11811.540008] [<806ab7da>] schedule_timeout+0x4a/0xd0
[11811.540008] [<801310e0>] process_timeout+0x0/0x40
[11811.540008] [<806ab7d5>] schedule_timeout+0x45/0xd0
[11811.540008] [<8019868c>] do_select+0x3dc/0x500
[11811.540008] [<801982b0>] do_select+0x0/0x500
[11811.540008] [<80198d60>] __pollwait+0x0/0xe0
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<805bdf56>] dev_queue_xmit+0xe6/0x3a0
[11811.540008] [<805bdeae>] dev_queue_xmit+0x3e/0x3a0
[11811.540008] [<805e2d7d>] ip_finish_output+0x13d/0x2b0
[11811.540008] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540008] [<805e3462>] ip_queue_xmit+0x1a2/0x350
[11811.540008] [<8013b7ad>] finish_wait+0x2d/0x70
[11811.540008] [<805b1734>] release_sock+0x24/0xc0
[11811.540008] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540008] [<805e9a14>] tcp_sendpage+0xb4/0x5d0
[11811.540008] [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540008] [<801987d1>] core_sys_select+0x21/0x350
[11811.540008] [<801aab54>] splice_from_pipe+0x64/0x80
[11811.540008] [<801a16cd>] mnt_drop_write+0x1d/0xd0
[11811.540008] [<806ad489>] _spin_lock+0x39/0x70
[11811.540008] [<806ad344>] _spin_unlock+0x14/0x20
[11811.540008] [<801aa506>] splice_direct_to_actor+0x136/0x170
[11811.540008] [<801aa260>] direct_splice_actor+0x0/0x40
[11811.540008] [<801aa581>] do_splice_direct+0x41/0x70
[11811.540008] [<80198f27>] sys_select+0xe7/0x1b0
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] named S e6b98f8c 0 24137 1
[11811.540008] e689ca10 00200092 00000002 e6b98f8c e6b98f94 00000000 e689ca10 806ad940
[11811.540008] 00200046 00000000 00000000 e19e7e14 00000000 01200046 e6b98f90 e689cb74
[11811.540008] 82cbb080 0001e2ae 00000000 00000000 00000000 00000000 000000ff 7fc53fa0
[11811.540008] Call Trace:
[11811.540008] [<806ad940>] _spin_unlock_irq+0x20/0x30
[11811.540008] [<801344af>] sys_rt_sigsuspend+0xaf/0xd0
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] named S 00200046 0 24138 1
[11811.540008] e689bb40 00200082 00000000 00200046 00200046 00000000 00000000 f2287e30
[11811.540008] 00200046 00000000 00000001 00000001 00000000 01200046 f2287e20 e689bca4
[11811.540008] 82cbb080 00200002 00000000 00000001 f2287e20 00200282 00000000 80d4e798
[11811.540008] Call Trace:
[11811.540008] [<8014a175>] futex_wait+0x345/0x3b0
[11811.540008] [<8018c534>] fget_light+0xc4/0xe0
[11811.540008] [<805b0172>] sys_recvmsg+0x222/0x230
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<8014b01c>] do_futex+0xbc/0x950
[11811.540008] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540008] [<8014b947>] sys_futex+0x97/0x120
[11811.540008] [<8018bbd8>] sys_write+0x98/0xa0
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] named S 00200046 0 24139 1
[11811.540008] eb69ac70 00200082 00000000 00200046 00200046 00000000 00000000 ee73ae30
[11811.540008] 00200046 00000000 00000001 00000001 00000000 01200046 ee73ae20 eb69add4
[11811.540008] 82cbb080 00200002 00000000 00000001 ee73ae20 00200282 00000000 80d4e798
[11811.540008] Call Trace:
[11811.540008] [<8014a175>] futex_wait+0x345/0x3b0
[11811.540008] [<8018c534>] fget_light+0xc4/0xe0
[11811.540008] [<805b0172>] sys_recvmsg+0x222/0x230
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<8014b01c>] do_futex+0xbc/0x950
[11811.540008] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540008] [<8014b947>] sys_futex+0x97/0x120
[11811.540008] [<8018bbd8>] sys_write+0x98/0xa0
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] named S e6990e64 0 24140 1
[11811.540008] eb699da0 00200082 80a6805c e6990e64 00200002 80a68058 00000001 82c1aae0
[11811.540008] 00200292 00000034 82c1ab34 00200292 00000000 01990e64 8013ef28 eb699f04
[11811.540008] 82c68080 809f7ae0 26fa1e00 00000b38 00000000 82c1aae0 00200292 e6990e64
[11811.540008] Call Trace:
[11811.540008] [<8013ef28>] hrtimer_start+0xd8/0x180
[11811.540008] [<8014a164>] futex_wait+0x334/0x3b0
[11811.540008] [<8013ea20>] hrtimer_wakeup+0x0/0x20
[11811.540008] [<8014a04b>] futex_wait+0x21b/0x3b0
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<8014b01c>] do_futex+0xbc/0x950
[11811.540008] [<80141077>] getnstimeofday+0x37/0x100
[11811.540008] [<8013f1be>] ktime_get_ts+0x1e/0x50
[11811.540008] [<8013f208>] ktime_get+0x18/0x40
[11811.540008] [<8014b947>] sys_futex+0x97/0x120
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] named S 00000000 0 24141 1
[11811.540008] eb69e7b0 00200086 00000002 00000000 f5fcbc28 eb69e7b0 00000000 00000000
[11811.540008] 00200046 00200046 00000000 f1f87328 00000000 01200046 00000000 eb69e914
[11811.540008] 82cbb080 00000000 00000000 f1f87328 00200046 00000000 00000001 00000000
[11811.540008] Call Trace:
[11811.540008] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540008] [<805e92a7>] tcp_poll+0x17/0x130
[11811.540008] [<8019868c>] do_select+0x3dc/0x500
[11811.540008] [<801982b0>] do_select+0x0/0x500
[11811.540008] [<80198d60>] __pollwait+0x0/0xe0
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540008] [<801987d1>] core_sys_select+0x21/0x350
[11811.540008] [<8018acc5>] do_sync_read+0xd5/0x120
[11811.540008] [<8014b6bd>] do_futex+0x75d/0x950
[11811.540008] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540008] [<802a53ec>] security_file_permission+0xc/0x10
[11811.540008] [<8018ad6e>] rw_verify_area+0x5e/0xd0
[11811.540008] [<80198e8b>] sys_select+0x4b/0x1b0
[11811.540008] [<8018bb38>] sys_read+0x98/0xa0
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] sshd S 00000001 0 25210 5085
[11811.540008] e6b40ed0 00000082 00000000 00000001 00000000 00000000 00000000 00000000
[11811.540008] f6b1fe2c e6b40ed0 802aa1b4 00100000 00000000 01000046 00000000 e6b41034
[11811.540008] 82c68080 00000046 00000000 00000046 00000046 00000000 00000000 f6b1fb80
[11811.540008] Call Trace:
[11811.540008] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540008] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540008] [<8013b87c>] prepare_to_wait+0x1c/0x60
[11811.540008] [<806ad9f5>] _spin_lock_irqsave+0x75/0x90
[11811.540008] [<80620496>] unix_stream_recvmsg+0x2f6/0x540
[11811.540008] [<802a9244>] avc_has_perm_noaudit+0x184/0x300
[11811.540008] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540008] [<805ae8e3>] sock_aio_read+0x123/0x130
[11811.540008] [<8018acc5>] do_sync_read+0xd5/0x120
[11811.540008] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540008] [<802a53ec>] security_file_permission+0xc/0x10
[11811.540008] [<8018ad6e>] rw_verify_area+0x5e/0xd0
[11811.540008] [<8018b5c7>] vfs_read+0x117/0x120
[11811.540008] [<80188d0e>] filp_close+0x3e/0x70
[11811.540008] [<8018baea>] sys_read+0x4a/0xa0
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] sshd S f2029b04 0 25214 25210
[11811.540008] e6b40000 00000086 00000002 f2029b04 f2029b0c 00000000 00000000 00000000
[11811.540008] 808f2d30 00000046 00000000 00000001 00000000 011202e4 f2029b08 e6b40164
[11811.540008] 82cbb080 001190b2 00000000 00000000 00000000 00000000 000000ff 00000000
[11811.540008] Call Trace:
[11811.540008] [<806ab80c>] schedule_timeout+0x7c/0xd0
[11811.540008] [<80346ec0>] normal_poll+0x0/0x160
[11811.540008] [<80346ec0>] normal_poll+0x0/0x160
[11811.540008] [<803424d6>] tty_poll+0x76/0x80
[11811.540008] [<8019868c>] do_select+0x3dc/0x500
[11811.540008] [<801982b0>] do_select+0x0/0x500
[11811.540008] [<80198d60>] __pollwait+0x0/0xe0
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<805bdeae>] dev_queue_xmit+0x3e/0x3a0
[11811.540008] [<805e2d7d>] ip_finish_output+0x13d/0x2b0
[11811.540008] [<805e28e5>] ip_local_out+0x15/0x20
[11811.540008] [<805e3462>] ip_queue_xmit+0x1a2/0x350
[11811.540008] [<803461ae>] n_tty_receive_buf+0x34e/0x1060
[11811.540008] [<801312f7>] lock_timer_base+0x27/0x60
[11811.540008] [<8016138b>] marker_probe_cb+0x3b/0x90
[11811.540008] [<80130c53>] internal_add_timer+0x103/0x110
[11811.540008] [<801314c4>] __mod_timer+0x94/0xe0
[11811.540008] [<805f5da8>] __tcp_push_pending_frames+0x118/0x770
[11811.540008] [<805b1734>] release_sock+0x24/0xc0
[11811.540008] [<806ad4f8>] _spin_lock_bh+0x38/0x60
[11811.540008] [<805ea632>] tcp_sendmsg+0x702/0xa90
[11811.540008] [<801989a8>] core_sys_select+0x1f8/0x350
[11811.540008] [<801987d1>] core_sys_select+0x21/0x350
[11811.540008] [<8018aba5>] do_sync_write+0xd5/0x120
[11811.540008] [<8013b6e0>] autoremove_wake_function+0x0/0x40
[11811.540008] [<80198e8b>] sys_select+0x4b/0x1b0
[11811.540008] [<8018bb8a>] sys_write+0x4a/0xa0
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] bash S e185decc 0 25217 25214
[11811.540008] f63f3b40 00000092 00000000 e185decc 00000001 802aa1b4 00000004 e185decc
[11811.540008] 00000000 00000000 00000001 ffffffff 00000000 01000000 00000246 f63f3ca4
[11811.540008] 82cbb080 00000003 0000000e 00000000 00000001 8098c800 00000000 f63f3b38
[11811.540008] Call Trace:
[11811.540008] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540008] [<8012b865>] do_wait+0x705/0xa80
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<8012bc5c>] sys_wait4+0x7c/0xc0
[11811.540008] [<8012bcc7>] sys_waitpid+0x27/0x30
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] su S ee4e4ecc 0 25266 25217
[11811.540008] eb400000 00000092 00000000 ee4e4ecc 00000001 802aa1b4 00000004 ee4e4ecc
[11811.540008] 00000000 00000000 00000001 ffffffff 00000000 01000000 00000246 eb400164
[11811.540008] 82cbb080 00000003 00000006 00000000 00000001 8098c800 00000000 eb3ffff8
[11811.540008] Call Trace:
[11811.540008] [<802aa1b4>] avc_has_perm+0x64/0x80
[11811.540008] [<8012b865>] do_wait+0x705/0xa80
[11811.540008] [<80120070>] default_wake_function+0x0/0x10
[11811.540008] [<8012bc5c>] sys_wait4+0x7c/0xc0
[11811.540008] [<8012bcc7>] sys_waitpid+0x27/0x30
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
[11811.540008] bash R running 0 25268 25266
[11811.540008] 00000034 00000000 80129157 00000086 fffc10c6 80b69820 00000000 80b69829
[11811.540008] 00000000 00000046 e6bc4ee0 80819378 00002e23 00083d68 e6bc4e82 00000004
[11811.540008] 00000046 00000000 00000009 315b4ea6 31313831 3034352e 5d383030 e6bc0020
[11811.540008] Call Trace:
[11811.540008] [<80129157>] release_console_sem+0x1b7/0x1f0
[11811.540008] [<80140dab>] sched_clock_cpu+0xbb/0x140
[11811.540008] [<801395d8>] __kernel_text_address+0x18/0x30
[11811.540008] [<80104f3e>] dump_trace+0x5e/0xf0
[11811.540008] [<80104ff7>] show_trace_log_lvl+0x27/0x50
[11811.540008] [<801050d4>] show_stack_log_lvl+0xb4/0xe0
[11811.540008] [<80105ab5>] show_stack+0x35/0x50
[11811.540008] [<80124e36>] show_state_filter+0x56/0x90
[11811.540008] [<8035421a>] __handle_sysrq+0x7a/0x120
[11811.540008] [<801c7fe0>] write_sysrq_trigger+0x0/0x30
[11811.540008] [<801c800b>] write_sysrq_trigger+0x2b/0x30
[11811.540008] [<801c0ed4>] proc_reg_write+0x64/0xa0
[11811.540008] [<801c0e70>] proc_reg_write+0x0/0xa0
[11811.540008] [<8018b42b>] vfs_write+0x9b/0x120
[11811.540008] [<8018bb8a>] sys_write+0x4a/0xa0
[11811.540008] [<80119fb0>] do_page_fault+0x0/0x710
[11811.540008] [<80103e48>] sysenter_past_esp+0x78/0xc4
[11811.540008] =======================
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 16:54 ` Ingo Molnar
@ 2008-05-26 17:08 ` Ilpo Järvinen
2008-05-26 18:12 ` Ingo Molnar
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-26 17:08 UTC (permalink / raw)
To: Ingo Molnar
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
[-- Attachment #1: Type: TEXT/PLAIN, Size: 2256 bytes --]
On Mon, 26 May 2008, Ingo Molnar wrote:
>
> * Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
>
> > > > Hmm, readfds is NULL isn't it?!? Are you sure you straced the
> > > > right process?
> > >
> > > yes, i'm stracing the task that is hung unexpectedly.
> >
> > But that wasn't the receiving process? (I didn't quickly find into
> > which direction distcc ports go, so I couldn't confirm this). If you
> > still have that situation at hand, could you check which is the
> > receiving process (e.g., using netstat -p, the end which has Recv-Q is
> > the right one) and where it's stuck?
>
> it wasnt the receiving process. There's no receiving process - which is
> weird:
>
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp 0 207232 europe:37198 europe:distcc ESTABLISHED 19578/distcc
> tcp 0 0 europe:ssh dione:36284 ESTABLISHED -
> tcp 0 0 europe:ssh e2:45910 ESTABLISHED -
> tcp 72283 0 europe:distcc europe:37198 ESTABLISHED -
Just to be sure (please forgive me if you find this nearly an insult :-)),
did you have enough rights to find out the pid (ie., if that process not
owned by you then you need superuser privs for that)?
> i just gave it as a general example of why sometimes stracing a task can
> 'disturb' the observed system and can kick the TCP state machine out of
> a stall. I did not say it's occuring here.
Yeah, I understood that earlier. Similarly, I just wanted to point out
the end where the problem lies :-).
> > ...It may still be that the receiving process is stuck due to the
> > non-net related changes you have there.
>
> the socket does not seem to be owned. It should have closed down?
> Refcounting issue?
It's well possible that e.g., net namespaces have some bug in handling
of orphaned tcp.
> find below the sysrq-t dump.
...I'll have a look into that as well (though with such I'm on a more
unfamiliar territory, so it will take a moment).
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 17:08 ` Ilpo Järvinen
@ 2008-05-26 18:12 ` Ingo Molnar
2008-05-26 20:41 ` Ingo Molnar
0 siblings, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-05-26 18:12 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
* Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> > Active Internet connections (w/o servers)
> > Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> > tcp 0 207232 europe:37198 europe:distcc ESTABLISHED 19578/distcc
> > tcp 0 0 europe:ssh dione:36284 ESTABLISHED -
> > tcp 0 0 europe:ssh e2:45910 ESTABLISHED -
> > tcp 72283 0 europe:distcc europe:37198 ESTABLISHED -
>
> Just to be sure (please forgive me if you find this nearly an insult
> :-)), did you have enough rights to find out the pid (ie., if that
> process not owned by you then you need superuser privs for that)?
yes, i ran netstat -p as root.
> > the socket does not seem to be owned. It should have closed down?
> > Refcounting issue?
>
> It's well possible that e.g., net namespaces have some bug in handling
> of orphaned tcp.
yes, that would match the symptoms i think. I half-assumed that it's a
state machine problem so i didnt even check what the reader does - and
in this case it appears to not exist at all anymore ;-)
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 18:12 ` Ingo Molnar
@ 2008-05-26 20:41 ` Ingo Molnar
2008-05-26 21:20 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-05-26 20:41 UTC (permalink / raw)
To: Ilpo J?rvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
* Ingo Molnar <mingo@elte.hu> wrote:
> > It's well possible that e.g., net namespaces have some bug in
> > handling of orphaned tcp.
>
> yes, that would match the symptoms i think. I half-assumed that it's a
> state machine problem so i didnt even check what the reader does - and
> in this case it appears to not exist at all anymore ;-)
after ~7 hours of uptime the networking code produced this assertion:
[25441.140000] KERNEL: assertion (!sk->sk_ack_backlog) failed at
net/ipv4/inet_connection_sock.c (642)
no further information in the logs. I kept the failing system booted up
for 8 hours and will reboot it now.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 20:41 ` Ingo Molnar
@ 2008-05-26 21:20 ` Ilpo Järvinen
0 siblings, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-26 21:20 UTC (permalink / raw)
To: Ingo Molnar, Pavel Emelyanov, Denis V. Lunev
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
...Added Pavel & Denis.
On Mon, 26 May 2008, Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
>
> > > It's well possible that e.g., net namespaces have some bug in
> > > handling of orphaned tcp.
> >
> > yes, that would match the symptoms i think. I half-assumed that it's a
> > state machine problem so i didnt even check what the reader does - and
> > in this case it appears to not exist at all anymore ;-)
I keep wondering why distcc in the first place has a connection orphaned
before it has read all data from it, I guess it should not yet be
speculatively executing anything or so :-). Anyway, there should be RST
due to that data because nobody is going read it after the process in gone
but since state is still ESTABLISHED, I doubt that tcp_close() was ever
executed which adds some strangeness.
> after ~7 hours of uptime the networking code produced this assertion:
>
> [25441.140000] KERNEL: assertion (!sk->sk_ack_backlog) failed at
> net/ipv4/inet_connection_sock.c (642)
...I checked /proc/net/tcp output you gave earlier, and indeed, it shows
1 in sk->sk_ack_backlog of the LISTENing socket at that time.
> no further information in the logs. I kept the failing system booted up
> for 8 hours and will reboot it now.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 11:56 [bug] stuck localhost TCP connections, v2.6.26-rc3+ Ingo Molnar
2008-05-26 13:28 ` Ilpo Järvinen
2008-05-26 16:24 ` Arjan van de Ven
@ 2008-05-28 9:27 ` Peter Zijlstra
2008-05-31 14:25 ` Håkon Løvdal
2008-05-29 8:45 ` Ingo Molnar
3 siblings, 1 reply; 96+ messages in thread
From: Peter Zijlstra @ 2008-05-28 9:27 UTC (permalink / raw)
To: Ingo Molnar
Cc: linux-kernel, netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton, Ilpo Järvinen
On Mon, 2008-05-26 at 13:56 +0200, Ingo Molnar wrote:
> in an overnight -tip testruns that is based on recent -git i got two
> stuck TCP connections:
>
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State
> tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632
> ESTABLISHED
> tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015
> ESTABLISHED
>
> on a previously reliable machine. That connection has been stuck for 9
> hours so it does not time out, etc. - and the distcc run that goes over
> that connection is stuck as well.
>
> kernel config is attached.
>
> in terms of debugging there's not much i can do i'm afraid. It's not
> possible to get a tcpdump of this incident, given the extreme amount of
> load these testboxes handle. This problem started sometime around rc3
> and it occured on two boxes (on a laptop and on a desktop), both are SMP
> Core2Duo based systems. I never saw this problem before on thousands of
> similar bootups, so i'm 99.9% sure the bug is either new or became
> easier to trigger.
>
> It's not possible to bisect it as it needs up to 12 hours of heavy
> workload to trigger. The incident happened about 5 times since the first
> incident a couple of days ago - 4 times on one box and once on another
> box. The first failing head i became aware of was 78b58e549a3098. (-tip
> has other changes beyond -git but changes nothing in networking.)
>
> One clue (which might or might not matter) is that distcc is one of the
> very few applications that makes use of sendfile().
Just a quick note to say, me too!!
same scenario: distcc on localhost.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 11:56 [bug] stuck localhost TCP connections, v2.6.26-rc3+ Ingo Molnar
` (2 preceding siblings ...)
2008-05-28 9:27 ` Peter Zijlstra
@ 2008-05-29 8:45 ` Ingo Molnar
2008-05-29 11:14 ` Ilpo Järvinen
3 siblings, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-05-29 8:45 UTC (permalink / raw)
To: linux-kernel; +Cc: netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
* Ingo Molnar <mingo@elte.hu> wrote:
> in an overnight -tip testruns that is based on recent -git i got two
> stuck TCP connections:
>
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED
> tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015 ESTABLISHED
update: in the past 5 days of -tip testing i've gathered about 10
randconfig kernel configs that all produced such failures.
Since the bug itself is very elusive (it takes up to 50 boot +
kernel-rebuild-via-distccc iterations to trigger) bisection was still
not an option - but with 10 configs statistical analysis of the configs
is now possible.
I made a histogram of all kernel options present in those configs, and
one networking related kernel option stood out:
5 CONFIG_TCP_CONG_ADVANCED=y
6 CONFIG_INET_TCP_DIAG=y
6 CONFIG_TCP_MD5SIG=y
9 CONFIG_TCP_CONG_CUBIC=y
that code is called in the bootlogs:
> [ 13.279410] calling cubictcp_register+0x0/0x80
> [ 13.279412] TCP cubic registered
the likelyhood of CONFIG_TCP_CONG_CUBIC=y being enabled in my randconfig
runs is 75%. The likelyhood of CONFIG_TCP_CONG_CUBIC=y being enabled in
10 configs in a row is 0.75^10, or 5.6%. So statistical analysis can say
it with a 95% confidence that the presence of this option correlates to
the hung sockets.
i have started testing this theory now, via the patch below, which turns
off TCP_CONG_CUBIC. It will take about 50 bootups on the affected
testsystems to confirm. (it will take a couple of hours today as not all
testsystems show these hung socket symptoms)
distributions enable TCP_CONG_CUBIC by default:
$ grep CUBIC /boot/config-2.6.24.7-92.fc8
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_CUBIC=y
which would explain why Arjan and Peter triggered similar hangs as well.
Ingo
---------------------->
Subject: qa: no TCP_CONG_CUBIC
From: Ingo Molnar <mingo@elte.hu>
Date: Thu May 29 09:45:51 CEST 2008
---
net/ipv4/Kconfig | 4 ++++
1 file changed, 4 insertions(+)
Index: tip/net/ipv4/Kconfig
===================================================================
--- tip.orig/net/ipv4/Kconfig
+++ tip/net/ipv4/Kconfig
@@ -454,6 +454,8 @@ config TCP_CONG_BIC
config TCP_CONG_CUBIC
tristate "CUBIC TCP"
default y
+ depends on BROKEN_BOOT_ALLOWED
+ select BROKEN_BOOT
---help---
This is version 2.0 of BIC-TCP which uses a cubic growth function
among other techniques.
@@ -608,6 +610,8 @@ endif
config TCP_CONG_CUBIC
tristate
depends on !TCP_CONG_ADVANCED
+ depends on BROKEN_BOOT_ALLOWED
+ select BROKEN_BOOT
default y
config DEFAULT_TCP_CONG
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-29 8:45 ` Ingo Molnar
@ 2008-05-29 11:14 ` Ilpo Järvinen
2008-05-29 11:22 ` Ingo Molnar
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-29 11:14 UTC (permalink / raw)
To: Ingo Molnar
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
On Thu, 29 May 2008, Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
>
> > in an overnight -tip testruns that is based on recent -git i got two
> > stuck TCP connections:
> >
> > Active Internet connections (w/o servers)
> > Proto Recv-Q Send-Q Local Address Foreign Address State
> > tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED
> > tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015 ESTABLISHED
>
> update: in the past 5 days of -tip testing i've gathered about 10
> randconfig kernel configs that all produced such failures.
...I tried yesterday some accept (& read some) & close/exit type
stressing but I couldn't get it to show up (though I'll try longer
time later on and also fault style exiting).
> Since the bug itself is very elusive (it takes up to 50 boot +
> kernel-rebuild-via-distccc iterations to trigger) bisection was still
> not an option - but with 10 configs statistical analysis of the configs
> is now possible.
>
> I made a histogram of all kernel options present in those configs, and
> one networking related kernel option stood out:
>
> 5 CONFIG_TCP_CONG_ADVANCED=y
> 6 CONFIG_INET_TCP_DIAG=y
> 6 CONFIG_TCP_MD5SIG=y
> 9 CONFIG_TCP_CONG_CUBIC=y
>
> that code is called in the bootlogs:
>
> > [ 13.279410] calling cubictcp_register+0x0/0x80
> > [ 13.279412] TCP cubic registered
>
> the likelyhood of CONFIG_TCP_CONG_CUBIC=y being enabled in my randconfig
> runs is 75%. The likelyhood of CONFIG_TCP_CONG_CUBIC=y being enabled in
> 10 configs in a row is 0.75^10, or 5.6%. So statistical analysis can say
> it with a 95% confidence that the presence of this option correlates to
> the hung sockets.
Do I understand you correctly... it doesn't explain the tenth case out
of ten but just nine of them?
> i have started testing this theory now, via the patch below, which turns
> off TCP_CONG_CUBIC. It will take about 50 bootups on the affected
> testsystems to confirm. (it will take a couple of hours today as not all
> testsystems show these hung socket symptoms)
>
> distributions enable TCP_CONG_CUBIC by default:
>
> $ grep CUBIC /boot/config-2.6.24.7-92.fc8
> CONFIG_TCP_CONG_CUBIC=y
> CONFIG_DEFAULT_CUBIC=y
>
> which would explain why Arjan and Peter triggered similar hangs as well.
Main problem with this explanation is that congestion control modules are
only in use when TCP is in ESTABLISHED and transmitting normally, while it
has nothing to do how we enter or leave ESTABLISHED.
But if it's really that the process who owned the connection already went
away, I think we should end up into tcp_close() which changes the state
from established (and send RST too if there's still data to be received,
which would be picked up by the other end and that end would no longer
keep established either). ...A failure to send the reset would show up in
LINUX_MIB_TCPABORTFAILED. Because both ends remain in established, it kind
of excludes the possibility that something would have accidently allowed
the Recv-Q end to return back to established too (due to some bug).
To me there are mainly two weird things:
1) Why we see orphaning with data in the first place (I think distcc would
be interested to read everything, unless some worker crashed in early...
Though some timeout in distcc could explain it as well but I don't know
too well how distcc does everything)...
2) Why the connection is still in ESTABLISHED when it was orphaned and
it has some data to receive... If it had some unread data, should be in
CLOSE or in FIN_WAIT1 otherwise.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-29 11:14 ` Ilpo Järvinen
@ 2008-05-29 11:22 ` Ingo Molnar
2008-05-29 13:05 ` Evgeniy Polyakov
` (2 more replies)
0 siblings, 3 replies; 96+ messages in thread
From: Ingo Molnar @ 2008-05-29 11:22 UTC (permalink / raw)
To: Ilpo J?rvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
after about 50 bootups i got a hung test again:
titan:~/tip> netstat -nt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State
tcp 0 0 10.0.1.14:22 10.0.1.16:58062 ESTABLISHED
tcp 0 0 10.0.1.14:22 10.0.1.16:60109 ESTABLISHED
tcp 0 86368 10.0.1.14:43914 10.0.1.16:3632 ESTABLISHED
and this time with CUBIC_TCP disabled - so that was a red herring.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-29 11:22 ` Ingo Molnar
@ 2008-05-29 13:05 ` Evgeniy Polyakov
2008-05-29 13:43 ` Ingo Molnar
2008-05-29 13:08 ` Ingo Molnar
2008-05-30 18:18 ` Ingo Molnar
2 siblings, 1 reply; 96+ messages in thread
From: Evgeniy Polyakov @ 2008-05-29 13:05 UTC (permalink / raw)
To: Ingo Molnar
Cc: Ilpo J?rvinen, LKML, Netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton
Hi.
On Thu, May 29, 2008 at 01:22:57PM +0200, Ingo Molnar (mingo@elte.hu) wrote:
> titan:~/tip> netstat -nt
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State
> tcp 0 0 10.0.1.14:22 10.0.1.16:58062 ESTABLISHED
> tcp 0 0 10.0.1.14:22 10.0.1.16:60109 ESTABLISHED
> tcp 0 86368 10.0.1.14:43914 10.0.1.16:3632 ESTABLISHED
Are namespaces enabled? It is pretty non-trivial to make established
socket without process from userspace :)
--
Evgeniy Polyakov
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-29 11:22 ` Ingo Molnar
2008-05-29 13:05 ` Evgeniy Polyakov
@ 2008-05-29 13:08 ` Ingo Molnar
2008-05-29 13:48 ` Ilpo Järvinen
2008-05-30 11:09 ` Ingo Molnar
2008-05-30 18:18 ` Ingo Molnar
2 siblings, 2 replies; 96+ messages in thread
From: Ingo Molnar @ 2008-05-29 13:08 UTC (permalink / raw)
To: Ilpo J?rvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
btw., i now also have a hung socket over real network:
titan:~> netstat -nt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 86752 10.0.1.14:49921 10.0.1.16:3632 ESTABLISHED
(the scenario is similar to the incident i've analyzed before, so i wont
repeat the tcpdump, etc.)
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-29 13:05 ` Evgeniy Polyakov
@ 2008-05-29 13:43 ` Ingo Molnar
0 siblings, 0 replies; 96+ messages in thread
From: Ingo Molnar @ 2008-05-29 13:43 UTC (permalink / raw)
To: Evgeniy Polyakov
Cc: Ilpo J?rvinen, LKML, Netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton
* Evgeniy Polyakov <johnpol@2ka.mipt.ru> wrote:
> Hi.
>
> On Thu, May 29, 2008 at 01:22:57PM +0200, Ingo Molnar (mingo@elte.hu) wrote:
> > titan:~/tip> netstat -nt
> > Active Internet connections (w/o servers)
> > Proto Recv-Q Send-Q Local Address Foreign Address
> > State
> > tcp 0 0 10.0.1.14:22 10.0.1.16:58062 ESTABLISHED
> > tcp 0 0 10.0.1.14:22 10.0.1.16:60109 ESTABLISHED
> > tcp 0 86368 10.0.1.14:43914 10.0.1.16:3632 ESTABLISHED
>
> Are namespaces enabled? It is pretty non-trivial to make established
> socket without process from userspace :)
they are enabled in this particular kernel config, but there were a
couple of other failures where they were not enabled.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-29 13:08 ` Ingo Molnar
@ 2008-05-29 13:48 ` Ilpo Järvinen
2008-05-30 11:09 ` Ingo Molnar
1 sibling, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-29 13:48 UTC (permalink / raw)
To: Ingo Molnar
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton,
Evgeniy Polyakov
On Thu, 29 May 2008, Ingo Molnar wrote:
> btw., i now also have a hung socket over real network:
>
> titan:~> netstat -nt
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 86752 10.0.1.14:49921 10.0.1.16:3632 ESTABLISHED
>
> (the scenario is similar to the incident i've analyzed before, so i wont
> repeat the tcpdump, etc.)
...Yes, also the previous one you quoted was already over the network.
But I think the key is in the receiving end rather than in the sending
end. Could try to extract any information about these I mentioned earlier,
to me there are mainly two weird things:
1) Why we see orphaning with data in the first place. I think distcc would
be interested to read everything, unless some worker crashed in early...
Though some timeout in distcc could explain it as well but I don't know
too well how distcc does everything...
2) Why the receiving end of the connection is still in ESTABLISHED
without an owner... If it had some unread data, should be in CLOSE or in
FIN_WAIT1 otherwise. Ie., tcp_close() would change the state of the flow.
These both basically happen at the receiving end, though if there was
unread data the RST would also reflected state change to the sending side
and since there's window, also FIN would be sent right away and
cause sender to leave ESTABLISHED rather than that FIN would get stuck
into the tail of the TCP queue.
How to collect the information, I'm not too sure, tcp_close might well
work as a plain printk because I think it shouldn't be that noisy. But
that would likely just show that it wasn't called for a stuck receiver
at all, so it would probably end up being a dead-end anyway.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-29 13:08 ` Ingo Molnar
2008-05-29 13:48 ` Ilpo Järvinen
@ 2008-05-30 11:09 ` Ingo Molnar
2008-05-30 21:12 ` Ilpo Järvinen
1 sibling, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-05-30 11:09 UTC (permalink / raw)
To: Ilpo J?rvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
* Ingo Molnar <mingo@elte.hu> wrote:
> btw., i now also have a hung socket over real network:
last night i turned off distcc support, and got about 200 successful
bootups and zero TCP hangs (as expected - there's not much TCP traffic
if the distcc cluster is not utilized).
but that's 200 overnight tests instead of the expected 600, so this is a
major and rather crippling bug to me.
There's no good way to detect these hung sockets by me from userspace
and get rid of them. Has anyone before thought of the obvious: to write
a kernel-space "TCP socket watchdog" kernel feature that detects them
and tries to free them so that people can become aware of it?
Hung sockets is a re-occuring bug in the TCP stack after all. (and it's
a natural property of it: state machine designs are always vulnerable to
lost event problems.)
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-26 13:28 ` Ilpo Järvinen
2008-05-26 13:59 ` Ingo Molnar
@ 2008-05-30 16:23 ` Ray Lee
1 sibling, 0 replies; 96+ messages in thread
From: Ray Lee @ 2008-05-30 16:23 UTC (permalink / raw)
To: Ingo Molnar
Cc: Ilpo Järvinen, LKML, Netdev, David S. Miller,
Rafael J. Wysocki, Andrew Morton
On Mon, May 26, 2008 at 6:28 AM, Ilpo Järvinen
<ilpo.jarvinen@helsinki.fi> wrote:
> On Mon, 26 May 2008, Ingo Molnar wrote:
>
>> in an overnight -tip testruns that is based on recent -git i got two
>> stuck TCP connections:
>>
>> Active Internet connections (w/o servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State
>> tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED
>> tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015 ESTABLISHED
>>
>> on a previously reliable machine. That connection has been stuck for 9
>> hours so it does not time out, etc. - and the distcc run that goes over
>> that connection is stuck as well.
>>
>> kernel config is attached.
>>
>> in terms of debugging there's not much i can do i'm afraid. It's not
>> possible to get a tcpdump of this incident, given the extreme amount of
>> load these testboxes handle.
>
> ...but you can still tcpdump that particular flow once the situation is
> discovered to see if TCP still tries to do something, no? One needs to
> tcpdump couple of minutes at minimum. Also please get /proc/net/tcp for
> that flow around the same time.
>
>> This problem started sometime around rc3
>> and it occured on two boxes (on a laptop and on a desktop), both are SMP
>> Core2Duo based systems. I never saw this problem before on thousands of
>> similar bootups, so i'm 99.9% sure the bug is either new or became
>> easier to trigger.
>>
>> It's not possible to bisect it as it needs up to 12 hours of heavy
>> workload to trigger. The incident happened about 5 times since the first
>> incident a couple of days ago - 4 times on one box and once on another
>> box. The first failing head i became aware of was 78b58e549a3098. (-tip
>> has other changes beyond -git but changes nothing in networking.)
Okay, but in some sense you've already bisected this somewhat. I'm
assuming that your testing uses the latest tip and is refreshed daily.
If that's the case, then I would (possibly naively) expect the culprit
to show up in a:
git log -p v2.6.26-rc1..78b58e549a3098
net/{compat.c,core,ipv4,netfilter,packet,sched,socket.c}
There are only a few commits in there that appear to touch network behavior:
79d44516b4b178ffb6e2159c75584cfcfc097914
a1c1f281b84a751fdb5ff919da3b09df7297619f
62ab22278308a40bcb7f4079e9719ab8b7fe11b5
Reverting just those three and running overnight might provide a clue.
OTOH, I'm in no way a net/ expert, so if you are already working a
debugging strategy then feel free to ignore this. I'm only piping up
as it appears that the troubleshooting has stalled.
> (but there were
> some recent fixes to FRTO and retrans_stamp change could have some
> significance here)?
>
> Other than that, nothing since -rc1 seems suspicious to me (though
> I hardly understand every part of networking).
^ permalink raw reply [flat|nested] 96+ messages in thread
* [bug] stuck localhost TCP connections, v2.6.26-rc3+
@ 2008-05-30 16:31 Ray Lee
2008-05-30 21:11 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Ray Lee @ 2008-05-30 16:31 UTC (permalink / raw)
To: Ingo Molnar
Cc: Ilpo Järvinen, LKML, Netdev, David S. Miller,
Rafael J. Wysocki, Andrew Morton
(Oy, resending. Freakin' gmail; sorry for the base64 encoded noise.)
On Mon, May 26, 2008 at 6:28 AM, Ilpo <ilpo.jarvinen@helsinki.fi> wrote:
> On Mon, 26 May 2008, Ingo Molnar wrote:
>
>> in an overnight -tip testruns that is based on recent -git i got two
>> stuck TCP connections:
>>
>> Active Internet connections (w/o servers)
>> Proto Recv-Q Send-Q Local Address Foreign Address State
>> tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED
>> tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015 ESTABLISHED
>>
>> on a previously reliable machine. That connection has been stuck for 9
>> hours so it does not time out, etc. - and the distcc run that goes over
>> that connection is stuck as well.
>>
>> kernel config is attached.
>>
>> in terms of debugging there's not much i can do i'm afraid. It's not
>> possible to get a tcpdump of this incident, given the extreme amount of
>> load these testboxes handle.
>
> ...but you can still tcpdump that particular flow once the situation is
> discovered to see if TCP still tries to do something, no? One needs to
> tcpdump couple of minutes at minimum. Also please get /proc/net/tcp for
> that flow around the same time.
>
>> This problem started sometime around rc3
>> and it occured on two boxes (on a laptop and on a desktop), both are SMP
>> Core2Duo based systems. I never saw this problem before on thousands of
>> similar bootups, so i'm 99.9% sure the bug is either new or became
>> easier to trigger.
>>
>> It's not possible to bisect it as it needs up to 12 hours of heavy
>> workload to trigger. The incident happened about 5 times since the first
>> incident a couple of days ago - 4 times on one box and once on another
>> box. The first failing head i became aware of was 78b58e549a3098. (-tip
>> has other changes beyond -git but changes nothing in networking.)
Okay, but in some sense you've already bisected this somewhat. I'm
assuming that your testing uses the latest tip and is refreshed daily.
If that's the case, then I would (possibly naively) expect the culprit
to show up in a:
git log -p v2.6.26-rc1..78b58e549a3098 net/{compat.c,core,ipv4,netfilter,packet,sched,socket.c}
There are only a few commits in there that appear to touch network behavior:
79d44516b4b178ffb6e2159c75584cfcfc097914
a1c1f281b84a751fdb5ff919da3b09df7297619f
62ab22278308a40bcb7f4079e9719ab8b7fe11b5
Reverting just those three and running overnight might provide a clue.
OTOH, I'm in no way a net/ expert, so if you are already working a
debugging strategy then feel free to ignore this. I'm only piping up
as it appears that the troubleshooting has stalled.
> (but there were
> some recent fixes to FRTO and retrans_stamp change could have some
> significance here)?
>
> Other than that, nothing since -rc1 seems suspicious to me (though
> I hardly understand every part of networking).
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-29 11:22 ` Ingo Molnar
2008-05-29 13:05 ` Evgeniy Polyakov
2008-05-29 13:08 ` Ingo Molnar
@ 2008-05-30 18:18 ` Ingo Molnar
2008-05-31 6:09 ` Ingo Molnar
2 siblings, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-05-30 18:18 UTC (permalink / raw)
To: Ilpo J?rvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
* Ingo Molnar <mingo@elte.hu> wrote:
> after about 50 bootups i got a hung test again:
>
> titan:~/tip> netstat -nt
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State
> tcp 0 0 10.0.1.14:22 10.0.1.16:58062 ESTABLISHED
> tcp 0 0 10.0.1.14:22 10.0.1.16:60109 ESTABLISHED
> tcp 0 86368 10.0.1.14:43914 10.0.1.16:3632 ESTABLISHED
>
> and this time with CUBIC_TCP disabled - so that was a red herring.
ah, in retrospect i realized that this test had one flaw: some of the
systems i the build cluster already ran a newer kernel and hence were
targets for this bug.
so i turned off CONFIG_TCP_CONG_CUBIC on all the testboxes and rebooted
the cluster boxes into 2.6.25, and the hung sockets are now gone. (about
150 successful iterations)
i did another change as well: i removed the localhost distcc component.
I'll reinstate that now to make sure it's really related to
TCP_CONG_CUBIC and not to localhost networking.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-30 16:31 Ray Lee
@ 2008-05-30 21:11 ` Ilpo Järvinen
2008-05-31 6:03 ` Evgeniy Polyakov
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-30 21:11 UTC (permalink / raw)
To: Ray Lee
Cc: Ingo Molnar, LKML, Netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton
Hi Ray,
...I reorganized it a bit.
On Fri, 30 May 2008, Ray Lee wrote:
> (Oy, resending. Freakin' gmail; sorry for the base64 encoded noise.)
>
> On Mon, May 26, 2008 at 6:28 AM, Ilpo <ilpo.jarvinen@helsinki.fi> wrote:
> > On Mon, 26 May 2008, Ingo Molnar wrote:
> > >
> > > on a previously reliable machine. That connection has been stuck for 9
> > > hours so it does not time out, etc. - and the distcc run that goes over
> > > that connection is stuck as well.
> > >
> > > kernel config is attached.
> > >
> > > in terms of debugging there's not much i can do i'm afraid. It's not
> > > possible to get a tcpdump of this incident, given the extreme amount of
> > > load these testboxes handle.
> >
> > ...but you can still tcpdump that particular flow once the situation is
> > discovered to see if TCP still tries to do something, no? One needs to
> > tcpdump couple of minutes at minimum. Also please get /proc/net/tcp for
> > that flow around the same time.
> >
> > > This problem started sometime around rc3
> > > and it occured on two boxes (on a laptop and on a desktop), both are SMP
> > > Core2Duo based systems. I never saw this problem before on thousands of
> > > similar bootups, so i'm 99.9% sure the bug is either new or became
> > > easier to trigger.
> > >
> > > It's not possible to bisect it as it needs up to 12 hours of heavy
> > > workload to trigger. The incident happened about 5 times since the first
> > > incident a couple of days ago - 4 times on one box and once on another
> > > box. The first failing head i became aware of was 78b58e549a3098. (-tip
> > > has other changes beyond -git but changes nothing in networking.)
>
> Okay, but in some sense you've already bisected this somewhat. I'm
> assuming that your testing uses the latest tip and is refreshed daily.
>
> If that's the case, then I would (possibly naively) expect the culprit
> to show up in a:
> git log -p v2.6.26-rc1..78b58e549a3098
> net/{compat.c,core,ipv4,netfilter,packet,sched,socket.c}
>
> There are only a few commits in there that appear to touch network behavior:
>
> 79d44516b4b178ffb6e2159c75584cfcfc097914
> a1c1f281b84a751fdb5ff919da3b09df7297619f
> 62ab22278308a40bcb7f4079e9719ab8b7fe11b5
I think you miss here a lot of clues. First I suspected my FRTO changes as
well but later discoveries pointed elsewhere... Those fixes are for sender
behavior, which is not the problem here. It's just that once you have flow
control, the sending TCP obviously gets stuck once all "buffering"
capacity downstream is used up, and that's _correct_ sender behavior
rather than a bug in itself. Therefore both FRTO and Ingo's theory
about Cubic (though his test with 2.6.25 will definately seems currently a
useful result with or without Cubic :-)) completely fails to explain why
receiver didn't read the portion that was sitting there waiting (see
below).
Also, I think you missed one (though it's commit message seems to say
that it isn't relevant here but who knows):
1ac06e0306d0192a7a4d9ea1c9e06d355ce7e7d3
...but still that hardly would explains why the receiver queue was not
consumed.
> Reverting just those three and running overnight might provide a clue.
Of course Ingo could easily test without FRTO by playing with the sysctl,
all those three patches are not in use if tcp_frto is set to zero (he
probably didn't because I "cancelled" that request...?), but I find it
very unlikely to help any.
> OTOH, I'm in no way a net/ expert,
Me neither, I just know some about TCP, so I probably have as much
problems as you do in understanding this :-).
> so if you are already working a
> debugging strategy then feel free to ignore this. I'm only piping up
> as it appears that the troubleshooting has stalled.
...Thanks, I definately don't mind any help here. Though it probably
partially seems completely "stalled" because figuring this out leads me
more and more to a territory which is previously unknown to me (plus the
time constraints I have), not that it's a bad thing to learn & read a lot
of other code too but it just takes more time and I cannot do anything
while off-line like I could with a code that I'm familiar with.
Would you perhaps have any clue about two clearly strange things I listed
here:
http://marc.info/?l=linux-kernel&m=121207001329497&w=2
...
> > > in an overnight -tip testruns that is based on recent -git i got two
> > > stuck TCP connections:
...i.e., one connection, two endpoints:
> > > Active Internet connections (w/o servers)
> > > Proto Recv-Q Send-Q Local Address Foreign Address
> > > State
> > > tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED
> > > tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015 ESTABLISHED
^^^^^
Can you perhaps find/guess/think some explanation for this _receiver
queue_...? This was a trick question :-), as we already know that the
receiving process is no longer there and therefore obviously won't be
reading anything anymore. But that opened another question, why TCP is
then still in ESTABLISHED as orphaned TCP shouldn't be in establised state
anymore, tcp_close should have changed the state (either at close or at
process exit). I guess once it becomes known why tcp_close either wasn't
called at all or it didn't change the state of the flow (it's quite
simple, see for yourself), the cause of the bug is found (it might even be
that the process went away when it shouldn't have, either a bookkeeping
bug somewhere or real death, or something along those lines).
I was thinking of storing some info about old owner while orphaning to
struct sock and collecting that once one of the flows gets stuck but
this requires me to figure a lot unknowns out before I can just code
it.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-30 11:09 ` Ingo Molnar
@ 2008-05-30 21:12 ` Ilpo Järvinen
0 siblings, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-30 21:12 UTC (permalink / raw)
To: Ingo Molnar
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
On Fri, 30 May 2008, Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
>
> > btw., i now also have a hung socket over real network:
>
> last night i turned off distcc support, and got about 200 successful
> bootups and zero TCP hangs (as expected - there's not much TCP traffic
> if the distcc cluster is not utilized).
>
> but that's 200 overnight tests instead of the expected 600, so this is a
> major and rather crippling bug to me.
>
> There's no good way to detect these hung sockets by me from userspace
> and get rid of them. Has anyone before thought of the obvious: to write
> a kernel-space "TCP socket watchdog" kernel feature that detects them
> and tries to free them so that people can become aware of it?
>
> Hung sockets is a re-occuring bug in the TCP stack after all. (and it's
> a natural property of it: state machine designs are always vulnerable to
> lost event problems.)
...It is quite easy to kill a TCP flow with enough information (seqnos
which one could collect e.g., from a tcpdump), just a RST is necessary,
so user-space could do that too.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-30 21:11 ` Ilpo Järvinen
@ 2008-05-31 6:03 ` Evgeniy Polyakov
2008-05-31 10:05 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Evgeniy Polyakov @ 2008-05-31 6:03 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: Ray Lee, Ingo Molnar, LKML, Netdev, David S. Miller,
Rafael J. Wysocki, Andrew Morton
On Sat, May 31, 2008 at 12:11:32AM +0300, Ilpo Järvinen (ilpo.jarvinen@helsinki.fi) wrote:
> > > > Active Internet connections (w/o servers)
> > > > Proto Recv-Q Send-Q Local Address Foreign Address
> > > > State
> > > > tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED
> > > > tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015 ESTABLISHED
>
> ^^^^^
>
> Can you perhaps find/guess/think some explanation for this _receiver
> queue_...? This was a trick question :-), as we already know that the
> receiving process is no longer there and therefore obviously won't be
> reading anything anymore. But that opened another question, why TCP is
> then still in ESTABLISHED as orphaned TCP shouldn't be in establised state
> anymore, tcp_close should have changed the state (either at close or at
> process exit). I guess once it becomes known why tcp_close either wasn't
> called at all or it didn't change the state of the flow (it's quite
> simple, see for yourself), the cause of the bug is found (it might even be
> that the process went away when it shouldn't have, either a bookkeeping
> bug somewhere or real death, or something along those lines).
In that case pid Ingo straced was still valid and existed (ans stuck
probably in tcp_sendpage() - sysrq-t dump was messy), so process still
exist, but sock somewhat lost its reference to socket. AFICS all (three)
calls for sk_orphan() can not happen without state change to something
closing (like fin wait or close), except some linger things and
listening socket, which frees its connection requests.
--
Evgeniy Polyakov
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-30 18:18 ` Ingo Molnar
@ 2008-05-31 6:09 ` Ingo Molnar
2008-05-31 11:46 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-05-31 6:09 UTC (permalink / raw)
To: Ilpo J?rvinen
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton
* Ingo Molnar <mingo@elte.hu> wrote:
> ah, in retrospect i realized that this test had one flaw: some of the
> systems i the build cluster already ran a newer kernel and hence were
> targets for this bug.
>
> so i turned off CONFIG_TCP_CONG_CUBIC on all the testboxes and
> rebooted the cluster boxes into 2.6.25, and the hung sockets are now
> gone. (about 150 successful iterations)
>
> i did another change as well: i removed the localhost distcc
> component. I'll reinstate that now to make sure it's really related to
> TCP_CONG_CUBIC and not to localhost networking.
ok, once i added back the localhost distcc component and the hung kernel
build + stuck TCP socket bug happened again overnight:
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 72187 0 10.0.1.14:3632 10.0.1.14:47910 ESTABLISHED
tcp 0 174464 10.0.1.14:47910 10.0.1.14:3632 ESTABLISHED
so it seems distcc over localhost was the aspect that made it fail.
_Perhaps_ what matters is to have the new post-rc3 TCP code on _both_
sides of the connection. But that is just a theory - it could be timing,
etc.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 6:03 ` Evgeniy Polyakov
@ 2008-05-31 10:05 ` Ilpo Järvinen
2008-06-02 6:19 ` Herbert Xu
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-31 10:05 UTC (permalink / raw)
To: Evgeniy Polyakov
Cc: Ray Lee, Ingo Molnar, LKML, Netdev, David S. Miller,
Rafael J. Wysocki, Andrew Morton
[-- Attachment #1: Type: TEXT/PLAIN, Size: 4168 bytes --]
On Sat, 31 May 2008, Evgeniy Polyakov wrote:
> On Sat, May 31, 2008 at 12:11:32AM +0300, Ilpo Järvinen (ilpo.jarvinen@helsinki.fi) wrote:
> > > > > Active Internet connections (w/o servers)
> > > > > Proto Recv-Q Send-Q Local Address Foreign Address
> > > > > State
> > > > > tcp 0 174592 10.0.1.14:58015 10.0.1.14:3632 ESTABLISHED
> > > > > tcp 72134 0 10.0.1.14:3632 10.0.1.14:58015 ESTABLISHED
> >
> > ^^^^^
> >
> > Can you perhaps find/guess/think some explanation for this _receiver
> > queue_...? This was a trick question :-), as we already know that the
> > receiving process is no longer there and therefore obviously won't be
> > reading anything anymore. But that opened another question, why TCP is
> > then still in ESTABLISHED as orphaned TCP shouldn't be in establised state
> > anymore, tcp_close should have changed the state (either at close or at
> > process exit). I guess once it becomes known why tcp_close either wasn't
> > called at all or it didn't change the state of the flow (it's quite
> > simple, see for yourself), the cause of the bug is found (it might even be
> > that the process went away when it shouldn't have, either a bookkeeping
> > bug somewhere or real death, or something along those lines).
>
> In that case pid Ingo straced was still valid and existed (ans stuck
> probably in tcp_sendpage() - sysrq-t dump was messy),
...No, I think it was the sender, not the receiver that Ingo straced!
A _correctly_ behaving TCP sender will just get "stuck" once the window
is exhausted. It as was proven that advertized window is zero (tcpdump).
So I find that hardly a bug.
> so process still exist, but sock somewhat lost its reference to socket.
...this is something that seems possible explanation.
> AFICS all (three)
> calls for sk_orphan() can not happen without state change to something
> closing (like fin wait or close), except some linger things and
> listening socket, which frees its connection requests.
Yes, this what I figured out as well (distcc didn't seem to play with
linger, I checked, nor would it have any reason to)... The LISTENING
socket was still there in /proc/net/tcp, so closing that won't explain it.
In addition, it seems that tcp_close would notify the other end as well,
either with RST or with FIN, which would cause the other end to leave
ESTABLISHED (unless tcp_transmit_skb would fail but I guess that would end
up into some MIB in either of the cases). Here RST is the way to go
because there's unread data (though in theory having yet another bug could
allow building up the receiver queue afterwards).
Which then boils down to two alternatives:
1) The receiving process lost its reference though the process
still exists and therefore cannot read
2) tcp_close wasn't called when it should have, e.g., process exited
but the tcp_close wasn't called because the reference wasn't there
Btw, Ingo also got this warning later on which might be helpful in
understanding all this:
>>>>> after ~7 hours of uptime the networking code produced this assertion:
>>>>>
>>>>> [25441.140000] KERNEL: assertion (!sk->sk_ack_backlog) failed at
>>>>> net/ipv4/inet_connection_sock.c (642)
I guess that sk_ack_backlog was already visible at the time when the TCP
got stuck because listening socket already had 1 there in /proc/net/tcp
(though this depends if there was some other connection attempts still
going on).
...Some new info (at least to me). I tried with ./stress_accept 5000 2
(attached), I could easily produce a number of ESTABLISHED connections
which won't yet have an owner in netstat (tested in 2.6.25.3) while
connect.sh is running but it seems some kind of transitional state as once
I do killall -TERM connect.sh (attached), they get the owning
stress_accept. So third explanation would be some race during that
transition that would allow process to never get the reference...
Before:
tcp 65436 0 lo:5000 lo:59980 ESTABLISHED -
After killall:
tcp 76444 0 lo:5000 lo:59980 ESTABLISHED 9013/stress_accept
--
i.
[-- Attachment #2: Type: TEXT/PLAIN, Size: 1657 bytes --]
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/wait.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
void error(const char *str)
{
perror(str);
exit(1);
}
void worker(int listenfd)
{
int fd;
struct sockaddr_in peeraddr;
socklen_t len = sizeof(peeraddr);
int pid;
char buf[100];
while(1) {
memset(&peeraddr, 0, sizeof(peeraddr));
fd = accept(listenfd, (struct sockaddr *)&peeraddr, &len);
if (fd < 0)
error("accept");
pid = fork();
if (pid < 0) {
error("fork");
} else if (pid != 0) {
close(fd);
/* while(1)
sleep(1); */
} else {
read(fd, &buf, sizeof(buf));
while(1)
sleep(1);
exit(0);
}
}
exit(0);
}
int main(int argc, char **argv)
{
int listenfd;
struct sockaddr_in myaddr;
int res;
pid_t pid;
int i;
if (argc < 3) {
fprintf(stderr, "%s serverport processes\n", argv[0]);
exit(1);
}
listenfd = socket(PF_INET, SOCK_STREAM, 0);
if (listenfd < 0)
error("socket");
memset(&myaddr, 0, sizeof(myaddr));
myaddr.sin_family = PF_INET;
myaddr.sin_addr.s_addr = inet_addr("127.0.0.1");
myaddr.sin_port = htons(atoi(argv[1]));
res = bind(listenfd, (struct sockaddr *)&myaddr, sizeof(myaddr));
if (res < 0)
error("bind");
/* distccd uses 10 here */
if (listen(listenfd, 10) < 0)
error("listen");
for (i = 0; i < atoi(argv[2]); i++) {
pid = fork();
if (pid < 0)
error("fork");
if (pid == 0) {
worker(listenfd);
}
}
while (1) {
sleep(1);
}
return 0;
}
[-- Attachment #3: Type: APPLICATION/x-sh, Size: 85 bytes --]
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 6:09 ` Ingo Molnar
@ 2008-05-31 11:46 ` Ilpo Järvinen
2008-05-31 12:18 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-31 11:46 UTC (permalink / raw)
To: Ingo Molnar
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton,
Evgeniy Polyakov
On Sat, 31 May 2008, Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
>
> > ah, in retrospect i realized that this test had one flaw: some of the
> > systems i the build cluster already ran a newer kernel and hence were
> > targets for this bug.
> >
> > so i turned off CONFIG_TCP_CONG_CUBIC on all the testboxes and
> > rebooted the cluster boxes into 2.6.25, and the hung sockets are now
> > gone. (about 150 successful iterations)
> >
> > i did another change as well: i removed the localhost distcc
> > component. I'll reinstate that now to make sure it's really related to
> > TCP_CONG_CUBIC and not to localhost networking.
>
> ok, once i added back the localhost distcc component and the hung kernel
> build + stuck TCP socket bug happened again overnight:
>
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 72187 0 10.0.1.14:3632 10.0.1.14:47910 ESTABLISHED
> tcp 0 174464 10.0.1.14:47910 10.0.1.14:3632 ESTABLISHED
>
> so it seems distcc over localhost was the aspect that made it fail.
>
> _Perhaps_ what matters is to have the new post-rc3 TCP code on _both_
> sides of the connection. But that is just a theory - it could be timing,
> etc.
Btw, does your distcc perhaps happen enable TCP_DEFER_ACCEPT (there were
some post 2.6.25 changes into it)?
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 11:46 ` Ilpo Järvinen
@ 2008-05-31 12:18 ` Ilpo Järvinen
2008-05-31 12:54 ` Ingo Molnar
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-31 12:18 UTC (permalink / raw)
To: Ingo Molnar, Peter Zijlstra
Cc: LKML, Netdev, David S. Miller, Rafael J. Wysocki, Andrew Morton,
Evgeniy Polyakov
[-- Attachment #1: Type: TEXT/PLAIN, Size: 11376 bytes --]
On Sat, 31 May 2008, Ilpo Järvinen wrote:
> On Sat, 31 May 2008, Ingo Molnar wrote:
>
> > ok, once i added back the localhost distcc component and the hung kernel
> > build + stuck TCP socket bug happened again overnight:
> >
> > Active Internet connections (w/o servers)
> > Proto Recv-Q Send-Q Local Address Foreign Address State
> > tcp 72187 0 10.0.1.14:3632 10.0.1.14:47910 ESTABLISHED
> > tcp 0 174464 10.0.1.14:47910 10.0.1.14:3632 ESTABLISHED
> >
> > so it seems distcc over localhost was the aspect that made it fail.
> >
> > _Perhaps_ what matters is to have the new post-rc3 TCP code on _both_
> > sides of the connection. But that is just a theory - it could be timing,
> > etc.
>
> Btw, does your distcc perhaps happen enable TCP_DEFER_ACCEPT (there were
> some post 2.6.25 changes into it)?
Hmm, if so, please try this revert below (I hope I got everything that is
related there, gitk is currently broken for me so that viewing many
commits quickly is a pain, so I might have accidently missed something)...
--
i.
--
[PATCH] tcp: Reverts DEFER_ACCEPT modifications
Reverts these commits:
[TCP]: TCP_DEFER_ACCEPT updates - defer timeout conflicts with max_t
[TCP]: TCP_DEFER_ACCEPT updates - dont retxmt synack
[TCP]: TCP_DEFER_ACCEPT updates - process as established
tcp: Fix slab corruption with ipv6 and tcp6fuzz
..., ie.,
539fae89bebd16ebeafd57a87169bc56eb530d76,
e4c78840284f3f51b1896cf3936d60a6033c4d2c,
ec3c0982a2dd1e671bad8e9d26c28dcba0039d87 and
9ae27e0adbf471c7a6b80102e38e1d5a346b3b38.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
---
include/linux/tcp.h | 7 ------
include/net/request_sock.h | 4 +-
include/net/tcp.h | 1 -
net/ipv4/inet_connection_sock.c | 11 +++++++--
net/ipv4/tcp.c | 18 +++++++++------
net/ipv4/tcp_input.c | 45 ---------------------------------------
net/ipv4/tcp_ipv4.c | 8 -------
net/ipv4/tcp_minisocks.c | 32 ++++++++++-----------------
net/ipv4/tcp_timer.c | 5 ----
9 files changed, 33 insertions(+), 98 deletions(-)
diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index 18e62e3..b31b6b7 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -239,11 +239,6 @@ static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
return (struct tcp_request_sock *)req;
}
-struct tcp_deferred_accept_info {
- struct sock *listen_sk;
- struct request_sock *request;
-};
-
struct tcp_sock {
/* inet_connection_sock has to be the first member of tcp_sock */
struct inet_connection_sock inet_conn;
@@ -379,8 +374,6 @@ struct tcp_sock {
unsigned int keepalive_intvl; /* time interval between keep alive probes */
int linger2;
- struct tcp_deferred_accept_info defer_tcp_accept;
-
unsigned long last_synq_overflow;
u32 tso_deferred;
diff --git a/include/net/request_sock.h b/include/net/request_sock.h
index b220b5f..0c96e7b 100644
--- a/include/net/request_sock.h
+++ b/include/net/request_sock.h
@@ -115,8 +115,8 @@ struct request_sock_queue {
struct request_sock *rskq_accept_head;
struct request_sock *rskq_accept_tail;
rwlock_t syn_wait_lock;
- u16 rskq_defer_accept;
- /* 2 bytes hole, try to pack */
+ u8 rskq_defer_accept;
+ /* 3 bytes hole, try to pack */
struct listen_sock *listen_opt;
};
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 633147c..981d5ba 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -139,7 +139,6 @@ extern void tcp_time_wait(struct sock *sk, int state, int timeo);
#define MAX_TCP_KEEPINTVL 32767
#define MAX_TCP_KEEPCNT 127
#define MAX_TCP_SYNCNT 127
-#define MAX_TCP_ACCEPT_DEFERRED 65535
#define TCP_SYNQ_INTERVAL (HZ/5) /* Period of SYNACK timer */
diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index 828ea21..ec83448 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -419,7 +419,8 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
struct inet_connection_sock *icsk = inet_csk(parent);
struct request_sock_queue *queue = &icsk->icsk_accept_queue;
struct listen_sock *lopt = queue->listen_opt;
- int thresh = icsk->icsk_syn_retries ? : sysctl_tcp_synack_retries;
+ int max_retries = icsk->icsk_syn_retries ? : sysctl_tcp_synack_retries;
+ int thresh = max_retries;
unsigned long now = jiffies;
struct request_sock **reqp, *req;
int i, budget;
@@ -455,6 +456,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
}
}
+ if (queue->rskq_defer_accept)
+ max_retries = queue->rskq_defer_accept;
+
budget = 2 * (lopt->nr_table_entries / (timeout / interval));
i = lopt->clock_hand;
@@ -462,8 +466,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
reqp=&lopt->syn_table[i];
while ((req = *reqp) != NULL) {
if (time_after_eq(now, req->expires)) {
- if (req->retrans < thresh &&
- !req->rsk_ops->rtx_syn_ack(parent, req)) {
+ if ((req->retrans < thresh ||
+ (inet_rsk(req)->acked && req->retrans < max_retries))
+ && !req->rsk_ops->rtx_syn_ack(parent, req)) {
unsigned long timeo;
if (req->retrans++ == 0)
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index f886531..a0deead 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -2105,12 +2105,15 @@ static int do_tcp_setsockopt(struct sock *sk, int level,
break;
case TCP_DEFER_ACCEPT:
- if (val < 0) {
- err = -EINVAL;
- } else {
- if (val > MAX_TCP_ACCEPT_DEFERRED)
- val = MAX_TCP_ACCEPT_DEFERRED;
- icsk->icsk_accept_queue.rskq_defer_accept = val;
+ icsk->icsk_accept_queue.rskq_defer_accept = 0;
+ if (val > 0) {
+ /* Translate value in seconds to number of
+ * retransmits */
+ while (icsk->icsk_accept_queue.rskq_defer_accept < 32 &&
+ val > ((TCP_TIMEOUT_INIT / HZ) <<
+ icsk->icsk_accept_queue.rskq_defer_accept))
+ icsk->icsk_accept_queue.rskq_defer_accept++;
+ icsk->icsk_accept_queue.rskq_defer_accept++;
}
break;
@@ -2292,7 +2295,8 @@ static int do_tcp_getsockopt(struct sock *sk, int level,
val = (val ? : sysctl_tcp_fin_timeout) / HZ;
break;
case TCP_DEFER_ACCEPT:
- val = icsk->icsk_accept_queue.rskq_defer_accept;
+ val = !icsk->icsk_accept_queue.rskq_defer_accept ? 0 :
+ ((TCP_TIMEOUT_INIT / HZ) << (icsk->icsk_accept_queue.rskq_defer_accept - 1));
break;
case TCP_WINDOW_CLAMP:
val = tp->window_clamp;
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index b54d9d3..ddc2e89 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4532,49 +4532,6 @@ static void tcp_urg(struct sock *sk, struct sk_buff *skb, struct tcphdr *th)
}
}
-static int tcp_defer_accept_check(struct sock *sk)
-{
- struct tcp_sock *tp = tcp_sk(sk);
-
- if (tp->defer_tcp_accept.request) {
- int queued_data = tp->rcv_nxt - tp->copied_seq;
- int hasfin = !skb_queue_empty(&sk->sk_receive_queue) ?
- tcp_hdr((struct sk_buff *)
- sk->sk_receive_queue.prev)->fin : 0;
-
- if (queued_data && hasfin)
- queued_data--;
-
- if (queued_data &&
- tp->defer_tcp_accept.listen_sk->sk_state == TCP_LISTEN) {
- if (sock_flag(sk, SOCK_KEEPOPEN)) {
- inet_csk_reset_keepalive_timer(sk,
- keepalive_time_when(tp));
- } else {
- inet_csk_delete_keepalive_timer(sk);
- }
-
- inet_csk_reqsk_queue_add(
- tp->defer_tcp_accept.listen_sk,
- tp->defer_tcp_accept.request,
- sk);
-
- tp->defer_tcp_accept.listen_sk->sk_data_ready(
- tp->defer_tcp_accept.listen_sk, 0);
-
- sock_put(tp->defer_tcp_accept.listen_sk);
- sock_put(sk);
- tp->defer_tcp_accept.listen_sk = NULL;
- tp->defer_tcp_accept.request = NULL;
- } else if (hasfin ||
- tp->defer_tcp_accept.listen_sk->sk_state != TCP_LISTEN) {
- tcp_reset(sk);
- return -1;
- }
- }
- return 0;
-}
-
static int tcp_copy_to_iovec(struct sock *sk, struct sk_buff *skb, int hlen)
{
struct tcp_sock *tp = tcp_sk(sk);
@@ -4935,8 +4892,6 @@ step5:
tcp_data_snd_check(sk);
tcp_ack_snd_check(sk);
-
- tcp_defer_accept_check(sk);
return 0;
csum_error:
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index cd601a8..b698b5b 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1918,14 +1918,6 @@ int tcp_v4_destroy_sock(struct sock *sk)
sk->sk_sndmsg_page = NULL;
}
- if (tp->defer_tcp_accept.request) {
- reqsk_free(tp->defer_tcp_accept.request);
- sock_put(tp->defer_tcp_accept.listen_sk);
- sock_put(sk);
- tp->defer_tcp_accept.listen_sk = NULL;
- tp->defer_tcp_accept.request = NULL;
- }
-
atomic_dec(&tcp_sockets_allocated);
return 0;
diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c
index 019c8c1..8245247 100644
--- a/net/ipv4/tcp_minisocks.c
+++ b/net/ipv4/tcp_minisocks.c
@@ -571,8 +571,10 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
does sequence test, SYN is truncated, and thus we consider
it a bare ACK.
- Both ends (listening sockets) accept the new incoming
- connection and try to talk to each other. 8-)
+ If icsk->icsk_accept_queue.rskq_defer_accept, we silently drop this
+ bare ACK. Otherwise, we create an established connection. Both
+ ends (listening sockets) accept the new incoming connection and try
+ to talk to each other. 8-)
Note: This case is both harmless, and rare. Possibility is about the
same as us discovering intelligent life on another plant tomorrow.
@@ -640,6 +642,13 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
if (!(flg & TCP_FLAG_ACK))
return NULL;
+ /* If TCP_DEFER_ACCEPT is set, drop bare ACK. */
+ if (inet_csk(sk)->icsk_accept_queue.rskq_defer_accept &&
+ TCP_SKB_CB(skb)->end_seq == tcp_rsk(req)->rcv_isn + 1) {
+ inet_rsk(req)->acked = 1;
+ return NULL;
+ }
+
/* OK, ACK is valid, create big socket and
* feed this segment to it. It will repeat all
* the tests. THIS SEGMENT MUST MOVE SOCKET TO
@@ -678,24 +687,7 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
inet_csk_reqsk_queue_unlink(sk, req, prev);
inet_csk_reqsk_queue_removed(sk, req);
- if (inet_csk(sk)->icsk_accept_queue.rskq_defer_accept &&
- TCP_SKB_CB(skb)->end_seq == tcp_rsk(req)->rcv_isn + 1) {
-
- /* the accept queue handling is done is est recv slow
- * path so lets make sure to start there
- */
- tcp_sk(child)->pred_flags = 0;
- sock_hold(sk);
- sock_hold(child);
- tcp_sk(child)->defer_tcp_accept.listen_sk = sk;
- tcp_sk(child)->defer_tcp_accept.request = req;
-
- inet_csk_reset_keepalive_timer(child,
- inet_csk(sk)->icsk_accept_queue.rskq_defer_accept * HZ);
- } else {
- inet_csk_reqsk_queue_add(sk, req, child);
- }
-
+ inet_csk_reqsk_queue_add(sk, req, child);
return child;
listen_overflow:
diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
index 4de68cf..63ed9d6 100644
--- a/net/ipv4/tcp_timer.c
+++ b/net/ipv4/tcp_timer.c
@@ -489,11 +489,6 @@ static void tcp_keepalive_timer (unsigned long data)
goto death;
}
- if (tp->defer_tcp_accept.request && sk->sk_state == TCP_ESTABLISHED) {
- tcp_send_active_reset(sk, GFP_ATOMIC);
- goto death;
- }
-
if (!sock_flag(sk, SOCK_KEEPOPEN) || sk->sk_state == TCP_CLOSE)
goto out;
--
1.5.5.3
^ permalink raw reply related [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 12:18 ` Ilpo Järvinen
@ 2008-05-31 12:54 ` Ingo Molnar
2008-05-31 12:58 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-05-31 12:54 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: Peter Zijlstra, LKML, Netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton, Evgeniy Polyakov
* Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> > Btw, does your distcc perhaps happen enable TCP_DEFER_ACCEPT (there
> > were some post 2.6.25 changes into it)?
>
> Hmm, if so, please try this revert below (I hope I got everything that
> is related there, gitk is currently broken for me so that viewing many
> commits quickly is a pain, so I might have accidently missed
> something)...
what is the easiest way to figure out whether my version of distcc
enables TCP_DEFER_ACCEPT?
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 12:54 ` Ingo Molnar
@ 2008-05-31 12:58 ` Ilpo Järvinen
2008-05-31 16:35 ` Ingo Molnar
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-31 12:58 UTC (permalink / raw)
To: Ingo Molnar
Cc: Peter Zijlstra, LKML, Netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton, Evgeniy Polyakov
[-- Attachment #1: Type: TEXT/PLAIN, Size: 781 bytes --]
On Sat, 31 May 2008, Ingo Molnar wrote:
>
> * Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
>
> > > Btw, does your distcc perhaps happen enable TCP_DEFER_ACCEPT (there
> > > were some post 2.6.25 changes into it)?
> >
> > Hmm, if so, please try this revert below (I hope I got everything that
> > is related there, gitk is currently broken for me so that viewing many
> > commits quickly is a pain, so I might have accidently missed
> > something)...
>
> what is the easiest way to figure out whether my version of distcc
> enables TCP_DEFER_ACCEPT?
strace distccd (the listening party is the interesting one here, not the
distcc)?
...setsockopt(listenfd, SOL_TCP, TCP_DEFER_ACCEPT, &val, sizeof(val))
seems to be the magic trick that is interestion here.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-28 9:27 ` Peter Zijlstra
@ 2008-05-31 14:25 ` Håkon Løvdal
2008-05-31 16:09 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Håkon Løvdal @ 2008-05-31 14:25 UTC (permalink / raw)
To: linux-kernel, netdev
Cc: Ingo Molnar, David S. Miller, Rafael J. Wysocki, Andrew Morton,
Ilpo Järvinen
2008/5/28 Peter Zijlstra <peterz@infradead.org>:
> Just a quick note to say, me too!!
>
> same scenario: distcc on localhost.
Me too, however with a completely different scenario; my hung connections
are not related to distcc at all. The output from /proc/net/tcp that Ingo
posted a few days ago are somewhat different from mine, however I believe
this is the same problem or at least related. Just as Ingo experienced,
netstat -p only shows PID/program as '-' for the hung connections while
for other connections it shows the expected results.
I have recently bought a new PC and have started the process of copying
stuff from my old PC to the new PC. During this I have experienced this
hang several times. I started copying by using tar on both ends over a ssh
pipe but in order to eliminate possible ssh problems I also have tried tar
over a ttcp connection which also fails. There is no obvious pattern of
when this happens, I have experienced failures after transferring 1.15GB,
51.4GB and 23.6GB.
Here is the output from netstat -n -o filtered for port 22 and slightly
edited. All the lines started with Proto == tcp and Recv-Q == 0.
Send-Q Local Addr Foreign Addr State Timer
0 old_pc:22 new_pc:52667 ESTABLISHED keepalive (3513.93/0/0)
0 old_pc:22 new_pc:43825 ESTABLISHED keepalive (5467.38/0/0)
2896 old_pc:22 new_pc:58601 ESTABLISHED on (21020884.65/0/0)
4344 old_pc:22 new_pc:54105 ESTABLISHED on (21017016.33/0/0)
2896 old_pc:22 new_pc:34149 ESTABLISHED on (20986889.24/0/0)
The first two connections are ongoing, working, interactive ssh
connections. The other three connections died days ago on my new PC.
One thing that caught my eyes was these very high timer values.
Checking the netstat source reveals that the value printed is "(double)
time_len / HZ" and that time_len is extracted from /proc/net/tcp. While
my CONFIG_HZ is 1000, I assume netstat has picked up HZ as 100 from
/usr/include/asm/param.h, and then things really seems to imply that
there is some integer overflow since 2^31 = 2147483648.
Looking into get_tcp4_sock in net/ipv4/tcp_ipv4.c I see that timer_expires
is initialized with icsk->icsk_timeout for the troublesome cases. But
here my competence to trace this further stops, so I have no idea of
how icsk->icsk_timeout gets such high values.
My old PC is currently still running with these stalled connections
present so let me know if there is something I should try to investigate
further. I can post output from /proc/net/tcp and my .config if you want
to have a look. My old PC is 32 bit/Celeron single core, kernel 2.6.24,
while my new is 64 bit/Q9300 quad core, kernel 2.6.25.3. The ethernet
cards are the following:
02:0d.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL-8139/8139C/8139C+ (rev 10)
02:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056
PCI-E Gigabit Ethernet Controller (rev 12)
BR Håkon Løvdal
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 14:25 ` Håkon Løvdal
@ 2008-05-31 16:09 ` Ilpo Järvinen
2008-05-31 17:22 ` Ilpo Järvinen
2008-05-31 17:58 ` Håkon Løvdal
0 siblings, 2 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-31 16:09 UTC (permalink / raw)
To: Håkon Løvdal
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
[-- Attachment #1: Type: TEXT/PLAIN, Size: 3203 bytes --]
Thanks for reporting!
On Sat, 31 May 2008, Håkon Løvdal wrote:
> posted a few days ago are somewhat different from mine, however I believe
> this is the same problem or at least related. Just as Ingo experienced,
> netstat -p only shows PID/program as '-' for the hung connections while
> for other connections it shows the expected results.
Hmm, are the other end's processes still there? ...I'd be interested to
know what they're doing at the moment...
> I have recently bought a new PC and have started the process of copying
> stuff from my old PC to the new PC. During this I have experienced this
> hang several times. I started copying by using tar on both ends over a ssh
> pipe but in order to eliminate possible ssh problems I also have tried tar
> over a ttcp connection which also fails. There is no obvious pattern of
> when this happens, I have experienced failures after transferring
> 1.15GB, 51.4GB and 23.6GB.
>
> Here is the output from netstat -n -o filtered for port 22 and slightly
> edited. All the lines started with Proto == tcp and Recv-Q == 0.
...The receiving end's state would be more interesting.
> Send-Q Local Addr Foreign Addr State Timer
> 0 old_pc:22 new_pc:52667 ESTABLISHED keepalive (3513.93/0/0)
> 0 old_pc:22 new_pc:43825 ESTABLISHED keepalive (5467.38/0/0)
> 2896 old_pc:22 new_pc:58601 ESTABLISHED on (21020884.65/0/0)
> 4344 old_pc:22 new_pc:54105 ESTABLISHED on (21017016.33/0/0)
> 2896 old_pc:22 new_pc:34149 ESTABLISHED on (20986889.24/0/0)
>
> The first two connections are ongoing, working, interactive ssh
> connections. The other three connections died days ago on my new PC.
Died? Do you mean that they don't exist all at the other end anymore?
> One thing that caught my eyes was these very high timer values.
> Checking the netstat source reveals that the value printed is "(double)
> time_len / HZ" and that time_len is extracted from /proc/net/tcp. While
> my CONFIG_HZ is 1000, I assume netstat has picked up HZ as 100 from
> /usr/include/asm/param.h, and then things really seems to imply that
> there is some integer overflow since 2^31 = 2147483648.
...plain /proc/net/tcp would be much nicer to read and without all such
conversion troubles ;-).
> Looking into get_tcp4_sock in net/ipv4/tcp_ipv4.c I see that timer_expires
> is initialized with icsk->icsk_timeout for the troublesome cases. But
> here my competence to trace this further stops, so I have no idea of
> how icsk->icsk_timeout gets such high values.
>
> My old PC is currently still running with these stalled connections
> present so let me know if there is something I should try to investigate
> further.
>
> I can post output from /proc/net/tcp
For both ends that would be great.
> and my .config if you want to have a look.
Not needed I think.
> My old PC is 32 bit/Celeron single core, kernel 2.6.24,
> while my new is 64 bit/Q9300 quad core, kernel 2.6.25.3.
> The ethernet cards are the following:
>
> 02:0d.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
> RTL-8139/8139C/8139C+ (rev 10)
> 02:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056
> PCI-E Gigabit Ethernet Controller (rev 12)
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 12:58 ` Ilpo Järvinen
@ 2008-05-31 16:35 ` Ingo Molnar
2008-05-31 22:46 ` Patrick McManus
2008-06-03 9:40 ` [fixed] [patch] " Ingo Molnar
0 siblings, 2 replies; 96+ messages in thread
From: Ingo Molnar @ 2008-05-31 16:35 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: Peter Zijlstra, LKML, Netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton, Evgeniy Polyakov
* Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> > what is the easiest way to figure out whether my version of distcc
> > enables TCP_DEFER_ACCEPT?
>
> strace distccd (the listening party is the interesting one here, not
> the distcc)?
>
> ...setsockopt(listenfd, SOL_TCP, TCP_DEFER_ACCEPT, &val, sizeof(val))
> seems to be the magic trick that is interestion here.
seems to be used:
22003 write(3, "distccd[22003] (dcc_listen_by_ad"..., 62) = 62
22003 listen(4, 10) = 0
22003 setsockopt(4, SOL_TCP, TCP_DEFER_ACCEPT, [1], 4) = 0
i'll queue up your reverts for testing in -tip.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 16:09 ` Ilpo Järvinen
@ 2008-05-31 17:22 ` Ilpo Järvinen
2008-05-31 17:58 ` Håkon Løvdal
1 sibling, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-31 17:22 UTC (permalink / raw)
To: Håkon Løvdal
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
[-- Attachment #1: Type: TEXT/PLAIN, Size: 718 bytes --]
On Sat, 31 May 2008, Ilpo Järvinen wrote:
> On Sat, 31 May 2008, Håkon Løvdal wrote:
>
> > posted a few days ago are somewhat different from mine, however I believe
> > this is the same problem or at least related. Just as Ingo experienced,
> > netstat -p only shows PID/program as '-' for the hung connections while
> > for other connections it shows the expected results.
>
> Hmm, are the other end's processes still there? ...I'd be interested to
> know what they're doing at the moment...
I meant that end where you see this '-'. I suppose it's easy for you to
figure out which process is the right one, something that wouldn't be so
easy with the Ingo's test case which forks/exits numerous times.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 16:09 ` Ilpo Järvinen
2008-05-31 17:22 ` Ilpo Järvinen
@ 2008-05-31 17:58 ` Håkon Løvdal
2008-05-31 18:37 ` Ilpo Järvinen
1 sibling, 1 reply; 96+ messages in thread
From: Håkon Løvdal @ 2008-05-31 17:58 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
Ilpo Järvinen wrote:
> Hmm, are the other end's processes still there? ...I'd be interested to
> know what they're doing at the moment...
> I meant that end where you see this '-'. I suppose it's easy for you to
> figure out which process is the right one, something that wouldn't be so
> easy with the Ingo's test case which forks/exits numerous times.
> Died? Do you mean that they don't exist all at the other end anymore?
The ssh connection used for copying (using the command <ssh old_pc "cd
/directory; tar cvf - *" | pv | tar xvf ->) died in the following way:
...
Read from remote host old_pc: Connection timed out
51.4GB 4:26:19 [3.29MB/s] [<=> ]
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
and there are currently no traces of those ssh processes any longer on
the new PC, only these two active ssh interactive connections are present:
new_pc>ps -elf | sed -n '1p;/ssh/p'
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
0 S hlovdal 1310 1195 0 80 0 - 14139 select 09:28 pts/14 00:00:00 ssh old_pc
5 S root 2477 1 0 80 0 - 15203 - May26 ? 00:00:00 /usr/sbin/sshd
1 S hlovdal 2727 2701 0 80 0 - 12466 select May26 ? 00:00:00 /usr/bin/ssh-agent /bin/sh -c exec -l /bin/bash -c "/usr/bin/startkde"
0 S hlovdal 26436 19976 0 80 0 - 20375 pipe_w 19:43 pts/9 00:00:00 sed -n 1p;/ssh/p
0 S hlovdal 30065 18970 0 80 0 - 14174 select May28 pts/6 00:00:00 ssh old_pc
new_pc>
On the old PC there are three corresponding zombies:
old_pc>ps -elf | sed -n '1p;/ssh/p'
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
5 S root 4402 1 0 80 0 - 1600 - May25 ? 00:00:02 /usr/sbin/sshd
1 S root 5418 1 0 80 0 - 1347 - May25 ? 00:00:00 ssh-agent
1 S hlovdal 5752 1 0 80 0 - 1347 - May25 ? 00:00:00 ssh-agent
4 S root 6824 4402 0 80 0 - 2315 - 09:28 ? 00:00:00 sshd: hlovdal [priv]
5 S hlovdal 6828 6824 0 80 0 - 2355 - 09:28 ? 00:00:01 sshd: hlovdal@pts/8
0 S hlovdal 17186 25557 0 80 0 - 1191 - 19:45 pts/0 00:00:00 sed -n 1p;/ssh/p
4 D root 24444 4402 0 80 0 - 0 - May27 ? 00:00:00 [sshd]
5 Z hlovdal 24448 24444 0 80 0 - 0 - May27 ? 00:58:33 [sshd] <defunct>
4 S root 25552 4402 0 80 0 - 2315 - May28 ? 00:00:00 sshd: hlovdal [priv]
5 S hlovdal 25556 25552 0 80 0 - 2383 - May28 ? 00:00:01 sshd: hlovdal@pts/0
1 S hlovdal 28504 28328 0 80 0 - 1347 - May28 ? 00:00:01 /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session /usr/bin/startkde
4 D root 31338 4402 0 80 0 - 0 - May27 ? 00:00:00 [sshd]
5 Z hlovdal 31342 31338 0 80 0 - 0 - May27 ? 00:07:55 [sshd] <defunct>
4 D root 31564 4402 0 80 0 - 0 - May27 ? 00:00:00 [sshd]
5 Z hlovdal 31568 31564 0 80 0 - 0 - May27 ? 00:43:53 [sshd] <defunct>
old_pc>
>> I can post output from /proc/net/tcp
>
> For both ends that would be great.
111111AC is the old PC.
480111AC is the new PC.
new_pc>sed -n '1p; /:0016/p' /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
3: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 8850 1 ffff81022ddd2680 3000 0 0 2 -1
7: 480111AC:AB31 111111AC:0016 01 00000000:00000000 02:000A2A2B 00000000 500 0 138582 2 ffff81022d9d7380 202 40 22 3 -1
8: 480111AC:CDBB 111111AC:0016 01 00000000:00000000 02:0001BBAD 00000000 500 0 723741 2 ffff81017560c000 202 40 18 3 -1
new_pc>
old_pc>sed -n '1p; /:0016/p' /proc/net/tcp
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
12: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7627 1 f384b080 3000 0 0 2 -1
17: 111111AC:0016 480111AC:CDBB 01 00000000:00000000 02:0001AA7B 00000000 0 0 1110320 2 f02e6580 201 53 7 3 -1
20: 111111AC:0016 480111AC:AB31 01 00000000:00000000 02:000A18F1 00000000 0 0 583506 4 f71a8080 201 40 29 3 -1
21: 111111AC:0016 480111AC:E4E9 01 00000B50:00000000 01:7D1F8746 00000000 0 0 398713 5 f71a8580 205 40 1 36 -1
23: 111111AC:0016 480111AC:D359 01 000010F8:00000000 01:7D19A035 00000000 0 0 396426 5 f71a8a80 202 42 1 144 -1
25: 111111AC:0016 480111AC:8565 01 00000B50:00000000 01:7CEBA7D1 00000000 0 0 349113 5 eeeaf580 204 40 1 26 -1
old_pc>
BR Håkon Løvdal
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 17:58 ` Håkon Løvdal
@ 2008-05-31 18:37 ` Ilpo Järvinen
2008-05-31 20:25 ` Håkon Løvdal
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-31 18:37 UTC (permalink / raw)
To: Håkon Løvdal
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
[-- Attachment #1: Type: TEXT/PLAIN, Size: 2609 bytes --]
On Sat, 31 May 2008, Håkon Løvdal wrote:
> Ilpo Järvinen wrote:
> > Hmm, are the other end's processes still there? ...I'd be interested to know
> > what they're doing at the moment...
>
> > I meant that end where you see this '-'. I suppose it's easy for you to
> > figure out which process is the right one, something that wouldn't be so
> > easy with the Ingo's test case which forks/exits numerous times.
>
> > Died? Do you mean that they don't exist all at the other end anymore?
Ok, like you said, this is not exactly the same, though it might be due to
the same bug. In Ingo's case both endpoints were doing pretty healtily,
with periodic window probes as expected. In your case TCP is not doing
window probes but got that interesting RTO value.
So you had that '-' earlier and you checked at that time but the
connection is now already dead?
> The ssh connection used for copying (using the command <ssh old_pc "cd
> /directory; tar cvf - *" | pv | tar xvf ->) died in the following way:
> ...
> Read from remote host old_pc: Connection timed out
> 51.4GB 4:26:19 [3.29MB/s] [<=> ]
> tar: Unexpected EOF in archive
> tar: Unexpected EOF in archive
> tar: Error is not recoverable: exiting now
>
> and there are currently no traces of those ssh processes any longer on
> the new PC, only these two active ssh interactive connections are present:
:-(, I would some much liked to see what they were doing.
> old_pc>sed -n '1p; /:0016/p' /proc/net/tcp
> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt
> uid timeout inode
> 12: 00000000:0016 00000000:0000 0A 00000000:00000000 00:00000000 00000000
> 0 0 7627 1 f384b080 3000 0 0 2 -1
> 17: 111111AC:0016 480111AC:CDBB 01 00000000:00000000 02:0001AA7B 00000000
> 0 0 1110320 2 f02e6580 201 53 7 3 -1
> 20: 111111AC:0016 480111AC:AB31 01 00000000:00000000 02:000A18F1 00000000
> 0 0 583506 4 f71a8080 201 40 29 3 -1
> 21: 111111AC:0016 480111AC:E4E9 01 00000B50:00000000 01:7D1F8746 00000000
> 0 0 398713 5 f71a8580 205 40 1 36 -1
> 23: 111111AC:0016 480111AC:D359 01 000010F8:00000000 01:7D19A035 00000000
> 0 0 396426 5 f71a8a80 202 42 1 144 -1
> 25: 111111AC:0016 480111AC:8565 01 00000B50:00000000 01:7CEBA7D1 00000000
> 0 0 349113 5 eeeaf580 204 40 1 26 -1
> old_pc>
These 7C/D... certainly seem strange values. Which TCP variant you have in
use (cat /proc/sys/net/ipv4/tcp_congestion_control)? It seems that vegas,
veno and yeah at least contain 0x7fffffff there for some rtt, which could
perhaps somehow leak.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 18:37 ` Ilpo Järvinen
@ 2008-05-31 20:25 ` Håkon Løvdal
2008-05-31 21:39 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Håkon Løvdal @ 2008-05-31 20:25 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
2008/5/31 Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>:
> Ok, like you said, this is not exactly the same, though it might be due to
> the same bug. In Ingo's case both endpoints were doing pretty healtily,
> with periodic window probes as expected. In your case TCP is not doing
> window probes but got that interesting RTO value.
>
> So you had that '-' earlier and you checked at that time but the
> connection is now already dead?
This is only from checking after the connection was dead. By the way,
I just had to remotely reboot the new machine because the window
manager locked up, however the old PC are still listing the defunct
connections after this.
> :-(, I would some much liked to see what they were doing.
I can of course keep on copying for testing purposes, but then I would like to
be able to dump only that single tcp connection, any tips of how to do that?
I found nothing specific in the manuals of wireshark and tcpdump. Of course
it is possible to capture everything and filter afterwards, but since I will be
transferring lots of data the logs will get huge and I would not like to have
even additional traffic inside...
> These 7C/D... certainly seem strange values. Which TCP variant you have in
> use (cat /proc/sys/net/ipv4/tcp_congestion_control)? It seems that vegas,
> veno and yeah at least contain 0x7fffffff there for some rtt, which could
> perhaps somehow leak.
I have not done any specific selection myself. On old_pc: bic, new_pc: cubic.
BR Håkon Løvdal
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 20:25 ` Håkon Løvdal
@ 2008-05-31 21:39 ` Ilpo Järvinen
2008-05-31 21:45 ` Håkon Løvdal
2008-06-04 0:10 ` Håkon Løvdal
0 siblings, 2 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-05-31 21:39 UTC (permalink / raw)
To: Håkon Løvdal
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
[-- Attachment #1: Type: TEXT/PLAIN, Size: 2815 bytes --]
On Sat, 31 May 2008, Håkon Løvdal wrote:
> 2008/5/31 Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>:
>
> > So you had that '-' earlier and you checked at that time but the
> > connection is now already dead?
>
> This is only from checking after the connection was dead.
Could you please rephrase the answer, I failed to understand it... :-)
...You said earlier that you had '-' owned connections like Ingo, when did
that happen (now the connections won't exists anymore, so at what point of
time you saw those non-owned connections)?
> By the way,
> I just had to remotely reboot the new machine because the window
> manager locked up, however the old PC are still listing the defunct
> connections after this.
Ok.
> > :-(, I would some much liked to see what they were doing.
>
> I can of course keep on copying for testing purposes, but then I would
> like to be able to dump only that single tcp connection, any tips of how
> to do that?
> I found nothing specific in the manuals of wireshark and tcpdump. Of
> cours it is possible to capture everything and filter afterwards, but
> since I will be transferring lots of data the logs will get huge and I
> would not like to have even additional traffic inside...
I didn't really mean tcpdump, I was more thinking of syscall what is the
syscall where the process is waiting. Though tcpdump might reveal
something as well about the behavior when nearing the problem,
tcpdump -n -i <iface> host <blahblah> and port <portno> and ...
Host & port as written above matches for either src and dst, I don't
remember how one could specify just one of them but it's not usually
necessary (won't be here either).
> > These 7C/D... certainly seem strange values. Which TCP variant you
> > have in use (cat /proc/sys/net/ipv4/tcp_congestion_control)? It seems
> > that vegas, veno and yeah at least contain 0x7fffffff there for some
> > rtt, which could perhaps somehow leak.
>
> I have not done any specific selection myself. On old_pc: bic, new_pc:
> cubic.
Ok, after some searching it also seems that it was a dead-end anyway:
- icsk_retransmit_timer is only set to icsk->icsk_timeout or
jiffies + (HZ / 20)
- icsk_timeout is only set after if (when > max_when) limiting (in
unsigned quantities)
- max_when is always given TCP_RTO_MAX by TCP...
...I'm currently out of ideas with this one then, I think I checked all
types too and nothing came up :-(.
Hmm, perhaps periodically checking /proc/net/tcp (e.g., once per 10s) if
the timeout is larger than TCP_RTO_MAX might allow some script to
immediately notice when things broke while reproducing it. Storing all
those once per 10s values shouldn't be a too big either, it could even be
done in both ends for a single flow (but I'll leave a script to do that on
Monday).
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 21:39 ` Ilpo Järvinen
@ 2008-05-31 21:45 ` Håkon Løvdal
2008-06-04 0:10 ` Håkon Løvdal
1 sibling, 0 replies; 96+ messages in thread
From: Håkon Løvdal @ 2008-05-31 21:45 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
2008/5/31 Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>:
> On Sat, 31 May 2008, Håkon Løvdal wrote:
>> 2008/5/31 Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>:
>> > So you had that '-' earlier and you checked at that time but the
>> > connection is now already dead?
>>
>> This is only from checking after the connection was dead.
>
> Could you please rephrase the answer, I failed to understand it... :-)
> ...You said earlier that you had '-' owned connections like Ingo, when did
> that happen (now the connections won't exists anymore, so at what point of
> time you saw those non-owned connections)?
What I mean is that I just tested "netstat -p" and noticed similarities
with Ingo's result today while replying to this thread, which is days
after the connections ended.
BR Håkon Løvdal
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 16:35 ` Ingo Molnar
@ 2008-05-31 22:46 ` Patrick McManus
2008-06-01 5:51 ` Ilpo Järvinen
2008-06-01 6:04 ` Eric Dumazet
2008-06-03 9:40 ` [fixed] [patch] " Ingo Molnar
1 sibling, 2 replies; 96+ messages in thread
From: Patrick McManus @ 2008-05-31 22:46 UTC (permalink / raw)
To: Ingo Molnar
Cc: Ilpo Järvinen, Peter Zijlstra, LKML, Netdev, David S. Miller,
Rafael J. Wysocki, Andrew Morton, Evgeniy Polyakov
On Sat, 2008-05-31 at 18:35 +0200, Ingo Molnar wrote:
> * Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
>
> > ...setsockopt(listenfd, SOL_TCP, TCP_DEFER_ACCEPT, &val, sizeof(val))
> > seems to be the magic trick that is interestion here.
>
> seems to be used:
>
> 22003 write(3, "distccd[22003] (dcc_listen_by_ad"..., 62) = 62
> 22003 listen(4, 10) = 0
> 22003 setsockopt(4, SOL_TCP, TCP_DEFER_ACCEPT, [1], 4) = 0
>
> i'll queue up your reverts for testing in -tip.
So the code you will revert came from my fingers. The circumstances here
make me nervous; while I'm at a loss to explain what might be going on
in particular, let me offer an apology in advance should the revert help
resolve the issue.
Here's what makes me nervous:
* not a lot of code uses DEFER_ACCEPT.. frankly it was pretty broken
before 26 - but not broken this way .. the correlation of your bug using
it is significant.
* in 26, a server TCP socket (with DA) goes to ESTABLISHED when the 3rd
part of the handshake is received (as normal without DA), but the socket
isn't put on the accept queue until a real data packet arrives. (That's
the point of DA). In <= 25 this socket would have syn-recv until the
data packet arrived.
- I did run tests where the server died in between the handshake being
completed and first data packet arriving - the client should see RST and
the server socket should disappear. But maybe something was missed?
Do I understand this correctly, the server process is gone but the
socket is still in the table? And the client process is still there
waiting for the server to do something - having sent a bunch of data?
Do we know if any data bytes (not handshake bytes) have been consumed by
the server side? If they were, that would seem to vindicate DA.
Also pointing away from DA is that you started seeing this with rc3 -
that code was included in rc1.Is that a firm observation, or maybe there
weren't enough datapoints to conclude that rc1 and rc2 were clean?
The most interesting patch is ec3c0982a2dd1e671bad8e9d26c28dcba0039d87
if anyone wants to eyeball it.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 22:46 ` Patrick McManus
@ 2008-06-01 5:51 ` Ilpo Järvinen
2008-06-01 6:04 ` Eric Dumazet
1 sibling, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-01 5:51 UTC (permalink / raw)
To: Patrick McManus
Cc: Ingo Molnar, Peter Zijlstra, LKML, Netdev, David S. Miller,
Rafael J. Wysocki, Andrew Morton, Evgeniy Polyakov
[-- Attachment #1: Type: TEXT/PLAIN, Size: 3905 bytes --]
On Sat, 31 May 2008, Patrick McManus wrote:
> On Sat, 2008-05-31 at 18:35 +0200, Ingo Molnar wrote:
> > * Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> >
>
> > > ...setsockopt(listenfd, SOL_TCP, TCP_DEFER_ACCEPT, &val, sizeof(val))
> > > seems to be the magic trick that is interestion here.
> >
> > seems to be used:
> >
> > 22003 write(3, "distccd[22003] (dcc_listen_by_ad"..., 62) = 62
> > 22003 listen(4, 10) = 0
> > 22003 setsockopt(4, SOL_TCP, TCP_DEFER_ACCEPT, [1], 4) = 0
> >
> > i'll queue up your reverts for testing in -tip.
>
>
> So the code you will revert came from my fingers. The circumstances here
> make me nervous; while I'm at a loss to explain what might be going on
> in particular, let me offer an apology in advance should the revert help
> resolve the issue.
Yes, don't worry just yet. It far from proven yet that this is the cause
(or contributes to easiness of reproducal in any way). The patch was just
for Ingo's testing in his -tip branch. I didn't even bother to cc you yet
because it's more or less a stab into dark, but it's definately worth of
testing still even though Ingo probably comes back soon and tells that it
didn't help any because it's clearly related :-).
> Here's what makes me nervous:
>
> * not a lot of code uses DEFER_ACCEPT.. frankly it was pretty broken
> before 26 - but not broken this way .. the correlation of your bug using
> it is significant.
>
> * in 26, a server TCP socket (with DA) goes to ESTABLISHED when the 3rd
> part of the handshake is received (as normal without DA), but the socket
> isn't put on the accept queue until a real data packet arrives. (That's
> the point of DA). In <= 25 this socket would have syn-recv until the
> data packet arrived.
>
> - I did run tests where the server died in between the handshake being
> completed and first data packet arriving - the client should see RST and
> the server socket should disappear. But maybe something was missed?
Also in this Ingo's case RST seems to be missing, ie., there's unread data
and both ends remain ESTABLISHED while the receiver is already gone (or
not referencing to the connection correctly).
> Do I understand this correctly, the server process is gone but the
> socket is still in the table? And the client process is still there
> waiting for the server to do something - having sent a bunch of data?
Yes, this seems to be the case, sender was doing window probes because
window became to zero.
Because it's distcc, tracking a particular process is not that simple
task. Either the process is gone or it doesn't correctly reference to the
connection.
> Do we know if any data bytes (not handshake bytes) have been consumed by
> the server side? If they were, that would seem to vindicate DA.
We don't know. We cannot currently track the particular process which
would definately be helpful here.
> Also pointing away from DA is that you started seeing this with rc3 -
> that code was included in rc1.Is that a firm observation, or maybe there
> weren't enough datapoints to conclude that rc1 and rc2 were clean?
Timeline won't match too well yes. I also find it quite unlikely, but
still worth of test because it's hard to know when this begun, luck might
have just played some role there because it's quite evasive in Ingo's
case anyway.
Anything you find suspicious between rc1..rc3?
...I suspected my rc3 FRTO fixes first but they have nothing to do with
window probing and orphan handling.
> The most interesting patch is ec3c0982a2dd1e671bad8e9d26c28dcba0039d87
> if anyone wants to eyeball it.
I personally think it might as well be some other issue which just become
more visible after DA but lets wait until Ingo has some results which may
well result in that DA is not making it to become visible in his case.
...Also, I doubt Arjan's mua has nothing to do with DA.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 22:46 ` Patrick McManus
2008-06-01 5:51 ` Ilpo Järvinen
@ 2008-06-01 6:04 ` Eric Dumazet
2008-06-02 9:23 ` Ingo Molnar
1 sibling, 1 reply; 96+ messages in thread
From: Eric Dumazet @ 2008-06-01 6:04 UTC (permalink / raw)
To: Patrick McManus
Cc: Ingo Molnar, Ilpo Järvinen, Peter Zijlstra, LKML, Netdev,
David S. Miller, Rafael J. Wysocki, Andrew Morton,
Evgeniy Polyakov
Patrick McManus a écrit :
> On Sat, 2008-05-31 at 18:35 +0200, Ingo Molnar wrote:
>
>> * Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
>>
>>
>
>
>>> ...setsockopt(listenfd, SOL_TCP, TCP_DEFER_ACCEPT, &val, sizeof(val))
>>> seems to be the magic trick that is interestion here.
>>>
>> seems to be used:
>>
>> 22003 write(3, "distccd[22003] (dcc_listen_by_ad"..., 62) = 62
>> 22003 listen(4, 10) = 0
>> 22003 setsockopt(4, SOL_TCP, TCP_DEFER_ACCEPT, [1], 4) = 0
>>
>> i'll queue up your reverts for testing in -tip.
>>
>
>
> So the code you will revert came from my fingers. The circumstances here
> make me nervous; while I'm at a loss to explain what might be going on
> in particular, let me offer an apology in advance should the revert help
> resolve the issue.
>
> Here's what makes me nervous:
>
> * not a lot of code uses DEFER_ACCEPT.. frankly it was pretty broken
> before 26 - but not broken this way .. the correlation of your bug using
> it is significant.
>
> * in 26, a server TCP socket (with DA) goes to ESTABLISHED when the 3rd
> part of the handshake is received (as normal without DA), but the socket
> isn't put on the accept queue until a real data packet arrives. (That's
> the point of DA). In <= 25 this socket would have syn-recv until the
> data packet arrived.
>
> - I did run tests where the server died in between the handshake being
> completed and first data packet arriving - the client should see RST and
> the server socket should disappear. But maybe something was missed?
>
> Do I understand this correctly, the server process is gone but the
> socket is still in the table? And the client process is still there
> waiting for the server to do something - having sent a bunch of data?
>
> Do we know if any data bytes (not handshake bytes) have been consumed by
> the server side? If they were, that would seem to vindicate DA.
>
> Also pointing away from DA is that you started seeing this with rc3 -
> that code was included in rc1.Is that a firm observation, or maybe there
> weren't enough datapoints to conclude that rc1 and rc2 were clean?
>
> The most interesting patch is ec3c0982a2dd1e671bad8e9d26c28dcba0039d87
> if anyone wants to eyeball it.
>
>
>
>
I believe Ingo problems come on long lived sockets (were many bytes were
exchanged between the peers), so I dont think DEFER_ACCEPT is the cullprit.
I suggest to enable CONFIG_TIMER_STATS and to check timers, because
/proc/net/tcp
can display apparently large timer values when the timer is elapsed
(jiffies > icsk->icsk_timeout)
and jiffies_to_clock_t(timer_expires - jiffies) is then overflowing
doing a multiply and a divide.
On a 64bits server running linux-2.6.24-rc2, I can see *strange* timers
values too in /proc/net/tcp, but not
stuck TCP sessions. On 64 bits, these strange values have 1AD7F
grep 1AD7F /proc/net/tcp | obfuscate_IP_and_ports
2017: local_peer remote_peer 03 00000000:00000000 01:1AD7F29ABBA
00000001 0 0 0 2 ffff81067e7520c0
2019: local_peer remote_peer 03 00000000:00000000 01:1AD7F29ABBA
00000003 0 0 0 2 ffff8106c580bcc0
2029: local_peer remote_peer 03 00000000:00000000 01:1AD7F29ABBA
00000002 0 0 0 2 ffff81067313fe40
2032: local_peer remote_peer 03 00000000:00000000 01:1AD7F29ABBA
00000003 0 0 0 2 ffff8106c716c340
2039: local_peer remote_peer 03 00000000:00000000 01:1AD7F29ABBA
00000002 0 0 0 2 ffff8107d45b3f40
2041: local_peer remote_peer 03 00000000:00000000 01:1AD7F29AB37
00000000 0 0 0 2 ffff810718e221c0
6610: local_peer remote_peer 01 00000000:00000000 00:1AD7F29ABCA
00000000 0 0 136594789 1 ffff8107183fb940 94 10 16 2 -1
9925: local_peer remote_peer 01 00000000:00000000 00:1AD7F29ABCA
00000000 0 0 144451161 1 ffff8107051a9840 351 10 0 2 -1
On TCP_SYN_RECV (03) sockets, timer can apparently be elapsed by many
ticks, while on TCP_ESTABLISHED (01) one, I get
jiffies_to_clock(-1) -> 1AD7F29ABCA value because the way
get_tcp4_sock() is coded (jiffies can change while running this function).
Note the 00: that means that no timer in my case.
Running again the command one second later gives completely different
results (other sockets are displayed)
Maybe on 2.6.26-rc3+ we miss some timer correctness or we expose a
latent NET bug.
void sk_reset_timer(struct sock *sk, struct timer_list* timer,
unsigned long expires)
{
if (!mod_timer(timer, expires))
sock_hold(sk);
}
Note that arming a timer also increase socket refcount and could explain
why Ingo have sockets
apparently not owned by a process but still referenced (by a timer or
many ones (I see refcnt=5) on following snapshots)
> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt
> uid timeout inode
> 21: 111111AC:0016 480111AC:E4E9 01 00000B50:00000000 01:7D1F8746 00000000
> 0 0 398713 5 f71a8580 205 40 1 36 -1
> 23: 111111AC:0016 480111AC:D359 01 000010F8:00000000 01:7D19A035 00000000
> 0 0 396426 5 f71a8a80 202 42 1 144 -1
> 25: 111111AC:0016 480111AC:8565 01 00000B50:00000000 01:7CEBA7D1 00000000
> 0 0 349113 5 eeeaf580 204 40 1 26 -1
Just my initial thoughts, sorry I currently cannot spend much time to
diagnose the problem.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 10:05 ` Ilpo Järvinen
@ 2008-06-02 6:19 ` Herbert Xu
2008-06-02 11:53 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Herbert Xu @ 2008-06-02 6:19 UTC (permalink / raw)
To: Ilpo J??rvinen
Cc: johnpol, ray-lk, mingo, linux-kernel, netdev, davem, rjw, akpm
Ilpo J??rvinen <ilpo.jarvinen@helsinki.fi> wrote:
>
> Before:
> tcp 65436 0 lo:5000 lo:59980 ESTABLISHED -
This is what happens before accept(2) is called. Ingo, please
strace the process that owns the listening socket to make sure
that it is actually calling accept(2). If it ever stops calling
accept(2) then this is expected behaviour.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-01 6:04 ` Eric Dumazet
@ 2008-06-02 9:23 ` Ingo Molnar
0 siblings, 0 replies; 96+ messages in thread
From: Ingo Molnar @ 2008-06-02 9:23 UTC (permalink / raw)
To: Eric Dumazet
Cc: Patrick McManus, Ilpo Järvinen, Peter Zijlstra, LKML, Netdev,
David S. Miller, Rafael J. Wysocki, Andrew Morton,
Evgeniy Polyakov
* Eric Dumazet <dada1@cosmosbay.com> wrote:
>>> 22003 write(3, "distccd[22003] (dcc_listen_by_ad"..., 62) = 62
>>> 22003 listen(4, 10) = 0
>>> 22003 setsockopt(4, SOL_TCP, TCP_DEFER_ACCEPT, [1], 4) = 0
>>>
>>> i'll queue up your reverts for testing in -tip.
I turned off localhost distcc two days ago and there has not been a
single hung socket since then, so we now know it for sure that without
localhost distcc connections, -tip's QA will not produce any hung
sockets in about 1000 random-kernel-build+boot iterations.
i've added those reverts this morning and added back the localhost
distcc rules - we'll see whether the hung sockets are back.
> I believe Ingo problems come on long lived sockets (were many bytes
> were exchanged between the peers), so I dont think DEFER_ACCEPT is the
> cullprit.
>
> I suggest to enable CONFIG_TIMER_STATS and to check timers, because
> /proc/net/tcp can display apparently large timer values when the timer
> is elapsed (jiffies > icsk->icsk_timeout) and
> jiffies_to_clock_t(timer_expires - jiffies) is then overflowing doing
> a multiply and a divide.
i'm wondering whether your suspicion on broken TCP timers is consistent
with the symptoms i've seen: the hung sockets clearly produced periodic
packet activity every 180 seconds, up to 8 hours, without ever changing
their receive of send queue. So at least a part of the TCP timer
mechanism for that specific stuck socket was working fine.
is there no sysctl or other debug mechanism to somehow get its full TCP
state and the reasons for why it is stuck? I'm wondering how you debug
broken TCP state machines without enabling testers to be able to dump
all state and passing it to developers.
I have a clearly reproducable testcase and i'd like to help out, but the
whole effort is stalled on 'not enough information' it appears. Doing
random reverts might help in truly helpless situations where a bug has
no debuggable state - but this situation seems really routine to me:
it's very difficult to trigger the bug but once it triggers the bug
scenario is stable and analyzable. I'd be glad to test any
instrumentation patch that makes similar scenarios more analyzable.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-02 6:19 ` Herbert Xu
@ 2008-06-02 11:53 ` Ilpo Järvinen
2008-06-02 14:08 ` Herbert Xu
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-02 11:53 UTC (permalink / raw)
To: Herbert Xu
Cc: johnpol, ray-lk, mingo, LKML, Netdev, David Miller, rjw,
Andrew Morton
On Mon, 2 Jun 2008, Herbert Xu wrote:
> Ilpo J??rvinen <ilpo.jarvinen@helsinki.fi> wrote:
> >
> > Before:
> > tcp 65436 0 lo:5000 lo:59980 ESTABLISHED -
>
> This is what happens before accept(2) is called. Ingo, please
> strace the process that owns the listening socket to make sure
> that it is actually calling accept(2). If it ever stops calling
> accept(2) then this is expected behaviour.
Shouldn't that be already visible in the ingo's sysrq-t dump, no? There:
http://marc.info/?l=linux-kernel&m=121182099131520&w=2
There are many sys_accepts at least (if I understand that dump at all).
There were some ip_local_outs there as well, which made me wonder if the
totally innocent looking commit 1ac06e0306d0192a7a4d9ea1c9e06d355ce7e7d3
(ipsec: Use the correct ip_local_out function) has something to do with
this (I definately don't understand that part at all).
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-02 11:53 ` Ilpo Järvinen
@ 2008-06-02 14:08 ` Herbert Xu
0 siblings, 0 replies; 96+ messages in thread
From: Herbert Xu @ 2008-06-02 14:08 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: johnpol, ray-lk, mingo, LKML, Netdev, David Miller, rjw,
Andrew Morton
On Mon, Jun 02, 2008 at 02:53:09PM +0300, Ilpo Järvinen wrote:
>
> Shouldn't that be already visible in the ingo's sysrq-t dump, no? There:
>
> http://marc.info/?l=linux-kernel&m=121182099131520&w=2
No, we need an actual strace rather than the static stack back-trace.
So start with netstat -lntp to locate the process listening on that
port, and then strace it to see if it's calling accept.
> There are many sys_accepts at least (if I understand that dump at all).
> There were some ip_local_outs there as well, which made me wonder if the
> totally innocent looking commit 1ac06e0306d0192a7a4d9ea1c9e06d355ce7e7d3
> (ipsec: Use the correct ip_local_out function) has something to do with
> this (I definately don't understand that part at all).
This patch only has an effect for IPsec.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 96+ messages in thread
* [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 16:35 ` Ingo Molnar
2008-05-31 22:46 ` Patrick McManus
@ 2008-06-03 9:40 ` Ingo Molnar
2008-06-03 14:41 ` Patrick McManus
2008-06-03 21:46 ` Ilpo Järvinen
1 sibling, 2 replies; 96+ messages in thread
From: Ingo Molnar @ 2008-06-03 9:40 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: Peter Zijlstra, LKML, Netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton, Evgeniy Polyakov
* Ingo Molnar <mingo@elte.hu> wrote:
> > ...setsockopt(listenfd, SOL_TCP, TCP_DEFER_ACCEPT, &val,
> > sizeof(val)) seems to be the magic trick that is interestion here.
>
> seems to be used:
>
> 22003 write(3, "distccd[22003] (dcc_listen_by_ad"..., 62) = 62
> 22003 listen(4, 10) = 0
> 22003 setsockopt(4, SOL_TCP, TCP_DEFER_ACCEPT, [1], 4) = 0
>
> i'll queue up your reverts for testing in -tip.
update: your 3 reverts in tip/out-of-tree [commit dad98991c] definitely
fixed the hangs!
Here is the testing i did:
first i ran about 500+ successful iterations on the affected testboxes
with your revert patch applied, on multiple systems. Then today, without
changing anything else on one of the testsystems i reverted your revert
on that single system. After about an hour of testing, in 20 iterations
i got a hang again over localhost:
titan:~> netstat -nt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 174592 10.0.1.14:34710 10.0.1.14:3632 ESTABLISHED
tcp 72145 0 10.0.1.14:3632 10.0.1.14:34710 ESTABLISHED
so i hereby conclude that your revert works :) I've repeated the commit
below that resolves this nasty regression.
phew ...
Ingo
---------------->
commit dad98991c137c089d64a3057dddc3a12bd6866b0
Author: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Date: Sat May 31 15:18:56 2008 +0300
tcp: revert DEFER_ACCEPT modifications
On Sat, 31 May 2008, Ilpo Järvinen wrote:
> On Sat, 31 May 2008, Ingo Molnar wrote:
>
> > ok, once i added back the localhost distcc component and the hung kernel
> > build + stuck TCP socket bug happened again overnight:
> >
> > Active Internet connections (w/o servers)
> > Proto Recv-Q Send-Q Local Address Foreign Address State
> > tcp 72187 0 10.0.1.14:3632 10.0.1.14:47910 ESTABLISHED
> > tcp 0 174464 10.0.1.14:47910 10.0.1.14:3632 ESTABLISHED
> >
> > so it seems distcc over localhost was the aspect that made it fail.
> >
> > _Perhaps_ what matters is to have the new post-rc3 TCP code on _both_
> > sides of the connection. But that is just a theory - it could be timing,
> > etc.
>
> Btw, does your distcc perhaps happen enable TCP_DEFER_ACCEPT (there were
> some post 2.6.25 changes into it)?
Hmm, if so, please try this revert below (I hope I got everything that is
related there, gitk is currently broken for me so that viewing many
commits quickly is a pain, so I might have accidently missed something)...
Reverts these commits:
[TCP]: TCP_DEFER_ACCEPT updates - defer timeout conflicts with max_t
[TCP]: TCP_DEFER_ACCEPT updates - dont retxmt synack
[TCP]: TCP_DEFER_ACCEPT updates - process as established
tcp: Fix slab corruption with ipv6 and tcp6fuzz
..., ie.,
539fae89bebd16ebeafd57a87169bc56eb530d76,
e4c78840284f3f51b1896cf3936d60a6033c4d2c,
ec3c0982a2dd1e671bad8e9d26c28dcba0039d87 and
9ae27e0adbf471c7a6b80102e38e1d5a346b3b38.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index 18e62e3..b31b6b7 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -239,11 +239,6 @@ static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
return (struct tcp_request_sock *)req;
}
-struct tcp_deferred_accept_info {
- struct sock *listen_sk;
- struct request_sock *request;
-};
-
struct tcp_sock {
/* inet_connection_sock has to be the first member of tcp_sock */
struct inet_connection_sock inet_conn;
@@ -379,8 +374,6 @@ struct tcp_sock {
unsigned int keepalive_intvl; /* time interval between keep alive probes */
int linger2;
- struct tcp_deferred_accept_info defer_tcp_accept;
-
unsigned long last_synq_overflow;
u32 tso_deferred;
diff --git a/include/net/request_sock.h b/include/net/request_sock.h
index b220b5f..0c96e7b 100644
--- a/include/net/request_sock.h
+++ b/include/net/request_sock.h
@@ -115,8 +115,8 @@ struct request_sock_queue {
struct request_sock *rskq_accept_head;
struct request_sock *rskq_accept_tail;
rwlock_t syn_wait_lock;
- u16 rskq_defer_accept;
- /* 2 bytes hole, try to pack */
+ u8 rskq_defer_accept;
+ /* 3 bytes hole, try to pack */
struct listen_sock *listen_opt;
};
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 633147c..981d5ba 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -139,7 +139,6 @@ extern void tcp_time_wait(struct sock *sk, int state, int timeo);
#define MAX_TCP_KEEPINTVL 32767
#define MAX_TCP_KEEPCNT 127
#define MAX_TCP_SYNCNT 127
-#define MAX_TCP_ACCEPT_DEFERRED 65535
#define TCP_SYNQ_INTERVAL (HZ/5) /* Period of SYNACK timer */
diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index 828ea21..ec83448 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -419,7 +419,8 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
struct inet_connection_sock *icsk = inet_csk(parent);
struct request_sock_queue *queue = &icsk->icsk_accept_queue;
struct listen_sock *lopt = queue->listen_opt;
- int thresh = icsk->icsk_syn_retries ? : sysctl_tcp_synack_retries;
+ int max_retries = icsk->icsk_syn_retries ? : sysctl_tcp_synack_retries;
+ int thresh = max_retries;
unsigned long now = jiffies;
struct request_sock **reqp, *req;
int i, budget;
@@ -455,6 +456,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
}
}
+ if (queue->rskq_defer_accept)
+ max_retries = queue->rskq_defer_accept;
+
budget = 2 * (lopt->nr_table_entries / (timeout / interval));
i = lopt->clock_hand;
@@ -462,8 +466,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
reqp=&lopt->syn_table[i];
while ((req = *reqp) != NULL) {
if (time_after_eq(now, req->expires)) {
- if (req->retrans < thresh &&
- !req->rsk_ops->rtx_syn_ack(parent, req)) {
+ if ((req->retrans < thresh ||
+ (inet_rsk(req)->acked && req->retrans < max_retries))
+ && !req->rsk_ops->rtx_syn_ack(parent, req)) {
unsigned long timeo;
if (req->retrans++ == 0)
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index f886531..a0deead 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -2105,12 +2105,15 @@ static int do_tcp_setsockopt(struct sock *sk, int level,
break;
case TCP_DEFER_ACCEPT:
- if (val < 0) {
- err = -EINVAL;
- } else {
- if (val > MAX_TCP_ACCEPT_DEFERRED)
- val = MAX_TCP_ACCEPT_DEFERRED;
- icsk->icsk_accept_queue.rskq_defer_accept = val;
+ icsk->icsk_accept_queue.rskq_defer_accept = 0;
+ if (val > 0) {
+ /* Translate value in seconds to number of
+ * retransmits */
+ while (icsk->icsk_accept_queue.rskq_defer_accept < 32 &&
+ val > ((TCP_TIMEOUT_INIT / HZ) <<
+ icsk->icsk_accept_queue.rskq_defer_accept))
+ icsk->icsk_accept_queue.rskq_defer_accept++;
+ icsk->icsk_accept_queue.rskq_defer_accept++;
}
break;
@@ -2292,7 +2295,8 @@ static int do_tcp_getsockopt(struct sock *sk, int level,
val = (val ? : sysctl_tcp_fin_timeout) / HZ;
break;
case TCP_DEFER_ACCEPT:
- val = icsk->icsk_accept_queue.rskq_defer_accept;
+ val = !icsk->icsk_accept_queue.rskq_defer_accept ? 0 :
+ ((TCP_TIMEOUT_INIT / HZ) << (icsk->icsk_accept_queue.rskq_defer_accept - 1));
break;
case TCP_WINDOW_CLAMP:
val = tp->window_clamp;
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index b54d9d3..ddc2e89 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4532,49 +4532,6 @@ static void tcp_urg(struct sock *sk, struct sk_buff *skb, struct tcphdr *th)
}
}
-static int tcp_defer_accept_check(struct sock *sk)
-{
- struct tcp_sock *tp = tcp_sk(sk);
-
- if (tp->defer_tcp_accept.request) {
- int queued_data = tp->rcv_nxt - tp->copied_seq;
- int hasfin = !skb_queue_empty(&sk->sk_receive_queue) ?
- tcp_hdr((struct sk_buff *)
- sk->sk_receive_queue.prev)->fin : 0;
-
- if (queued_data && hasfin)
- queued_data--;
-
- if (queued_data &&
- tp->defer_tcp_accept.listen_sk->sk_state == TCP_LISTEN) {
- if (sock_flag(sk, SOCK_KEEPOPEN)) {
- inet_csk_reset_keepalive_timer(sk,
- keepalive_time_when(tp));
- } else {
- inet_csk_delete_keepalive_timer(sk);
- }
-
- inet_csk_reqsk_queue_add(
- tp->defer_tcp_accept.listen_sk,
- tp->defer_tcp_accept.request,
- sk);
-
- tp->defer_tcp_accept.listen_sk->sk_data_ready(
- tp->defer_tcp_accept.listen_sk, 0);
-
- sock_put(tp->defer_tcp_accept.listen_sk);
- sock_put(sk);
- tp->defer_tcp_accept.listen_sk = NULL;
- tp->defer_tcp_accept.request = NULL;
- } else if (hasfin ||
- tp->defer_tcp_accept.listen_sk->sk_state != TCP_LISTEN) {
- tcp_reset(sk);
- return -1;
- }
- }
- return 0;
-}
-
static int tcp_copy_to_iovec(struct sock *sk, struct sk_buff *skb, int hlen)
{
struct tcp_sock *tp = tcp_sk(sk);
@@ -4935,8 +4892,6 @@ step5:
tcp_data_snd_check(sk);
tcp_ack_snd_check(sk);
-
- tcp_defer_accept_check(sk);
return 0;
csum_error:
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index cd601a8..b698b5b 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1918,14 +1918,6 @@ int tcp_v4_destroy_sock(struct sock *sk)
sk->sk_sndmsg_page = NULL;
}
- if (tp->defer_tcp_accept.request) {
- reqsk_free(tp->defer_tcp_accept.request);
- sock_put(tp->defer_tcp_accept.listen_sk);
- sock_put(sk);
- tp->defer_tcp_accept.listen_sk = NULL;
- tp->defer_tcp_accept.request = NULL;
- }
-
atomic_dec(&tcp_sockets_allocated);
return 0;
diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c
index 019c8c1..8245247 100644
--- a/net/ipv4/tcp_minisocks.c
+++ b/net/ipv4/tcp_minisocks.c
@@ -571,8 +571,10 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
does sequence test, SYN is truncated, and thus we consider
it a bare ACK.
- Both ends (listening sockets) accept the new incoming
- connection and try to talk to each other. 8-)
+ If icsk->icsk_accept_queue.rskq_defer_accept, we silently drop this
+ bare ACK. Otherwise, we create an established connection. Both
+ ends (listening sockets) accept the new incoming connection and try
+ to talk to each other. 8-)
Note: This case is both harmless, and rare. Possibility is about the
same as us discovering intelligent life on another plant tomorrow.
@@ -640,6 +642,13 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
if (!(flg & TCP_FLAG_ACK))
return NULL;
+ /* If TCP_DEFER_ACCEPT is set, drop bare ACK. */
+ if (inet_csk(sk)->icsk_accept_queue.rskq_defer_accept &&
+ TCP_SKB_CB(skb)->end_seq == tcp_rsk(req)->rcv_isn + 1) {
+ inet_rsk(req)->acked = 1;
+ return NULL;
+ }
+
/* OK, ACK is valid, create big socket and
* feed this segment to it. It will repeat all
* the tests. THIS SEGMENT MUST MOVE SOCKET TO
@@ -678,24 +687,7 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
inet_csk_reqsk_queue_unlink(sk, req, prev);
inet_csk_reqsk_queue_removed(sk, req);
- if (inet_csk(sk)->icsk_accept_queue.rskq_defer_accept &&
- TCP_SKB_CB(skb)->end_seq == tcp_rsk(req)->rcv_isn + 1) {
-
- /* the accept queue handling is done is est recv slow
- * path so lets make sure to start there
- */
- tcp_sk(child)->pred_flags = 0;
- sock_hold(sk);
- sock_hold(child);
- tcp_sk(child)->defer_tcp_accept.listen_sk = sk;
- tcp_sk(child)->defer_tcp_accept.request = req;
-
- inet_csk_reset_keepalive_timer(child,
- inet_csk(sk)->icsk_accept_queue.rskq_defer_accept * HZ);
- } else {
- inet_csk_reqsk_queue_add(sk, req, child);
- }
-
+ inet_csk_reqsk_queue_add(sk, req, child);
return child;
listen_overflow:
diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
index 4de68cf..63ed9d6 100644
--- a/net/ipv4/tcp_timer.c
+++ b/net/ipv4/tcp_timer.c
@@ -489,11 +489,6 @@ static void tcp_keepalive_timer (unsigned long data)
goto death;
}
- if (tp->defer_tcp_accept.request && sk->sk_state == TCP_ESTABLISHED) {
- tcp_send_active_reset(sk, GFP_ATOMIC);
- goto death;
- }
-
if (!sock_flag(sk, SOCK_KEEPOPEN) || sk->sk_state == TCP_CLOSE)
goto out;
^ permalink raw reply related [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 9:40 ` [fixed] [patch] " Ingo Molnar
@ 2008-06-03 14:41 ` Patrick McManus
2008-06-03 21:46 ` Ilpo Järvinen
1 sibling, 0 replies; 96+ messages in thread
From: Patrick McManus @ 2008-06-03 14:41 UTC (permalink / raw)
To: Ingo Molnar
Cc: Ilpo Järvinen, Peter Zijlstra, LKML, Netdev, David S. Miller,
Rafael J. Wysocki, Andrew Morton, Evgeniy Polyakov
On Tue, 2008-06-03 at 11:40 +0200, Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
>
> > > ...setsockopt(listenfd, SOL_TCP, TCP_DEFER_ACCEPT, &val,
> > > sizeof(val)) seems to be the magic trick that is interestion here.
> >
> > seems to be used:
> >
> > 22003 write(3, "distccd[22003] (dcc_listen_by_ad"..., 62) = 62
> > 22003 listen(4, 10) = 0
> > 22003 setsockopt(4, SOL_TCP, TCP_DEFER_ACCEPT, [1], 4) = 0
> >
> > i'll queue up your reverts for testing in -tip.
>
> update: your 3 reverts in tip/out-of-tree [commit dad98991c] definitely
> fixed the hangs!
>
> Here is the testing i did:
hm.
Here's a theory - the DA code put the socket in the accept queue after
data has been queued on the socket, instead of the other way around
which is normal.
Could this be confusing the select() mechanism?
-Pat
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 9:40 ` [fixed] [patch] " Ingo Molnar
2008-06-03 14:41 ` Patrick McManus
@ 2008-06-03 21:46 ` Ilpo Järvinen
2008-06-03 22:01 ` Ilpo Järvinen
2008-06-04 7:23 ` Ingo Molnar
1 sibling, 2 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-03 21:46 UTC (permalink / raw)
To: Ingo Molnar
Cc: Peter Zijlstra, LKML, Netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton, Evgeniy Polyakov, Patrick McManus
[-- Attachment #1: Type: TEXT/PLAIN, Size: 3897 bytes --]
On Tue, 3 Jun 2008, Ingo Molnar wrote:
> * Ingo Molnar <mingo@elte.hu> wrote:
>
> > > ...setsockopt(listenfd, SOL_TCP, TCP_DEFER_ACCEPT, &val,
> > > sizeof(val)) seems to be the magic trick that is interestion here.
> >
> > seems to be used:
> >
> > 22003 write(3, "distccd[22003] (dcc_listen_by_ad"..., 62) = 62
> > 22003 listen(4, 10) = 0
> > 22003 setsockopt(4, SOL_TCP, TCP_DEFER_ACCEPT, [1], 4) = 0
> >
> > i'll queue up your reverts for testing in -tip.
>
> update: your 3 reverts in tip/out-of-tree [commit dad98991c] definitely
> fixed the hangs!
...It wasn't exactly out-of-tree, Evgeniy fixed a problem that was found
in "TCP_DEFER_ACCEPT updates - process as established", perhaps it just
wasn't in your testing tree yet.
$ git-log -n 1 --pretty=full 9ae27e0adbf471c7a6b80102e38e1d5a346b3b38 |
grep "Commit:"
Commit: David S. Miller <davem@davemloft.net>
> Here is the testing i did:
>
> first i ran about 500+ successful iterations on the affected testboxes
> with your revert patch applied, on multiple systems.
Are you sure this is enough to conclude the results? Seems quite small
number to me to rule out luck. Especially considering that it was some
amount of time in the tree already until you noticed it for the first
time.
Anyway, nice that it seems to be helping. It was almost the only
possibility on TCP side, I don't think there were any other state machine
related changes. So it wasn't just "random revert" in that sense like you
were implying :-), I just didn't have any theory how it would cause the
problem... ...I even first disregarded DA that because of timeline
in-exactness and because I wrongly assumed that distcc probably won't use
it anyway, but then I checked later on and found out that it was present
at least in the source I had lying around.
Anyway, it might be that the revert was a bit overkill, I'm not fully sure
if 539fae and e4c7884 need to be reverted to fix it since main changes
are in ec3c098. I just didn't want to take chances at first and put them
all to the revert list.
> Then today, without
> changing anything else on one of the testsystems i reverted your revert
> on that single system. After about an hour of testing, in 20 iterations
> i got a hang again over localhost:
>
> titan:~> netstat -nt
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 174592 10.0.1.14:34710 10.0.1.14:3632 ESTABLISHED
> tcp 72145 0 10.0.1.14:3632 10.0.1.14:34710 ESTABLISHED
>
> so i hereby conclude that your revert works :) I've repeated the commit
> below that resolves this nasty regression.
...I couldn't immediately find anything obviously wrong with those changes
but the patch below might be worth of a try (without the revert of
course). If it ever spits out that WARN_ON for you, we were playing with
fire too much and it's better to return on the safe side there...
--
i.
[PATCH] tcp DEFER_ACCEPT: see if header prediction got turned on
If header prediction is turned on under some circumstances,
DA can deadlock though I have great trouble in figuring out
how it could ever happen while ending up into that else
branch (but I've been wrong before as well :-)).
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
---
net/ipv4/tcp_input.c | 3 +++
1 files changed, 3 insertions(+), 0 deletions(-)
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index c9454f0..0d9a3fe 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4595,6 +4595,9 @@ static int tcp_defer_accept_check(struct sock *sk)
tp->defer_tcp_accept.listen_sk->sk_state != TCP_LISTEN) {
tcp_reset(sk);
return -1;
+ } else {
+ WARN_ON(tp->pred_flags);
+ tp->pred_flags = 0;
}
}
return 0;
--
1.5.2.2
^ permalink raw reply related [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 21:46 ` Ilpo Järvinen
@ 2008-06-03 22:01 ` Ilpo Järvinen
2008-06-03 22:03 ` David Miller
2008-06-04 7:23 ` Ingo Molnar
1 sibling, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-03 22:01 UTC (permalink / raw)
To: Ingo Molnar
Cc: Peter Zijlstra, LKML, Netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton, Evgeniy Polyakov, Patrick McManus
[-- Attachment #1: Type: TEXT/PLAIN, Size: 647 bytes --]
On Wed, 4 Jun 2008, Ilpo Järvinen wrote:
> ...I couldn't immediately find anything obviously wrong with those changes
> but the patch below might be worth of a try (without the revert of
> course). If it ever spits out that WARN_ON for you, we were playing with
> fire too much and it's better to return on the safe side there...
> [PATCH] tcp DEFER_ACCEPT: see if header prediction got turned on
>
> If header prediction is turned on under some circumstances,
> DA can deadlock though I have great trouble in figuring out
...Nah, keepalive timer would then eventually kill it then, so no
deadlock seems possible through that one.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 22:01 ` Ilpo Järvinen
@ 2008-06-03 22:03 ` David Miller
2008-06-03 22:10 ` Ilpo Järvinen
2008-06-03 23:22 ` Ilpo Järvinen
0 siblings, 2 replies; 96+ messages in thread
From: David Miller @ 2008-06-03 22:03 UTC (permalink / raw)
To: ilpo.jarvinen
Cc: mingo, peterz, linux-kernel, netdev, rjw, akpm, johnpol, mcmanus
From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi>
Date: Wed, 4 Jun 2008 01:01:25 +0300 (EEST)
> On Wed, 4 Jun 2008, Ilpo Järvinen wrote:
>
> > ...I couldn't immediately find anything obviously wrong with those changes
> > but the patch below might be worth of a try (without the revert of
> > course). If it ever spits out that WARN_ON for you, we were playing with
> > fire too much and it's better to return on the safe side there...
>
>
> > [PATCH] tcp DEFER_ACCEPT: see if header prediction got turned on
> >
> > If header prediction is turned on under some circumstances,
> > DA can deadlock though I have great trouble in figuring out
>
> ...Nah, keepalive timer would then eventually kill it then, so no
> deadlock seems possible through that one.
Keepalive is very long, it might still "seem" like a deadlock for
someone without much patience :-)
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 22:03 ` David Miller
@ 2008-06-03 22:10 ` Ilpo Järvinen
2008-06-03 23:22 ` Ilpo Järvinen
1 sibling, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-03 22:10 UTC (permalink / raw)
To: David Miller
Cc: mingo, peterz, LKML, Netdev, rjw, Andrew Morton, johnpol, mcmanus
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1161 bytes --]
On Tue, 3 Jun 2008, David Miller wrote:
> From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi>
> Date: Wed, 4 Jun 2008 01:01:25 +0300 (EEST)
>
> > On Wed, 4 Jun 2008, Ilpo Järvinen wrote:
> >
> > > ...I couldn't immediately find anything obviously wrong with those changes
> > > but the patch below might be worth of a try (without the revert of
> > > course). If it ever spits out that WARN_ON for you, we were playing with
> > > fire too much and it's better to return on the safe side there...
> >
> >
> > > [PATCH] tcp DEFER_ACCEPT: see if header prediction got turned on
> > >
> > > If header prediction is turned on under some circumstances,
> > > DA can deadlock though I have great trouble in figuring out
> >
> > ...Nah, keepalive timer would then eventually kill it then, so no
> > deadlock seems possible through that one.
>
> Keepalive is very long, it might still "seem" like a deadlock for
> someone without much patience :-)
Sure, but I think Ingo kept it once hang for hours, ~8h or so...
...Though I'm not sure if that qualifies as a proof of Ingo's patience
but rather shows his willingness to help solving the matter... ;-)
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 22:03 ` David Miller
2008-06-03 22:10 ` Ilpo Järvinen
@ 2008-06-03 23:22 ` Ilpo Järvinen
2008-06-03 23:54 ` Joe Perches
` (3 more replies)
1 sibling, 4 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-03 23:22 UTC (permalink / raw)
To: David Miller, mingo, mcmanus, peterz
Cc: LKML, Netdev, rjw, Andrew Morton, johnpol
[-- Attachment #1: Type: TEXT/PLAIN, Size: 3981 bytes --]
On Tue, 3 Jun 2008, David Miller wrote:
> From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi>
> Date: Wed, 4 Jun 2008 01:01:25 +0300 (EEST)
>
> > On Wed, 4 Jun 2008, Ilpo Järvinen wrote:
> >
> > > ...I couldn't immediately find anything obviously wrong with those changes
> > > but the patch below might be worth of a try (without the revert of
> > > course). If it ever spits out that WARN_ON for you, we were playing with
> > > fire too much and it's better to return on the safe side there...
> >
> >
> > > [PATCH] tcp DEFER_ACCEPT: see if header prediction got turned on
> > >
> > > If header prediction is turned on under some circumstances,
> > > DA can deadlock though I have great trouble in figuring out
> >
> > ...Nah, keepalive timer would then eventually kill it then, so no
> > deadlock seems possible through that one.
>
> Keepalive is very long, it might still "seem" like a deadlock for
> someone without much patience :-)
I think we want that clearing there, it's better to be safe than sorry
there and to not put any trust on the keepalive thingie which tears down
rather than results in a connection.
But here's somewhat more likely explanation... Only compile tested...
It probably needs some commenting from people who understand locking
variants & details (I don't).
--
i.
--
[PATCH] tcp DEFER_ACCEPT: fix racy access to listen_sk
It seems that replacement of DA code also moved parts outside
of appropriate locking. The Ingo's problem seems to come from
the fact that two flows could now race in
(inet_csk_)reqsk_queue_add corrupting the queue. ...This can
leave dangling socks around which won't resolve themselves
without stimuli from outside (e.g., external RST would help
I think).
Then some details I'm not too sure of:
I guess we want to put listen_sk->sk_state checking under the
lock as well. I've not evaluated if ->sk_data_ready too
requires locking but assumed it does.
I'm by no means familiar with all locking variants, requirements,
etc.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
---
net/ipv4/tcp_input.c | 23 +++++++++++++----------
1 files changed, 13 insertions(+), 10 deletions(-)
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index c9454f0..d21d2b9 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4562,6 +4562,7 @@ static int tcp_defer_accept_check(struct sock *sk)
struct tcp_sock *tp = tcp_sk(sk);
if (tp->defer_tcp_accept.request) {
+ struct sock *listen_sk = tp->defer_tcp_accept.listen_sk;
int queued_data = tp->rcv_nxt - tp->copied_seq;
int hasfin = !skb_queue_empty(&sk->sk_receive_queue) ?
tcp_hdr((struct sk_buff *)
@@ -4570,8 +4571,9 @@ static int tcp_defer_accept_check(struct sock *sk)
if (queued_data && hasfin)
queued_data--;
- if (queued_data &&
- tp->defer_tcp_accept.listen_sk->sk_state == TCP_LISTEN) {
+ bh_lock_sock(listen_sk);
+
+ if (queued_data && listen_sk->sk_state == TCP_LISTEN) {
if (sock_flag(sk, SOCK_KEEPOPEN)) {
inet_csk_reset_keepalive_timer(sk,
keepalive_time_when(tp));
@@ -4579,23 +4581,24 @@ static int tcp_defer_accept_check(struct sock *sk)
inet_csk_delete_keepalive_timer(sk);
}
- inet_csk_reqsk_queue_add(
- tp->defer_tcp_accept.listen_sk,
- tp->defer_tcp_accept.request,
- sk);
+ inet_csk_reqsk_queue_add(listen_sk,
+ tp->defer_tcp_accept.request,
+ sk);
tp->defer_tcp_accept.listen_sk->sk_data_ready(
- tp->defer_tcp_accept.listen_sk, 0);
+ listen_sk, 0);
- sock_put(tp->defer_tcp_accept.listen_sk);
+ sock_put(listen_sk);
sock_put(sk);
tp->defer_tcp_accept.listen_sk = NULL;
tp->defer_tcp_accept.request = NULL;
- } else if (hasfin ||
- tp->defer_tcp_accept.listen_sk->sk_state != TCP_LISTEN) {
+ } else if (hasfin || listen_sk->sk_state != TCP_LISTEN) {
+ bh_unlock_sock(listen_sk);
tcp_reset(sk);
return -1;
}
+
+ bh_unlock_sock(listen_sk);
}
return 0;
}
--
1.5.2.2
^ permalink raw reply related [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 23:22 ` Ilpo Järvinen
@ 2008-06-03 23:54 ` Joe Perches
2008-06-04 6:25 ` Ilpo Järvinen
2008-06-04 2:54 ` Patrick McManus
` (2 subsequent siblings)
3 siblings, 1 reply; 96+ messages in thread
From: Joe Perches @ 2008-06-03 23:54 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: David Miller, mingo, mcmanus, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
On Wed, 2008-06-04 at 02:22 +0300, Ilpo Järvinen wrote:
> But here's somewhat more likely explanation... Only compile tested...
> It probably needs some commenting from people who understand locking
> variants & details (I don't).
> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
> diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> index c9454f0..d21d2b9 100644
> --- a/net/ipv4/tcp_input.c
> +++ b/net/ipv4/tcp_input.c
> @@ -4562,6 +4562,7 @@ static int tcp_defer_accept_check(struct sock *sk)
> struct tcp_sock *tp = tcp_sk(sk);
>
> if (tp->defer_tcp_accept.request) {
> + struct sock *listen_sk = tp->defer_tcp_accept.listen_sk;
Not commenting on the locking, but I think the code
would be clearer if tp->defer_tcp_accept was used in
a temporary instead.
Perhaps:
net/ipv4/tcp_input.c | 33 ++++++++++++++++-----------------
1 files changed, 16 insertions(+), 17 deletions(-)
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index b54d9d3..f846e11 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4535,18 +4535,18 @@ static void tcp_urg(struct sock *sk, struct sk_buff *skb, struct tcphdr *th)
static int tcp_defer_accept_check(struct sock *sk)
{
struct tcp_sock *tp = tcp_sk(sk);
-
- if (tp->defer_tcp_accept.request) {
- int queued_data = tp->rcv_nxt - tp->copied_seq;
- int hasfin = !skb_queue_empty(&sk->sk_receive_queue) ?
+ struct tcp_deferred_accept_info *tdai = &tp->defer_tcp_accept;
+ if (tdai->request) {
+ int queued_data = tp->rcv_nxt - tp->copied_seq;
+ int hasfin = !skb_queue_empty(&sk->sk_receive_queue) ?
tcp_hdr((struct sk_buff *)
sk->sk_receive_queue.prev)->fin : 0;
if (queued_data && hasfin)
queued_data--;
- if (queued_data &&
- tp->defer_tcp_accept.listen_sk->sk_state == TCP_LISTEN) {
+ bh_lock_sock(tdai->listen_sk);
+ if (queued_data && tdai->listen_sk->sk_state == TCP_LISTEN) {
if (sock_flag(sk, SOCK_KEEPOPEN)) {
inet_csk_reset_keepalive_timer(sk,
keepalive_time_when(tp));
@@ -4554,23 +4554,22 @@ static int tcp_defer_accept_check(struct sock *sk)
inet_csk_delete_keepalive_timer(sk);
}
- inet_csk_reqsk_queue_add(
- tp->defer_tcp_accept.listen_sk,
- tp->defer_tcp_accept.request,
- sk);
+ inet_csk_reqsk_queue_add(tdai->listen_sk,
+ tdai->request,
+ sk);
- tp->defer_tcp_accept.listen_sk->sk_data_ready(
- tp->defer_tcp_accept.listen_sk, 0);
+ tdai->listen_sk->sk_data_ready(tdai->listen_sk, 0);
- sock_put(tp->defer_tcp_accept.listen_sk);
+ sock_put(tdai->listen_sk);
sock_put(sk);
- tp->defer_tcp_accept.listen_sk = NULL;
- tp->defer_tcp_accept.request = NULL;
- } else if (hasfin ||
- tp->defer_tcp_accept.listen_sk->sk_state != TCP_LISTEN) {
+ tdai->listen_sk = NULL;
+ tdai->request = NULL;
+ } else if (hasfin || tdai->listen_sk->sk_state != TCP_LISTEN) {
+ bh_unlock_sock(tdai->listen_sk);
tcp_reset(sk);
return -1;
}
+ bh_unlock_sock(tdai->listen_sk);
}
return 0;
}
^ permalink raw reply related [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-05-31 21:39 ` Ilpo Järvinen
2008-05-31 21:45 ` Håkon Løvdal
@ 2008-06-04 0:10 ` Håkon Løvdal
2008-06-04 11:14 ` Ilpo Järvinen
1 sibling, 1 reply; 96+ messages in thread
From: Håkon Løvdal @ 2008-06-04 0:10 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
2008/5/31 Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>:
> I didn't really mean tcpdump, I was more thinking of syscall what is the
> syscall where the process is waiting. Though tcpdump might reveal
> something as well about the behavior when nearing the problem,
> Hmm, perhaps periodically checking /proc/net/tcp (e.g., once per 10s) if
> the timeout is larger than TCP_RTO_MAX might allow some script to
> immediately notice when things broke while reproducing it. Storing all
> those once per 10s values shouldn't be a too big either, it could even be
> done in both ends for a single flow (but I'll leave a script to do that on
> Monday).
Hi. I have now tested and taken various logs before/under/after (ps, netstat,
tcpdump, strace, /proc/net/tcp). The logs did not grow as big as I feared
however they are way too big attach here. Any tips on how to distribute?
I mentioned before that the ssh connection died on the new PC, but sometimes
it just hangs as well, i.e. the ssh connection is alive but there is
no data transferred.
The last test I did hang after just 292MB transferred. In this case pv
just hangs
on the new PC, waiting for data that never arrives. On the sending
side tar which
is producing data have died and the corresponding ssh is locked in some
uninterruptible sleep:
//netstat before the hang:
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name Timer
tcp 0 504 new_pc:22 old_pc:39464
ESTABLISHED 17054/sshd: hlovdal on (0.20/0/0)
// after the hang:
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name Timer
tcp 0 2896 old_pc:22 new_pc:39464
ESTABLISHED - on (21329441.19/0/0)
old_pc>.ps -elf | sed -n '1p; /17054.*ssh/p
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
4 D root 17054 4402 0 80 0 - 0 - 00:08 ?
00:00:00 [sshd]
5 Z hlovdal 17058 17054 0 80 0 - 0 - 00:08 ?
00:00:33 [sshd] <defunct>
old_pc>strace -p 17054
attach: ptrace(PTRACE_ATTACH, ...): Operation not permitted
old_pc>
Many things in /proc/17054/ are in a non-normal state (exe is an
empty symlink, fd is empty, etc)
but the stat* files seem normal:
/proc/17054/stat
17054 (sshd) D 4402 17054 17054 0 -1 4202756 1007 395 11 2 1 6 0 1 20
0 1 0 80373564 0 0 4294967295 0 0 0 0 0 0 0 4096 16387 0 0 0 17 0 0 0
37 0 0
/proc/17054/statm
0 0 0 0 0 0 0
/proc/17054/status
Name: sshd
State: D (disk sleep)
Tgid: 17054
Pid: 17054
PPid: 4402
TracerPid: 0
Uid: 0 0 500 0
Gid: 0 0 500 0
FDSize: 0
Groups:
Threads: 1
SigQ: 0/32639
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001000
SigCgt: 0000000180004003
CapInh: 0000000000000000
CapPrm: 00000000ffffffff
CapEff: 00000000ffffffff
voluntary_ctxt_switches: 86
nonvoluntary_ctxt_switches: 43
I also took a sysrq-t dump and here is the the output related to 17054:
sshd D 000110ab 1972 17054 4402
f0628aa0 00200046 a3fbe728 000110ab f3788640 a4088c34 000110ab effb1a80
effb1ab0 effb1ac4 e769a380 c05be260 00000001 f0628aa0 c042f321 effb1ad4
effb1ad4 effb1a80 e769a3b4 00000000 e769a380 c05bb9aa 00000000 c9e82c00
Call Trace:
[<c05be260>] lock_sock_nested+0x6c/0x9d
[<c042f321>] autoremove_wake_function+0x0/0x35
[<c05bb9aa>] sock_fasync+0x5e/0x113
[<c05bcc68>] sock_close+0x22/0x2f
[<c0468b07>] __fput+0xaf/0x180
[<c046623a>] filp_close+0x51/0x58
[<c0421318>] put_files_struct+0x5f/0xa7
[<c0422443>] do_exit+0x208/0x6b5
[<c045990b>] do_munmap+0x193/0x1ac
[<c042295b>] sys_exit_group+0x0/0xd
[<c0404daa>] sysenter_past_esp+0x5f/0x85
=======================
sshd ? c04dc1a6 1296 17058 17054
d240d740 00200046 c0626ea3 c04dc1a6 c7886380 d240d740 00000000 00000010
d240d740 f0628aa0 f783f440 c04228ec c06d0b50 d240d9fd 000042a2 00000002
d240d8c4 00000000 f394eb60 f394eb60 f394ebe8 f394ec20 c06c0068 00200006
Call Trace:
[<c0626ea3>] _spin_unlock+0xf/0x23
[<c04dc1a6>] _atomic_dec_and_lock+0x22/0x2c
[<c04228ec>] do_exit+0x6b1/0x6b5
[<c04060d0>] die+0x1d8/0x1e0
[<c062838e>] do_page_fault+0x0/0x615
[<c06288b9>] do_page_fault+0x52b/0x615
[<c062838e>] do_page_fault+0x0/0x615
[<c062710a>] error_code+0x6a/0x70
[<f8c77b39>] nf_conntrack_find_get+0x36/0x3a [nf_conntrack]
[<f8c78379>] nf_conntrack_in+0x135/0x3e8 [nf_conntrack]
[<c0423c5d>] local_bh_disable+0xa/0xb
[<c0626d54>] _read_lock_bh+0x12/0x1a
[<c0423d2c>] local_bh_enable_ip+0x37/0x4b
[<f8a4a3a3>] ipt_do_table+0x2af/0x2d8 [ip_tables]
[<f8c88161>] ipv4_conntrack_local+0x0/0x51 [nf_conntrack_ipv4]
[<c05dbde6>] nf_iterate+0x3a/0x6e
[<c05e33b0>] dst_output+0x0/0x7
[<c05dc12f>] nf_hook_slow+0x57/0xde
[<c05e33b0>] dst_output+0x0/0x7
[<c05e5678>] ip_queue_xmit+0x2ff/0x340
[<c05e33b0>] dst_output+0x0/0x7
[<c05f34d7>] tcp_transmit_skb+0x6cc/0x6ff
[<c05f34d7>] tcp_transmit_skb+0x6cc/0x6ff
[<c04057bb>] common_interrupt+0x23/0x28
[<c05f2acc>] tcp_snd_test+0x13/0xe1
[<c05f554c>] tcp_push_one+0xb3/0xd4
[<c05eb11e>] tcp_sendmsg+0x7f1/0xa3b
[<c05ebae4>] tcp_recvmsg+0x60b/0x717
[<c04bcb75>] avc_has_perm_noaudit+0x389/0x431
[<c04c57e4>] selinux_netlbl_inode_permission+0x40/0xa3
[<c05bbd73>] sock_aio_write+0xe3/0xef
[<c04678e1>] do_sync_write+0xc6/0x109
[<c0626f11>] _spin_unlock_irqrestore+0x14/0x2a
[<c042f321>] autoremove_wake_function+0x0/0x35
[<c0468135>] vfs_write+0xb5/0x14b
[<c046869f>] sys_write+0x41/0x67
[<c0404daa>] sysenter_past_esp+0x5f/0x85
=======================
All the other ssh zombies are also stuck in lock_sock_nested/_spin_unlock
(except one pair of lock_sock_nested/do_exit).
Any other tests I should do/logs you want?
BR Håkon Løvdal
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 23:22 ` Ilpo Järvinen
2008-06-03 23:54 ` Joe Perches
@ 2008-06-04 2:54 ` Patrick McManus
2008-06-04 6:42 ` Ilpo Järvinen
2008-06-05 14:22 ` Ingo Molnar
2008-06-10 22:32 ` David Miller
3 siblings, 1 reply; 96+ messages in thread
From: Patrick McManus @ 2008-06-04 2:54 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: David Miller, mingo, peterz, LKML, Netdev, rjw, Andrew Morton,
johnpol
On Wed, 2008-06-04 at 02:22 +0300, Ilpo Järvinen wrote:
==
> --
> [PATCH] tcp DEFER_ACCEPT: fix racy access to listen_sk
>
> It seems that replacement of DA code also moved parts outside
> of appropriate locking. The Ingo's problem seems to come from
> the fact that two flows could now race in
> (inet_csk_)reqsk_queue_add corrupting the queue. ...This can
> leave dangling socks around which won't resolve themselves
> without stimuli from outside (e.g., external RST would help
> I think).
Ilpo, has anyone told you today that you rock? Well allow me - you
rock.
>
do_rcv() clearly has the listening socket locked in the non-DA case, and
in the DA case it is the 'child' ESTABLISHED socket that is locked -
leaving the accept queue unprotected. So simple.
-Pat
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 23:54 ` Joe Perches
@ 2008-06-04 6:25 ` Ilpo Järvinen
0 siblings, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-04 6:25 UTC (permalink / raw)
To: Joe Perches
Cc: David Miller, mingo, mcmanus, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
[-- Attachment #1: Type: TEXT/PLAIN, Size: 3532 bytes --]
On Tue, 3 Jun 2008, Joe Perches wrote:
> On Wed, 2008-06-04 at 02:22 +0300, Ilpo Järvinen wrote:
> > But here's somewhat more likely explanation... Only compile tested...
> > It probably needs some commenting from people who understand locking
> > variants & details (I don't).
> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
> > diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> > index c9454f0..d21d2b9 100644
> > --- a/net/ipv4/tcp_input.c
> > +++ b/net/ipv4/tcp_input.c
> > @@ -4562,6 +4562,7 @@ static int tcp_defer_accept_check(struct sock *sk)
> > struct tcp_sock *tp = tcp_sk(sk);
> >
> > if (tp->defer_tcp_accept.request) {
> > + struct sock *listen_sk = tp->defer_tcp_accept.listen_sk;
>
> Not commenting on the locking, but I think the code
> would be clearer if tp->defer_tcp_accept was used in
> a temporary instead.
Only fixes to net-2.6 now, as agreed. ...So I try to leave syntax cleanup
to net-next.
Besides, there's a trap which makes your patch a lot worse than
mine... ;-) I too once tried a version without temporary until
noticed that trap.
> Perhaps:
>
> net/ipv4/tcp_input.c | 33 ++++++++++++++++-----------------
> 1 files changed, 16 insertions(+), 17 deletions(-)
>
> diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> index b54d9d3..f846e11 100644
> --- a/net/ipv4/tcp_input.c
> +++ b/net/ipv4/tcp_input.c
> @@ -4535,18 +4535,18 @@ static void tcp_urg(struct sock *sk, struct sk_buff *skb, struct tcphdr *th)
> static int tcp_defer_accept_check(struct sock *sk)
> {
> struct tcp_sock *tp = tcp_sk(sk);
> -
> - if (tp->defer_tcp_accept.request) {
> - int queued_data = tp->rcv_nxt - tp->copied_seq;
> - int hasfin = !skb_queue_empty(&sk->sk_receive_queue) ?
> + struct tcp_deferred_accept_info *tdai = &tp->defer_tcp_accept;
> + if (tdai->request) {
> + int queued_data = tp->rcv_nxt - tp->copied_seq;
> + int hasfin = !skb_queue_empty(&sk->sk_receive_queue) ?
> tcp_hdr((struct sk_buff *)
> sk->sk_receive_queue.prev)->fin : 0;
>
> if (queued_data && hasfin)
> queued_data--;
>
> - if (queued_data &&
> - tp->defer_tcp_accept.listen_sk->sk_state == TCP_LISTEN) {
> + bh_lock_sock(tdai->listen_sk);
> + if (queued_data && tdai->listen_sk->sk_state == TCP_LISTEN) {
> if (sock_flag(sk, SOCK_KEEPOPEN)) {
> inet_csk_reset_keepalive_timer(sk,
> keepalive_time_when(tp));
> @@ -4554,23 +4554,22 @@ static int tcp_defer_accept_check(struct sock *sk)
> inet_csk_delete_keepalive_timer(sk);
> }
>
> - inet_csk_reqsk_queue_add(
> - tp->defer_tcp_accept.listen_sk,
> - tp->defer_tcp_accept.request,
> - sk);
> + inet_csk_reqsk_queue_add(tdai->listen_sk,
> + tdai->request,
> + sk);
>
> - tp->defer_tcp_accept.listen_sk->sk_data_ready(
> - tp->defer_tcp_accept.listen_sk, 0);
> + tdai->listen_sk->sk_data_ready(tdai->listen_sk, 0);
>
> - sock_put(tp->defer_tcp_accept.listen_sk);
> + sock_put(tdai->listen_sk);
> sock_put(sk);
> - tp->defer_tcp_accept.listen_sk = NULL;
> - tp->defer_tcp_accept.request = NULL;
> - } else if (hasfin ||
> - tp->defer_tcp_accept.listen_sk->sk_state != TCP_LISTEN) {
> + tdai->listen_sk = NULL;
> + tdai->request = NULL;
uh-oh...
> + } else if (hasfin || tdai->listen_sk->sk_state != TCP_LISTEN) {
> + bh_unlock_sock(tdai->listen_sk);
> tcp_reset(sk);
> return -1;
> }
> + bh_unlock_sock(tdai->listen_sk);
...which will crash in here.
NAK.
> }
> return 0;
> }
>
>
>
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-04 2:54 ` Patrick McManus
@ 2008-06-04 6:42 ` Ilpo Järvinen
0 siblings, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-04 6:42 UTC (permalink / raw)
To: Patrick McManus
Cc: David Miller, mingo, peterz, LKML, Netdev, rjw, Andrew Morton,
johnpol
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1045 bytes --]
On Tue, 3 Jun 2008, Patrick McManus wrote:
> On Wed, 2008-06-04 at 02:22 +0300, Ilpo Järvinen wrote:
> ==
> > --
> > [PATCH] tcp DEFER_ACCEPT: fix racy access to listen_sk
> >
> > It seems that replacement of DA code also moved parts outside
> > of appropriate locking. The Ingo's problem seems to come from
> > the fact that two flows could now race in
> > (inet_csk_)reqsk_queue_add corrupting the queue. ...This can
> > leave dangling socks around which won't resolve themselves
> > without stimuli from outside (e.g., external RST would help
> > I think).
> do_rcv() clearly has the listening socket locked in the non-DA case, and
> in the DA case it is the 'child' ESTABLISHED socket that is locked -
> leaving the accept queue unprotected. So simple.
It also well explains why Ingo finally got the KERNEL: assertion
(!sk->sk_ack_backlog) which is adjusted in the very same reqsk_queue_add
making it and the actual queue come out of sync.
...too bad this has no relevance to Håkon's case, so more digging
is necessary... :-)
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 21:46 ` Ilpo Järvinen
2008-06-03 22:01 ` Ilpo Järvinen
@ 2008-06-04 7:23 ` Ingo Molnar
2008-06-04 18:24 ` David Miller
1 sibling, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-06-04 7:23 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: Peter Zijlstra, LKML, Netdev, David S. Miller, Rafael J. Wysocki,
Andrew Morton, Evgeniy Polyakov, Patrick McManus
* Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> > > i'll queue up your reverts for testing in -tip.
> >
> > update: your 3 reverts in tip/out-of-tree [commit dad98991c] definitely
> > fixed the hangs!
>
> ...It wasn't exactly out-of-tree, Evgeniy fixed a problem that was
> found in "TCP_DEFER_ACCEPT updates - process as established", perhaps
> it just wasn't in your testing tree yet.
out of the -tip tree :-) The -tip tree has 75+ topic branches at the
moment, but TCP topics are not in its scope - so any TCP change is "out
of tree" for the -tip tree.
People got confused in the past when they saw similar test patches show
up in sched.git and x86.git before, so we wanted to make it very clear
in -tip (with is the successor of sched.git, x86.git and a couple of
other git trees) that these are commits we dont want to push anywhere.
Commits in tip/out-of-tree dont get propagated into the tip/auto-*-next
topic branches that linux-next and -mm picks up, they are purely a
courtesy to help the testing/fixing of bugs in subsystems that are
maintained in other git trees.
See attached below the current shortlog of the tip/out-of-tree topic
branch - it contains changes all around the tree for various things that
we triggered in -tip and are not yet upstream or are in flight somewhere
in another git tree.
> > Here is the testing i did:
> >
> > first i ran about 500+ successful iterations on the affected
> > testboxes with your revert patch applied, on multiple systems.
>
> Are you sure this is enough to conclude the results? Seems quite small
> number to me to rule out luck. Especially considering that it was some
> amount of time in the tree already until you noticed it for the first
> time.
a full day of testing on a testsystem with 500 random kernel builds and
bootups (the kernel build done on the testsystem utilizing distcc and
make -j100, so it's rather heavy and parallel TCP traffic per iteration)
with no hang, compared to the same system with your reverts not applied
that hung after an hour with 20-30 iterations.
And that count increased to 1000 successful test iterations since
yesterday.
So i think yes, it seems rather conclusive, given the circumstances ;-)
These random kernel boots found many 'impossible to trigger' bugs and
races in the past. The reason for its race finding capability is the
timing randomness of the resulting random kernel image: the delays
caused by random combination of debugging facilities, build variants,
kernel subsystem variants we have. This -tip qa method - as a
side-effect of its coverage testing - simulates timing variantions that
are otherwise only observable via hardware variations.
I.e. this is not the same kernel booted up a 1000 times - that would be
a very narrow test. This is 1000 _different_ kernels built and booted
up. Each kernel having subtly different timings and ordering. And it's
more than just externally injected random kernel: the test-system itself
builds its "next version" (and uses the network for that as well), so
it's a self-hosting recursive random test in essence.
This method is also amazingly good at finding compiler/linker trouble:
it found 3-4 real gcc bugs so far. (For example i triggered an ancient
bug in gcc 4.0.2 just yesterday. For the record, the testsystem with the
TCP hang utilizes gcc-4.2.2.)
> > so i hereby conclude that your revert works :) I've repeated the
> > commit below that resolves this nasty regression.
>
> ...I couldn't immediately find anything obviously wrong with those
> changes but the patch below might be worth of a try (without the
> revert of course). If it ever spits out that WARN_ON for you, we were
> playing with fire too much and it's better to return on the safe side
> there...
i'll queue it up for testing, but no promises about speedy action here -
the test cycle is really long with this bug.
Ingo
------{ tip/out-of-tree shortlog: }----------->
Alexander van Heukelum (1):
uml: cleanup: use def_bool in Kconfig files
Bjorn Helgaas (1):
PNPACPI: use _CRS IRQ descriptor length for _SRS
Ilpo Järvinen (1):
tcp: revert DEFER_ACCEPT modifications
Ingo Molnar (7):
video/dvb: fix MEDIA_TUNER && FW_LOADER build error
dvb: input layer dependencies fixes
drivers/media/video build fix for modular builds
drivers/watchdog/geodewdt.c: build fix
USB: fix build bug in USB_ISIGHTFW
acpi-acpi_numa_init-build-fix
acpi: fix drivers/acpi/glue.c build error
Michael Krufky (1):
dib7000p: fix dib7000p_attach when !CONFIG_DVB_DIB7000P
Russ Anderson (1):
acpi: fix boot breakage on Altix
Yinghai Lu (2):
net: use numa_node in net_devcice->dev instead of parent
ide: use dev_to_node instead of pcibus_to_node
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-04 0:10 ` Håkon Løvdal
@ 2008-06-04 11:14 ` Ilpo Järvinen
2008-06-04 14:00 ` Håkon Løvdal
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-04 11:14 UTC (permalink / raw)
To: Håkon Løvdal
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1714 bytes --]
On Wed, 4 Jun 2008, Håkon Løvdal wrote:
> Hi. I have now tested and taken various logs before/under/after (ps,
> netstat, tcpdump, strace, /proc/net/tcp). The logs did not grow as big
> as I feared however they are way too big attach here. Any tips on how to
> distribute?
Usually such logs compress quite well. A best thing might be to put
somewhere (if possible) and http:// reference them.
> I mentioned before that the ssh connection died on the new PC, but
> sometimes it just hangs as well, i.e. the ssh connection is alive but
> there is no data transferred.
> The last test I did hang after just 292MB transferred. In this case pv
> just hangs on the new PC, waiting for data that never arrives. On the
> sending side tar which is producing data have died and the corresponding
> ssh is locked in some uninterruptible sleep:
> //netstat before the hang:
> Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name Timer
> tcp 0 504 new_pc:22 old_pc:39464
> ESTABLISHED 17054/sshd: hlovdal on (0.20/0/0)
>
> // after the hang:
> Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name Timer
> tcp 0 2896 old_pc:22 new_pc:39464
> ESTABLISHED - on (21329441.19/0/0)
??? Are these on the same machine? How come the ip-addresses got reversed
(while ports remain the same)...?!?
...Something could be corrupting more than a little, addresses, reference
to the process, timers all seem mixed up here...
> Many things in /proc/17054/ are in a non-normal state (exe is an
> empty symlink, fd is empty, etc)
...
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-04 11:14 ` Ilpo Järvinen
@ 2008-06-04 14:00 ` Håkon Løvdal
2008-06-04 15:09 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Håkon Løvdal @ 2008-06-04 14:00 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
2008/6/4 Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>:
> On Wed, 4 Jun 2008, Håkon Løvdal wrote:
>
>> Hi. I have now tested and taken various logs before/under/after (ps,
>> netstat, tcpdump, strace, /proc/net/tcp). The logs did not grow as big
>> as I feared however they are way too big attach here. Any tips on how to
>> distribute?
>
> Usually such logs compress quite well. A best thing might be to put
> somewhere (if possible) and http:// reference them.
The largest file was a strace log of 1.5GB uncompressed. I will try to
sort out something.
>> //netstat before the hang:
>> Proto Recv-Q Send-Q Local Address Foreign Address
>> State PID/Program name Timer
>> tcp 0 504 new_pc:22 old_pc:39464
>> ESTABLISHED 17054/sshd: hlovdal on (0.20/0/0)
>>
>> // after the hang:
>> Proto Recv-Q Send-Q Local Address Foreign Address
>> State PID/Program name Timer
>> tcp 0 2896 old_pc:22 new_pc:39464
>> ESTABLISHED - on (21329441.19/0/0)
>
> ??? Are these on the same machine? How come the ip-addresses got reversed
> (while ports remain the same)...?!?
Oops, that was an editing mistake on my part, old and new are mixed up
in the last line.
BR Håkon Løvdal
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-04 14:00 ` Håkon Løvdal
@ 2008-06-04 15:09 ` Ilpo Järvinen
2008-06-06 9:32 ` Håkon Løvdal
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-04 15:09 UTC (permalink / raw)
To: Håkon Løvdal
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
[-- Attachment #1: Type: TEXT/PLAIN, Size: 470 bytes --]
On Wed, 4 Jun 2008, Håkon Løvdal wrote:
> 2008/6/4 Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>:
>
> > Usually such logs compress quite well. A best thing might be to put
> > somewhere (if possible) and http:// reference them.
> The largest file was a strace log of 1.5GB uncompressed. I will try to
> sort out something.
I guess not all of the strace will be necessary anyway. You could first
leave that out completely and concentrate on the other logs first.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-04 7:23 ` Ingo Molnar
@ 2008-06-04 18:24 ` David Miller
2008-06-04 20:56 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: David Miller @ 2008-06-04 18:24 UTC (permalink / raw)
To: mingo
Cc: ilpo.jarvinen, peterz, linux-kernel, netdev, rjw, akpm, johnpol,
mcmanus
From: Ingo Molnar <mingo@elte.hu>
Date: Wed, 4 Jun 2008 09:23:11 +0200
> * Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
>
> > ...I couldn't immediately find anything obviously wrong with those
> > changes but the patch below might be worth of a try (without the
> > revert of course). If it ever spits out that WARN_ON for you, we were
> > playing with fire too much and it's better to return on the safe side
> > there...
>
> i'll queue it up for testing, but no promises about speedy action here -
> the test cycle is really long with this bug.
Ilpo posted another patch which fixes a locking bug in the
code, please test with that patch. I include it below so
that you know exactly which one I am referring to.
The quicker you test this, the faster I can merge it to
Linus and get the bug fixed for good.
[PATCH] tcp DEFER_ACCEPT: fix racy access to listen_sk
It seems that replacement of DA code also moved parts outside
of appropriate locking. The Ingo's problem seems to come from
the fact that two flows could now race in
(inet_csk_)reqsk_queue_add corrupting the queue. ...This can
leave dangling socks around which won't resolve themselves
without stimuli from outside (e.g., external RST would help
I think).
Then some details I'm not too sure of:
I guess we want to put listen_sk->sk_state checking under the
lock as well. I've not evaluated if ->sk_data_ready too
requires locking but assumed it does.
I'm by no means familiar with all locking variants, requirements,
etc.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
---
net/ipv4/tcp_input.c | 23 +++++++++++++----------
1 files changed, 13 insertions(+), 10 deletions(-)
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index c9454f0..d21d2b9 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4562,6 +4562,7 @@ static int tcp_defer_accept_check(struct sock *sk)
struct tcp_sock *tp = tcp_sk(sk);
if (tp->defer_tcp_accept.request) {
+ struct sock *listen_sk = tp->defer_tcp_accept.listen_sk;
int queued_data = tp->rcv_nxt - tp->copied_seq;
int hasfin = !skb_queue_empty(&sk->sk_receive_queue) ?
tcp_hdr((struct sk_buff *)
@@ -4570,8 +4571,9 @@ static int tcp_defer_accept_check(struct sock *sk)
if (queued_data && hasfin)
queued_data--;
- if (queued_data &&
- tp->defer_tcp_accept.listen_sk->sk_state == TCP_LISTEN) {
+ bh_lock_sock(listen_sk);
+
+ if (queued_data && listen_sk->sk_state == TCP_LISTEN) {
if (sock_flag(sk, SOCK_KEEPOPEN)) {
inet_csk_reset_keepalive_timer(sk,
keepalive_time_when(tp));
@@ -4579,23 +4581,24 @@ static int tcp_defer_accept_check(struct sock *sk)
inet_csk_delete_keepalive_timer(sk);
}
- inet_csk_reqsk_queue_add(
- tp->defer_tcp_accept.listen_sk,
- tp->defer_tcp_accept.request,
- sk);
+ inet_csk_reqsk_queue_add(listen_sk,
+ tp->defer_tcp_accept.request,
+ sk);
tp->defer_tcp_accept.listen_sk->sk_data_ready(
- tp->defer_tcp_accept.listen_sk, 0);
+ listen_sk, 0);
- sock_put(tp->defer_tcp_accept.listen_sk);
+ sock_put(listen_sk);
sock_put(sk);
tp->defer_tcp_accept.listen_sk = NULL;
tp->defer_tcp_accept.request = NULL;
- } else if (hasfin ||
- tp->defer_tcp_accept.listen_sk->sk_state != TCP_LISTEN) {
+ } else if (hasfin || listen_sk->sk_state != TCP_LISTEN) {
+ bh_unlock_sock(listen_sk);
tcp_reset(sk);
return -1;
}
+
+ bh_unlock_sock(listen_sk);
}
return 0;
}
^ permalink raw reply related [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-04 18:24 ` David Miller
@ 2008-06-04 20:56 ` Ilpo Järvinen
2008-06-04 21:55 ` David Miller
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-04 20:56 UTC (permalink / raw)
To: David Miller
Cc: mingo, peterz, LKML, Netdev, rjw, Andrew Morton, johnpol, mcmanus
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1021 bytes --]
On Wed, 4 Jun 2008, David Miller wrote:
> From: Ingo Molnar <mingo@elte.hu>
> Date: Wed, 4 Jun 2008 09:23:11 +0200
>
> > * Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> >
> > > ...I couldn't immediately find anything obviously wrong with those
> > > changes but the patch below might be worth of a try (without the
> > > revert of course). If it ever spits out that WARN_ON for you, we were
> > > playing with fire too much and it's better to return on the safe side
> > > there...
> >
> > i'll queue it up for testing, but no promises about speedy action here -
> > the test cycle is really long with this bug.
>
> Ilpo posted another patch which fixes a locking bug in the
> code, please test with that patch. I include it below so
> that you know exactly which one I am referring to.
Do you know if TCP ever needs to lock the child sk again (after creating
it and it has started to live on its own) from a context that has already
locked the listen_sk (it could deadlock after this change)?
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-04 20:56 ` Ilpo Järvinen
@ 2008-06-04 21:55 ` David Miller
0 siblings, 0 replies; 96+ messages in thread
From: David Miller @ 2008-06-04 21:55 UTC (permalink / raw)
To: ilpo.jarvinen
Cc: mingo, peterz, linux-kernel, netdev, rjw, akpm, johnpol, mcmanus
From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi>
Date: Wed, 4 Jun 2008 23:56:34 +0300 (EEST)
> Do you know if TCP ever needs to lock the child sk again (after creating
> it and it has started to live on its own) from a context that has already
> locked the listen_sk (it could deadlock after this change)?
I don't know, and since I'm getting back onto an airplane again tomorrow
morning I'm the last person who is likely do be able to do an audit
of these cases for you :-)
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 23:22 ` Ilpo Järvinen
2008-06-03 23:54 ` Joe Perches
2008-06-04 2:54 ` Patrick McManus
@ 2008-06-05 14:22 ` Ingo Molnar
2008-06-05 18:00 ` Ilpo Järvinen
2008-06-10 22:32 ` David Miller
3 siblings, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-06-05 14:22 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: David Miller, mcmanus, peterz, LKML, Netdev, rjw, Andrew Morton,
johnpol
* Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> [PATCH] tcp DEFER_ACCEPT: fix racy access to listen_sk
i unapplied the reverts then applied your fix, and got this hang after
1.5 hours of testing:
titan:~> netstat -nt
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 65032 10.0.1.14:39454 10.0.1.19:3632 ESTABLISHED
10.0.1.14 is this box, Core2Duo dual-core. 10.0.1.19 is another box with
16 CPUs.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-05 14:22 ` Ingo Molnar
@ 2008-06-05 18:00 ` Ilpo Järvinen
2008-06-05 21:13 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-05 18:00 UTC (permalink / raw)
To: Ingo Molnar
Cc: David Miller, mcmanus, peterz, LKML, Netdev, rjw, Andrew Morton,
johnpol
[-- Attachment #1: Type: TEXT/PLAIN, Size: 669 bytes --]
On Thu, 5 Jun 2008, Ingo Molnar wrote:
>
> * Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
>
> > [PATCH] tcp DEFER_ACCEPT: fix racy access to listen_sk
>
> i unapplied the reverts then applied your fix, and got this hang after
> 1.5 hours of testing:
>
> titan:~> netstat -nt
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> tcp 0 65032 10.0.1.14:39454 10.0.1.19:3632 ESTABLISHED
>
> 10.0.1.14 is this box, Core2Duo dual-core. 10.0.1.19 is another box with
> 16 CPUs.
Did 10.0.1.19 too have the fix (the receiving end is the important
one to have this fix)?
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-05 18:00 ` Ilpo Järvinen
@ 2008-06-05 21:13 ` Ilpo Järvinen
2008-06-05 23:29 ` Patrick McManus
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-05 21:13 UTC (permalink / raw)
To: Ingo Molnar
Cc: David Miller, mcmanus, peterz, LKML, Netdev, rjw, Andrew Morton,
johnpol
[-- Attachment #1: Type: TEXT/PLAIN, Size: 11941 bytes --]
On Thu, 5 Jun 2008, Ilpo Järvinen wrote:
> On Thu, 5 Jun 2008, Ingo Molnar wrote:
>
> >
> > * Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> >
> > > [PATCH] tcp DEFER_ACCEPT: fix racy access to listen_sk
> >
> > i unapplied the reverts then applied your fix, and got this hang after
> > 1.5 hours of testing:
> >
> > titan:~> netstat -nt
> > Active Internet connections (w/o servers)
> > Proto Recv-Q Send-Q Local Address Foreign Address State
> > tcp 0 65032 10.0.1.14:39454 10.0.1.19:3632 ESTABLISHED
> >
> > 10.0.1.14 is this box, Core2Duo dual-core. 10.0.1.19 is another box with
> > 16 CPUs.
>
> Did 10.0.1.19 too have the fix (the receiving end is the important
> one to have this fix)?
Since you're probably going to tell that it was, here's a try which just
reverts the most questionable commit out of those three DA related
changes that were included in the first revert you found to be working
(the other two are quite useful fixes).
I'm out of new ideas what could be still wrong (I got confused and lost
track number of times while I tried to verify socket locking today and
probably don't have more time for that now)... Unless somebody else
(Patrick?) comes up with something quickly, I propose you Ingo try if
this smaller revert is enough to solve it and we leave those two fixes
there and postpone the rest. If it isn't enough then things would again
get quite interesting but that's unlikely to happen.
--
i.
--
[PATCH] tcp: revert DEFER_ACCEPT updates - process as established
Reverts "[TCP]: TCP_DEFER_ACCEPT updates - process as established"
(ec3c0982a2dd1e6) and a fix made into it ("tcp: Fix slab corruption
with ipv6 and tcp6fuzz"; 9ae27e0adbf471c7).
Ingo Molnar found out that TCP flows got stuck into ESTABLISHED
state without receiving process. I tried to fix the locking for
listen_sk:
http://marc.info/?l=linux-kernel&m=121253537018216&w=2
...but Ingo's tests still got stuck TCP flow with that fix.
Two other DA related fixes which were among the reverted commits
in the first revert attempt are not reverted here.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
---
include/linux/tcp.h | 7 ------
include/net/request_sock.h | 4 +-
include/net/tcp.h | 1 -
net/ipv4/inet_connection_sock.c | 11 +++++++--
net/ipv4/tcp.c | 18 +++++++++------
net/ipv4/tcp_input.c | 45 ---------------------------------------
net/ipv4/tcp_ipv4.c | 8 -------
net/ipv4/tcp_minisocks.c | 32 ++++++++++-----------------
net/ipv4/tcp_timer.c | 5 ----
9 files changed, 33 insertions(+), 98 deletions(-)
diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index 9881295..07e79bd 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -239,11 +239,6 @@ static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
return (struct tcp_request_sock *)req;
}
-struct tcp_deferred_accept_info {
- struct sock *listen_sk;
- struct request_sock *request;
-};
-
struct tcp_sock {
/* inet_connection_sock has to be the first member of tcp_sock */
struct inet_connection_sock inet_conn;
@@ -379,8 +374,6 @@ struct tcp_sock {
unsigned int keepalive_time; /* time before keep alive takes place */
unsigned int keepalive_intvl; /* time interval between keep alive probes */
- struct tcp_deferred_accept_info defer_tcp_accept;
-
unsigned long last_synq_overflow;
/* Receiver side RTT estimation */
diff --git a/include/net/request_sock.h b/include/net/request_sock.h
index b220b5f..0c96e7b 100644
--- a/include/net/request_sock.h
+++ b/include/net/request_sock.h
@@ -115,8 +115,8 @@ struct request_sock_queue {
struct request_sock *rskq_accept_head;
struct request_sock *rskq_accept_tail;
rwlock_t syn_wait_lock;
- u16 rskq_defer_accept;
- /* 2 bytes hole, try to pack */
+ u8 rskq_defer_accept;
+ /* 3 bytes hole, try to pack */
struct listen_sock *listen_opt;
};
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 633147c..981d5ba 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -139,7 +139,6 @@ extern void tcp_time_wait(struct sock *sk, int state, int timeo);
#define MAX_TCP_KEEPINTVL 32767
#define MAX_TCP_KEEPCNT 127
#define MAX_TCP_SYNCNT 127
-#define MAX_TCP_ACCEPT_DEFERRED 65535
#define TCP_SYNQ_INTERVAL (HZ/5) /* Period of SYNACK timer */
diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index 828ea21..045e799 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -419,7 +419,8 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
struct inet_connection_sock *icsk = inet_csk(parent);
struct request_sock_queue *queue = &icsk->icsk_accept_queue;
struct listen_sock *lopt = queue->listen_opt;
- int thresh = icsk->icsk_syn_retries ? : sysctl_tcp_synack_retries;
+ int max_retries = icsk->icsk_syn_retries ? : sysctl_tcp_synack_retries;
+ int thresh = max_retries;
unsigned long now = jiffies;
struct request_sock **reqp, *req;
int i, budget;
@@ -455,6 +456,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
}
}
+ if (queue->rskq_defer_accept)
+ max_retries = queue->rskq_defer_accept;
+
budget = 2 * (lopt->nr_table_entries / (timeout / interval));
i = lopt->clock_hand;
@@ -462,8 +466,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
reqp=&lopt->syn_table[i];
while ((req = *reqp) != NULL) {
if (time_after_eq(now, req->expires)) {
- if (req->retrans < thresh &&
- !req->rsk_ops->rtx_syn_ack(parent, req)) {
+ if ((req->retrans < (inet_rsk(req)->acked ? max_retries : thresh)) &&
+ (inet_rsk(req)->acked ||
+ !req->rsk_ops->rtx_syn_ack(parent, req))) {
unsigned long timeo;
if (req->retrans++ == 0)
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index ab66683..fc54a48 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -2112,12 +2112,15 @@ static int do_tcp_setsockopt(struct sock *sk, int level,
break;
case TCP_DEFER_ACCEPT:
- if (val < 0) {
- err = -EINVAL;
- } else {
- if (val > MAX_TCP_ACCEPT_DEFERRED)
- val = MAX_TCP_ACCEPT_DEFERRED;
- icsk->icsk_accept_queue.rskq_defer_accept = val;
+ icsk->icsk_accept_queue.rskq_defer_accept = 0;
+ if (val > 0) {
+ /* Translate value in seconds to number of
+ * retransmits */
+ while (icsk->icsk_accept_queue.rskq_defer_accept < 32 &&
+ val > ((TCP_TIMEOUT_INIT / HZ) <<
+ icsk->icsk_accept_queue.rskq_defer_accept))
+ icsk->icsk_accept_queue.rskq_defer_accept++;
+ icsk->icsk_accept_queue.rskq_defer_accept++;
}
break;
@@ -2299,7 +2302,8 @@ static int do_tcp_getsockopt(struct sock *sk, int level,
val = (val ? : sysctl_tcp_fin_timeout) / HZ;
break;
case TCP_DEFER_ACCEPT:
- val = icsk->icsk_accept_queue.rskq_defer_accept;
+ val = !icsk->icsk_accept_queue.rskq_defer_accept ? 0 :
+ ((TCP_TIMEOUT_INIT / HZ) << (icsk->icsk_accept_queue.rskq_defer_accept - 1));
break;
case TCP_WINDOW_CLAMP:
val = tp->window_clamp;
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 1322fa1..fb1ab3f 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4557,49 +4557,6 @@ static void tcp_urg(struct sock *sk, struct sk_buff *skb, struct tcphdr *th)
}
}
-static int tcp_defer_accept_check(struct sock *sk)
-{
- struct tcp_sock *tp = tcp_sk(sk);
-
- if (tp->defer_tcp_accept.request) {
- int queued_data = tp->rcv_nxt - tp->copied_seq;
- int hasfin = !skb_queue_empty(&sk->sk_receive_queue) ?
- tcp_hdr((struct sk_buff *)
- sk->sk_receive_queue.prev)->fin : 0;
-
- if (queued_data && hasfin)
- queued_data--;
-
- if (queued_data &&
- tp->defer_tcp_accept.listen_sk->sk_state == TCP_LISTEN) {
- if (sock_flag(sk, SOCK_KEEPOPEN)) {
- inet_csk_reset_keepalive_timer(sk,
- keepalive_time_when(tp));
- } else {
- inet_csk_delete_keepalive_timer(sk);
- }
-
- inet_csk_reqsk_queue_add(
- tp->defer_tcp_accept.listen_sk,
- tp->defer_tcp_accept.request,
- sk);
-
- tp->defer_tcp_accept.listen_sk->sk_data_ready(
- tp->defer_tcp_accept.listen_sk, 0);
-
- sock_put(tp->defer_tcp_accept.listen_sk);
- sock_put(sk);
- tp->defer_tcp_accept.listen_sk = NULL;
- tp->defer_tcp_accept.request = NULL;
- } else if (hasfin ||
- tp->defer_tcp_accept.listen_sk->sk_state != TCP_LISTEN) {
- tcp_reset(sk);
- return -1;
- }
- }
- return 0;
-}
-
static int tcp_copy_to_iovec(struct sock *sk, struct sk_buff *skb, int hlen)
{
struct tcp_sock *tp = tcp_sk(sk);
@@ -4974,8 +4931,6 @@ step5:
tcp_data_snd_check(sk);
tcp_ack_snd_check(sk);
-
- tcp_defer_accept_check(sk);
return 0;
csum_error:
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index cd601a8..b698b5b 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1918,14 +1918,6 @@ int tcp_v4_destroy_sock(struct sock *sk)
sk->sk_sndmsg_page = NULL;
}
- if (tp->defer_tcp_accept.request) {
- reqsk_free(tp->defer_tcp_accept.request);
- sock_put(tp->defer_tcp_accept.listen_sk);
- sock_put(sk);
- tp->defer_tcp_accept.listen_sk = NULL;
- tp->defer_tcp_accept.request = NULL;
- }
-
atomic_dec(&tcp_sockets_allocated);
return 0;
diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c
index 019c8c1..8245247 100644
--- a/net/ipv4/tcp_minisocks.c
+++ b/net/ipv4/tcp_minisocks.c
@@ -571,8 +571,10 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
does sequence test, SYN is truncated, and thus we consider
it a bare ACK.
- Both ends (listening sockets) accept the new incoming
- connection and try to talk to each other. 8-)
+ If icsk->icsk_accept_queue.rskq_defer_accept, we silently drop this
+ bare ACK. Otherwise, we create an established connection. Both
+ ends (listening sockets) accept the new incoming connection and try
+ to talk to each other. 8-)
Note: This case is both harmless, and rare. Possibility is about the
same as us discovering intelligent life on another plant tomorrow.
@@ -640,6 +642,13 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
if (!(flg & TCP_FLAG_ACK))
return NULL;
+ /* If TCP_DEFER_ACCEPT is set, drop bare ACK. */
+ if (inet_csk(sk)->icsk_accept_queue.rskq_defer_accept &&
+ TCP_SKB_CB(skb)->end_seq == tcp_rsk(req)->rcv_isn + 1) {
+ inet_rsk(req)->acked = 1;
+ return NULL;
+ }
+
/* OK, ACK is valid, create big socket and
* feed this segment to it. It will repeat all
* the tests. THIS SEGMENT MUST MOVE SOCKET TO
@@ -678,24 +687,7 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
inet_csk_reqsk_queue_unlink(sk, req, prev);
inet_csk_reqsk_queue_removed(sk, req);
- if (inet_csk(sk)->icsk_accept_queue.rskq_defer_accept &&
- TCP_SKB_CB(skb)->end_seq == tcp_rsk(req)->rcv_isn + 1) {
-
- /* the accept queue handling is done is est recv slow
- * path so lets make sure to start there
- */
- tcp_sk(child)->pred_flags = 0;
- sock_hold(sk);
- sock_hold(child);
- tcp_sk(child)->defer_tcp_accept.listen_sk = sk;
- tcp_sk(child)->defer_tcp_accept.request = req;
-
- inet_csk_reset_keepalive_timer(child,
- inet_csk(sk)->icsk_accept_queue.rskq_defer_accept * HZ);
- } else {
- inet_csk_reqsk_queue_add(sk, req, child);
- }
-
+ inet_csk_reqsk_queue_add(sk, req, child);
return child;
listen_overflow:
diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
index 4de68cf..63ed9d6 100644
--- a/net/ipv4/tcp_timer.c
+++ b/net/ipv4/tcp_timer.c
@@ -489,11 +489,6 @@ static void tcp_keepalive_timer (unsigned long data)
goto death;
}
- if (tp->defer_tcp_accept.request && sk->sk_state == TCP_ESTABLISHED) {
- tcp_send_active_reset(sk, GFP_ATOMIC);
- goto death;
- }
-
if (!sock_flag(sk, SOCK_KEEPOPEN) || sk->sk_state == TCP_CLOSE)
goto out;
--
1.5.2.2
^ permalink raw reply related [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-05 21:13 ` Ilpo Järvinen
@ 2008-06-05 23:29 ` Patrick McManus
2008-06-06 10:03 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Patrick McManus @ 2008-06-05 23:29 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: Ingo Molnar, David Miller, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
On Fri, 2008-06-06 at 00:13 +0300, Ilpo Järvinen wrote:
>
> I'm out of new ideas what could be still wrong (I got confused and
> lost
> track number of times while I tried to verify socket locking today and
> probably don't have more time for that now)... Unless somebody else
> (Patrick?) comes up with something quickly,
Sorry, I don't see anything - it seems to boil down to the same code in
the DA and non-DA case as far as I can tell, but after a while all the
twisty passages seem to look alike.
If Ingo confirms that the recv end was running the locking patch code,
it would be interesting to just confirm the sysreq+t looks the same as
before - it is possible the patch turned the race into a non-obvious
deadlock.
I'm sure your smaller revert will make the problem go away just as the
larger one did, fwiw.
The other odd thing is that Ingo did a lot of experimentation and was
only making this happen on localhost before (though I agree there is
nothing inherent about that lock and localhost) - isn't it odd that the
first trigger of it now is between two hosts? What do you make of that?
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-04 15:09 ` Ilpo Järvinen
@ 2008-06-06 9:32 ` Håkon Løvdal
2008-06-09 19:24 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Håkon Løvdal @ 2008-06-06 9:32 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
On 04/06/2008, Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> I guess not all of the strace will be necessary anyway. You could first
> leave that out completely and concentrate on the other logs first.
I have now set up a server on my machine and made all logs available.
I will follow up with the adress in private email to those that already
have posted in this thread. For others, please ask to receive it.
BR Håkon Løvdal
PS
I will be away for this weekend.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-05 23:29 ` Patrick McManus
@ 2008-06-06 10:03 ` Ilpo Järvinen
2008-06-06 17:11 ` Patrick McManus
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-06 10:03 UTC (permalink / raw)
To: Patrick McManus
Cc: Ingo Molnar, David Miller, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1779 bytes --]
On Thu, 5 Jun 2008, Patrick McManus wrote:
> On Fri, 2008-06-06 at 00:13 +0300, Ilpo Järvinen wrote:
>
> >
> > I'm out of new ideas what could be still wrong (I got confused and
> > lost
> > track number of times while I tried to verify socket locking today and
> > probably don't have more time for that now)... Unless somebody else
> > (Patrick?) comes up with something quickly,
>
> Sorry, I don't see anything - it seems to boil down to the same code in
> the DA and non-DA case as far as I can tell, but after a while all the
> twisty passages seem to look alike.
:-)
This Ingo's testcase should anyway be quite "simple", I mean that distcc
shouldn't do anything unexpected in a sense it shouldn't abort the flows
by not sending data, close the listening socket or other things like that.
> If Ingo confirms that the recv end was running the locking patch code,
> it would be interesting to just confirm the sysreq+t looks the same as
> before - it is possible the patch turned the race into a non-obvious
> deadlock.
...Yes, but we want that from the receiver's host rather than from the
sender end. Also checking that sender is still doing window probes once
per 2min is probably worth of it though a change in that is quite
unlikely.
> I'm sure your smaller revert will make the problem go away just as the
> larger one did, fwiw.
I'd very much except it to.
> The other odd thing is that Ingo did a lot of experimentation and was
> only making this happen on localhost before (though I agree there is
> nothing inherent about that lock and localhost) - isn't it odd that the
> first trigger of it now is between two hosts? What do you make of that?
...No, it occurred couple of times in the past between host as well,
nothing new in that.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 10:03 ` Ilpo Järvinen
@ 2008-06-06 17:11 ` Patrick McManus
2008-06-06 17:33 ` Ingo Molnar
2008-06-06 18:25 ` Ilpo Järvinen
0 siblings, 2 replies; 96+ messages in thread
From: Patrick McManus @ 2008-06-06 17:11 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: Ingo Molnar, David Miller, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
> This Ingo's testcase should anyway be quite "simple", I mean that distcc
> shouldn't do anything unexpected in a sense it shouldn't abort the flows
> by not sending data, close the listening socket or other things like that.
>
maybe - I've noted that I can get the distcc server to crash with just a
little fuzz (telnet to it and close the telnet) - but it is true I
haven't seen anything odd using the distcc client.
Anyhow, my news is that using rc5 I have managed to reproduce it on
localhost - so it isn't just ingo anymore ! ;)
I didn't have a 16 cpu machine at my disposal, so I was concerned I
wouldn't be able to make it happen - but I setup a 64 bit kvm image with
-smp 4, running on my core2-duo and created a makefile with 3 src files
that I just compiled as "while true; do make -j3 all; done" - the
makefile uses distcc to localhost and has intentionally broken
dependencies so it just keeps recompiling stuff. The input files are
approximately 135k, 98k, and 16k after running gcc -E on them (which I
what I assume distcc does before putting them down the socket).
On rc5 I could get the lockup in under 20 minutes.. usually 10. I think
I did it 4 times. My compile test is probably a better trigger than the
kernel compile because the distcc connects are never staggered like they
would be in a large directory of files. (3 files, -j4).
When I apply the locking patch you (Ilpo) wrote, I cannot reproduce the
error at all in the first 90 minutes of testing. I'll let the test run
and update the list.
I'm holding out hope that Ingo's report did not have the locking patch
on the distcc server end - because it certainly makes a difference for
me.
-Patrick
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 17:11 ` Patrick McManus
@ 2008-06-06 17:33 ` Ingo Molnar
2008-06-06 18:19 ` Ilpo Järvinen
2008-06-06 18:25 ` Ilpo Järvinen
1 sibling, 1 reply; 96+ messages in thread
From: Ingo Molnar @ 2008-06-06 17:33 UTC (permalink / raw)
To: Patrick McManus
Cc: Ilpo Järvinen, David Miller, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
* Patrick McManus <mcmanus@ducksong.com> wrote:
> When I apply the locking patch you (Ilpo) wrote, I cannot reproduce
> the error at all in the first 90 minutes of testing. I'll let the test
> run and update the list.
>
> I'm holding out hope that Ingo's report did not have the locking patch
> on the distcc server end - because it certainly makes a difference for
> me.
Hm, the distcc server had the full 3-patch-revert from Ilpo, was that
supposed to fix the problem too, indirectly?
The box is running that 3-patch revert right now as well:
phoenix:~> uptime
19:20:28 up 9:58, 2 users, load average: 7.75, 13.88, 30.95
phoenix:~> uname -a
Linux phoenix 2.6.26-rc4 #2352 SMP Fri Jun 6 09:18:07 CEST 2008 x86_64 x86_64 x86_64 GNU/Linux
... and i never saw a single hang today in the 10 hours of uptime this
box has. (and it built a good 500 kernel today) Nor any hang yesterday,
and that was a good 500 kernels too.
You can see it that the box built more than two thousand kernels in the
past few days alone, so it's a rather busy little bee. The other
testboxes built even more kernels - a quad box built and booted 2500
kernels:
#define UTS_VERSION "#2524 SMP PREEMPT Fri Jun 6 19:22:21 CEST 2008"
and i never saw a hang on that box either.
a third box has:
titan:~> uname -a
Linux titan 2.6.26-rc5-00002-g737697d-dirty #2557 SMP PREEMPT Fri Jun 6
19:24:00 CEST 2008 x86_64 x86_64 x86_64 GNU/Linux
(this is the one that showed the hang for the first time)
The total count of kernel bootups i did this week for -tip QA was
somewhere between five and ten thousand random build+bootups - and the
only time i got a hang was when i removed the 3-patch-revert
intentionally, on one of the boxes.
Maybe that 3-patch-revert just makes this locking bug a bit less likely
to trigger, by accident? Out tip test-setup is specialized to find
arch/x86 and scheduler bugs, not primarily to find networking bugs. (but
at this test volume, and given that it makes use of distcc, it will
trigger them too.)
i have a rather accurate timeline of when the hang first occured, do we
know the timeline of the introduction of the locking bug by any chance?
Which commit introduced it? (Ilpo's commit log does not say it)
Your test results are compelling nevertheless so i'll do a retest in any
case, with all boxes either running an older kernel or a kernel with the
locking fix.
Ingo
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 17:33 ` Ingo Molnar
@ 2008-06-06 18:19 ` Ilpo Järvinen
2008-06-06 18:39 ` Ingo Molnar
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-06 18:19 UTC (permalink / raw)
To: Ingo Molnar
Cc: Patrick McManus, David Miller, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
...I kind of fail to follow in general in this mail which patch have been
tested and where and when... But I understand that it's just due to number
of tests & hosts & kernels & what-else you use and know by heart (and we
don't do all that well :-)). But I'll try to still sort it out below...
On Fri, 6 Jun 2008, Ingo Molnar wrote:
>
> * Patrick McManus <mcmanus@ducksong.com> wrote:
>
> > When I apply the locking patch you (Ilpo) wrote, I cannot reproduce
> > the error at all in the first 90 minutes of testing. I'll let the test
> > run and update the list.
> >
> > I'm holding out hope that Ingo's report did not have the locking patch
> > on the distcc server end - because it certainly makes a difference for
> > me.
>
> Hm, the distcc server had the full 3-patch-revert from Ilpo, was that
> supposed to fix the problem too, indirectly?
Yes, the problematic outside of locking portion shouldn't be there
without those DA changes.
> The box is running that 3-patch revert right now as well:
>
> phoenix:~> uptime
> 19:20:28 up 9:58, 2 users, load average: 7.75, 13.88, 30.95
> phoenix:~> uname -a
> Linux phoenix 2.6.26-rc4 #2352 SMP Fri Jun 6 09:18:07 CEST 2008 x86_64 x86_64 x86_64 GNU/Linux
>
> ... and i never saw a single hang today in the 10 hours of uptime this
> box has. (and it built a good 500 kernel today) Nor any hang yesterday,
> and that was a good 500 kernels too.
>
> You can see it that the box built more than two thousand kernels in the
> past few days alone, so it's a rather busy little bee. The other
> testboxes built even more kernels - a quad box built and booted 2500
> kernels:
>
> #define UTS_VERSION "#2524 SMP PREEMPT Fri Jun 6 19:22:21 CEST 2008"
>
> and i never saw a hang on that box either.
>
> a third box has:
>
> titan:~> uname -a
> Linux titan 2.6.26-rc5-00002-g737697d-dirty #2557 SMP PREEMPT Fri Jun 6
> 19:24:00 CEST 2008 x86_64 x86_64 x86_64 GNU/Linux
>
> (this is the one that showed the hang for the first time)
>
> The total count of kernel bootups i did this week for -tip QA was
> somewhere between five and ten thousand random build+bootups - and the
> only time i got a hang was when i removed the 3-patch-revert
> intentionally, on one of the boxes.
...and you added the locking fix there instead? Or was this a removal?
> Maybe that 3-patch-revert just makes this locking bug a bit less likely
> to trigger, by accident?
No, part of the DEFER_ACCEPT stuff was postponed in 2.6.25..2.6.26-rc1
timeframe (ec3c0982a2dd1e671bad8e9d26c28dcba0039d87) so that one portion
of it ended up being added outside of the socket lock of the listening
socket, while touching its datastructures. Without
ec3c0982a2dd1e671bad8e9d26c28dcba0039d87 the deferred accept related
things happen earlier, ie., while we still are under the lock of the
listening socket. So that particular locking bug was _introduced_ by that
ec3c change, not made more likely or so.
...Of course software is known to have bugs, so we might always be
(un?)lucky and hit another one and confuse... :-)
> Out tip test-setup is specialized to find
> arch/x86 and scheduler bugs, not primarily to find networking bugs. (but
> at this test volume, and given that it makes use of distcc, it will
> trigger them too.)
It seems to work quite well actually for this kind of networking related
bugs too which hardly depend on network at all :-).
> i have a rather accurate timeline of when the hang first occured, do we
> know the timeline of the introduction of the locking bug by any chance?
> Which commit introduced it? (Ilpo's commit log does not say it)
Ah, sorry I forgot to add that one there, it was sent quite late in the
night and I just couldn't get sleep until sending the fix... :-) It was
one of the reverted ones that did it:
ec3c0982a2dd1e671bad8e9d26c28dcba0039d87.
> Your test results are compelling nevertheless so i'll do a retest in any
> case, with all boxes either running an older kernel or a kernel with the
> locking fix.
If you want an older kernel, you would have to go basically to 2.6.25 or
so.
To summarize. Both 3changes+1fix revert (you refer to it only as 3-patch
revert) _and_ the locking fix I made should fix the problem (obviously
they exclude each other). ...And end which is significant is the one which
has LISTENing sockets (please keep this in mind if you still get the hang
and provide some info).
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 17:11 ` Patrick McManus
2008-06-06 17:33 ` Ingo Molnar
@ 2008-06-06 18:25 ` Ilpo Järvinen
1 sibling, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-06 18:25 UTC (permalink / raw)
To: Patrick McManus
Cc: Ingo Molnar, David Miller, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
On Fri, 6 Jun 2008, Patrick McManus wrote:
> > This Ingo's testcase should anyway be quite "simple", I mean that distcc
> > shouldn't do anything unexpected in a sense it shouldn't abort the flows
> > by not sending data, close the listening socket or other things like that.
>
> maybe - I've noted that I can get the distcc server to crash with just a
> little fuzz (telnet to it and close the telnet) - but it is true I
> haven't seen anything odd using the distcc client.
In addition I think I've also seen some bits floating around that
occassionally distcc does something weird in a correct setup too.
I briefly looked how distcc behaved while doing the stress_accept. Distcc
basically seems to have n processes each accept()ing and some kind of
memleak killer by limiting number of successive accepts then exit, while
the parent who did the listen is only periodically (had some sleep(1)s)
collecting dead ones & respawning them.
> Anyhow, my news is that using rc5 I have managed to reproduce it on
> localhost - so it isn't just ingo anymore ! ;)
Also Peter Z has reported it earlier, it was distcc+localhost for him as
well.
> and has intentionally broken dependencies so it just keeps recompiling
> stuff.
...Trying to invent perpetual motion machine? :-/
> The input files are
> approximately 135k, 98k, and 16k after running gcc -E on them (which I
> what I assume distcc does before putting them down the socket).
>
> On rc5 I could get the lockup in under 20 minutes.. usually 10. I think
> I did it 4 times. My compile test is probably a better trigger than the
> kernel compile because the distcc connects are never staggered like they
> would be in a large directory of files. (3 files, -j4).
It could be even easier if you make next in path gcc to play with
nice, trying a number of different values might reveal some really fast
to reproduce scenario.
> When I apply the locking patch you (Ilpo) wrote, I cannot reproduce the
> error at all in the first 90 minutes of testing. I'll let the test run
> and update the list.
At least it helps some :-), like it should.
> I'm holding out hope that Ingo's report did not have the locking patch
> on the distcc server end - because it certainly makes a difference for
> me.
...He had some issue with different versions being deployed at least in
the past, and I failed to follow his latest answer :-).
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 18:19 ` Ilpo Järvinen
@ 2008-06-06 18:39 ` Ingo Molnar
2008-06-06 19:49 ` Ilpo Järvinen
2008-06-06 20:08 ` Patrick McManus
0 siblings, 2 replies; 96+ messages in thread
From: Ingo Molnar @ 2008-06-06 18:39 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: Patrick McManus, David Miller, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
* Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> If you want an older kernel, you would have to go basically to 2.6.25
> or so.
correct, that's what i use as fallback, some distro kernel which is
2.6.25 or older.
but i'm confused a bit, you say v2.6.25-rc6-475-gec3c098 introduced the
locking problem - so 2.6.25 is affected as well? This is a significant
question because the fallback kernel is kernel-2.6.25.3-18.fc9.x86_64 on
the 16-way box. (all other build-boxes have 2.6.24 or older as a
fallback kernel)
> To summarize. Both 3changes+1fix revert (you refer to it only as
> 3-patch revert) _and_ the locking fix I made should fix the problem
> (obviously they exclude each other). ...And end which is significant
> is the one which has LISTENing sockets (please keep this in mind if
> you still get the hang and provide some info).
ok.
For completeness, let me repeat the patch i referred to as the
'3-patch-revert' below. (which indeed is 3+1 as you note)
this is the patch that appears to be working empirically. (Disclaimer:
it might just hide the problem, change timings, have a lucky code
layout, etc.)
Ingo
----------------->
commit 9e5b6ca8fa8e07aceaddbfb0f2a324449e5ef014
Author: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Date: Sat May 31 15:18:56 2008 +0300
tcp: revert DEFER_ACCEPT modifications
On Sat, 31 May 2008, Ilpo Järvinen wrote:
> On Sat, 31 May 2008, Ingo Molnar wrote:
>
> > ok, once i added back the localhost distcc component and the hung kernel
> > build + stuck TCP socket bug happened again overnight:
> >
> > Active Internet connections (w/o servers)
> > Proto Recv-Q Send-Q Local Address Foreign Address State
> > tcp 72187 0 10.0.1.14:3632 10.0.1.14:47910 ESTABLISHED
> > tcp 0 174464 10.0.1.14:47910 10.0.1.14:3632 ESTABLISHED
> >
> > so it seems distcc over localhost was the aspect that made it fail.
> >
> > _Perhaps_ what matters is to have the new post-rc3 TCP code on _both_
> > sides of the connection. But that is just a theory - it could be timing,
> > etc.
>
> Btw, does your distcc perhaps happen enable TCP_DEFER_ACCEPT (there were
> some post 2.6.25 changes into it)?
Hmm, if so, please try this revert below (I hope I got everything that is
related there, gitk is currently broken for me so that viewing many
commits quickly is a pain, so I might have accidently missed something)...
Reverts these commits:
[TCP]: TCP_DEFER_ACCEPT updates - defer timeout conflicts with max_t
[TCP]: TCP_DEFER_ACCEPT updates - dont retxmt synack
[TCP]: TCP_DEFER_ACCEPT updates - process as established
tcp: Fix slab corruption with ipv6 and tcp6fuzz
..., ie.,
539fae89bebd16ebeafd57a87169bc56eb530d76,
e4c78840284f3f51b1896cf3936d60a6033c4d2c,
ec3c0982a2dd1e671bad8e9d26c28dcba0039d87 and
9ae27e0adbf471c7a6b80102e38e1d5a346b3b38.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index 18e62e3..b31b6b7 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -239,11 +239,6 @@ static inline struct tcp_request_sock *tcp_rsk(const struct request_sock *req)
return (struct tcp_request_sock *)req;
}
-struct tcp_deferred_accept_info {
- struct sock *listen_sk;
- struct request_sock *request;
-};
-
struct tcp_sock {
/* inet_connection_sock has to be the first member of tcp_sock */
struct inet_connection_sock inet_conn;
@@ -379,8 +374,6 @@ struct tcp_sock {
unsigned int keepalive_intvl; /* time interval between keep alive probes */
int linger2;
- struct tcp_deferred_accept_info defer_tcp_accept;
-
unsigned long last_synq_overflow;
u32 tso_deferred;
diff --git a/include/net/request_sock.h b/include/net/request_sock.h
index b220b5f..0c96e7b 100644
--- a/include/net/request_sock.h
+++ b/include/net/request_sock.h
@@ -115,8 +115,8 @@ struct request_sock_queue {
struct request_sock *rskq_accept_head;
struct request_sock *rskq_accept_tail;
rwlock_t syn_wait_lock;
- u16 rskq_defer_accept;
- /* 2 bytes hole, try to pack */
+ u8 rskq_defer_accept;
+ /* 3 bytes hole, try to pack */
struct listen_sock *listen_opt;
};
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 633147c..981d5ba 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -139,7 +139,6 @@ extern void tcp_time_wait(struct sock *sk, int state, int timeo);
#define MAX_TCP_KEEPINTVL 32767
#define MAX_TCP_KEEPCNT 127
#define MAX_TCP_SYNCNT 127
-#define MAX_TCP_ACCEPT_DEFERRED 65535
#define TCP_SYNQ_INTERVAL (HZ/5) /* Period of SYNACK timer */
diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index 828ea21..ec83448 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -419,7 +419,8 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
struct inet_connection_sock *icsk = inet_csk(parent);
struct request_sock_queue *queue = &icsk->icsk_accept_queue;
struct listen_sock *lopt = queue->listen_opt;
- int thresh = icsk->icsk_syn_retries ? : sysctl_tcp_synack_retries;
+ int max_retries = icsk->icsk_syn_retries ? : sysctl_tcp_synack_retries;
+ int thresh = max_retries;
unsigned long now = jiffies;
struct request_sock **reqp, *req;
int i, budget;
@@ -455,6 +456,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
}
}
+ if (queue->rskq_defer_accept)
+ max_retries = queue->rskq_defer_accept;
+
budget = 2 * (lopt->nr_table_entries / (timeout / interval));
i = lopt->clock_hand;
@@ -462,8 +466,9 @@ void inet_csk_reqsk_queue_prune(struct sock *parent,
reqp=&lopt->syn_table[i];
while ((req = *reqp) != NULL) {
if (time_after_eq(now, req->expires)) {
- if (req->retrans < thresh &&
- !req->rsk_ops->rtx_syn_ack(parent, req)) {
+ if ((req->retrans < thresh ||
+ (inet_rsk(req)->acked && req->retrans < max_retries))
+ && !req->rsk_ops->rtx_syn_ack(parent, req)) {
unsigned long timeo;
if (req->retrans++ == 0)
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index ab66683..fc54a48 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -2112,12 +2112,15 @@ static int do_tcp_setsockopt(struct sock *sk, int level,
break;
case TCP_DEFER_ACCEPT:
- if (val < 0) {
- err = -EINVAL;
- } else {
- if (val > MAX_TCP_ACCEPT_DEFERRED)
- val = MAX_TCP_ACCEPT_DEFERRED;
- icsk->icsk_accept_queue.rskq_defer_accept = val;
+ icsk->icsk_accept_queue.rskq_defer_accept = 0;
+ if (val > 0) {
+ /* Translate value in seconds to number of
+ * retransmits */
+ while (icsk->icsk_accept_queue.rskq_defer_accept < 32 &&
+ val > ((TCP_TIMEOUT_INIT / HZ) <<
+ icsk->icsk_accept_queue.rskq_defer_accept))
+ icsk->icsk_accept_queue.rskq_defer_accept++;
+ icsk->icsk_accept_queue.rskq_defer_accept++;
}
break;
@@ -2299,7 +2302,8 @@ static int do_tcp_getsockopt(struct sock *sk, int level,
val = (val ? : sysctl_tcp_fin_timeout) / HZ;
break;
case TCP_DEFER_ACCEPT:
- val = icsk->icsk_accept_queue.rskq_defer_accept;
+ val = !icsk->icsk_accept_queue.rskq_defer_accept ? 0 :
+ ((TCP_TIMEOUT_INIT / HZ) << (icsk->icsk_accept_queue.rskq_defer_accept - 1));
break;
case TCP_WINDOW_CLAMP:
val = tp->window_clamp;
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index eba873e..cad73b7 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -4541,49 +4541,6 @@ static void tcp_urg(struct sock *sk, struct sk_buff *skb, struct tcphdr *th)
}
}
-static int tcp_defer_accept_check(struct sock *sk)
-{
- struct tcp_sock *tp = tcp_sk(sk);
-
- if (tp->defer_tcp_accept.request) {
- int queued_data = tp->rcv_nxt - tp->copied_seq;
- int hasfin = !skb_queue_empty(&sk->sk_receive_queue) ?
- tcp_hdr((struct sk_buff *)
- sk->sk_receive_queue.prev)->fin : 0;
-
- if (queued_data && hasfin)
- queued_data--;
-
- if (queued_data &&
- tp->defer_tcp_accept.listen_sk->sk_state == TCP_LISTEN) {
- if (sock_flag(sk, SOCK_KEEPOPEN)) {
- inet_csk_reset_keepalive_timer(sk,
- keepalive_time_when(tp));
- } else {
- inet_csk_delete_keepalive_timer(sk);
- }
-
- inet_csk_reqsk_queue_add(
- tp->defer_tcp_accept.listen_sk,
- tp->defer_tcp_accept.request,
- sk);
-
- tp->defer_tcp_accept.listen_sk->sk_data_ready(
- tp->defer_tcp_accept.listen_sk, 0);
-
- sock_put(tp->defer_tcp_accept.listen_sk);
- sock_put(sk);
- tp->defer_tcp_accept.listen_sk = NULL;
- tp->defer_tcp_accept.request = NULL;
- } else if (hasfin ||
- tp->defer_tcp_accept.listen_sk->sk_state != TCP_LISTEN) {
- tcp_reset(sk);
- return -1;
- }
- }
- return 0;
-}
-
static int tcp_copy_to_iovec(struct sock *sk, struct sk_buff *skb, int hlen)
{
struct tcp_sock *tp = tcp_sk(sk);
@@ -4944,8 +4901,6 @@ step5:
tcp_data_snd_check(sk);
tcp_ack_snd_check(sk);
-
- tcp_defer_accept_check(sk);
return 0;
csum_error:
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index cd601a8..b698b5b 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1918,14 +1918,6 @@ int tcp_v4_destroy_sock(struct sock *sk)
sk->sk_sndmsg_page = NULL;
}
- if (tp->defer_tcp_accept.request) {
- reqsk_free(tp->defer_tcp_accept.request);
- sock_put(tp->defer_tcp_accept.listen_sk);
- sock_put(sk);
- tp->defer_tcp_accept.listen_sk = NULL;
- tp->defer_tcp_accept.request = NULL;
- }
-
atomic_dec(&tcp_sockets_allocated);
return 0;
diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c
index 019c8c1..8245247 100644
--- a/net/ipv4/tcp_minisocks.c
+++ b/net/ipv4/tcp_minisocks.c
@@ -571,8 +571,10 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
does sequence test, SYN is truncated, and thus we consider
it a bare ACK.
- Both ends (listening sockets) accept the new incoming
- connection and try to talk to each other. 8-)
+ If icsk->icsk_accept_queue.rskq_defer_accept, we silently drop this
+ bare ACK. Otherwise, we create an established connection. Both
+ ends (listening sockets) accept the new incoming connection and try
+ to talk to each other. 8-)
Note: This case is both harmless, and rare. Possibility is about the
same as us discovering intelligent life on another plant tomorrow.
@@ -640,6 +642,13 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
if (!(flg & TCP_FLAG_ACK))
return NULL;
+ /* If TCP_DEFER_ACCEPT is set, drop bare ACK. */
+ if (inet_csk(sk)->icsk_accept_queue.rskq_defer_accept &&
+ TCP_SKB_CB(skb)->end_seq == tcp_rsk(req)->rcv_isn + 1) {
+ inet_rsk(req)->acked = 1;
+ return NULL;
+ }
+
/* OK, ACK is valid, create big socket and
* feed this segment to it. It will repeat all
* the tests. THIS SEGMENT MUST MOVE SOCKET TO
@@ -678,24 +687,7 @@ struct sock *tcp_check_req(struct sock *sk,struct sk_buff *skb,
inet_csk_reqsk_queue_unlink(sk, req, prev);
inet_csk_reqsk_queue_removed(sk, req);
- if (inet_csk(sk)->icsk_accept_queue.rskq_defer_accept &&
- TCP_SKB_CB(skb)->end_seq == tcp_rsk(req)->rcv_isn + 1) {
-
- /* the accept queue handling is done is est recv slow
- * path so lets make sure to start there
- */
- tcp_sk(child)->pred_flags = 0;
- sock_hold(sk);
- sock_hold(child);
- tcp_sk(child)->defer_tcp_accept.listen_sk = sk;
- tcp_sk(child)->defer_tcp_accept.request = req;
-
- inet_csk_reset_keepalive_timer(child,
- inet_csk(sk)->icsk_accept_queue.rskq_defer_accept * HZ);
- } else {
- inet_csk_reqsk_queue_add(sk, req, child);
- }
-
+ inet_csk_reqsk_queue_add(sk, req, child);
return child;
listen_overflow:
diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
index 4de68cf..63ed9d6 100644
--- a/net/ipv4/tcp_timer.c
+++ b/net/ipv4/tcp_timer.c
@@ -489,11 +489,6 @@ static void tcp_keepalive_timer (unsigned long data)
goto death;
}
- if (tp->defer_tcp_accept.request && sk->sk_state == TCP_ESTABLISHED) {
- tcp_send_active_reset(sk, GFP_ATOMIC);
- goto death;
- }
-
if (!sock_flag(sk, SOCK_KEEPOPEN) || sk->sk_state == TCP_CLOSE)
goto out;
^ permalink raw reply related [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 18:39 ` Ingo Molnar
@ 2008-06-06 19:49 ` Ilpo Järvinen
2008-06-06 20:08 ` Patrick McManus
1 sibling, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-06 19:49 UTC (permalink / raw)
To: Ingo Molnar
Cc: Patrick McManus, David Miller, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
[-- Attachment #1: Type: TEXT/PLAIN, Size: 3643 bytes --]
On Fri, 6 Jun 2008, Ingo Molnar wrote:
>
> * Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
>
> > If you want an older kernel, you would have to go basically to 2.6.25
> > or so.
>
> correct, that's what i use as fallback, some distro kernel which is
> 2.6.25 or older.
>
> but i'm confused a bit, you say v2.6.25-rc6-475-gec3c098 introduced the
> locking problem - so 2.6.25 is affected as well?
No, you're probably just falling into a git-describe trap I also used
to fall:
ijjarvin@pointhope:~/linux/mainline$ git-log -n 1 --pretty=oneline
ec3c0982a2dd1e671bad8e9d26c28dcba0039d87 ^v2.6.25 | cat -
ec3c0982a2dd1e671bad8e9d26c28dcba0039d87 [TCP]: TCP_DEFER_ACCEPT updates -
process as established
ijjarvin@pointhope:~/linux/mainline$ git-log -n 1 --pretty=oneline
ec3c0982a2dd1e671bad8e9d26c28dcba0039d87 ^v2.6.26-rc1 | cat -
ijjarvin@pointhope:~/linux/mainline$ git-describe
ec3c0982a2dd1e671bad8e9d26c28dcba0039d87
v2.6.25-rc6-475-gec3c098
ijjarvin@pointhope:~/linux/mainline$
The git-describe is not the way one can determine into which mainline
tag a commit was included, it basically just provides the closest tag
among ancestors, which can be a vastly different one and has _no_
relation whatsoever to the tag we'd desire to get. In here, Dave had
net-2.6 based on 2.5.25-rc6ish (or alternatively last merge to net-2.6
from Linus' tree's content came from that point of time), but Linus did
the merge from 2.6.25 but git-describe won't look anything that happens
after the asked commit. This is similar to the
bisect-lands-lower-tag-than-select-good-commit-was "mystery" that was
recently discussed extensively, again the Makefile only tracks ancestors,
not the future.
If somebody knows a trivial command to get that future information (to
where merged info), I'd pretty interested to hear.
> This is a significant
> question because the fallback kernel is kernel-2.6.25.3-18.fc9.x86_64 on
> the 16-way box. (all other build-boxes have 2.6.24 or older as a
> fallback kernel)
Please do get the receiver state if you still see such problem with it,
it is also relevant but it a different problem then (I'm yet to analyze
the data Håkan was collecting, dl it already by didn't even look into
that yet).
...Or also if you see stuck TCPs with other cases I've told should fix it:
1. 2.6.25 (pre-ec3c to be accurate)
2. 3+1 revert
3. ec3c+locking fix (this is the most unsure one because it still would
have the reversed socket lock taking order though nothing bad has been
found by some review neither by me nor Patrick)
Please collect at least /proc/net/tcp and the netstat -np, if there's
process associated to the flow with _Recv-Q_ (in localhost case there
are two of them, the other with Send-Q), also where the process is
waiting is useful. Hopefully clear enough now... :-)
> > To summarize. Both 3changes+1fix revert (you refer to it only as
> > 3-patch revert) _and_ the locking fix I made should fix the problem
> > (obviously they exclude each other). ...And end which is significant
> > is the one which has LISTENing sockets (please keep this in mind if
> > you still get the hang and provide some info).
>
> ok.
>
> For completeness, let me repeat the patch i referred to as the
> '3-patch-revert' below. (which indeed is 3+1 as you note)
...I know because there never have been any 3-patch-revert made... :-)
> this is the patch that appears to be working empirically. (Disclaimer:
> it might just hide the problem, change timings, have a lucky code
> layout, etc.)
Sure, but the revert also removes the obvious locking problem that was
introduced in ec3c.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 18:39 ` Ingo Molnar
2008-06-06 19:49 ` Ilpo Järvinen
@ 2008-06-06 20:08 ` Patrick McManus
2008-06-06 21:12 ` Ilpo Järvinen
2008-06-10 22:49 ` David Miller
1 sibling, 2 replies; 96+ messages in thread
From: Patrick McManus @ 2008-06-06 20:08 UTC (permalink / raw)
To: Ingo Molnar
Cc: Ilpo Järvinen, David Miller, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
This is all a bit confusing, but here are the conclusions I have drawn.
There definitely is a problem with the locking of the DA commit
ec3c0982a2dd1e671bad8e9d26c28dcba0039d87 . That code was part of 26-rc1
but it never appeared in 25. It exists in pretty much the same form in
rc5 (there was 1 patch to it over that time to fix a different problem).
We're certain this code has a problem with the accept queue both because
of code inspection and the fact that Ingo can back it out (as the
significant part of the 3-patch revert) and the problem goes away in his
testing.
I have run tests that can reproduce the hung socket with distcc over
localhost using 26-rc5. I can also apparently cure it using the locking
fix patch Ilpo sent (c9454f0..d21d2b9) on top of that. (My test of rc5
+lockpatch is at 4.5+ hrs and counting without failures, it fails 6
times an hour with vanilla rc5)
Based on all of that, the right thing to do seems to be to apply the
lockpatch (c9454f0..d21d2b9) to Linus's tree and not revert anything -
just fix the code and I'll send Ilpo and Ingo cookies at Christmas time
for being great guys. Alternatively, Ingo could run the distcc servers
and clients on -tip with the lockpatch (nothing reverted) for more
testing.
The only lingering problem is Ingo's report yesterday
http://marc.info/?l=linux-netdev&m=121267587715976&w=2
of a distcc hang. In this one it was not over localhost and the distcc
server had the ec3c DA changes totally reverted. (The server is really
the only stack that matters in this case - the client is not impacted by
the DA changes). This has to be a different issue, because the ec3c code
we're talking about here wasn't on the server at all. As Ilpo mentions,
Hakon is beleived to have a different problem and maybe you've tripped
over that too?
If we're sure of that conclusion we should just take Ilpo's DA patch as
that will narrow the field for finding Hakon's issue. Its just with all
of these data points I'm not sure if I'm reaching the right conclusion.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 20:08 ` Patrick McManus
@ 2008-06-06 21:12 ` Ilpo Järvinen
2008-06-06 21:23 ` Arjan van de Ven
2008-06-10 22:49 ` David Miller
1 sibling, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-06 21:12 UTC (permalink / raw)
To: Patrick McManus, Arjan van de Ven
Cc: Ingo Molnar, David Miller, peterz, LKML, Netdev, rjw,
Andrew Morton, johnpol
[-- Attachment #1: Type: TEXT/PLAIN, Size: 4077 bytes --]
...added Arjan.
On Fri, 6 Jun 2008, Patrick McManus wrote:
> This is all a bit confusing, but here are the conclusions I have drawn.
Your observations here match what I've understood :-).
> There definitely is a problem with the locking of the DA commit
> ec3c0982a2dd1e671bad8e9d26c28dcba0039d87 . That code was part of 26-rc1
> but it never appeared in 25. It exists in pretty much the same form in
> rc5 (there was 1 patch to it over that time to fix a different problem).
>
> We're certain this code has a problem with the accept queue both because
> of code inspection and the fact that Ingo can back it out (as the
> significant part of the 3-patch revert) and the problem goes away in his
> testing.
Problems were at least these:
- Accept queue addition was racy and could leave dangling items
- Dangling items caused inconsistent sk_ack_backlog
- Checking for still in LISTEN state was racy, could be changed after
the check was made (shouldn't happen with distcc though)
I didn't read ->sk_data_ready that carefully, it could have some
additional problems that are not listed (but they all should be fixed
by the added locking anyway).
AFAICT, rest of that ec3c change is safe wrt. locking, just holding sk is
enough for the rest and those bits mostly shouldn't anyway be executed
with a distcc setup.
> I have run tests that can reproduce the hung socket with distcc over
> localhost using 26-rc5. I can also apparently cure it using the locking
> fix patch Ilpo sent (c9454f0..d21d2b9) on top of that. (My test of rc5
> +lockpatch is at 4.5+ hrs and counting without failures, it fails 6
> times an hour with vanilla rc5)
>
> Based on all of that, the right thing to do seems to be to apply the
> lockpatch (c9454f0..d21d2b9) to Linus's tree and not revert anything -
> just fix the code and I'll send Ilpo and Ingo cookies at Christmas time
> for being great guys. Alternatively, Ingo could run the distcc servers
> and clients on -tip with the lockpatch (nothing reverted) for more
> testing.
Anyway, we still would have an option to revert both the DA change + the
locking fix later if the problem is still clearly more likely than with
stable-2.6.25.
> The only lingering problem is Ingo's report yesterday
> http://marc.info/?l=linux-netdev&m=121267587715976&w=2
> of a distcc hang. In this one it was not over localhost and the distcc
> server had the ec3c DA changes totally reverted. (The server is really
> the only stack that matters in this case - the client is not impacted by
> the DA changes).
It definately didn't fit to picture that well if we would be talking just
a single bug here.
...I wish Ingo would have provided the receiver state already then. :-)
> This has to be a different issue, because the ec3c code
> we're talking about here wasn't on the server at all. As Ilpo mentions,
> Hakon is beleived to have a different problem and maybe you've tripped
> over that too?
...The Håkon's case is definately different thing, also the symptoms
are quite different because there's no deadlock at all but the TCP flow
eventually dies, I don't yet know with what timescale that dying happens.
Only common denominator actually was this receiver process missing, though
it provably still was there.
Besides, I don't know how long Ingo waited in this case until concluding
that the TCP was stuck again?
> If we're sure of that conclusion we should just take Ilpo's DA patch as
> that will narrow the field for finding Hakon's issue. Its just with all
> of these data points I'm not sure if I'm reaching the right conclusion.
Lets widen the scope to two to three bugs then, one down already...
In case you missed btw, also Arjan reported some problem quite early, but
in his case claws mua+imap was the workload, so I doubt that DEFER_ACCEPT
would be involved but who knows without strace, here:
http://marc.info/?l=linux-kernel&m=121182171000434&w=2
Arjan, can you please check if your workload uses setsockopt
TCP_DEFER_ACCEPT for the LISTENing socket? ...If not, then your case
is different from Ingo's.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 21:12 ` Ilpo Järvinen
@ 2008-06-06 21:23 ` Arjan van de Ven
2008-06-06 21:28 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Arjan van de Ven @ 2008-06-06 21:23 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: Patrick McManus, Ingo Molnar, David Miller, peterz, LKML, Netdev,
rjw, Andrew Morton, johnpol
On Sat, 7 Jun 2008 00:12:55 +0300 (EEST)
"Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi> wrote:
> Arjan, can you please check if your workload uses setsockopt
> TCP_DEFER_ACCEPT for the LISTENing socket? ...If not, then your case
> is different from Ingo's.
at this point the observed hung I had seems to be the app doing
something bad (it's the app that hung) so for now please disregard my
previous comments on this topic.
>
>
--
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings,
visit http://www.lesswatts.org
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 21:23 ` Arjan van de Ven
@ 2008-06-06 21:28 ` Ilpo Järvinen
0 siblings, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-06 21:28 UTC (permalink / raw)
To: Arjan van de Ven
Cc: Patrick McManus, Ingo Molnar, David Miller, peterz, LKML, Netdev,
rjw, Andrew Morton, johnpol
[-- Attachment #1: Type: TEXT/PLAIN, Size: 530 bytes --]
On Fri, 6 Jun 2008, Arjan van de Ven wrote:
> On Sat, 7 Jun 2008 00:12:55 +0300 (EEST)
> "Ilpo Järvinen" <ilpo.jarvinen@helsinki.fi> wrote:
> > Arjan, can you please check if your workload uses setsockopt
> > TCP_DEFER_ACCEPT for the LISTENing socket? ...If not, then your case
> > is different from Ingo's.
>
> at this point the observed hung I had seems to be the app doing
> something bad (it's the app that hung) so for now please disregard my
> previous comments on this topic.
...Ok, thanks for letting us know.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 9:32 ` Håkon Løvdal
@ 2008-06-09 19:24 ` Ilpo Järvinen
2008-06-10 23:26 ` Håkon Løvdal
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-09 19:24 UTC (permalink / raw)
To: Håkon Løvdal
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
[-- Attachment #1: Type: TEXT/PLAIN, Size: 1169 bytes --]
On Fri, 6 Jun 2008, Håkon Løvdal wrote:
> On 04/06/2008, Ilpo Järvinen <ilpo.jarvinen@helsinki.fi> wrote:
> > I guess not all of the strace will be necessary anyway. You could first
> > leave that out completely and concentrate on the other logs first.
>
> I have now set up a server on my machine and made all logs available.
> I will follow up with the adress in private email to those that already
> have posted in this thread. For others, please ask to receive it.
Can you please explain this:
ijjarvin@wrl-59:~/tmp$ bunzip2 -t copytest003.capture.bz2
bunzip2: copytest003.capture.bz2: data integrity (CRC) error in data
You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.
ijjarvin@wrl-59:~/tmp$ ls -al copytest003.capture.bz2
-rw-r----- 1 ijjarvin tkol 275441620 9. kesä  16:39
copytest003.capture.bz2
ijjarvin@wrl-59:~/tmp$ md5sum copytest003.capture.bz2
4bbf801cf1fbd52399322b088c1a2058 copytest003.capture.bz2
ijjarvin@wrl-59:~/tmp$
...also the one from allfiles.zip is similarily damaged.
...If you don't have an explanation, them it might be time to check
your hardware....
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-03 23:22 ` Ilpo Järvinen
` (2 preceding siblings ...)
2008-06-05 14:22 ` Ingo Molnar
@ 2008-06-10 22:32 ` David Miller
2008-06-11 13:10 ` Patrick McManus
2008-06-11 15:13 ` Ilpo Järvinen
3 siblings, 2 replies; 96+ messages in thread
From: David Miller @ 2008-06-10 22:32 UTC (permalink / raw)
To: ilpo.jarvinen
Cc: mingo, mcmanus, peterz, linux-kernel, netdev, rjw, akpm, johnpol
From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi>
Date: Wed, 4 Jun 2008 02:22:16 +0300 (EEST)
> [PATCH] tcp DEFER_ACCEPT: fix racy access to listen_sk
>
> It seems that replacement of DA code also moved parts outside
> of appropriate locking. The Ingo's problem seems to come from
> the fact that two flows could now race in
> (inet_csk_)reqsk_queue_add corrupting the queue. ...This can
> leave dangling socks around which won't resolve themselves
> without stimuli from outside (e.g., external RST would help
> I think).
>
> Then some details I'm not too sure of:
> I guess we want to put listen_sk->sk_state checking under the
> lock as well. I've not evaluated if ->sk_data_ready too
> requires locking but assumed it does.
>
> I'm by no means familiar with all locking variants, requirements,
> etc.
>
> Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
I took a close look at this, it seems this patch adds
an ABBA deadlock. But I might be wrong.
Normally the locking order is:
listen_sk --> some_child_sk
And this can be seen by the code paths that flow down
to tcp_child_process() in net/ipv4/tcp_minisocks.c (via
tcp_v4_do_rcv() for example).
However here in this patch we will lock in the:
child_sk --> listen_sk
order.
Unless... these defer accepted sockets live outside of the
normal socket collection found by tcp_v{4,6}_hnd_req(). If
that is the case, that ought to make this locking order OK
but I fear that lockdep will likely complain because it has
no way to see this distinction.
If we cannot find a simple and easy way to deal with this locking
problem, I am reverting the defer-accept changes entirely. It's not
the end of the world if this feature has to be deferred to 2.6.27
and this bug has been known for several weeks already.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-06 20:08 ` Patrick McManus
2008-06-06 21:12 ` Ilpo Järvinen
@ 2008-06-10 22:49 ` David Miller
1 sibling, 0 replies; 96+ messages in thread
From: David Miller @ 2008-06-10 22:49 UTC (permalink / raw)
To: mcmanus
Cc: mingo, ilpo.jarvinen, peterz, linux-kernel, netdev, rjw, akpm,
johnpol
From: Patrick McManus <mcmanus@ducksong.com>
Date: Fri, 06 Jun 2008 16:08:57 -0400
> If we're sure of that conclusion we should just take Ilpo's DA patch as
> that will narrow the field for finding Hakon's issue.
While the conclusion might be accurate, Ilpo's patch adds a lock
ordering problem that might introduce potential deadlocks (worst
case) or at least some lockdep warnings. Both of which are
unacceptable.
I personally don't see any clear solution, and because this new
DA code moves an operation into a code block where the locking
order pretty much has to be inverted, I don't see any real short
term solution other than a revert of this particular DA commit.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-09 19:24 ` Ilpo Järvinen
@ 2008-06-10 23:26 ` Håkon Løvdal
2008-06-11 13:39 ` Ilpo Järvinen
0 siblings, 1 reply; 96+ messages in thread
From: Håkon Løvdal @ 2008-06-10 23:26 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
2008/6/9 Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>:
> On Fri, 6 Jun 2008, Håkon Løvdal wrote:
>> I have now set up a server on my machine and made all logs available.
>> I will follow up with the adress in private email to those that already
>> have posted in this thread. For others, please ask to receive it.
>
> Can you please explain this:
>
> ijjarvin@wrl-59:~/tmp$ bunzip2 -t copytest003.capture.bz2
> bunzip2: copytest003.capture.bz2: data integrity (CRC) error in data
>
> You can use the `bzip2recover' program to attempt to recover
> data from undamaged sections of corrupted files.
>
> ijjarvin@wrl-59:~/tmp$ ls -al copytest003.capture.bz2
> -rw-r----- 1 ijjarvin tkol 275441620 9. kesä  16:39
> copytest003.capture.bz2
> ijjarvin@wrl-59:~/tmp$ md5sum copytest003.capture.bz2
> 4bbf801cf1fbd52399322b088c1a2058 copytest003.capture.bz2
> ijjarvin@wrl-59:~/tmp$
>
> ...also the one from allfiles.zip is similarily damaged.
>
> ...If you don't have an explanation, them it might be time to check
> your hardware....
I do not have any good explanation (bad memory?). When checking the
copytest003.capture.bz2 file in the http server storage directory
I get the same results as you. However I also have the original file
stored still and comparing them with cmp -l gives
20558656 124 104
so obviously a bit has been flipped. I have now replaced the
erroneous file with a good copy, so you should be able to download again.
BR Håkon Løvdal
Btw, I got a "traditional" hang now on my first attempt to copy
copytest003.capture.bz2 from old_pc to new_pc just using scp.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-10 22:32 ` David Miller
@ 2008-06-11 13:10 ` Patrick McManus
2008-06-11 15:13 ` Ilpo Järvinen
1 sibling, 0 replies; 96+ messages in thread
From: Patrick McManus @ 2008-06-11 13:10 UTC (permalink / raw)
To: David Miller
Cc: ilpo.jarvinen, mingo, peterz, linux-kernel, netdev, rjw, akpm,
johnpol, vgusev
On Tue, 2008-06-10 at 15:32 -0700, David Miller wrote:
> If we cannot find a simple and easy way to deal with this locking
> problem, I am reverting the defer-accept changes entirely. It's not
> the end of the world if this feature has to be deferred to 2.6.27
based on this, plus the comments from vitaliy it is clear that moving
the DA socket to established is pretty tricky. I agree I should address
these things in the >= 27 time frame.
imho we don't need to revert the first two patches:
e4c78840284f3f51b1896cf3936d60a6033c4d2c and
539fae89bebd16ebeafd57a87169bc56eb530d76 .. they have nothing to do with
rearranging states or the locking and they do fix the more notable DA
behavior issues.
-Patrick
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-10 23:26 ` Håkon Løvdal
@ 2008-06-11 13:39 ` Ilpo Järvinen
2008-06-19 0:30 ` Håkon Løvdal
0 siblings, 1 reply; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-11 13:39 UTC (permalink / raw)
To: Håkon Løvdal
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
[-- Attachment #1: Type: TEXT/PLAIN, Size: 901 bytes --]
On Wed, 11 Jun 2008, Håkon Løvdal wrote:
> 2008/6/9 Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>:
> > ...also the one from allfiles.zip is similarily damaged.
> >
> > ...If you don't have an explanation, them it might be time to check
> > your hardware....
>
> I do not have any good explanation (bad memory?). When checking the
> copytest003.capture.bz2 file in the http server storage directory
> I get the same results as you. However I also have the original file
> stored still and comparing them with cmp -l gives
> 20558656 124 104
> so obviously a bit has been flipped.
...Yeah, which is a bad thing because now we don't know if that same error
causes the TCP stops too.
> I have now replaced the erroneous file with a good copy, so you should
> be able to download again.
I'll have a look if there's some common denominator but this whole issue
might well be due to hw failure.
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [fixed] [patch] Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-10 22:32 ` David Miller
2008-06-11 13:10 ` Patrick McManus
@ 2008-06-11 15:13 ` Ilpo Järvinen
1 sibling, 0 replies; 96+ messages in thread
From: Ilpo Järvinen @ 2008-06-11 15:13 UTC (permalink / raw)
To: David Miller
Cc: mingo, mcmanus, peterz, LKML, Netdev, rjw, Andrew Morton, johnpol
[-- Attachment #1: Type: TEXT/PLAIN, Size: 5129 bytes --]
On Tue, 10 Jun 2008, David Miller wrote:
> From: "Ilpo_Järvinen" <ilpo.jarvinen@helsinki.fi>
> Date: Wed, 4 Jun 2008 02:22:16 +0300 (EEST)
>
> > [PATCH] tcp DEFER_ACCEPT: fix racy access to listen_sk
> >
> > It seems that replacement of DA code also moved parts outside
> > of appropriate locking. The Ingo's problem seems to come from
> > the fact that two flows could now race in
> > (inet_csk_)reqsk_queue_add corrupting the queue. ...This can
> > leave dangling socks around which won't resolve themselves
> > without stimuli from outside (e.g., external RST would help
> > I think).
> >
> > Then some details I'm not too sure of:
> > I guess we want to put listen_sk->sk_state checking under the
> > lock as well. I've not evaluated if ->sk_data_ready too
> > requires locking but assumed it does.
> >
> > I'm by no means familiar with all locking variants, requirements,
> > etc.
> >
> > Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
>
> I took a close look at this, it seems this patch adds
> an ABBA deadlock. But I might be wrong.
>
> Normally the locking order is:
>
> listen_sk --> some_child_sk
>
> And this can be seen by the code paths that flow down
> to tcp_child_process() in net/ipv4/tcp_minisocks.c (via
> tcp_v4_do_rcv() for example).
>
> However here in this patch we will lock in the:
>
> child_sk --> listen_sk
>
> order.
>
> Unless... these defer accepted sockets live outside of the
> normal socket collection found by tcp_v{4,6}_hnd_req().
All these collection/queues make the analysis a bit complex :-), but
I think I finally got a hold of it anyway and my analysis agrees with
yours. To me it seems that when DAed TCP sk moves to established, there's
no longer a connection from listen_sk to child_sk, we keep them alive by
incrementing those refcounts and the child sk is responsible about the
binding between them (until the DA gets resolution). But the child_sk
locking during the creation is also a suspect, but only after
tcp_rcv_established() part can run for the child sock.
Am I right that after tcp_v4_syn_recv_sock() it is possible to get
tcp_rcv_established() to execute for the created child? It's called
by this path:
tcp_v4_hnd_req ->
tcp_check_req ->
tcp_v4_syn_recv_sock (inet_csk(sk)->icsk_af_ops->syn_recv_sock)
...and the listen_sk part acquires that child's lock only after that in
tcp_v4_hnd_req(). ...Allowing it to deadlock during that short window!
...But from that point onward nothing in listen_sk needs to lock the
child. Then this connection from listen_sk to child_sk comes back once
we've reinserted the DAed child_sk back to the queue (in that racy part I
was trying to fix) but at that point of time we won't ever need to lock in
child_sk -> listen_sk order again because we've already passed that
point.
...But like you, I don't understand all of it that well... Btw, your
sk_callback_lock notes explained some mystery part for me as I came
across with it too while looking around... :-)
> If that is the case, that ought to make this locking order OK
> but I fear that lockdep will likely complain because it has
> no way to see this distinction.
I definately agree that the locks are taken in different order no matter
what (I actually referring to that already in the first mail about the
deadlock) so probably lockdep is not going to be too friendly :-), whether
it could cause a real deadlock, I was not that sure at that point of time.
> If we cannot find a simple and easy way to deal with this locking
> problem, I am reverting the defer-accept changes entirely.
To avoid all problem, I was already thinking of another approach (though
my time constraints hardly allows finishing it any time soon):
Because DA code resides quite late on the TCP path it would be quite easy
to do some preparatory work, drop child sk's lock and re-acquire both
locks in the usual order (listen->child) but that would require handling
correctly reset & other things that could intervene (luckily that pesky
userspace isn't there yet to mess around so not that many things can
happen :-)). But ipv6 seems to do some additional processing after
tcp_rcv_established() which I'd expect to choke if child sk's lock was
dropped there for a moment, while ipv4 part seems quite doable. I don't
even know what exactly are the requirements for that ipv6 part (see the
stuff after ipv6_pktoptions label in tcp_v6_do_rcv). ...I tried to do
this but came across those two things mentioned above.
IMHO, changing locking this late in the release cycle would be quite
risky anyway... And we would also be fiddling with TCP state machine.
> It's not the end of the world if this feature has to be deferred to
> 2.6.27
Agreed. Especially in the light of the another issue that has been
raised.
> and this bug has been known for several weeks already.
...That's partially because Ingo didn't even test my fix on the receiver
which got stuck but used 2.6.25 and got some other bug which looked alike
but couldn't be the same because these DA problems weren't in 2.6.25. What
could I've done for that :-).
--
i.
^ permalink raw reply [flat|nested] 96+ messages in thread
* Re: [bug] stuck localhost TCP connections, v2.6.26-rc3+
2008-06-11 13:39 ` Ilpo Järvinen
@ 2008-06-19 0:30 ` Håkon Løvdal
0 siblings, 0 replies; 96+ messages in thread
From: Håkon Løvdal @ 2008-06-19 0:30 UTC (permalink / raw)
To: Ilpo Järvinen
Cc: LKML, Netdev, Ingo Molnar, David S. Miller, Rafael J. Wysocki,
Andrew Morton
2008/6/11 Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>:
> On Wed, 11 Jun 2008, Håkon Løvdal wrote:
>> I do not have any good explanation (bad memory?). When checking the
>> copytest003.capture.bz2 file in the http server storage directory
>> I get the same results as you. However I also have the original file
>> stored still and comparing them with cmp -l gives
>> 20558656 124 104
>> so obviously a bit has been flipped.
>
> ...Yeah, which is a bad thing because now we don't know if that same error
> causes the TCP stops too.
>
>> I have now replaced the erroneous file with a good copy, so you should
>> be able to download again.
>
> I'll have a look if there's some common denominator but this whole issue
> might well be due to hw failure.
>
Hi. I tested the old pc with memtest86 and memtest86+ for 16 and 50
hours without any errors detected, so apparently the ram is ok.
I now just did another test from a freshley booted old pc, standing
idle for around 2 hours before I started the testing (no network activity
before I plugged in the cable just before starting testing).
The copying failed after just 2.62GB, so the logs are short and
nice. Stored on same server as earlier, new 2008-06-19 directory. One
thing I noticed in the dmesg output on the old pc was the following:
BUG: unable to handle kernel paging request at virtual address ffff3cff
printing eip: ffff3cff *pde = 00793067 *pte = 00000000
Oops: 0000 [#1] PREEMPT
Modules linked in: ...
Pid: 18103, comm: sshd Not tainted (2.6.24-gafd564a8 #4)
...
Call Trace:
[<f8c45b39>] nf_conntrack_find_get+0x36/0x3a [nf_conntrack]
[<f8c46379>] nf_conntrack_in+0x135/0x3e8 [nf_conntrack]
[<c0423c5d>] local_bh_disable+0xa/0xb
[<c0626d54>] _read_lock_bh+0x12/0x1a
...
BR Håkon Løvdal
^ permalink raw reply [flat|nested] 96+ messages in thread
end of thread, other threads:[~2008-06-19 0:30 UTC | newest]
Thread overview: 96+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-05-26 11:56 [bug] stuck localhost TCP connections, v2.6.26-rc3+ Ingo Molnar
2008-05-26 13:28 ` Ilpo Järvinen
2008-05-26 13:59 ` Ingo Molnar
2008-05-26 14:12 ` Ingo Molnar
2008-05-26 14:17 ` Ingo Molnar
2008-05-26 14:29 ` Ingo Molnar
2008-05-26 14:43 ` Ilpo Järvinen
2008-05-26 14:58 ` Ilpo Järvinen
2008-05-26 16:23 ` Ingo Molnar
2008-05-26 16:32 ` Ilpo Järvinen
2008-05-26 16:54 ` Ingo Molnar
2008-05-26 17:08 ` Ilpo Järvinen
2008-05-26 18:12 ` Ingo Molnar
2008-05-26 20:41 ` Ingo Molnar
2008-05-26 21:20 ` Ilpo Järvinen
2008-05-30 16:23 ` Ray Lee
2008-05-26 16:24 ` Arjan van de Ven
2008-05-28 9:27 ` Peter Zijlstra
2008-05-31 14:25 ` Håkon Løvdal
2008-05-31 16:09 ` Ilpo Järvinen
2008-05-31 17:22 ` Ilpo Järvinen
2008-05-31 17:58 ` Håkon Løvdal
2008-05-31 18:37 ` Ilpo Järvinen
2008-05-31 20:25 ` Håkon Løvdal
2008-05-31 21:39 ` Ilpo Järvinen
2008-05-31 21:45 ` Håkon Løvdal
2008-06-04 0:10 ` Håkon Løvdal
2008-06-04 11:14 ` Ilpo Järvinen
2008-06-04 14:00 ` Håkon Løvdal
2008-06-04 15:09 ` Ilpo Järvinen
2008-06-06 9:32 ` Håkon Løvdal
2008-06-09 19:24 ` Ilpo Järvinen
2008-06-10 23:26 ` Håkon Løvdal
2008-06-11 13:39 ` Ilpo Järvinen
2008-06-19 0:30 ` Håkon Løvdal
2008-05-29 8:45 ` Ingo Molnar
2008-05-29 11:14 ` Ilpo Järvinen
2008-05-29 11:22 ` Ingo Molnar
2008-05-29 13:05 ` Evgeniy Polyakov
2008-05-29 13:43 ` Ingo Molnar
2008-05-29 13:08 ` Ingo Molnar
2008-05-29 13:48 ` Ilpo Järvinen
2008-05-30 11:09 ` Ingo Molnar
2008-05-30 21:12 ` Ilpo Järvinen
2008-05-30 18:18 ` Ingo Molnar
2008-05-31 6:09 ` Ingo Molnar
2008-05-31 11:46 ` Ilpo Järvinen
2008-05-31 12:18 ` Ilpo Järvinen
2008-05-31 12:54 ` Ingo Molnar
2008-05-31 12:58 ` Ilpo Järvinen
2008-05-31 16:35 ` Ingo Molnar
2008-05-31 22:46 ` Patrick McManus
2008-06-01 5:51 ` Ilpo Järvinen
2008-06-01 6:04 ` Eric Dumazet
2008-06-02 9:23 ` Ingo Molnar
2008-06-03 9:40 ` [fixed] [patch] " Ingo Molnar
2008-06-03 14:41 ` Patrick McManus
2008-06-03 21:46 ` Ilpo Järvinen
2008-06-03 22:01 ` Ilpo Järvinen
2008-06-03 22:03 ` David Miller
2008-06-03 22:10 ` Ilpo Järvinen
2008-06-03 23:22 ` Ilpo Järvinen
2008-06-03 23:54 ` Joe Perches
2008-06-04 6:25 ` Ilpo Järvinen
2008-06-04 2:54 ` Patrick McManus
2008-06-04 6:42 ` Ilpo Järvinen
2008-06-05 14:22 ` Ingo Molnar
2008-06-05 18:00 ` Ilpo Järvinen
2008-06-05 21:13 ` Ilpo Järvinen
2008-06-05 23:29 ` Patrick McManus
2008-06-06 10:03 ` Ilpo Järvinen
2008-06-06 17:11 ` Patrick McManus
2008-06-06 17:33 ` Ingo Molnar
2008-06-06 18:19 ` Ilpo Järvinen
2008-06-06 18:39 ` Ingo Molnar
2008-06-06 19:49 ` Ilpo Järvinen
2008-06-06 20:08 ` Patrick McManus
2008-06-06 21:12 ` Ilpo Järvinen
2008-06-06 21:23 ` Arjan van de Ven
2008-06-06 21:28 ` Ilpo Järvinen
2008-06-10 22:49 ` David Miller
2008-06-06 18:25 ` Ilpo Järvinen
2008-06-10 22:32 ` David Miller
2008-06-11 13:10 ` Patrick McManus
2008-06-11 15:13 ` Ilpo Järvinen
2008-06-04 7:23 ` Ingo Molnar
2008-06-04 18:24 ` David Miller
2008-06-04 20:56 ` Ilpo Järvinen
2008-06-04 21:55 ` David Miller
-- strict thread matches above, loose matches on Subject: below --
2008-05-30 16:31 Ray Lee
2008-05-30 21:11 ` Ilpo Järvinen
2008-05-31 6:03 ` Evgeniy Polyakov
2008-05-31 10:05 ` Ilpo Järvinen
2008-06-02 6:19 ` Herbert Xu
2008-06-02 11:53 ` Ilpo Järvinen
2008-06-02 14:08 ` Herbert Xu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).