* 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50
@ 2008-04-27 22:55 Alexey Dobriyan
2008-04-28 12:01 ` Andrew Morton
0 siblings, 1 reply; 37+ messages in thread
From: Alexey Dobriyan @ 2008-04-27 22:55 UTC (permalink / raw)
To: axboe; +Cc: linux-kernel
This happened while ~90 cross-compile jobs were running in parallel on
ext2/noatime partition (slowly -- much debugging was on)
general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
CPU 0
Modules linked in: ext2 nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp uhci_hcd ehci_hcd usbcore sr_mod cdrom
Pid: 16483, comm: as Not tainted 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48 #1
RIP: 0010:[<ffffffff80307525>] [<ffffffff80307525>] call_for_each_cic+0x25/0x50
RSP: 0018:ffff810170811e58 EFLAGS: 00010202
RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81010ff92000
RBP: ffff810170811e78 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000000000 R11: ffff8100010069d8 R12: ffff810138ada300
R13: ffffffff803075b0 R14: ffff81017fcd2000 R15: ffff81010ff92168
FS: 00002ac3462426f0(0000) GS:ffffffff805d0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002ab602550000 CR3: 000000013609d000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process as (pid: 16483, threadinfo ffff810170810000, task ffff81010ff92000)
Stack: ffff810170811e88 ffff810138ada300 0000000000000010 ffff81010ff92100
ffff810170811e88 ffffffff80307580 ffff810170811ea8 ffffffff80302a55
ffff81010ff92100 ffff810138ada300 ffff810170811ec8 ffffffff80302b1f
Call Trace:
[<ffffffff80307580>] cfq_free_io_context+0x10/0x20
[<ffffffff80302a55>] put_io_context+0x85/0x90
[<ffffffff80302b1f>] exit_io_context+0x8f/0xb0
[<ffffffff80235d19>] do_exit+0x549/0x780
[<ffffffff80235f8e>] do_group_exit+0x3e/0xb0
[<ffffffff80236012>] sys_exit_group+0x12/0x20
[<ffffffff8020b6db>] system_call_after_swapgs+0x7b/0x80
Code: 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 e8 18 e1 f5 ff 49 8b 44 24 68 48 85 c0 74 1e 48 89 c3 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
RIP [<ffffffff80307525>] call_for_each_cic+0x25/0x50
RSP <ffff810170811e58>
---[ end trace ca143223eefdc828 ]---
Fixing recursive fault but reboot is needed!
ffffffff80307500 <call_for_each_cic>:
ffffffff80307500: 55 push %rbp
ffffffff80307501: 48 89 e5 mov %rsp,%rbp
ffffffff80307504: 41 55 push %r13
ffffffff80307506: 49 89 f5 mov %rsi,%r13
ffffffff80307509: 41 54 push %r12
ffffffff8030750b: 49 89 fc mov %rdi,%r12
ffffffff8030750e: 53 push %rbx
ffffffff8030750f: 48 83 ec 08 sub $0x8,%rsp
ffffffff80307513: e8 18 e1 f5 ff callq ffffffff80265630 <__rcu_read_lock>
ffffffff80307518: 49 8b 44 24 68 mov 0x68(%r12),%rax
ffffffff8030751d: 48 85 c0 test %rax,%rax
ffffffff80307520: 74 1e je ffffffff80307540 <call_for_each_cic+0x40>
ffffffff80307522: 48 89 c3 mov %rax,%rbx
ffffffff80307525: 48 8b 03 mov (%rbx),%rax
ffffffff80307528: 48 8d 73 88 lea -0x78(%rbx),%rsi
ffffffff8030752c: 4c 89 e7 mov %r12,%rdi
ffffffff8030752f: 0f 18 08 prefetcht0 (%rax)
ffffffff80307532: 41 ff d5 callq *%r13
ffffffff80307535: 48 8b 03 mov (%rbx),%rax
ffffffff80307538: 48 85 c0 test %rax,%rax
ffffffff8030753b: 48 89 c3 mov %rax,%rbx
ffffffff8030753e: 75 e5 jne ffffffff80307525 <call_for_each_cic+0x25>
ffffffff80307540: e8 2b e0 f5 ff callq ffffffff80265570 <__rcu_read_unlock>
ffffffff80307545: 48 83 c4 08 add $0x8,%rsp
ffffffff80307549: 5b pop %rbx
ffffffff8030754a: 41 5c pop %r12
ffffffff8030754c: 41 5d pop %r13
ffffffff8030754e: c9 leaveq
ffffffff8030754f: c3 retq
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48
# Sun Apr 27 23:36:28 2008
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
# CONFIG_GENERIC_LOCKBREAK is not set
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
# CONFIG_GENERIC_GPIO is not set
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_DMI=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
# CONFIG_ARCH_HAS_ILOG2_U32 is not set
# CONFIG_ARCH_HAS_ILOG2_U64 is not set
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_AOUT=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_TRAMPOLINE=y
# CONFIG_KTIME_SCALAR is not set
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
# CONFIG_POSIX_MQUEUE is not set
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set
# CONFIG_AUDIT is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=17
# CONFIG_CGROUPS is not set
# CONFIG_GROUP_SCHED is not set
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
# CONFIG_RELAY is not set
# CONFIG_NAMESPACES is not set
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_EMBEDDED=y
# CONFIG_UID16 is not set
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
# CONFIG_COMPAT_BRK is not set
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
# CONFIG_EPOLL is not set
# CONFIG_SIGNALFD is not set
# CONFIG_TIMERFD is not set
# CONFIG_EVENTFD is not set
CONFIG_SHMEM=y
# CONFIG_VM_EVENT_COUNTERS is not set
CONFIG_SLUB_DEBUG=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
# CONFIG_MARKERS is not set
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
# CONFIG_PROC_PAGE_MONITOR is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
# CONFIG_BLK_DEV_IO_TRACE is not set
# CONFIG_BLK_DEV_BSG is not set
CONFIG_BLOCK_COMPAT=y
#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_AS is not set
# CONFIG_IOSCHED_DEADLINE is not set
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
# CONFIG_CLASSIC_RCU is not set
#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_X86_RDC321X is not set
# CONFIG_X86_VSMP is not set
# CONFIG_PARAVIRT_GUEST is not set
# CONFIG_MEMTEST_BOOTPARAM is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
CONFIG_MCORE2=y
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_CPU=y
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_INTERNODE_CACHE_BYTES=64
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_P6_NOP=y
CONFIG_X86_TSC=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
CONFIG_IOMMU_HELPER=y
CONFIG_SWIOTLB=y
CONFIG_NR_CPUS=2
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
CONFIG_PREEMPT_RCU=y
# CONFIG_RCU_TRACE is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_INTEL=y
# CONFIG_X86_MCE_AMD is not set
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
# CONFIG_NUMA is not set
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not set
# CONFIG_DISCONTIGMEM_MANUAL is not set
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
# CONFIG_SPARSEMEM_STATIC is not set
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y
# CONFIG_MEMORY_HOTPLUG is not set
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_RESOURCES_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_MTRR=y
CONFIG_X86_PAT=y
# CONFIG_EFI is not set
# CONFIG_SECCOMP is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000
CONFIG_SCHED_HRTICK=y
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x200000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x200000
# CONFIG_HOTPLUG_CPU is not set
# CONFIG_COMPAT_VDSO is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
#
# Power management options
#
CONFIG_PM=y
# CONFIG_PM_LEGACY is not set
# CONFIG_PM_DEBUG is not set
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
CONFIG_ACPI=y
# CONFIG_ACPI_PROCFS is not set
# CONFIG_ACPI_PROCFS_POWER is not set
# CONFIG_ACPI_SYSFS_POWER is not set
# CONFIG_ACPI_PROC_EVENT is not set
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
# CONFIG_ACPI_BUTTON is not set
# CONFIG_ACPI_FAN is not set
CONFIG_ACPI_DOCK=y
# CONFIG_ACPI_BAY is not set
# CONFIG_ACPI_PROCESSOR is not set
# CONFIG_ACPI_WMI is not set
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_TOSHIBA is not set
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
# CONFIG_X86_PM_TIMER is not set
# CONFIG_ACPI_CONTAINER is not set
# CONFIG_ACPI_SBS is not set
#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
# CONFIG_CPU_IDLE is not set
#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
# CONFIG_PCI_MMCONFIG is not set
CONFIG_PCI_DOMAINS=y
# CONFIG_DMAR is not set
# CONFIG_PCIEPORTBUS is not set
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
# CONFIG_PCI_LEGACY is not set
# CONFIG_PCI_DEBUG is not set
# CONFIG_HT_IRQ is not set
CONFIG_ISA_DMA_API=y
CONFIG_K8_NB=y
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set
#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
# CONFIG_BINFMT_MISC is not set
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
#
# Networking
#
CONFIG_NET=y
#
# Networking options
#
# CONFIG_PACKET is not set
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
# CONFIG_INET_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
# CONFIG_INET_LRO is not set
# CONFIG_INET_DIAG is not set
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IP_VS is not set
# CONFIG_IPV6 is not set
# CONFIG_NETWORK_SECMARK is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
CONFIG_NETFILTER_ADVANCED=y
#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=m
# CONFIG_NETFILTER_NETLINK_QUEUE is not set
# CONFIG_NETFILTER_NETLINK_LOG is not set
CONFIG_NF_CONNTRACK=m
CONFIG_NF_CT_ACCT=y
CONFIG_NF_CONNTRACK_MARK=y
# CONFIG_NF_CONNTRACK_EVENTS is not set
CONFIG_NF_CT_PROTO_DCCP=m
CONFIG_NF_CT_PROTO_GRE=m
# CONFIG_NF_CT_PROTO_SCTP is not set
CONFIG_NF_CT_PROTO_UDPLITE=m
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SANE=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_XTABLES=m
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_MARK=m
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_TARGET_RATEEST=m
CONFIG_NETFILTER_XT_TARGET_TRACE=m
CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=m
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=m
CONFIG_NETFILTER_XT_MATCH_OWNER=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_RATEEST=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=m
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_TIME=m
CONFIG_NETFILTER_XT_MATCH_U32=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
#
# IP: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV4=m
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_NF_NAT=m
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_DCCP=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_PROTO_UDPLITE=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_RAW=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
# CONFIG_IP_DCCP is not set
CONFIG_IP_SCTP=m
# CONFIG_SCTP_DBG_MSG is not set
# CONFIG_SCTP_DBG_OBJCNT is not set
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_SCHED is not set
CONFIG_NET_CLS_ROUTE=y
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_AF_RXRPC=m
# CONFIG_AF_RXRPC_DEBUG is not set
# CONFIG_RXKAD is not set
#
# Wireless
#
# CONFIG_CFG80211 is not set
# CONFIG_WIRELESS_EXT is not set
# CONFIG_MAC80211 is not set
# CONFIG_IEEE80211 is not set
# CONFIG_RFKILL is not set
# CONFIG_NET_9P is not set
#
# Device Drivers
#
#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
# CONFIG_FW_LOADER is not set
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_CONNECTOR is not set
# CONFIG_MTD is not set
# CONFIG_PARPORT is not set
CONFIG_PNP=y
# CONFIG_PNP_DEBUG is not set
#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
# CONFIG_BLK_DEV_RAM is not set
# CONFIG_CDROM_PKTCDVD is not set
# CONFIG_ATA_OVER_ETH is not set
# CONFIG_MISC_DEVICES is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set
#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
# CONFIG_SCSI_PROC_FS is not set
#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
# CONFIG_CHR_DEV_SG is not set
# CONFIG_CHR_DEV_SCH is not set
#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m
#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
# CONFIG_SCSI_LOWLEVEL is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
# CONFIG_SATA_PMP is not set
CONFIG_SATA_AHCI=y
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
# CONFIG_SATA_SVW is not set
CONFIG_ATA_PIIX=y
# CONFIG_SATA_MV is not set
# CONFIG_SATA_NV is not set
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_PATA_ACPI is not set
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_ATA_GENERIC is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
CONFIG_PATA_JMICRON=y
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_PLATFORM is not set
# CONFIG_MD is not set
# CONFIG_FUSION is not set
#
# IEEE 1394 (FireWire) support
#
# CONFIG_FIREWIRE is not set
# CONFIG_IEEE1394 is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
# CONFIG_NETDEVICES_MULTIQUEUE is not set
# CONFIG_DUMMY is not set
# CONFIG_BONDING is not set
# CONFIG_MACVLAN is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
# CONFIG_PHYLIB is not set
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNGEM is not set
# CONFIG_CASSINI is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_NET_TULIP is not set
# CONFIG_HP100 is not set
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_B44 is not set
# CONFIG_FORCEDETH is not set
# CONFIG_EEPRO100 is not set
# CONFIG_E100 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_8139CP is not set
CONFIG_8139TOO=y
CONFIG_8139TOO_PIO=y
# CONFIG_8139TOO_TUNE_TWISTER is not set
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_R6040 is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_SC92031 is not set
CONFIG_NETDEV_1000=y
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_E1000E is not set
# CONFIG_E1000E_ENABLED is not set
# CONFIG_IP1000 is not set
# CONFIG_IGB is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set
# CONFIG_QLA3XXX is not set
CONFIG_ATL1=y
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set
#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set
# CONFIG_IWLWIFI_LEDS is not set
#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
CONFIG_NETCONSOLE=y
# CONFIG_NETCONSOLE_DYNAMIC is not set
CONFIG_NETPOLL=y
# CONFIG_NETPOLL_TRAP is not set
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set
#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set
#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set
#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set
#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set
#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set
#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set
CONFIG_FIX_EARLYCON_MEM=y
#
# Non-8250 serial port support
#
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_NVRAM is not set
CONFIG_RTC=y
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
# CONFIG_HANGCHECK_TIMER is not set
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
# CONFIG_I2C is not set
#
# SPI support
#
# CONFIG_SPI is not set
# CONFIG_SPI_MASTER is not set
# CONFIG_W1 is not set
# CONFIG_POWER_SUPPLY is not set
# CONFIG_HWMON is not set
# CONFIG_THERMAL is not set
# CONFIG_WATCHDOG is not set
#
# Sonics Silicon Backplane
#
CONFIG_SSB_POSSIBLE=y
# CONFIG_SSB is not set
#
# Multifunction device drivers
#
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set
# CONFIG_DVB_CORE is not set
# CONFIG_DAB is not set
#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
# CONFIG_AGP_INTEL is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_DRM is not set
# CONFIG_VGASTATE is not set
# CONFIG_VIDEO_OUTPUT_CONTROL is not set
# CONFIG_FB is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set
#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set
#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
# CONFIG_VGACON_SOFT_SCROLLBACK is not set
# CONFIG_VIDEO_SELECT is not set
CONFIG_DUMMY_CONSOLE=y
#
# Sound
#
CONFIG_SOUND=y
#
# Advanced Linux Sound Architecture
#
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=y
CONFIG_SND_SEQ_DUMMY=y
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=y
CONFIG_SND_PCM_OSS=y
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_RTCTIMER=y
CONFIG_SND_SEQ_RTCTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
CONFIG_SND_SUPPORT_OLD_API=y
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
#
# Generic devices
#
# CONFIG_SND_PCSP is not set
CONFIG_SND_MPU401_UART=m
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_VIRMIDI is not set
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_SERIAL_U16550 is not set
CONFIG_SND_MPU401=m
#
# PCI devices
#
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
CONFIG_SND_HDA_INTEL=y
# CONFIG_SND_HDA_HWDEP is not set
CONFIG_SND_HDA_CODEC_REALTEK=y
CONFIG_SND_HDA_CODEC_ANALOG=y
CONFIG_SND_HDA_CODEC_SIGMATEL=y
CONFIG_SND_HDA_CODEC_VIA=y
CONFIG_SND_HDA_CODEC_ATIHDMI=y
CONFIG_SND_HDA_CODEC_CONEXANT=y
CONFIG_SND_HDA_CODEC_CMEDIA=y
CONFIG_SND_HDA_CODEC_SI3054=y
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_HIFIER is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
# CONFIG_SND_INTEL8X0 is not set
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
#
# USB devices
#
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set
# CONFIG_SND_USB_CAIAQ is not set
#
# System on Chip audio support
#
# CONFIG_SND_SOC is not set
#
# ALSA SoC audio for Freescale SOCs
#
#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HID_DEBUG is not set
# CONFIG_HIDRAW is not set
#
# USB Input Devices
#
CONFIG_USB_HID=m
# CONFIG_USB_HIDINPUT_POWERBOOK is not set
# CONFIG_HID_FF is not set
# CONFIG_USB_HIDDEV is not set
#
# USB HID Boot Protocol drivers
#
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=m
# CONFIG_USB_DEBUG is not set
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y
#
# Miscellaneous USB options
#
# CONFIG_USB_DEVICEFS is not set
# CONFIG_USB_DEVICE_CLASS is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
# CONFIG_USB_OTG_WHITELIST is not set
# CONFIG_USB_OTG_BLACKLIST_HUB is not set
#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_OHCI_HCD is not set
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
#
# USB Device Class drivers
#
# CONFIG_USB_ACM is not set
CONFIG_USB_PRINTER=m
#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support'
#
#
# may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_LIBUSUAL is not set
#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set
# CONFIG_USB_MON is not set
#
# USB port drivers
#
# CONFIG_USB_SERIAL is not set
#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_GADGET is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set
# CONFIG_UIO is not set
#
# Firmware Drivers
#
# CONFIG_EDD is not set
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
# CONFIG_DMIID is not set
# CONFIG_ISCSI_IBFT_FIND is not set
#
# File systems
#
CONFIG_EXT2_FS=m
# CONFIG_EXT2_FS_XATTR is not set
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=y
# CONFIG_EXT3_FS_XATTR is not set
# CONFIG_EXT4DEV_FS is not set
CONFIG_JBD=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
# CONFIG_FS_POSIX_ACL is not set
# CONFIG_XFS_FS is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
# CONFIG_DNOTIFY is not set
# CONFIG_INOTIFY is not set
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set
#
# CD-ROM/DVD Filesystems
#
# CONFIG_ISO9660_FS is not set
# CONFIG_UDF_FS is not set
#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
# CONFIG_MSDOS_FS is not set
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=866
CONFIG_FAT_DEFAULT_IOCHARSET="koi8-r"
# CONFIG_NTFS_FS is not set
#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
# CONFIG_PROC_KCORE is not set
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
# CONFIG_HUGETLBFS is not set
# CONFIG_HUGETLB_PAGE is not set
CONFIG_CONFIGFS_FS=m
#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_ECRYPT_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
CONFIG_NETWORK_FILESYSTEMS=y
# CONFIG_NFS_FS is not set
# CONFIG_NFSD is not set
# CONFIG_SMB_FS is not set
CONFIG_CIFS=m
# CONFIG_CIFS_STATS is not set
# CONFIG_CIFS_WEAK_PW_HASH is not set
# CONFIG_CIFS_XATTR is not set
# CONFIG_CIFS_DEBUG2 is not set
# CONFIG_CIFS_EXPERIMENTAL is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set
#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=m
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
CONFIG_NLS_CODEPAGE_855=m
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
CONFIG_NLS_CODEPAGE_866=m
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
CONFIG_NLS_ISO8859_5=m
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
CONFIG_NLS_KOI8_R=m
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=m
# CONFIG_DLM is not set
#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
# CONFIG_UNUSED_SYMBOLS is not set
# CONFIG_DEBUG_FS is not set
# CONFIG_HEADERS_CHECK is not set
CONFIG_DEBUG_KERNEL=y
CONFIG_DEBUG_SHIRQ=y
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_SCHED_DEBUG is not set
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
CONFIG_SLUB_DEBUG_ON=y
# CONFIG_SLUB_STATS is not set
CONFIG_DEBUG_PREEMPT=y
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_PI_LIST=y
# CONFIG_RT_MUTEX_TESTER is not set
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
CONFIG_LOCKDEP=y
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_LOCKDEP is not set
CONFIG_TRACE_IRQFLAGS=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
# CONFIG_DEBUG_INFO is not set
CONFIG_DEBUG_VM=y
CONFIG_DEBUG_WRITECOUNT=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_SG=y
CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_SAMPLES is not set
# CONFIG_KGDB is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_NONPROMISC_DEVMEM is not set
# CONFIG_EARLY_PRINTK is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_DEBUG_PER_CPU_MAPS=y
# CONFIG_X86_PTDUMP is not set
CONFIG_DEBUG_RODATA=y
# CONFIG_DIRECT_GBPAGES is not set
# CONFIG_DEBUG_RODATA_TEST is not set
# CONFIG_DEBUG_NX_TEST is not set
CONFIG_X86_MPPARSE=y
# CONFIG_IOMMU_DEBUG is not set
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_CPA_DEBUG=y
CONFIG_OPTIMIZE_INLINING=y
#
# Security options
#
CONFIG_KEYS=y
# CONFIG_KEYS_DEBUG_PROC_KEYS is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
CONFIG_CRYPTO=m
#
# Crypto core or helper
#
CONFIG_CRYPTO_ALGAPI=m
CONFIG_CRYPTO_HASH=m
CONFIG_CRYPTO_MANAGER=m
# CONFIG_CRYPTO_GF128MUL is not set
# CONFIG_CRYPTO_NULL is not set
# CONFIG_CRYPTO_CRYPTD is not set
# CONFIG_CRYPTO_AUTHENC is not set
# CONFIG_CRYPTO_TEST is not set
#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set
#
# Block modes
#
# CONFIG_CRYPTO_CBC is not set
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
# CONFIG_CRYPTO_ECB is not set
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_XTS is not set
#
# Hash modes
#
CONFIG_CRYPTO_HMAC=m
# CONFIG_CRYPTO_XCBC is not set
#
# Digest
#
# CONFIG_CRYPTO_CRC32C is not set
# CONFIG_CRYPTO_MD4 is not set
CONFIG_CRYPTO_MD5=m
# CONFIG_CRYPTO_MICHAEL_MIC is not set
CONFIG_CRYPTO_SHA1=m
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_WP512 is not set
#
# Ciphers
#
# CONFIG_CRYPTO_AES is not set
# CONFIG_CRYPTO_AES_X86_64 is not set
# CONFIG_CRYPTO_ANUBIS is not set
# CONFIG_CRYPTO_ARC4 is not set
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
# CONFIG_CRYPTO_DES is not set
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_TWOFISH_X86_64 is not set
#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_LZO is not set
# CONFIG_CRYPTO_HW is not set
CONFIG_HAVE_KVM=y
# CONFIG_VIRTUALIZATION is not set
#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
# CONFIG_CRC_CCITT is not set
CONFIG_CRC16=m
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=m
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_PLIST=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50
2008-04-27 22:55 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50 Alexey Dobriyan
@ 2008-04-28 12:01 ` Andrew Morton
2008-04-28 12:04 ` Jens Axboe
0 siblings, 1 reply; 37+ messages in thread
From: Andrew Morton @ 2008-04-28 12:01 UTC (permalink / raw)
To: Alexey Dobriyan; +Cc: axboe, linux-kernel
On Mon, 28 Apr 2008 02:55:53 +0400 Alexey Dobriyan <adobriyan@gmail.com> wrote:
> This happened while ~90 cross-compile jobs were running in parallel on
> ext2/noatime partition (slowly -- much debugging was on)
>
>
> general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> CPU 0
> Modules linked in: ext2 nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp uhci_hcd ehci_hcd usbcore sr_mod cdrom
> Pid: 16483, comm: as Not tainted 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48 #1
> RIP: 0010:[<ffffffff80307525>] [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> RSP: 0018:ffff810170811e58 EFLAGS: 00010202
> RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81010ff92000
> RBP: ffff810170811e78 R08: 0000000000000001 R09: 0000000000000000
> R10: 0000000000000000 R11: ffff8100010069d8 R12: ffff810138ada300
> R13: ffffffff803075b0 R14: ffff81017fcd2000 R15: ffff81010ff92168
> FS: 00002ac3462426f0(0000) GS:ffffffff805d0000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00002ab602550000 CR3: 000000013609d000 CR4: 0000000000000660
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process as (pid: 16483, threadinfo ffff810170810000, task ffff81010ff92000)
> Stack: ffff810170811e88 ffff810138ada300 0000000000000010 ffff81010ff92100
> ffff810170811e88 ffffffff80307580 ffff810170811ea8 ffffffff80302a55
> ffff81010ff92100 ffff810138ada300 ffff810170811ec8 ffffffff80302b1f
> Call Trace:
> [<ffffffff80307580>] cfq_free_io_context+0x10/0x20
> [<ffffffff80302a55>] put_io_context+0x85/0x90
> [<ffffffff80302b1f>] exit_io_context+0x8f/0xb0
> [<ffffffff80235d19>] do_exit+0x549/0x780
> [<ffffffff80235f8e>] do_group_exit+0x3e/0xb0
> [<ffffffff80236012>] sys_exit_group+0x12/0x20
> [<ffffffff8020b6db>] system_call_after_swapgs+0x7b/0x80
>
>
> Code: 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 e8 18 e1 f5 ff 49 8b 44 24 68 48 85 c0 74 1e 48 89 c3 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> RIP [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> RSP <ffff810170811e58>
> ---[ end trace ca143223eefdc828 ]---
> Fixing recursive fault but reboot is needed!
>
>
> ffffffff80307500 <call_for_each_cic>:
> ffffffff80307500: 55 push %rbp
> ffffffff80307501: 48 89 e5 mov %rsp,%rbp
> ffffffff80307504: 41 55 push %r13
> ffffffff80307506: 49 89 f5 mov %rsi,%r13
> ffffffff80307509: 41 54 push %r12
> ffffffff8030750b: 49 89 fc mov %rdi,%r12
> ffffffff8030750e: 53 push %rbx
> ffffffff8030750f: 48 83 ec 08 sub $0x8,%rsp
> ffffffff80307513: e8 18 e1 f5 ff callq ffffffff80265630 <__rcu_read_lock>
> ffffffff80307518: 49 8b 44 24 68 mov 0x68(%r12),%rax
> ffffffff8030751d: 48 85 c0 test %rax,%rax
> ffffffff80307520: 74 1e je ffffffff80307540 <call_for_each_cic+0x40>
> ffffffff80307522: 48 89 c3 mov %rax,%rbx
> ffffffff80307525: 48 8b 03 mov (%rbx),%rax
use-after-free.
> ffffffff80307528: 48 8d 73 88 lea -0x78(%rbx),%rsi
> ffffffff8030752c: 4c 89 e7 mov %r12,%rdi
> ffffffff8030752f: 0f 18 08 prefetcht0 (%rax)
> ffffffff80307532: 41 ff d5 callq *%r13
> ffffffff80307535: 48 8b 03 mov (%rbx),%rax
> ffffffff80307538: 48 85 c0 test %rax,%rax
> ffffffff8030753b: 48 89 c3 mov %rax,%rbx
> ffffffff8030753e: 75 e5 jne ffffffff80307525 <call_for_each_cic+0x25>
> ffffffff80307540: e8 2b e0 f5 ff callq ffffffff80265570 <__rcu_read_unlock>
> ffffffff80307545: 48 83 c4 08 add $0x8,%rsp
> ffffffff80307549: 5b pop %rbx
> ffffffff8030754a: 41 5c pop %r12
> ffffffff8030754c: 41 5d pop %r13
> ffffffff8030754e: c9 leaveq
> ffffffff8030754f: c3 retq
cfq-iosched.c hasn't been altered (yet) so it might not be a regression.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50
2008-04-28 12:01 ` Andrew Morton
@ 2008-04-28 12:04 ` Jens Axboe
2008-04-28 19:55 ` Alexey Dobriyan
0 siblings, 1 reply; 37+ messages in thread
From: Jens Axboe @ 2008-04-28 12:04 UTC (permalink / raw)
To: Andrew Morton; +Cc: Alexey Dobriyan, linux-kernel
On Mon, Apr 28 2008, Andrew Morton wrote:
> On Mon, 28 Apr 2008 02:55:53 +0400 Alexey Dobriyan <adobriyan@gmail.com> wrote:
>
> > This happened while ~90 cross-compile jobs were running in parallel on
> > ext2/noatime partition (slowly -- much debugging was on)
> >
> >
> > general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> > CPU 0
> > Modules linked in: ext2 nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp uhci_hcd ehci_hcd usbcore sr_mod cdrom
> > Pid: 16483, comm: as Not tainted 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48 #1
> > RIP: 0010:[<ffffffff80307525>] [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > RSP: 0018:ffff810170811e58 EFLAGS: 00010202
> > RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> > RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81010ff92000
> > RBP: ffff810170811e78 R08: 0000000000000001 R09: 0000000000000000
> > R10: 0000000000000000 R11: ffff8100010069d8 R12: ffff810138ada300
> > R13: ffffffff803075b0 R14: ffff81017fcd2000 R15: ffff81010ff92168
> > FS: 00002ac3462426f0(0000) GS:ffffffff805d0000(0000) knlGS:0000000000000000
> > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > CR2: 00002ab602550000 CR3: 000000013609d000 CR4: 0000000000000660
> > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > Process as (pid: 16483, threadinfo ffff810170810000, task ffff81010ff92000)
> > Stack: ffff810170811e88 ffff810138ada300 0000000000000010 ffff81010ff92100
> > ffff810170811e88 ffffffff80307580 ffff810170811ea8 ffffffff80302a55
> > ffff81010ff92100 ffff810138ada300 ffff810170811ec8 ffffffff80302b1f
> > Call Trace:
> > [<ffffffff80307580>] cfq_free_io_context+0x10/0x20
> > [<ffffffff80302a55>] put_io_context+0x85/0x90
> > [<ffffffff80302b1f>] exit_io_context+0x8f/0xb0
> > [<ffffffff80235d19>] do_exit+0x549/0x780
> > [<ffffffff80235f8e>] do_group_exit+0x3e/0xb0
> > [<ffffffff80236012>] sys_exit_group+0x12/0x20
> > [<ffffffff8020b6db>] system_call_after_swapgs+0x7b/0x80
> >
> >
> > Code: 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 e8 18 e1 f5 ff 49 8b 44 24 68 48 85 c0 74 1e 48 89 c3 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> > RIP [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > RSP <ffff810170811e58>
> > ---[ end trace ca143223eefdc828 ]---
> > Fixing recursive fault but reboot is needed!
> >
> >
> > ffffffff80307500 <call_for_each_cic>:
> > ffffffff80307500: 55 push %rbp
> > ffffffff80307501: 48 89 e5 mov %rsp,%rbp
> > ffffffff80307504: 41 55 push %r13
> > ffffffff80307506: 49 89 f5 mov %rsi,%r13
> > ffffffff80307509: 41 54 push %r12
> > ffffffff8030750b: 49 89 fc mov %rdi,%r12
> > ffffffff8030750e: 53 push %rbx
> > ffffffff8030750f: 48 83 ec 08 sub $0x8,%rsp
> > ffffffff80307513: e8 18 e1 f5 ff callq ffffffff80265630 <__rcu_read_lock>
> > ffffffff80307518: 49 8b 44 24 68 mov 0x68(%r12),%rax
> > ffffffff8030751d: 48 85 c0 test %rax,%rax
> > ffffffff80307520: 74 1e je ffffffff80307540 <call_for_each_cic+0x40>
> > ffffffff80307522: 48 89 c3 mov %rax,%rbx
> > ffffffff80307525: 48 8b 03 mov (%rbx),%rax
>
> use-after-free.
Yep, apparently a freed entry on the list. Not good...
> > ffffffff80307528: 48 8d 73 88 lea -0x78(%rbx),%rsi
> > ffffffff8030752c: 4c 89 e7 mov %r12,%rdi
> > ffffffff8030752f: 0f 18 08 prefetcht0 (%rax)
> > ffffffff80307532: 41 ff d5 callq *%r13
> > ffffffff80307535: 48 8b 03 mov (%rbx),%rax
> > ffffffff80307538: 48 85 c0 test %rax,%rax
> > ffffffff8030753b: 48 89 c3 mov %rax,%rbx
> > ffffffff8030753e: 75 e5 jne ffffffff80307525 <call_for_each_cic+0x25>
> > ffffffff80307540: e8 2b e0 f5 ff callq ffffffff80265570 <__rcu_read_unlock>
> > ffffffff80307545: 48 83 c4 08 add $0x8,%rsp
> > ffffffff80307549: 5b pop %rbx
> > ffffffff8030754a: 41 5c pop %r12
> > ffffffff8030754c: 41 5d pop %r13
> > ffffffff8030754e: c9 leaveq
> > ffffffff8030754f: c3 retq
>
> cfq-iosched.c hasn't been altered (yet) so it might not be a regression.
It's not a regression, it's definitely in 2.6.25 as well. So that's a
bit scary, I've been looking over this stuff this morning but haven't
pin pointed anything yet.
Alexey, is this something that reproduces for you?
--
Jens Axboe
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50
2008-04-28 12:04 ` Jens Axboe
@ 2008-04-28 19:55 ` Alexey Dobriyan
2008-04-29 6:21 ` Alexey Dobriyan
0 siblings, 1 reply; 37+ messages in thread
From: Alexey Dobriyan @ 2008-04-28 19:55 UTC (permalink / raw)
To: Jens Axboe; +Cc: Andrew Morton, linux-kernel
On Mon, Apr 28, 2008 at 02:04:13PM +0200, Jens Axboe wrote:
> On Mon, Apr 28 2008, Andrew Morton wrote:
> > On Mon, 28 Apr 2008 02:55:53 +0400 Alexey Dobriyan <adobriyan@gmail.com> wrote:
> >
> > > This happened while ~90 cross-compile jobs were running in parallel on
> > > ext2/noatime partition (slowly -- much debugging was on)
> > >
> > >
> > > general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> > > CPU 0
> > > Modules linked in: ext2 nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp uhci_hcd ehci_hcd usbcore sr_mod cdrom
> > > Pid: 16483, comm: as Not tainted 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48 #1
> > > RIP: 0010:[<ffffffff80307525>] [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > RSP: 0018:ffff810170811e58 EFLAGS: 00010202
> > > RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> > > RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81010ff92000
> > > RBP: ffff810170811e78 R08: 0000000000000001 R09: 0000000000000000
> > > R10: 0000000000000000 R11: ffff8100010069d8 R12: ffff810138ada300
> > > R13: ffffffff803075b0 R14: ffff81017fcd2000 R15: ffff81010ff92168
> > > FS: 00002ac3462426f0(0000) GS:ffffffff805d0000(0000) knlGS:0000000000000000
> > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > CR2: 00002ab602550000 CR3: 000000013609d000 CR4: 0000000000000660
> > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > Process as (pid: 16483, threadinfo ffff810170810000, task ffff81010ff92000)
> > > Stack: ffff810170811e88 ffff810138ada300 0000000000000010 ffff81010ff92100
> > > ffff810170811e88 ffffffff80307580 ffff810170811ea8 ffffffff80302a55
> > > ffff81010ff92100 ffff810138ada300 ffff810170811ec8 ffffffff80302b1f
> > > Call Trace:
> > > [<ffffffff80307580>] cfq_free_io_context+0x10/0x20
> > > [<ffffffff80302a55>] put_io_context+0x85/0x90
> > > [<ffffffff80302b1f>] exit_io_context+0x8f/0xb0
> > > [<ffffffff80235d19>] do_exit+0x549/0x780
> > > [<ffffffff80235f8e>] do_group_exit+0x3e/0xb0
> > > [<ffffffff80236012>] sys_exit_group+0x12/0x20
> > > [<ffffffff8020b6db>] system_call_after_swapgs+0x7b/0x80
> > >
> > >
> > > Code: 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 e8 18 e1 f5 ff 49 8b 44 24 68 48 85 c0 74 1e 48 89 c3 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> > > RIP [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > RSP <ffff810170811e58>
> > > ---[ end trace ca143223eefdc828 ]---
> > > Fixing recursive fault but reboot is needed!
> > >
> > >
> > > ffffffff80307500 <call_for_each_cic>:
> > > ffffffff80307500: 55 push %rbp
> > > ffffffff80307501: 48 89 e5 mov %rsp,%rbp
> > > ffffffff80307504: 41 55 push %r13
> > > ffffffff80307506: 49 89 f5 mov %rsi,%r13
> > > ffffffff80307509: 41 54 push %r12
> > > ffffffff8030750b: 49 89 fc mov %rdi,%r12
> > > ffffffff8030750e: 53 push %rbx
> > > ffffffff8030750f: 48 83 ec 08 sub $0x8,%rsp
> > > ffffffff80307513: e8 18 e1 f5 ff callq ffffffff80265630 <__rcu_read_lock>
> > > ffffffff80307518: 49 8b 44 24 68 mov 0x68(%r12),%rax
> > > ffffffff8030751d: 48 85 c0 test %rax,%rax
> > > ffffffff80307520: 74 1e je ffffffff80307540 <call_for_each_cic+0x40>
> > > ffffffff80307522: 48 89 c3 mov %rax,%rbx
> > > ffffffff80307525: 48 8b 03 mov (%rbx),%rax
> >
> > use-after-free.
>
> Yep, apparently a freed entry on the list. Not good...
>
> > > ffffffff80307528: 48 8d 73 88 lea -0x78(%rbx),%rsi
> > > ffffffff8030752c: 4c 89 e7 mov %r12,%rdi
> > > ffffffff8030752f: 0f 18 08 prefetcht0 (%rax)
> > > ffffffff80307532: 41 ff d5 callq *%r13
> > > ffffffff80307535: 48 8b 03 mov (%rbx),%rax
> > > ffffffff80307538: 48 85 c0 test %rax,%rax
> > > ffffffff8030753b: 48 89 c3 mov %rax,%rbx
> > > ffffffff8030753e: 75 e5 jne ffffffff80307525 <call_for_each_cic+0x25>
> > > ffffffff80307540: e8 2b e0 f5 ff callq ffffffff80265570 <__rcu_read_unlock>
> > > ffffffff80307545: 48 83 c4 08 add $0x8,%rsp
> > > ffffffff80307549: 5b pop %rbx
> > > ffffffff8030754a: 41 5c pop %r12
> > > ffffffff8030754c: 41 5d pop %r13
> > > ffffffff8030754e: c9 leaveq
> > > ffffffff8030754f: c3 retq
> >
> > cfq-iosched.c hasn't been altered (yet) so it might not be a regression.
>
> It's not a regression, it's definitely in 2.6.25 as well. So that's a
> bit scary, I've been looking over this stuff this morning but haven't
> pin pointed anything yet.
>
> Alexey, is this something that reproduces for you?
Not yet, second run of same workload went fine and I've never seen such
oopses before.
FWIW, the only unusual thing in .config is preemptible RCU. Before
2.6.25 it was turned off here.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50
2008-04-28 19:55 ` Alexey Dobriyan
@ 2008-04-29 6:21 ` Alexey Dobriyan
2008-04-29 9:06 ` Jens Axboe
0 siblings, 1 reply; 37+ messages in thread
From: Alexey Dobriyan @ 2008-04-29 6:21 UTC (permalink / raw)
To: Jens Axboe; +Cc: Andrew Morton, linux-kernel
On Mon, Apr 28, 2008 at 11:55:09PM +0400, Alexey Dobriyan wrote:
> On Mon, Apr 28, 2008 at 02:04:13PM +0200, Jens Axboe wrote:
> > On Mon, Apr 28 2008, Andrew Morton wrote:
> > > On Mon, 28 Apr 2008 02:55:53 +0400 Alexey Dobriyan <adobriyan@gmail.com> wrote:
> > >
> > > > This happened while ~90 cross-compile jobs were running in parallel on
> > > > ext2/noatime partition (slowly -- much debugging was on)
> > > >
> > > >
> > > > general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> > > > CPU 0
> > > > Modules linked in: ext2 nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp uhci_hcd ehci_hcd usbcore sr_mod cdrom
> > > > Pid: 16483, comm: as Not tainted 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48 #1
> > > > RIP: 0010:[<ffffffff80307525>] [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > RSP: 0018:ffff810170811e58 EFLAGS: 00010202
> > > > RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> > > > RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81010ff92000
> > > > RBP: ffff810170811e78 R08: 0000000000000001 R09: 0000000000000000
> > > > R10: 0000000000000000 R11: ffff8100010069d8 R12: ffff810138ada300
> > > > R13: ffffffff803075b0 R14: ffff81017fcd2000 R15: ffff81010ff92168
> > > > FS: 00002ac3462426f0(0000) GS:ffffffff805d0000(0000) knlGS:0000000000000000
> > > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > > CR2: 00002ab602550000 CR3: 000000013609d000 CR4: 0000000000000660
> > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > Process as (pid: 16483, threadinfo ffff810170810000, task ffff81010ff92000)
> > > > Stack: ffff810170811e88 ffff810138ada300 0000000000000010 ffff81010ff92100
> > > > ffff810170811e88 ffffffff80307580 ffff810170811ea8 ffffffff80302a55
> > > > ffff81010ff92100 ffff810138ada300 ffff810170811ec8 ffffffff80302b1f
> > > > Call Trace:
> > > > [<ffffffff80307580>] cfq_free_io_context+0x10/0x20
> > > > [<ffffffff80302a55>] put_io_context+0x85/0x90
> > > > [<ffffffff80302b1f>] exit_io_context+0x8f/0xb0
> > > > [<ffffffff80235d19>] do_exit+0x549/0x780
> > > > [<ffffffff80235f8e>] do_group_exit+0x3e/0xb0
> > > > [<ffffffff80236012>] sys_exit_group+0x12/0x20
> > > > [<ffffffff8020b6db>] system_call_after_swapgs+0x7b/0x80
> > > >
> > > >
> > > > Code: 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 e8 18 e1 f5 ff 49 8b 44 24 68 48 85 c0 74 1e 48 89 c3 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> > > > RIP [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > RSP <ffff810170811e58>
> > > > ---[ end trace ca143223eefdc828 ]---
> > > > Fixing recursive fault but reboot is needed!
> > > cfq-iosched.c hasn't been altered (yet) so it might not be a regression.
> > It's not a regression, it's definitely in 2.6.25 as well. So that's a
> > bit scary, I've been looking over this stuff this morning but haven't
> > pin pointed anything yet.
> >
> > Alexey, is this something that reproduces for you?
>
> Not yet, second run of same workload went fine and I've never seen such
> oopses before.
And it oopses the very same way on the third run. as(1) again.
So if there are any debugging patches, let me know.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50
2008-04-29 6:21 ` Alexey Dobriyan
@ 2008-04-29 9:06 ` Jens Axboe
2008-04-30 22:12 ` Alexey Dobriyan
0 siblings, 1 reply; 37+ messages in thread
From: Jens Axboe @ 2008-04-29 9:06 UTC (permalink / raw)
To: Alexey Dobriyan; +Cc: Andrew Morton, linux-kernel
On Tue, Apr 29 2008, Alexey Dobriyan wrote:
> On Mon, Apr 28, 2008 at 11:55:09PM +0400, Alexey Dobriyan wrote:
> > On Mon, Apr 28, 2008 at 02:04:13PM +0200, Jens Axboe wrote:
> > > On Mon, Apr 28 2008, Andrew Morton wrote:
> > > > On Mon, 28 Apr 2008 02:55:53 +0400 Alexey Dobriyan <adobriyan@gmail.com> wrote:
> > > >
> > > > > This happened while ~90 cross-compile jobs were running in parallel on
> > > > > ext2/noatime partition (slowly -- much debugging was on)
> > > > >
> > > > >
> > > > > general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> > > > > CPU 0
> > > > > Modules linked in: ext2 nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp uhci_hcd ehci_hcd usbcore sr_mod cdrom
> > > > > Pid: 16483, comm: as Not tainted 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48 #1
> > > > > RIP: 0010:[<ffffffff80307525>] [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > RSP: 0018:ffff810170811e58 EFLAGS: 00010202
> > > > > RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> > > > > RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81010ff92000
> > > > > RBP: ffff810170811e78 R08: 0000000000000001 R09: 0000000000000000
> > > > > R10: 0000000000000000 R11: ffff8100010069d8 R12: ffff810138ada300
> > > > > R13: ffffffff803075b0 R14: ffff81017fcd2000 R15: ffff81010ff92168
> > > > > FS: 00002ac3462426f0(0000) GS:ffffffff805d0000(0000) knlGS:0000000000000000
> > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > > > CR2: 00002ab602550000 CR3: 000000013609d000 CR4: 0000000000000660
> > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > > Process as (pid: 16483, threadinfo ffff810170810000, task ffff81010ff92000)
> > > > > Stack: ffff810170811e88 ffff810138ada300 0000000000000010 ffff81010ff92100
> > > > > ffff810170811e88 ffffffff80307580 ffff810170811ea8 ffffffff80302a55
> > > > > ffff81010ff92100 ffff810138ada300 ffff810170811ec8 ffffffff80302b1f
> > > > > Call Trace:
> > > > > [<ffffffff80307580>] cfq_free_io_context+0x10/0x20
> > > > > [<ffffffff80302a55>] put_io_context+0x85/0x90
> > > > > [<ffffffff80302b1f>] exit_io_context+0x8f/0xb0
> > > > > [<ffffffff80235d19>] do_exit+0x549/0x780
> > > > > [<ffffffff80235f8e>] do_group_exit+0x3e/0xb0
> > > > > [<ffffffff80236012>] sys_exit_group+0x12/0x20
> > > > > [<ffffffff8020b6db>] system_call_after_swapgs+0x7b/0x80
> > > > >
> > > > >
> > > > > Code: 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 e8 18 e1 f5 ff 49 8b 44 24 68 48 85 c0 74 1e 48 89 c3 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> > > > > RIP [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > RSP <ffff810170811e58>
> > > > > ---[ end trace ca143223eefdc828 ]---
> > > > > Fixing recursive fault but reboot is needed!
> > > > cfq-iosched.c hasn't been altered (yet) so it might not be a regression.
>
>
> > > It's not a regression, it's definitely in 2.6.25 as well. So that's a
> > > bit scary, I've been looking over this stuff this morning but haven't
> > > pin pointed anything yet.
> > >
> > > Alexey, is this something that reproduces for you?
> >
> > Not yet, second run of same workload went fine and I've never seen such
> > oopses before.
>
> And it oopses the very same way on the third run. as(1) again.
> So if there are any debugging patches, let me know.
There seems to be a small race in the destructor path, can you see if
this makes a difference?
diff --git a/block/blk-ioc.c b/block/blk-ioc.c
index e34df7c..012f065 100644
--- a/block/blk-ioc.c
+++ b/block/blk-ioc.c
@@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
rcu_read_lock();
if (ioc->aic && ioc->aic->dtor)
ioc->aic->dtor(ioc->aic);
- rcu_read_unlock();
cfq_dtor(ioc);
+ rcu_read_unlock();
kmem_cache_free(iocontext_cachep, ioc);
return 1;
--
Jens Axboe
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50
2008-04-29 9:06 ` Jens Axboe
@ 2008-04-30 22:12 ` Alexey Dobriyan
2008-05-04 19:08 ` Jens Axboe
0 siblings, 1 reply; 37+ messages in thread
From: Alexey Dobriyan @ 2008-04-30 22:12 UTC (permalink / raw)
To: Jens Axboe; +Cc: Andrew Morton, linux-kernel
On Tue, Apr 29, 2008 at 11:06:05AM +0200, Jens Axboe wrote:
> On Tue, Apr 29 2008, Alexey Dobriyan wrote:
> > On Mon, Apr 28, 2008 at 11:55:09PM +0400, Alexey Dobriyan wrote:
> > > On Mon, Apr 28, 2008 at 02:04:13PM +0200, Jens Axboe wrote:
> > > > On Mon, Apr 28 2008, Andrew Morton wrote:
> > > > > On Mon, 28 Apr 2008 02:55:53 +0400 Alexey Dobriyan <adobriyan@gmail.com> wrote:
> > > > >
> > > > > > This happened while ~90 cross-compile jobs were running in parallel on
> > > > > > ext2/noatime partition (slowly -- much debugging was on)
> > > > > >
> > > > > >
> > > > > > general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> > > > > > CPU 0
> > > > > > Modules linked in: ext2 nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp uhci_hcd ehci_hcd usbcore sr_mod cdrom
> > > > > > Pid: 16483, comm: as Not tainted 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48 #1
> > > > > > RIP: 0010:[<ffffffff80307525>] [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > > RSP: 0018:ffff810170811e58 EFLAGS: 00010202
> > > > > > RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> > > > > > RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81010ff92000
> > > > > > RBP: ffff810170811e78 R08: 0000000000000001 R09: 0000000000000000
> > > > > > R10: 0000000000000000 R11: ffff8100010069d8 R12: ffff810138ada300
> > > > > > R13: ffffffff803075b0 R14: ffff81017fcd2000 R15: ffff81010ff92168
> > > > > > FS: 00002ac3462426f0(0000) GS:ffffffff805d0000(0000) knlGS:0000000000000000
> > > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > > > > CR2: 00002ab602550000 CR3: 000000013609d000 CR4: 0000000000000660
> > > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > > > Process as (pid: 16483, threadinfo ffff810170810000, task ffff81010ff92000)
> > > > > > Stack: ffff810170811e88 ffff810138ada300 0000000000000010 ffff81010ff92100
> > > > > > ffff810170811e88 ffffffff80307580 ffff810170811ea8 ffffffff80302a55
> > > > > > ffff81010ff92100 ffff810138ada300 ffff810170811ec8 ffffffff80302b1f
> > > > > > Call Trace:
> > > > > > [<ffffffff80307580>] cfq_free_io_context+0x10/0x20
> > > > > > [<ffffffff80302a55>] put_io_context+0x85/0x90
> > > > > > [<ffffffff80302b1f>] exit_io_context+0x8f/0xb0
> > > > > > [<ffffffff80235d19>] do_exit+0x549/0x780
> > > > > > [<ffffffff80235f8e>] do_group_exit+0x3e/0xb0
> > > > > > [<ffffffff80236012>] sys_exit_group+0x12/0x20
> > > > > > [<ffffffff8020b6db>] system_call_after_swapgs+0x7b/0x80
> > > > > >
> > > > > >
> > > > > > Code: 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 e8 18 e1 f5 ff 49 8b 44 24 68 48 85 c0 74 1e 48 89 c3 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> > > > > > RIP [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > > RSP <ffff810170811e58>
> > > > > > ---[ end trace ca143223eefdc828 ]---
> > > > > > Fixing recursive fault but reboot is needed!
> > > > > cfq-iosched.c hasn't been altered (yet) so it might not be a regression.
> >
> >
> > > > It's not a regression, it's definitely in 2.6.25 as well. So that's a
> > > > bit scary, I've been looking over this stuff this morning but haven't
> > > > pin pointed anything yet.
> > > >
> > > > Alexey, is this something that reproduces for you?
> > >
> > > Not yet, second run of same workload went fine and I've never seen such
> > > oopses before.
> >
> > And it oopses the very same way on the third run. as(1) again.
> > So if there are any debugging patches, let me know.
>
> There seems to be a small race in the destructor path, can you see if
> this makes a difference?
>
> diff --git a/block/blk-ioc.c b/block/blk-ioc.c
> index e34df7c..012f065 100644
> --- a/block/blk-ioc.c
> +++ b/block/blk-ioc.c
> @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> rcu_read_lock();
> if (ioc->aic && ioc->aic->dtor)
> ioc->aic->dtor(ioc->aic);
> - rcu_read_unlock();
> cfq_dtor(ioc);
> + rcu_read_unlock();
>
> kmem_cache_free(iocontext_cachep, ioc);
> return 1;
This helps in sense that 3 times bulk cross-compiles finish to the end.
You'll hear me if another such oops will resurface.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50
2008-04-30 22:12 ` Alexey Dobriyan
@ 2008-05-04 19:08 ` Jens Axboe
2008-05-04 20:15 ` Alexey Dobriyan
0 siblings, 1 reply; 37+ messages in thread
From: Jens Axboe @ 2008-05-04 19:08 UTC (permalink / raw)
To: Alexey Dobriyan; +Cc: Andrew Morton, linux-kernel
On Thu, May 01 2008, Alexey Dobriyan wrote:
> On Tue, Apr 29, 2008 at 11:06:05AM +0200, Jens Axboe wrote:
> > On Tue, Apr 29 2008, Alexey Dobriyan wrote:
> > > On Mon, Apr 28, 2008 at 11:55:09PM +0400, Alexey Dobriyan wrote:
> > > > On Mon, Apr 28, 2008 at 02:04:13PM +0200, Jens Axboe wrote:
> > > > > On Mon, Apr 28 2008, Andrew Morton wrote:
> > > > > > On Mon, 28 Apr 2008 02:55:53 +0400 Alexey Dobriyan <adobriyan@gmail.com> wrote:
> > > > > >
> > > > > > > This happened while ~90 cross-compile jobs were running in parallel on
> > > > > > > ext2/noatime partition (slowly -- much debugging was on)
> > > > > > >
> > > > > > >
> > > > > > > general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> > > > > > > CPU 0
> > > > > > > Modules linked in: ext2 nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp uhci_hcd ehci_hcd usbcore sr_mod cdrom
> > > > > > > Pid: 16483, comm: as Not tainted 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48 #1
> > > > > > > RIP: 0010:[<ffffffff80307525>] [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > > > RSP: 0018:ffff810170811e58 EFLAGS: 00010202
> > > > > > > RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> > > > > > > RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81010ff92000
> > > > > > > RBP: ffff810170811e78 R08: 0000000000000001 R09: 0000000000000000
> > > > > > > R10: 0000000000000000 R11: ffff8100010069d8 R12: ffff810138ada300
> > > > > > > R13: ffffffff803075b0 R14: ffff81017fcd2000 R15: ffff81010ff92168
> > > > > > > FS: 00002ac3462426f0(0000) GS:ffffffff805d0000(0000) knlGS:0000000000000000
> > > > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > > > > > CR2: 00002ab602550000 CR3: 000000013609d000 CR4: 0000000000000660
> > > > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > > > > Process as (pid: 16483, threadinfo ffff810170810000, task ffff81010ff92000)
> > > > > > > Stack: ffff810170811e88 ffff810138ada300 0000000000000010 ffff81010ff92100
> > > > > > > ffff810170811e88 ffffffff80307580 ffff810170811ea8 ffffffff80302a55
> > > > > > > ffff81010ff92100 ffff810138ada300 ffff810170811ec8 ffffffff80302b1f
> > > > > > > Call Trace:
> > > > > > > [<ffffffff80307580>] cfq_free_io_context+0x10/0x20
> > > > > > > [<ffffffff80302a55>] put_io_context+0x85/0x90
> > > > > > > [<ffffffff80302b1f>] exit_io_context+0x8f/0xb0
> > > > > > > [<ffffffff80235d19>] do_exit+0x549/0x780
> > > > > > > [<ffffffff80235f8e>] do_group_exit+0x3e/0xb0
> > > > > > > [<ffffffff80236012>] sys_exit_group+0x12/0x20
> > > > > > > [<ffffffff8020b6db>] system_call_after_swapgs+0x7b/0x80
> > > > > > >
> > > > > > >
> > > > > > > Code: 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 e8 18 e1 f5 ff 49 8b 44 24 68 48 85 c0 74 1e 48 89 c3 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> > > > > > > RIP [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > > > RSP <ffff810170811e58>
> > > > > > > ---[ end trace ca143223eefdc828 ]---
> > > > > > > Fixing recursive fault but reboot is needed!
> > > > > > cfq-iosched.c hasn't been altered (yet) so it might not be a regression.
> > >
> > >
> > > > > It's not a regression, it's definitely in 2.6.25 as well. So that's a
> > > > > bit scary, I've been looking over this stuff this morning but haven't
> > > > > pin pointed anything yet.
> > > > >
> > > > > Alexey, is this something that reproduces for you?
> > > >
> > > > Not yet, second run of same workload went fine and I've never seen such
> > > > oopses before.
> > >
> > > And it oopses the very same way on the third run. as(1) again.
> > > So if there are any debugging patches, let me know.
> >
> > There seems to be a small race in the destructor path, can you see if
> > this makes a difference?
> >
> > diff --git a/block/blk-ioc.c b/block/blk-ioc.c
> > index e34df7c..012f065 100644
> > --- a/block/blk-ioc.c
> > +++ b/block/blk-ioc.c
> > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > rcu_read_lock();
> > if (ioc->aic && ioc->aic->dtor)
> > ioc->aic->dtor(ioc->aic);
> > - rcu_read_unlock();
> > cfq_dtor(ioc);
> > + rcu_read_unlock();
> >
> > kmem_cache_free(iocontext_cachep, ioc);
> > return 1;
>
> This helps in sense that 3 times bulk cross-compiles finish to the end.
> You'll hear me if another such oops will resurface.
Still looking good?
--
Jens Axboe
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50
2008-05-04 20:15 ` Alexey Dobriyan
@ 2008-05-04 19:25 ` Jens Axboe
2008-05-04 21:17 ` Alexey Dobriyan
2008-05-10 10:37 ` 2.6.25-$sha1: RIP __call_for_each_cic+0x20/0x50 Alexey Dobriyan
1 sibling, 1 reply; 37+ messages in thread
From: Jens Axboe @ 2008-05-04 19:25 UTC (permalink / raw)
To: Alexey Dobriyan; +Cc: Andrew Morton, linux-kernel
On Mon, May 05 2008, Alexey Dobriyan wrote:
> On Sun, May 04, 2008 at 09:08:11PM +0200, Jens Axboe wrote:
> > On Thu, May 01 2008, Alexey Dobriyan wrote:
> > > On Tue, Apr 29, 2008 at 11:06:05AM +0200, Jens Axboe wrote:
> > > > On Tue, Apr 29 2008, Alexey Dobriyan wrote:
> > > > > On Mon, Apr 28, 2008 at 11:55:09PM +0400, Alexey Dobriyan wrote:
> > > > > > On Mon, Apr 28, 2008 at 02:04:13PM +0200, Jens Axboe wrote:
> > > > > > > On Mon, Apr 28 2008, Andrew Morton wrote:
> > > > > > > > On Mon, 28 Apr 2008 02:55:53 +0400 Alexey Dobriyan <adobriyan@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > This happened while ~90 cross-compile jobs were running in parallel on
> > > > > > > > > ext2/noatime partition (slowly -- much debugging was on)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> > > > > > > > > CPU 0
> > > > > > > > > Modules linked in: ext2 nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp uhci_hcd ehci_hcd usbcore sr_mod cdrom
> > > > > > > > > Pid: 16483, comm: as Not tainted 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48 #1
> > > > > > > > > RIP: 0010:[<ffffffff80307525>] [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > > > > > RSP: 0018:ffff810170811e58 EFLAGS: 00010202
> > > > > > > > > RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> > > > > > > > > RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81010ff92000
> > > > > > > > > RBP: ffff810170811e78 R08: 0000000000000001 R09: 0000000000000000
> > > > > > > > > R10: 0000000000000000 R11: ffff8100010069d8 R12: ffff810138ada300
> > > > > > > > > R13: ffffffff803075b0 R14: ffff81017fcd2000 R15: ffff81010ff92168
> > > > > > > > > FS: 00002ac3462426f0(0000) GS:ffffffff805d0000(0000) knlGS:0000000000000000
> > > > > > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > > > > > > > CR2: 00002ab602550000 CR3: 000000013609d000 CR4: 0000000000000660
> > > > > > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > > > > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > > > > > > Process as (pid: 16483, threadinfo ffff810170810000, task ffff81010ff92000)
> > > > > > > > > Stack: ffff810170811e88 ffff810138ada300 0000000000000010 ffff81010ff92100
> > > > > > > > > ffff810170811e88 ffffffff80307580 ffff810170811ea8 ffffffff80302a55
> > > > > > > > > ffff81010ff92100 ffff810138ada300 ffff810170811ec8 ffffffff80302b1f
> > > > > > > > > Call Trace:
> > > > > > > > > [<ffffffff80307580>] cfq_free_io_context+0x10/0x20
> > > > > > > > > [<ffffffff80302a55>] put_io_context+0x85/0x90
> > > > > > > > > [<ffffffff80302b1f>] exit_io_context+0x8f/0xb0
> > > > > > > > > [<ffffffff80235d19>] do_exit+0x549/0x780
> > > > > > > > > [<ffffffff80235f8e>] do_group_exit+0x3e/0xb0
> > > > > > > > > [<ffffffff80236012>] sys_exit_group+0x12/0x20
> > > > > > > > > [<ffffffff8020b6db>] system_call_after_swapgs+0x7b/0x80
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Code: 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 e8 18 e1 f5 ff 49 8b 44 24 68 48 85 c0 74 1e 48 89 c3 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> > > > > > > > > RIP [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > > > > > RSP <ffff810170811e58>
> > > > > > > > > ---[ end trace ca143223eefdc828 ]---
> > > > > > > > > Fixing recursive fault but reboot is needed!
> > > > > > > > cfq-iosched.c hasn't been altered (yet) so it might not be a regression.
> > > > >
> > > > >
> > > > > > > It's not a regression, it's definitely in 2.6.25 as well. So that's a
> > > > > > > bit scary, I've been looking over this stuff this morning but haven't
> > > > > > > pin pointed anything yet.
> > > > > > >
> > > > > > > Alexey, is this something that reproduces for you?
> > > > > >
> > > > > > Not yet, second run of same workload went fine and I've never seen such
> > > > > > oopses before.
> > > > >
> > > > > And it oopses the very same way on the third run. as(1) again.
> > > > > So if there are any debugging patches, let me know.
> > > >
> > > > There seems to be a small race in the destructor path, can you see if
> > > > this makes a difference?
> > > >
> > > > diff --git a/block/blk-ioc.c b/block/blk-ioc.c
> > > > index e34df7c..012f065 100644
> > > > --- a/block/blk-ioc.c
> > > > +++ b/block/blk-ioc.c
> > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > rcu_read_lock();
> > > > if (ioc->aic && ioc->aic->dtor)
> > > > ioc->aic->dtor(ioc->aic);
> > > > - rcu_read_unlock();
> > > > cfq_dtor(ioc);
> > > > + rcu_read_unlock();
> > > >
> > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > return 1;
> > >
> > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > You'll hear me if another such oops will resurface.
> >
> > Still looking good?
>
> Yup!
Great! I'll clean the path up and submit a patch for 2.6.26-rc1 as well
as 2.6.25.stable. Thanks a lot for reporting and testing.
--
Jens Axboe
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50
2008-05-04 19:08 ` Jens Axboe
@ 2008-05-04 20:15 ` Alexey Dobriyan
2008-05-04 19:25 ` Jens Axboe
2008-05-10 10:37 ` 2.6.25-$sha1: RIP __call_for_each_cic+0x20/0x50 Alexey Dobriyan
0 siblings, 2 replies; 37+ messages in thread
From: Alexey Dobriyan @ 2008-05-04 20:15 UTC (permalink / raw)
To: Jens Axboe; +Cc: Andrew Morton, linux-kernel
On Sun, May 04, 2008 at 09:08:11PM +0200, Jens Axboe wrote:
> On Thu, May 01 2008, Alexey Dobriyan wrote:
> > On Tue, Apr 29, 2008 at 11:06:05AM +0200, Jens Axboe wrote:
> > > On Tue, Apr 29 2008, Alexey Dobriyan wrote:
> > > > On Mon, Apr 28, 2008 at 11:55:09PM +0400, Alexey Dobriyan wrote:
> > > > > On Mon, Apr 28, 2008 at 02:04:13PM +0200, Jens Axboe wrote:
> > > > > > On Mon, Apr 28 2008, Andrew Morton wrote:
> > > > > > > On Mon, 28 Apr 2008 02:55:53 +0400 Alexey Dobriyan <adobriyan@gmail.com> wrote:
> > > > > > >
> > > > > > > > This happened while ~90 cross-compile jobs were running in parallel on
> > > > > > > > ext2/noatime partition (slowly -- much debugging was on)
> > > > > > > >
> > > > > > > >
> > > > > > > > general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> > > > > > > > CPU 0
> > > > > > > > Modules linked in: ext2 nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp uhci_hcd ehci_hcd usbcore sr_mod cdrom
> > > > > > > > Pid: 16483, comm: as Not tainted 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48 #1
> > > > > > > > RIP: 0010:[<ffffffff80307525>] [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > > > > RSP: 0018:ffff810170811e58 EFLAGS: 00010202
> > > > > > > > RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> > > > > > > > RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81010ff92000
> > > > > > > > RBP: ffff810170811e78 R08: 0000000000000001 R09: 0000000000000000
> > > > > > > > R10: 0000000000000000 R11: ffff8100010069d8 R12: ffff810138ada300
> > > > > > > > R13: ffffffff803075b0 R14: ffff81017fcd2000 R15: ffff81010ff92168
> > > > > > > > FS: 00002ac3462426f0(0000) GS:ffffffff805d0000(0000) knlGS:0000000000000000
> > > > > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > > > > > > CR2: 00002ab602550000 CR3: 000000013609d000 CR4: 0000000000000660
> > > > > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > > > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > > > > > Process as (pid: 16483, threadinfo ffff810170810000, task ffff81010ff92000)
> > > > > > > > Stack: ffff810170811e88 ffff810138ada300 0000000000000010 ffff81010ff92100
> > > > > > > > ffff810170811e88 ffffffff80307580 ffff810170811ea8 ffffffff80302a55
> > > > > > > > ffff81010ff92100 ffff810138ada300 ffff810170811ec8 ffffffff80302b1f
> > > > > > > > Call Trace:
> > > > > > > > [<ffffffff80307580>] cfq_free_io_context+0x10/0x20
> > > > > > > > [<ffffffff80302a55>] put_io_context+0x85/0x90
> > > > > > > > [<ffffffff80302b1f>] exit_io_context+0x8f/0xb0
> > > > > > > > [<ffffffff80235d19>] do_exit+0x549/0x780
> > > > > > > > [<ffffffff80235f8e>] do_group_exit+0x3e/0xb0
> > > > > > > > [<ffffffff80236012>] sys_exit_group+0x12/0x20
> > > > > > > > [<ffffffff8020b6db>] system_call_after_swapgs+0x7b/0x80
> > > > > > > >
> > > > > > > >
> > > > > > > > Code: 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 e8 18 e1 f5 ff 49 8b 44 24 68 48 85 c0 74 1e 48 89 c3 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> > > > > > > > RIP [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > > > > RSP <ffff810170811e58>
> > > > > > > > ---[ end trace ca143223eefdc828 ]---
> > > > > > > > Fixing recursive fault but reboot is needed!
> > > > > > > cfq-iosched.c hasn't been altered (yet) so it might not be a regression.
> > > >
> > > >
> > > > > > It's not a regression, it's definitely in 2.6.25 as well. So that's a
> > > > > > bit scary, I've been looking over this stuff this morning but haven't
> > > > > > pin pointed anything yet.
> > > > > >
> > > > > > Alexey, is this something that reproduces for you?
> > > > >
> > > > > Not yet, second run of same workload went fine and I've never seen such
> > > > > oopses before.
> > > >
> > > > And it oopses the very same way on the third run. as(1) again.
> > > > So if there are any debugging patches, let me know.
> > >
> > > There seems to be a small race in the destructor path, can you see if
> > > this makes a difference?
> > >
> > > diff --git a/block/blk-ioc.c b/block/blk-ioc.c
> > > index e34df7c..012f065 100644
> > > --- a/block/blk-ioc.c
> > > +++ b/block/blk-ioc.c
> > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > rcu_read_lock();
> > > if (ioc->aic && ioc->aic->dtor)
> > > ioc->aic->dtor(ioc->aic);
> > > - rcu_read_unlock();
> > > cfq_dtor(ioc);
> > > + rcu_read_unlock();
> > >
> > > kmem_cache_free(iocontext_cachep, ioc);
> > > return 1;
> >
> > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > You'll hear me if another such oops will resurface.
>
> Still looking good?
Yup!
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50
2008-05-04 19:25 ` Jens Axboe
@ 2008-05-04 21:17 ` Alexey Dobriyan
0 siblings, 0 replies; 37+ messages in thread
From: Alexey Dobriyan @ 2008-05-04 21:17 UTC (permalink / raw)
To: Jens Axboe; +Cc: Andrew Morton, linux-kernel
On Sun, May 04, 2008 at 09:25:00PM +0200, Jens Axboe wrote:
> On Mon, May 05 2008, Alexey Dobriyan wrote:
> > On Sun, May 04, 2008 at 09:08:11PM +0200, Jens Axboe wrote:
> > > On Thu, May 01 2008, Alexey Dobriyan wrote:
> > > > On Tue, Apr 29, 2008 at 11:06:05AM +0200, Jens Axboe wrote:
> > > > > On Tue, Apr 29 2008, Alexey Dobriyan wrote:
> > > > > > On Mon, Apr 28, 2008 at 11:55:09PM +0400, Alexey Dobriyan wrote:
> > > > > > > On Mon, Apr 28, 2008 at 02:04:13PM +0200, Jens Axboe wrote:
> > > > > > > > On Mon, Apr 28 2008, Andrew Morton wrote:
> > > > > > > > > On Mon, 28 Apr 2008 02:55:53 +0400 Alexey Dobriyan <adobriyan@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > This happened while ~90 cross-compile jobs were running in parallel on
> > > > > > > > > > ext2/noatime partition (slowly -- much debugging was on)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> > > > > > > > > > CPU 0
> > > > > > > > > > Modules linked in: ext2 nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp uhci_hcd ehci_hcd usbcore sr_mod cdrom
> > > > > > > > > > Pid: 16483, comm: as Not tainted 2.6.25-c3bf9bc243092c53946fd6d8ebd6dc2f4e572d48 #1
> > > > > > > > > > RIP: 0010:[<ffffffff80307525>] [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > > > > > > RSP: 0018:ffff810170811e58 EFLAGS: 00010202
> > > > > > > > > > RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> > > > > > > > > > RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff81010ff92000
> > > > > > > > > > RBP: ffff810170811e78 R08: 0000000000000001 R09: 0000000000000000
> > > > > > > > > > R10: 0000000000000000 R11: ffff8100010069d8 R12: ffff810138ada300
> > > > > > > > > > R13: ffffffff803075b0 R14: ffff81017fcd2000 R15: ffff81010ff92168
> > > > > > > > > > FS: 00002ac3462426f0(0000) GS:ffffffff805d0000(0000) knlGS:0000000000000000
> > > > > > > > > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > > > > > > > > > CR2: 00002ab602550000 CR3: 000000013609d000 CR4: 0000000000000660
> > > > > > > > > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > > > > > > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > > > > > > > Process as (pid: 16483, threadinfo ffff810170810000, task ffff81010ff92000)
> > > > > > > > > > Stack: ffff810170811e88 ffff810138ada300 0000000000000010 ffff81010ff92100
> > > > > > > > > > ffff810170811e88 ffffffff80307580 ffff810170811ea8 ffffffff80302a55
> > > > > > > > > > ffff81010ff92100 ffff810138ada300 ffff810170811ec8 ffffffff80302b1f
> > > > > > > > > > Call Trace:
> > > > > > > > > > [<ffffffff80307580>] cfq_free_io_context+0x10/0x20
> > > > > > > > > > [<ffffffff80302a55>] put_io_context+0x85/0x90
> > > > > > > > > > [<ffffffff80302b1f>] exit_io_context+0x8f/0xb0
> > > > > > > > > > [<ffffffff80235d19>] do_exit+0x549/0x780
> > > > > > > > > > [<ffffffff80235f8e>] do_group_exit+0x3e/0xb0
> > > > > > > > > > [<ffffffff80236012>] sys_exit_group+0x12/0x20
> > > > > > > > > > [<ffffffff8020b6db>] system_call_after_swapgs+0x7b/0x80
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Code: 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 e8 18 e1 f5 ff 49 8b 44 24 68 48 85 c0 74 1e 48 89 c3 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> > > > > > > > > > RIP [<ffffffff80307525>] call_for_each_cic+0x25/0x50
> > > > > > > > > > RSP <ffff810170811e58>
> > > > > > > > > > ---[ end trace ca143223eefdc828 ]---
> > > > > > > > > > Fixing recursive fault but reboot is needed!
> > > > > > > > > cfq-iosched.c hasn't been altered (yet) so it might not be a regression.
> > > > > >
> > > > > >
> > > > > > > > It's not a regression, it's definitely in 2.6.25 as well. So that's a
> > > > > > > > bit scary, I've been looking over this stuff this morning but haven't
> > > > > > > > pin pointed anything yet.
> > > > > > > >
> > > > > > > > Alexey, is this something that reproduces for you?
> > > > > > >
> > > > > > > Not yet, second run of same workload went fine and I've never seen such
> > > > > > > oopses before.
> > > > > >
> > > > > > And it oopses the very same way on the third run. as(1) again.
> > > > > > So if there are any debugging patches, let me know.
> > > > >
> > > > > There seems to be a small race in the destructor path, can you see if
> > > > > this makes a difference?
> > > > >
> > > > > diff --git a/block/blk-ioc.c b/block/blk-ioc.c
> > > > > index e34df7c..012f065 100644
> > > > > --- a/block/blk-ioc.c
> > > > > +++ b/block/blk-ioc.c
> > > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > > rcu_read_lock();
> > > > > if (ioc->aic && ioc->aic->dtor)
> > > > > ioc->aic->dtor(ioc->aic);
> > > > > - rcu_read_unlock();
> > > > > cfq_dtor(ioc);
> > > > > + rcu_read_unlock();
> > > > >
> > > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > > return 1;
> > > >
> > > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > > You'll hear me if another such oops will resurface.
> > >
> > > Still looking good?
> >
> > Yup!
>
> Great! I'll clean the path up and submit a patch for 2.6.26-rc1 as well
> as 2.6.25.stable. Thanks a lot for reporting and testing.
And please include side-by-side diagram in changelog :^!
Alexey, who still doesn't quite got it.
^ permalink raw reply [flat|nested] 37+ messages in thread
* 2.6.25-$sha1: RIP __call_for_each_cic+0x20/0x50
2008-05-04 20:15 ` Alexey Dobriyan
2008-05-04 19:25 ` Jens Axboe
@ 2008-05-10 10:37 ` Alexey Dobriyan
2008-05-27 5:27 ` 2.6.26-rc4: " Alexey Dobriyan
1 sibling, 1 reply; 37+ messages in thread
From: Alexey Dobriyan @ 2008-05-10 10:37 UTC (permalink / raw)
To: Jens Axboe; +Cc: Andrew Morton, linux-kernel
> > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > rcu_read_lock();
> > > > if (ioc->aic && ioc->aic->dtor)
> > > > ioc->aic->dtor(ioc->aic);
> > > > - rcu_read_unlock();
> > > > cfq_dtor(ioc);
> > > > + rcu_read_unlock();
> > > >
> > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > return 1;
> > >
> > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > You'll hear me if another such oops will resurface.
> >
> > Still looking good?
>
> Yup!
And this with patch in mainline, again with PREEMPT_RCU.
general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
CPU 0
Modules linked in: ext2 nls_utf8 cifs nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp ehci_hcd uhci_hcd usbcore sr_mod cdrom
Pid: 26971, comm: fixdep Not tainted 2.6.26-rc1-28a4acb48586dc21d2d14a75a7aab7be78b7c83b #4
RIP: 0010:[<ffffffff80304b30>] [<ffffffff80304b30>] __call_for_each_cic+0x20/0x50
RSP: 0018:ffff810104335e58 EFLAGS: 00010202
RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
RDX: 0000000000002827 RSI: 0000000000000000 RDI: ffff810105a62280
RBP: ffff810104335e78 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000001 R12: ffff8101715bde40
R13: ffffffff80304ba0 R14: ffff81017fcc0000 R15: ffff810105a623e8
FS: 00002b50c7e566f0(0000) GS:ffffffff805c6000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002b4d7f8b9b80 CR3: 0000000000201000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process fixdep (pid: 26971, threadinfo ffff810104334000, task ffff810105a62280)
Stack: ffff810104335e88 ffff8101715bde40 0000000000000010 ffff810105a62380
ffff810104335e88 ffffffff80304b70 ffff810104335ea8 ffffffff8030000b
ffff810105a62380 ffff8101715bde40 ffff810104335ec8 ffffffff803000df
Call Trace:
[<ffffffff80304b70>] cfq_free_io_context+0x10/0x20
[<ffffffff8030000b>] put_io_context+0x7b/0x90
[<ffffffff803000df>] exit_io_context+0x8f/0xb0
[<ffffffff80236533>] do_exit+0x543/0x770
[<ffffffff802367a1>] do_group_exit+0x41/0xb0
[<ffffffff80236822>] sys_exit_group+0x12/0x20
[<ffffffff8020b60b>] system_call_after_swapgs+0x7b/0x80
Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 48 8b 47 68 48 85 c0 74 1f 48 89 c3 90 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
RIP [<ffffffff80304b30>] __call_for_each_cic+0x20/0x50
RSP <ffff810104335e58>
---[ end trace ff773ec56ab2930e ]---
Fixing recursive fault but reboot is needed!
^ permalink raw reply [flat|nested] 37+ messages in thread
* 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-10 10:37 ` 2.6.25-$sha1: RIP __call_for_each_cic+0x20/0x50 Alexey Dobriyan
@ 2008-05-27 5:27 ` Alexey Dobriyan
2008-05-27 13:35 ` Jens Axboe
0 siblings, 1 reply; 37+ messages in thread
From: Alexey Dobriyan @ 2008-05-27 5:27 UTC (permalink / raw)
To: Jens Axboe, torvalds; +Cc: Andrew Morton, linux-kernel
On Sat, May 10, 2008 at 02:37:19PM +0400, Alexey Dobriyan wrote:
> > > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > > rcu_read_lock();
> > > > > if (ioc->aic && ioc->aic->dtor)
> > > > > ioc->aic->dtor(ioc->aic);
> > > > > - rcu_read_unlock();
> > > > > cfq_dtor(ioc);
> > > > > + rcu_read_unlock();
> > > > >
> > > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > > return 1;
> > > >
> > > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > > You'll hear me if another such oops will resurface.
> > >
> > > Still looking good?
> >
> > Yup!
>
> And this with patch in mainline, again with PREEMPT_RCU.
Ping, this happened again with 2.6.26-rc4 and PREEMPT_RCU.
> general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> CPU 0
> Modules linked in: ext2 nls_utf8 cifs nf_conntrack_irc ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables usblp ehci_hcd uhci_hcd usbcore sr_mod cdrom
> Pid: 26971, comm: fixdep Not tainted 2.6.26-rc1-28a4acb48586dc21d2d14a75a7aab7be78b7c83b #4
> RIP: 0010:[<ffffffff80304b30>] [<ffffffff80304b30>] __call_for_each_cic+0x20/0x50
> RSP: 0018:ffff810104335e58 EFLAGS: 00010202
> RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> RDX: 0000000000002827 RSI: 0000000000000000 RDI: ffff810105a62280
> RBP: ffff810104335e78 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000001 R12: ffff8101715bde40
> R13: ffffffff80304ba0 R14: ffff81017fcc0000 R15: ffff810105a623e8
> FS: 00002b50c7e566f0(0000) GS:ffffffff805c6000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00002b4d7f8b9b80 CR3: 0000000000201000 CR4: 0000000000000660
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process fixdep (pid: 26971, threadinfo ffff810104334000, task ffff810105a62280)
> Stack: ffff810104335e88 ffff8101715bde40 0000000000000010 ffff810105a62380
> ffff810104335e88 ffffffff80304b70 ffff810104335ea8 ffffffff8030000b
> ffff810105a62380 ffff8101715bde40 ffff810104335ec8 ffffffff803000df
> Call Trace:
> [<ffffffff80304b70>] cfq_free_io_context+0x10/0x20
> [<ffffffff8030000b>] put_io_context+0x7b/0x90
> [<ffffffff803000df>] exit_io_context+0x8f/0xb0
> [<ffffffff80236533>] do_exit+0x543/0x770
> [<ffffffff802367a1>] do_group_exit+0x41/0xb0
> [<ffffffff80236822>] sys_exit_group+0x12/0x20
> [<ffffffff8020b60b>] system_call_after_swapgs+0x7b/0x80
>
>
> Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 48 8b 47 68 48 85 c0 74 1f 48 89 c3 90 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> RIP [<ffffffff80304b30>] __call_for_each_cic+0x20/0x50
> RSP <ffff810104335e58>
> ---[ end trace ff773ec56ab2930e ]---
> Fixing recursive fault but reboot is needed!
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-27 5:27 ` 2.6.26-rc4: " Alexey Dobriyan
@ 2008-05-27 13:35 ` Jens Axboe
2008-05-27 15:18 ` Paul E. McKenney
0 siblings, 1 reply; 37+ messages in thread
From: Jens Axboe @ 2008-05-27 13:35 UTC (permalink / raw)
To: Alexey Dobriyan; +Cc: torvalds, Andrew Morton, linux-kernel, paulmck
On Tue, May 27 2008, Alexey Dobriyan wrote:
> On Sat, May 10, 2008 at 02:37:19PM +0400, Alexey Dobriyan wrote:
> > > > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > > > rcu_read_lock();
> > > > > > if (ioc->aic && ioc->aic->dtor)
> > > > > > ioc->aic->dtor(ioc->aic);
> > > > > > - rcu_read_unlock();
> > > > > > cfq_dtor(ioc);
> > > > > > + rcu_read_unlock();
> > > > > >
> > > > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > > > return 1;
> > > > >
> > > > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > > > You'll hear me if another such oops will resurface.
> > > >
> > > > Still looking good?
> > >
> > > Yup!
> >
> > And this with patch in mainline, again with PREEMPT_RCU.
>
> Ping, this happened again with 2.6.26-rc4 and PREEMPT_RCU.
Worrisome... Paul, would you mind taking a quick look at cfq
and see if you can detect why breaks with preempt rcu? It's
clearly a use-after-free symptom, but I don't see how it can
happen.
--
Jens Axboe
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-27 13:35 ` Jens Axboe
@ 2008-05-27 15:18 ` Paul E. McKenney
2008-05-28 10:07 ` Jens Axboe
0 siblings, 1 reply; 37+ messages in thread
From: Paul E. McKenney @ 2008-05-27 15:18 UTC (permalink / raw)
To: Jens Axboe; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Tue, May 27, 2008 at 03:35:10PM +0200, Jens Axboe wrote:
> On Tue, May 27 2008, Alexey Dobriyan wrote:
> > On Sat, May 10, 2008 at 02:37:19PM +0400, Alexey Dobriyan wrote:
> > > > > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > > > > rcu_read_lock();
> > > > > > > if (ioc->aic && ioc->aic->dtor)
> > > > > > > ioc->aic->dtor(ioc->aic);
> > > > > > > - rcu_read_unlock();
> > > > > > > cfq_dtor(ioc);
> > > > > > > + rcu_read_unlock();
> > > > > > >
> > > > > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > > > > return 1;
> > > > > >
> > > > > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > > > > You'll hear me if another such oops will resurface.
> > > > >
> > > > > Still looking good?
> > > >
> > > > Yup!
> > >
> > > And this with patch in mainline, again with PREEMPT_RCU.
> >
> > Ping, this happened again with 2.6.26-rc4 and PREEMPT_RCU.
>
> Worrisome... Paul, would you mind taking a quick look at cfq
> and see if you can detect why breaks with preempt rcu? It's
> clearly a use-after-free symptom, but I don't see how it can
> happen.
Some quick and probably off-the-mark questions...
o What is the purpose of __call_for_each_cic()? When called
from call_for_each_cic(), it is under rcu_read_lock(), as
required, but it is also called from cfq_free_io_context(),
which is assigned to the ->dtor and ->exit members of the
cfq_io_context struct. What protects calls through these
members?
(This is for the ->cic_list field of the cfq_io_context structure.
One possibility is that the io_context's ->lock member is held,
but I don't see this. Not that I looked all that hard...)
My suggestion would be to simply change all invocations of
__call_for_each_cic() to instead invoke call_for_each_cic().
The rcu_read_lock()/rcu_read_unlock() pair is pretty
lightweight, even in CONFIG_PREEMPT_RCU.
o When calling cfq_slab_kill(), for example from cfq_exit(),
what ensures that all previous RCU callbacks have completed?
I suspect that you need an rcu_barrier() at the beginning
of cfq_slab_kill(), but I could be missing something.
o Updates to the ->ioc_data field of the cfq_io_context
seem to be protected by the request_queue ->queue_lock
field. This seems very strange to me. It is OK if every
cfq_io_context is associated with only one request_queue
structure -- is this the case?
o What protects the first rcu_dereference() in cfq_cic_lookup()?
There needs to be either an enclose rcu_read_lock() on the
one hand or the ->queue_lock needs to be held.
(My guess is the latter, given the later rcu_assign_pointer()
in this same function, but I don't see a lock acquisition
in the immediate vicinity -- might be further up the function
call stack, though.)
o Why is there no grace period associated with the ioc_data?
For example, what happens to the old value of ->ioc_data
after the rcu_assign_pointer() in cfq_cic_lookup()? Readers
might still be referencing the old version, right? If so,
how do we avoid messing them up?
Or are we somehow leveraging the call_rcu() in cfq_cic_free()?
Any of this at all helpful?
Thanx, Paul
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-27 15:18 ` Paul E. McKenney
@ 2008-05-28 10:07 ` Jens Axboe
2008-05-28 10:30 ` Paul E. McKenney
2008-05-28 11:52 ` Fabio Checconi
0 siblings, 2 replies; 37+ messages in thread
From: Jens Axboe @ 2008-05-28 10:07 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Tue, May 27 2008, Paul E. McKenney wrote:
> On Tue, May 27, 2008 at 03:35:10PM +0200, Jens Axboe wrote:
> > On Tue, May 27 2008, Alexey Dobriyan wrote:
> > > On Sat, May 10, 2008 at 02:37:19PM +0400, Alexey Dobriyan wrote:
> > > > > > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > > > > > rcu_read_lock();
> > > > > > > > if (ioc->aic && ioc->aic->dtor)
> > > > > > > > ioc->aic->dtor(ioc->aic);
> > > > > > > > - rcu_read_unlock();
> > > > > > > > cfq_dtor(ioc);
> > > > > > > > + rcu_read_unlock();
> > > > > > > >
> > > > > > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > > > > > return 1;
> > > > > > >
> > > > > > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > > > > > You'll hear me if another such oops will resurface.
> > > > > >
> > > > > > Still looking good?
> > > > >
> > > > > Yup!
> > > >
> > > > And this with patch in mainline, again with PREEMPT_RCU.
> > >
> > > Ping, this happened again with 2.6.26-rc4 and PREEMPT_RCU.
> >
> > Worrisome... Paul, would you mind taking a quick look at cfq
> > and see if you can detect why breaks with preempt rcu? It's
> > clearly a use-after-free symptom, but I don't see how it can
> > happen.
>
> Some quick and probably off-the-mark questions...
Thanks!
> o What is the purpose of __call_for_each_cic()? When called
> from call_for_each_cic(), it is under rcu_read_lock(), as
> required, but it is also called from cfq_free_io_context(),
> which is assigned to the ->dtor and ->exit members of the
> cfq_io_context struct. What protects calls through these
> members?
>
> (This is for the ->cic_list field of the cfq_io_context structure.
> One possibility is that the io_context's ->lock member is held,
> but I don't see this. Not that I looked all that hard...)
>
> My suggestion would be to simply change all invocations of
> __call_for_each_cic() to instead invoke call_for_each_cic().
> The rcu_read_lock()/rcu_read_unlock() pair is pretty
> lightweight, even in CONFIG_PREEMPT_RCU.
__call_for_each_cic() is always called under rcu_read_lock(), it merely
exists to avoid a double rcu_read_lock(). Even if it is cheap. The
convention follows the usual __lock_is_already_held() double under
score, but I guess it could do with a comment! There are only two
callers of the function, call_for_each_cic() which does the
rcu_read_lock(), and cfq_free_io_context() which is called from ->dtor
(and holds the rcu_read_lock() and ->trim which actually does not. That
looks like it could be problematic, but it's only called when an io
scheduler module is removed so not really critical. I'll add it, though!
Actually, the task_lock() should be enough there. So no bug, but (again)
it could do with a comment.
> o When calling cfq_slab_kill(), for example from cfq_exit(),
> what ensures that all previous RCU callbacks have completed?
>
> I suspect that you need an rcu_barrier() at the beginning
> of cfq_slab_kill(), but I could be missing something.
So we have two callers of that, one is from the error path at init time
and is obviously ok. The other does need rcu_barrier()! I'll add that.
> o Updates to the ->ioc_data field of the cfq_io_context
> seem to be protected by the request_queue ->queue_lock
> field. This seems very strange to me. It is OK if every
> cfq_io_context is associated with only one request_queue
> structure -- is this the case?
->ioc_data is part of the io_context, not cfq_io_context. And it can be
shared now, so the correct locking for that would be ioc->lock and not
the queue lock. __cfq_exit_single_io_context() is serialized in the
sense that only one process gets to call the exit path.
> o What protects the first rcu_dereference() in cfq_cic_lookup()?
> There needs to be either an enclose rcu_read_lock() on the
> one hand or the ->queue_lock needs to be held.
>
> (My guess is the latter, given the later rcu_assign_pointer()
> in this same function, but I don't see a lock acquisition
> in the immediate vicinity -- might be further up the function
> call stack, though.)
There's no locking going into that function when coming from
cfq_get_io_context(), the other paths (happen) to hold the queue lock
already though.
> o Why is there no grace period associated with the ioc_data?
> For example, what happens to the old value of ->ioc_data
> after the rcu_assign_pointer() in cfq_cic_lookup()? Readers
> might still be referencing the old version, right? If so,
> how do we avoid messing them up?
>
> Or are we somehow leveraging the call_rcu() in cfq_cic_free()?
The data belonging to ->ioc_data (the cic, or per-process per-queue
context) is only freed through call_rcu().
> Any of this at all helpful?
Very, perhaps with a few more rounds we can find some more bugs :-). I'm
attaching a patch below, how does that look?
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 4df3f05..75db529 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -1142,6 +1142,9 @@ static void cfq_put_queue(struct cfq_queue *cfqq)
kmem_cache_free(cfq_pool, cfqq);
}
+/*
+ * Must always be called with the rcu_read_lock() held
+ */
static void
__call_for_each_cic(struct io_context *ioc,
void (*func)(struct io_context *, struct cfq_io_context *))
@@ -1197,6 +1200,11 @@ static void cic_free_func(struct io_context *ioc, struct cfq_io_context *cic)
cfq_cic_free(cic);
}
+/*
+ * Must be called with rcu_read_lock() held or preemption otherwise disabled.
+ * Only two callers of this - ->dtor() which is called with the rcu_read_lock(),
+ * and ->trim() which is called with the task lock held
+ */
static void cfq_free_io_context(struct io_context *ioc)
{
/*
@@ -1502,20 +1510,24 @@ static struct cfq_io_context *
cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
{
struct cfq_io_context *cic;
+ unsigned long flags;
void *k;
if (unlikely(!ioc))
return NULL;
+ rcu_read_lock();
+
/*
* we maintain a last-hit cache, to avoid browsing over the tree
*/
cic = rcu_dereference(ioc->ioc_data);
- if (cic && cic->key == cfqd)
+ if (cic && cic->key == cfqd) {
+ rcu_read_unlock();
return cic;
+ }
do {
- rcu_read_lock();
cic = radix_tree_lookup(&ioc->radix_root, (unsigned long) cfqd);
rcu_read_unlock();
if (!cic)
@@ -1524,10 +1536,13 @@ cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
k = cic->key;
if (unlikely(!k)) {
cfq_drop_dead_cic(cfqd, ioc, cic);
+ rcu_read_lock();
continue;
}
+ spin_lock_irqsave(&ioc->lock, flags);
rcu_assign_pointer(ioc->ioc_data, cic);
+ spin_unlock_irqrestore(&ioc->lock, flags);
break;
} while (1);
@@ -2134,6 +2149,11 @@ static void *cfq_init_queue(struct request_queue *q)
static void cfq_slab_kill(void)
{
+ /*
+ * Make sure that all existing RCU callbacks have been processed
+ */
+ rcu_barrier();
+
if (cfq_pool)
kmem_cache_destroy(cfq_pool);
if (cfq_ioc_pool)
--
Jens Axboe
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-28 10:07 ` Jens Axboe
@ 2008-05-28 10:30 ` Paul E. McKenney
2008-05-28 12:44 ` Jens Axboe
2008-05-28 11:52 ` Fabio Checconi
1 sibling, 1 reply; 37+ messages in thread
From: Paul E. McKenney @ 2008-05-28 10:30 UTC (permalink / raw)
To: Jens Axboe; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Wed, May 28, 2008 at 12:07:21PM +0200, Jens Axboe wrote:
> On Tue, May 27 2008, Paul E. McKenney wrote:
> > On Tue, May 27, 2008 at 03:35:10PM +0200, Jens Axboe wrote:
> > > On Tue, May 27 2008, Alexey Dobriyan wrote:
> > > > On Sat, May 10, 2008 at 02:37:19PM +0400, Alexey Dobriyan wrote:
> > > > > > > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > > > > > > rcu_read_lock();
> > > > > > > > > if (ioc->aic && ioc->aic->dtor)
> > > > > > > > > ioc->aic->dtor(ioc->aic);
> > > > > > > > > - rcu_read_unlock();
> > > > > > > > > cfq_dtor(ioc);
> > > > > > > > > + rcu_read_unlock();
> > > > > > > > >
> > > > > > > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > > > > > > return 1;
> > > > > > > >
> > > > > > > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > > > > > > You'll hear me if another such oops will resurface.
> > > > > > >
> > > > > > > Still looking good?
> > > > > >
> > > > > > Yup!
> > > > >
> > > > > And this with patch in mainline, again with PREEMPT_RCU.
> > > >
> > > > Ping, this happened again with 2.6.26-rc4 and PREEMPT_RCU.
> > >
> > > Worrisome... Paul, would you mind taking a quick look at cfq
> > > and see if you can detect why breaks with preempt rcu? It's
> > > clearly a use-after-free symptom, but I don't see how it can
> > > happen.
> >
> > Some quick and probably off-the-mark questions...
>
> Thanks!
Glad it actually was of help! ;-)
> > o What is the purpose of __call_for_each_cic()? When called
> > from call_for_each_cic(), it is under rcu_read_lock(), as
> > required, but it is also called from cfq_free_io_context(),
> > which is assigned to the ->dtor and ->exit members of the
> > cfq_io_context struct. What protects calls through these
> > members?
> >
> > (This is for the ->cic_list field of the cfq_io_context structure.
> > One possibility is that the io_context's ->lock member is held,
> > but I don't see this. Not that I looked all that hard...)
> >
> > My suggestion would be to simply change all invocations of
> > __call_for_each_cic() to instead invoke call_for_each_cic().
> > The rcu_read_lock()/rcu_read_unlock() pair is pretty
> > lightweight, even in CONFIG_PREEMPT_RCU.
>
> __call_for_each_cic() is always called under rcu_read_lock(), it merely
> exists to avoid a double rcu_read_lock(). Even if it is cheap. The
> convention follows the usual __lock_is_already_held() double under
> score, but I guess it could do with a comment! There are only two
> callers of the function, call_for_each_cic() which does the
> rcu_read_lock(), and cfq_free_io_context() which is called from ->dtor
> (and holds the rcu_read_lock() and ->trim which actually does not. That
> looks like it could be problematic, but it's only called when an io
> scheduler module is removed so not really critical. I'll add it, though!
> Actually, the task_lock() should be enough there. So no bug, but (again)
> it could do with a comment.
Sounds good!
> > o When calling cfq_slab_kill(), for example from cfq_exit(),
> > what ensures that all previous RCU callbacks have completed?
> >
> > I suspect that you need an rcu_barrier() at the beginning
> > of cfq_slab_kill(), but I could be missing something.
>
> So we have two callers of that, one is from the error path at init time
> and is obviously ok. The other does need rcu_barrier()! I'll add that.
OK, that does make my brain hurt less. ;-)
> > o Updates to the ->ioc_data field of the cfq_io_context
> > seem to be protected by the request_queue ->queue_lock
> > field. This seems very strange to me. It is OK if every
> > cfq_io_context is associated with only one request_queue
> > structure -- is this the case?
>
> ->ioc_data is part of the io_context, not cfq_io_context. And it can be
> shared now, so the correct locking for that would be ioc->lock and not
> the queue lock. __cfq_exit_single_io_context() is serialized in the
> sense that only one process gets to call the exit path.
Makes sense to me!
> > o What protects the first rcu_dereference() in cfq_cic_lookup()?
> > There needs to be either an enclose rcu_read_lock() on the
> > one hand or the ->queue_lock needs to be held.
> >
> > (My guess is the latter, given the later rcu_assign_pointer()
> > in this same function, but I don't see a lock acquisition
> > in the immediate vicinity -- might be further up the function
> > call stack, though.)
>
> There's no locking going into that function when coming from
> cfq_get_io_context(), the other paths (happen) to hold the queue lock
> already though.
So the call from cfq_get_io_context() needs an rcu_read_lock()?
Not seeing this in the patch below, but maybe you have it up a
function-call level or two?
> > o Why is there no grace period associated with the ioc_data?
> > For example, what happens to the old value of ->ioc_data
> > after the rcu_assign_pointer() in cfq_cic_lookup()? Readers
> > might still be referencing the old version, right? If so,
> > how do we avoid messing them up?
> >
> > Or are we somehow leveraging the call_rcu() in cfq_cic_free()?
>
> The data belonging to ->ioc_data (the cic, or per-process per-queue
> context) is only freed through call_rcu().
Ah, OK, got it.
> > Any of this at all helpful?
>
> Very, perhaps with a few more rounds we can find some more bugs :-). I'm
> attaching a patch below, how does that look?
Looks much improved! Very interested to hear how it does with the testing.
Thanx, Paul
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> index 4df3f05..75db529 100644
> --- a/block/cfq-iosched.c
> +++ b/block/cfq-iosched.c
> @@ -1142,6 +1142,9 @@ static void cfq_put_queue(struct cfq_queue *cfqq)
> kmem_cache_free(cfq_pool, cfqq);
> }
>
> +/*
> + * Must always be called with the rcu_read_lock() held
> + */
> static void
> __call_for_each_cic(struct io_context *ioc,
> void (*func)(struct io_context *, struct cfq_io_context *))
> @@ -1197,6 +1200,11 @@ static void cic_free_func(struct io_context *ioc, struct cfq_io_context *cic)
> cfq_cic_free(cic);
> }
>
> +/*
> + * Must be called with rcu_read_lock() held or preemption otherwise disabled.
> + * Only two callers of this - ->dtor() which is called with the rcu_read_lock(),
> + * and ->trim() which is called with the task lock held
> + */
> static void cfq_free_io_context(struct io_context *ioc)
> {
> /*
> @@ -1502,20 +1510,24 @@ static struct cfq_io_context *
> cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
> {
> struct cfq_io_context *cic;
> + unsigned long flags;
> void *k;
>
> if (unlikely(!ioc))
> return NULL;
>
> + rcu_read_lock();
> +
> /*
> * we maintain a last-hit cache, to avoid browsing over the tree
> */
> cic = rcu_dereference(ioc->ioc_data);
> - if (cic && cic->key == cfqd)
> + if (cic && cic->key == cfqd) {
> + rcu_read_unlock();
> return cic;
> + }
>
> do {
> - rcu_read_lock();
> cic = radix_tree_lookup(&ioc->radix_root, (unsigned long) cfqd);
> rcu_read_unlock();
> if (!cic)
> @@ -1524,10 +1536,13 @@ cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
> k = cic->key;
> if (unlikely(!k)) {
> cfq_drop_dead_cic(cfqd, ioc, cic);
> + rcu_read_lock();
> continue;
> }
>
> + spin_lock_irqsave(&ioc->lock, flags);
> rcu_assign_pointer(ioc->ioc_data, cic);
> + spin_unlock_irqrestore(&ioc->lock, flags);
> break;
> } while (1);
>
> @@ -2134,6 +2149,11 @@ static void *cfq_init_queue(struct request_queue *q)
>
> static void cfq_slab_kill(void)
> {
> + /*
> + * Make sure that all existing RCU callbacks have been processed
> + */
> + rcu_barrier();
> +
> if (cfq_pool)
> kmem_cache_destroy(cfq_pool);
> if (cfq_ioc_pool)
>
> --
> Jens Axboe
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-28 10:07 ` Jens Axboe
2008-05-28 10:30 ` Paul E. McKenney
@ 2008-05-28 11:52 ` Fabio Checconi
2008-05-28 11:58 ` Jens Axboe
1 sibling, 1 reply; 37+ messages in thread
From: Fabio Checconi @ 2008-05-28 11:52 UTC (permalink / raw)
To: Jens Axboe
Cc: Paul E. McKenney, Alexey Dobriyan, torvalds, Andrew Morton,
linux-kernel
> From: Jens Axboe <jens.axboe@oracle.com>
> Date: Wed, May 28, 2008 12:07:21PM +0200
>
> On Tue, May 27 2008, Paul E. McKenney wrote:
> > o When calling cfq_slab_kill(), for example from cfq_exit(),
> > what ensures that all previous RCU callbacks have completed?
> >
> > I suspect that you need an rcu_barrier() at the beginning
> > of cfq_slab_kill(), but I could be missing something.
>
> So we have two callers of that, one is from the error path at init time
> and is obviously ok. The other does need rcu_barrier()! I'll add that.
>
But isn't the ioc_gone completion (notified only when there are no more
cic allocated) assuring that cfq_slab_kill() is called only after all
the rcu callbacks are completed? This should avoid the need for the
rcu_barrier().
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-28 11:52 ` Fabio Checconi
@ 2008-05-28 11:58 ` Jens Axboe
0 siblings, 0 replies; 37+ messages in thread
From: Jens Axboe @ 2008-05-28 11:58 UTC (permalink / raw)
To: Fabio Checconi
Cc: Paul E. McKenney, Alexey Dobriyan, torvalds, Andrew Morton,
linux-kernel
On Wed, May 28 2008, Fabio Checconi wrote:
> > From: Jens Axboe <jens.axboe@oracle.com>
> > Date: Wed, May 28, 2008 12:07:21PM +0200
> >
> > On Tue, May 27 2008, Paul E. McKenney wrote:
> > > o When calling cfq_slab_kill(), for example from cfq_exit(),
> > > what ensures that all previous RCU callbacks have completed?
> > >
> > > I suspect that you need an rcu_barrier() at the beginning
> > > of cfq_slab_kill(), but I could be missing something.
> >
> > So we have two callers of that, one is from the error path at init time
> > and is obviously ok. The other does need rcu_barrier()! I'll add that.
> >
>
> But isn't the ioc_gone completion (notified only when there are no more
> cic allocated) assuring that cfq_slab_kill() is called only after all
> the rcu callbacks are completed? This should avoid the need for the
> rcu_barrier().
Good point, I was thinking we decremented the mod count on call_rcu(),
but we don't actually do it before the rcu callback has completed. So
that part is actually OK already.
--
Jens Axboe
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-28 10:30 ` Paul E. McKenney
@ 2008-05-28 12:44 ` Jens Axboe
2008-05-28 13:20 ` Paul E. McKenney
2008-05-30 18:34 ` Alexey Dobriyan
0 siblings, 2 replies; 37+ messages in thread
From: Jens Axboe @ 2008-05-28 12:44 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Wed, May 28 2008, Paul E. McKenney wrote:
> On Wed, May 28, 2008 at 12:07:21PM +0200, Jens Axboe wrote:
> > On Tue, May 27 2008, Paul E. McKenney wrote:
> > > On Tue, May 27, 2008 at 03:35:10PM +0200, Jens Axboe wrote:
> > > > On Tue, May 27 2008, Alexey Dobriyan wrote:
> > > > > On Sat, May 10, 2008 at 02:37:19PM +0400, Alexey Dobriyan wrote:
> > > > > > > > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > > > > > > > rcu_read_lock();
> > > > > > > > > > if (ioc->aic && ioc->aic->dtor)
> > > > > > > > > > ioc->aic->dtor(ioc->aic);
> > > > > > > > > > - rcu_read_unlock();
> > > > > > > > > > cfq_dtor(ioc);
> > > > > > > > > > + rcu_read_unlock();
> > > > > > > > > >
> > > > > > > > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > > > > > > > return 1;
> > > > > > > > >
> > > > > > > > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > > > > > > > You'll hear me if another such oops will resurface.
> > > > > > > >
> > > > > > > > Still looking good?
> > > > > > >
> > > > > > > Yup!
> > > > > >
> > > > > > And this with patch in mainline, again with PREEMPT_RCU.
> > > > >
> > > > > Ping, this happened again with 2.6.26-rc4 and PREEMPT_RCU.
> > > >
> > > > Worrisome... Paul, would you mind taking a quick look at cfq
> > > > and see if you can detect why breaks with preempt rcu? It's
> > > > clearly a use-after-free symptom, but I don't see how it can
> > > > happen.
> > >
> > > Some quick and probably off-the-mark questions...
> >
> > Thanks!
>
> Glad it actually was of help! ;-)
Your reviews are ALWAYS greatly appreciated!
> > > o What is the purpose of __call_for_each_cic()? When called
> > > from call_for_each_cic(), it is under rcu_read_lock(), as
> > > required, but it is also called from cfq_free_io_context(),
> > > which is assigned to the ->dtor and ->exit members of the
> > > cfq_io_context struct. What protects calls through these
> > > members?
> > >
> > > (This is for the ->cic_list field of the cfq_io_context structure.
> > > One possibility is that the io_context's ->lock member is held,
> > > but I don't see this. Not that I looked all that hard...)
> > >
> > > My suggestion would be to simply change all invocations of
> > > __call_for_each_cic() to instead invoke call_for_each_cic().
> > > The rcu_read_lock()/rcu_read_unlock() pair is pretty
> > > lightweight, even in CONFIG_PREEMPT_RCU.
> >
> > __call_for_each_cic() is always called under rcu_read_lock(), it merely
> > exists to avoid a double rcu_read_lock(). Even if it is cheap. The
> > convention follows the usual __lock_is_already_held() double under
> > score, but I guess it could do with a comment! There are only two
> > callers of the function, call_for_each_cic() which does the
> > rcu_read_lock(), and cfq_free_io_context() which is called from ->dtor
> > (and holds the rcu_read_lock() and ->trim which actually does not. That
> > looks like it could be problematic, but it's only called when an io
> > scheduler module is removed so not really critical. I'll add it, though!
> > Actually, the task_lock() should be enough there. So no bug, but (again)
> > it could do with a comment.
>
> Sounds good!
>
> > > o When calling cfq_slab_kill(), for example from cfq_exit(),
> > > what ensures that all previous RCU callbacks have completed?
> > >
> > > I suspect that you need an rcu_barrier() at the beginning
> > > of cfq_slab_kill(), but I could be missing something.
> >
> > So we have two callers of that, one is from the error path at init time
> > and is obviously ok. The other does need rcu_barrier()! I'll add that.
>
> OK, that does make my brain hurt less. ;-)
So that one was also OK, as Fabio pointed out. If you follow the
ioc_gone and user tracking, the:
if (elv_ioc_count_read(ioc_count))
wait_for_completion(ioc_gone);
also has the side effect of waiting for RCU callbacks calling
kmem_cache_free() to have finished as well.
> > > o What protects the first rcu_dereference() in cfq_cic_lookup()?
> > > There needs to be either an enclose rcu_read_lock() on the
> > > one hand or the ->queue_lock needs to be held.
> > >
> > > (My guess is the latter, given the later rcu_assign_pointer()
> > > in this same function, but I don't see a lock acquisition
> > > in the immediate vicinity -- might be further up the function
> > > call stack, though.)
> >
> > There's no locking going into that function when coming from
> > cfq_get_io_context(), the other paths (happen) to hold the queue lock
> > already though.
>
> So the call from cfq_get_io_context() needs an rcu_read_lock()?
> Not seeing this in the patch below, but maybe you have it up a
> function-call level or two?
It's in there, it now does:
rcu_read_lock();
cic = rcu_dereference(ioc->ioc_data);
if (cic && cic->key == cfqd) {
rcu_read_unlock();
return cic;
}
...
OK? Which is basically what remains of the patch now, except for the
comment additions. Oh, and the ioc->lock protecting setting of
->ioc_data as well. New version below. Alexey, care to give this a
spin? Seems your box is very well suited for finding RCU preempt
problems :-)
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 4df3f05..d01b411 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -1142,6 +1142,9 @@ static void cfq_put_queue(struct cfq_queue *cfqq)
kmem_cache_free(cfq_pool, cfqq);
}
+/*
+ * Must always be called with the rcu_read_lock() held
+ */
static void
__call_for_each_cic(struct io_context *ioc,
void (*func)(struct io_context *, struct cfq_io_context *))
@@ -1197,6 +1200,11 @@ static void cic_free_func(struct io_context *ioc, struct cfq_io_context *cic)
cfq_cic_free(cic);
}
+/*
+ * Must be called with rcu_read_lock() held or preemption otherwise disabled.
+ * Only two callers of this - ->dtor() which is called with the rcu_read_lock(),
+ * and ->trim() which is called with the task lock held
+ */
static void cfq_free_io_context(struct io_context *ioc)
{
/*
@@ -1502,20 +1510,24 @@ static struct cfq_io_context *
cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
{
struct cfq_io_context *cic;
+ unsigned long flags;
void *k;
if (unlikely(!ioc))
return NULL;
+ rcu_read_lock();
+
/*
* we maintain a last-hit cache, to avoid browsing over the tree
*/
cic = rcu_dereference(ioc->ioc_data);
- if (cic && cic->key == cfqd)
+ if (cic && cic->key == cfqd) {
+ rcu_read_unlock();
return cic;
+ }
do {
- rcu_read_lock();
cic = radix_tree_lookup(&ioc->radix_root, (unsigned long) cfqd);
rcu_read_unlock();
if (!cic)
@@ -1524,10 +1536,13 @@ cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
k = cic->key;
if (unlikely(!k)) {
cfq_drop_dead_cic(cfqd, ioc, cic);
+ rcu_read_lock();
continue;
}
+ spin_lock_irqsave(&ioc->lock, flags);
rcu_assign_pointer(ioc->ioc_data, cic);
+ spin_unlock_irqrestore(&ioc->lock, flags);
break;
} while (1);
@@ -2134,6 +2149,10 @@ static void *cfq_init_queue(struct request_queue *q)
static void cfq_slab_kill(void)
{
+ /*
+ * Caller already ensured that pending RCU callbacks are completed,
+ * so we should have no busy allocations at this point.
+ */
if (cfq_pool)
kmem_cache_destroy(cfq_pool);
if (cfq_ioc_pool)
@@ -2292,6 +2311,11 @@ static void __exit cfq_exit(void)
ioc_gone = &all_gone;
/* ioc_gone's update must be visible before reading ioc_count */
smp_wmb();
+
+ /*
+ * this also protects us from entering cfq_slab_kill() with
+ * pending RCU callbacks
+ */
if (elv_ioc_count_read(ioc_count))
wait_for_completion(ioc_gone);
cfq_slab_kill();
--
Jens Axboe
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-28 12:44 ` Jens Axboe
@ 2008-05-28 13:20 ` Paul E. McKenney
2008-05-29 4:38 ` Paul E. McKenney
2008-05-30 18:34 ` Alexey Dobriyan
1 sibling, 1 reply; 37+ messages in thread
From: Paul E. McKenney @ 2008-05-28 13:20 UTC (permalink / raw)
To: Jens Axboe; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Wed, May 28, 2008 at 02:44:24PM +0200, Jens Axboe wrote:
> On Wed, May 28 2008, Paul E. McKenney wrote:
> > On Wed, May 28, 2008 at 12:07:21PM +0200, Jens Axboe wrote:
> > > On Tue, May 27 2008, Paul E. McKenney wrote:
> > > > On Tue, May 27, 2008 at 03:35:10PM +0200, Jens Axboe wrote:
> > > > > On Tue, May 27 2008, Alexey Dobriyan wrote:
> > > > > > On Sat, May 10, 2008 at 02:37:19PM +0400, Alexey Dobriyan wrote:
> > > > > > > > > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > > > > > > > > rcu_read_lock();
> > > > > > > > > > > if (ioc->aic && ioc->aic->dtor)
> > > > > > > > > > > ioc->aic->dtor(ioc->aic);
> > > > > > > > > > > - rcu_read_unlock();
> > > > > > > > > > > cfq_dtor(ioc);
> > > > > > > > > > > + rcu_read_unlock();
> > > > > > > > > > >
> > > > > > > > > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > > > > > > > > return 1;
> > > > > > > > > >
> > > > > > > > > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > > > > > > > > You'll hear me if another such oops will resurface.
> > > > > > > > >
> > > > > > > > > Still looking good?
> > > > > > > >
> > > > > > > > Yup!
> > > > > > >
> > > > > > > And this with patch in mainline, again with PREEMPT_RCU.
> > > > > >
> > > > > > Ping, this happened again with 2.6.26-rc4 and PREEMPT_RCU.
> > > > >
> > > > > Worrisome... Paul, would you mind taking a quick look at cfq
> > > > > and see if you can detect why breaks with preempt rcu? It's
> > > > > clearly a use-after-free symptom, but I don't see how it can
> > > > > happen.
> > > >
> > > > Some quick and probably off-the-mark questions...
> > >
> > > Thanks!
> >
> > Glad it actually was of help! ;-)
>
> Your reviews are ALWAYS greatly appreciated!
:-)
> > > > o What is the purpose of __call_for_each_cic()? When called
> > > > from call_for_each_cic(), it is under rcu_read_lock(), as
> > > > required, but it is also called from cfq_free_io_context(),
> > > > which is assigned to the ->dtor and ->exit members of the
> > > > cfq_io_context struct. What protects calls through these
> > > > members?
> > > >
> > > > (This is for the ->cic_list field of the cfq_io_context structure.
> > > > One possibility is that the io_context's ->lock member is held,
> > > > but I don't see this. Not that I looked all that hard...)
> > > >
> > > > My suggestion would be to simply change all invocations of
> > > > __call_for_each_cic() to instead invoke call_for_each_cic().
> > > > The rcu_read_lock()/rcu_read_unlock() pair is pretty
> > > > lightweight, even in CONFIG_PREEMPT_RCU.
> > >
> > > __call_for_each_cic() is always called under rcu_read_lock(), it merely
> > > exists to avoid a double rcu_read_lock(). Even if it is cheap. The
> > > convention follows the usual __lock_is_already_held() double under
> > > score, but I guess it could do with a comment! There are only two
> > > callers of the function, call_for_each_cic() which does the
> > > rcu_read_lock(), and cfq_free_io_context() which is called from ->dtor
> > > (and holds the rcu_read_lock() and ->trim which actually does not. That
> > > looks like it could be problematic, but it's only called when an io
> > > scheduler module is removed so not really critical. I'll add it, though!
> > > Actually, the task_lock() should be enough there. So no bug, but (again)
> > > it could do with a comment.
> >
> > Sounds good!
> >
> > > > o When calling cfq_slab_kill(), for example from cfq_exit(),
> > > > what ensures that all previous RCU callbacks have completed?
> > > >
> > > > I suspect that you need an rcu_barrier() at the beginning
> > > > of cfq_slab_kill(), but I could be missing something.
> > >
> > > So we have two callers of that, one is from the error path at init time
> > > and is obviously ok. The other does need rcu_barrier()! I'll add that.
> >
> > OK, that does make my brain hurt less. ;-)
>
> So that one was also OK, as Fabio pointed out. If you follow the
> ioc_gone and user tracking, the:
>
> if (elv_ioc_count_read(ioc_count))
> wait_for_completion(ioc_gone);
>
> also has the side effect of waiting for RCU callbacks calling
> kmem_cache_free() to have finished as well.
I stand corrected.
> > > > o What protects the first rcu_dereference() in cfq_cic_lookup()?
> > > > There needs to be either an enclose rcu_read_lock() on the
> > > > one hand or the ->queue_lock needs to be held.
> > > >
> > > > (My guess is the latter, given the later rcu_assign_pointer()
> > > > in this same function, but I don't see a lock acquisition
> > > > in the immediate vicinity -- might be further up the function
> > > > call stack, though.)
> > >
> > > There's no locking going into that function when coming from
> > > cfq_get_io_context(), the other paths (happen) to hold the queue lock
> > > already though.
> >
> > So the call from cfq_get_io_context() needs an rcu_read_lock()?
> > Not seeing this in the patch below, but maybe you have it up a
> > function-call level or two?
>
> It's in there, it now does:
>
> rcu_read_lock();
> cic = rcu_dereference(ioc->ioc_data);
> if (cic && cic->key == cfqd) {
> rcu_read_unlock();
> return cic;
> }
> ...
>
> OK? Which is basically what remains of the patch now, except for the
> comment additions. Oh, and the ioc->lock protecting setting of
> ->ioc_data as well. New version below. Alexey, care to give this a
> spin? Seems your box is very well suited for finding RCU preempt
> problems :-)
OK, looks good.
Thanx, Paul
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> index 4df3f05..d01b411 100644
> --- a/block/cfq-iosched.c
> +++ b/block/cfq-iosched.c
> @@ -1142,6 +1142,9 @@ static void cfq_put_queue(struct cfq_queue *cfqq)
> kmem_cache_free(cfq_pool, cfqq);
> }
>
> +/*
> + * Must always be called with the rcu_read_lock() held
> + */
> static void
> __call_for_each_cic(struct io_context *ioc,
> void (*func)(struct io_context *, struct cfq_io_context *))
> @@ -1197,6 +1200,11 @@ static void cic_free_func(struct io_context *ioc, struct cfq_io_context *cic)
> cfq_cic_free(cic);
> }
>
> +/*
> + * Must be called with rcu_read_lock() held or preemption otherwise disabled.
> + * Only two callers of this - ->dtor() which is called with the rcu_read_lock(),
> + * and ->trim() which is called with the task lock held
> + */
> static void cfq_free_io_context(struct io_context *ioc)
> {
> /*
> @@ -1502,20 +1510,24 @@ static struct cfq_io_context *
> cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
> {
> struct cfq_io_context *cic;
> + unsigned long flags;
> void *k;
>
> if (unlikely(!ioc))
> return NULL;
>
> + rcu_read_lock();
> +
> /*
> * we maintain a last-hit cache, to avoid browsing over the tree
> */
> cic = rcu_dereference(ioc->ioc_data);
> - if (cic && cic->key == cfqd)
> + if (cic && cic->key == cfqd) {
> + rcu_read_unlock();
> return cic;
> + }
>
> do {
> - rcu_read_lock();
> cic = radix_tree_lookup(&ioc->radix_root, (unsigned long) cfqd);
> rcu_read_unlock();
> if (!cic)
> @@ -1524,10 +1536,13 @@ cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
> k = cic->key;
> if (unlikely(!k)) {
> cfq_drop_dead_cic(cfqd, ioc, cic);
> + rcu_read_lock();
> continue;
> }
>
> + spin_lock_irqsave(&ioc->lock, flags);
> rcu_assign_pointer(ioc->ioc_data, cic);
> + spin_unlock_irqrestore(&ioc->lock, flags);
> break;
> } while (1);
>
> @@ -2134,6 +2149,10 @@ static void *cfq_init_queue(struct request_queue *q)
>
> static void cfq_slab_kill(void)
> {
> + /*
> + * Caller already ensured that pending RCU callbacks are completed,
> + * so we should have no busy allocations at this point.
> + */
> if (cfq_pool)
> kmem_cache_destroy(cfq_pool);
> if (cfq_ioc_pool)
> @@ -2292,6 +2311,11 @@ static void __exit cfq_exit(void)
> ioc_gone = &all_gone;
> /* ioc_gone's update must be visible before reading ioc_count */
> smp_wmb();
> +
> + /*
> + * this also protects us from entering cfq_slab_kill() with
> + * pending RCU callbacks
> + */
> if (elv_ioc_count_read(ioc_count))
> wait_for_completion(ioc_gone);
> cfq_slab_kill();
>
> --
> Jens Axboe
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-28 13:20 ` Paul E. McKenney
@ 2008-05-29 4:38 ` Paul E. McKenney
2008-05-29 6:26 ` Jens Axboe
0 siblings, 1 reply; 37+ messages in thread
From: Paul E. McKenney @ 2008-05-29 4:38 UTC (permalink / raw)
To: Jens Axboe; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Wed, May 28, 2008 at 06:20:12AM -0700, Paul E. McKenney wrote:
> On Wed, May 28, 2008 at 02:44:24PM +0200, Jens Axboe wrote:
> > On Wed, May 28 2008, Paul E. McKenney wrote:
> > > On Wed, May 28, 2008 at 12:07:21PM +0200, Jens Axboe wrote:
> > > > On Tue, May 27 2008, Paul E. McKenney wrote:
> > > > > On Tue, May 27, 2008 at 03:35:10PM +0200, Jens Axboe wrote:
> > > > > > On Tue, May 27 2008, Alexey Dobriyan wrote:
> > > > > > > On Sat, May 10, 2008 at 02:37:19PM +0400, Alexey Dobriyan wrote:
> > > > > > > > > > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > > > > > > > > > rcu_read_lock();
> > > > > > > > > > > > if (ioc->aic && ioc->aic->dtor)
> > > > > > > > > > > > ioc->aic->dtor(ioc->aic);
> > > > > > > > > > > > - rcu_read_unlock();
> > > > > > > > > > > > cfq_dtor(ioc);
> > > > > > > > > > > > + rcu_read_unlock();
> > > > > > > > > > > >
> > > > > > > > > > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > > > > > > > > > return 1;
> > > > > > > > > > >
> > > > > > > > > > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > > > > > > > > > You'll hear me if another such oops will resurface.
> > > > > > > > > >
> > > > > > > > > > Still looking good?
> > > > > > > > >
> > > > > > > > > Yup!
> > > > > > > >
> > > > > > > > And this with patch in mainline, again with PREEMPT_RCU.
> > > > > > >
> > > > > > > Ping, this happened again with 2.6.26-rc4 and PREEMPT_RCU.
> > > > > >
> > > > > > Worrisome... Paul, would you mind taking a quick look at cfq
> > > > > > and see if you can detect why breaks with preempt rcu? It's
> > > > > > clearly a use-after-free symptom, but I don't see how it can
> > > > > > happen.
> > > > >
> > > > > Some quick and probably off-the-mark questions...
> > > >
> > > > Thanks!
> > >
> > > Glad it actually was of help! ;-)
> >
> > Your reviews are ALWAYS greatly appreciated!
>
> :-)
>
> > > > > o What is the purpose of __call_for_each_cic()? When called
> > > > > from call_for_each_cic(), it is under rcu_read_lock(), as
> > > > > required, but it is also called from cfq_free_io_context(),
> > > > > which is assigned to the ->dtor and ->exit members of the
> > > > > cfq_io_context struct. What protects calls through these
> > > > > members?
> > > > >
> > > > > (This is for the ->cic_list field of the cfq_io_context structure.
> > > > > One possibility is that the io_context's ->lock member is held,
> > > > > but I don't see this. Not that I looked all that hard...)
> > > > >
> > > > > My suggestion would be to simply change all invocations of
> > > > > __call_for_each_cic() to instead invoke call_for_each_cic().
> > > > > The rcu_read_lock()/rcu_read_unlock() pair is pretty
> > > > > lightweight, even in CONFIG_PREEMPT_RCU.
> > > >
> > > > __call_for_each_cic() is always called under rcu_read_lock(), it merely
> > > > exists to avoid a double rcu_read_lock(). Even if it is cheap. The
> > > > convention follows the usual __lock_is_already_held() double under
> > > > score, but I guess it could do with a comment! There are only two
> > > > callers of the function, call_for_each_cic() which does the
> > > > rcu_read_lock(), and cfq_free_io_context() which is called from ->dtor
> > > > (and holds the rcu_read_lock() and ->trim which actually does not. That
> > > > looks like it could be problematic, but it's only called when an io
> > > > scheduler module is removed so not really critical. I'll add it, though!
> > > > Actually, the task_lock() should be enough there. So no bug, but (again)
> > > > it could do with a comment.
> > >
> > > Sounds good!
> > >
> > > > > o When calling cfq_slab_kill(), for example from cfq_exit(),
> > > > > what ensures that all previous RCU callbacks have completed?
> > > > >
> > > > > I suspect that you need an rcu_barrier() at the beginning
> > > > > of cfq_slab_kill(), but I could be missing something.
> > > >
> > > > So we have two callers of that, one is from the error path at init time
> > > > and is obviously ok. The other does need rcu_barrier()! I'll add that.
> > >
> > > OK, that does make my brain hurt less. ;-)
> >
> > So that one was also OK, as Fabio pointed out. If you follow the
> > ioc_gone and user tracking, the:
> >
> > if (elv_ioc_count_read(ioc_count))
> > wait_for_completion(ioc_gone);
> >
> > also has the side effect of waiting for RCU callbacks calling
> > kmem_cache_free() to have finished as well.
>
> I stand corrected.
But one additional question...
static void cfq_cic_free_rcu(struct rcu_head *head)
{
struct cfq_io_context *cic;
cic = container_of(head, struct cfq_io_context, rcu_head);
kmem_cache_free(cfq_ioc_pool, cic);
elv_ioc_count_dec(ioc_count);
if (ioc_gone && !elv_ioc_count_read(ioc_count))
complete(ioc_gone);
}
Suppose that a pair of tasks both execute the elv_ioc_count_dec()
at the same time, so that all counters are now zero. Both then
find that there is still an ioc_gone, and that the count is
now zero. One of the tasks invokes complete(ioc_gone). This
awakens the corresponding cfq_exit(), which now returns, getting
rid of its stack frame -- and corrupting the all_gone auto variable
that ioc_gone references.
Now the second task gets a big surprise when it tries to invoke
complete(ioc_gone).
Or is there something else that I am missing here?
Thanx, Paul
> > > > > o What protects the first rcu_dereference() in cfq_cic_lookup()?
> > > > > There needs to be either an enclose rcu_read_lock() on the
> > > > > one hand or the ->queue_lock needs to be held.
> > > > >
> > > > > (My guess is the latter, given the later rcu_assign_pointer()
> > > > > in this same function, but I don't see a lock acquisition
> > > > > in the immediate vicinity -- might be further up the function
> > > > > call stack, though.)
> > > >
> > > > There's no locking going into that function when coming from
> > > > cfq_get_io_context(), the other paths (happen) to hold the queue lock
> > > > already though.
> > >
> > > So the call from cfq_get_io_context() needs an rcu_read_lock()?
> > > Not seeing this in the patch below, but maybe you have it up a
> > > function-call level or two?
> >
> > It's in there, it now does:
> >
> > rcu_read_lock();
> > cic = rcu_dereference(ioc->ioc_data);
> > if (cic && cic->key == cfqd) {
> > rcu_read_unlock();
> > return cic;
> > }
> > ...
> >
> > OK? Which is basically what remains of the patch now, except for the
> > comment additions. Oh, and the ioc->lock protecting setting of
> > ->ioc_data as well. New version below. Alexey, care to give this a
> > spin? Seems your box is very well suited for finding RCU preempt
> > problems :-)
>
> OK, looks good.
>
> Thanx, Paul
>
> > diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> > index 4df3f05..d01b411 100644
> > --- a/block/cfq-iosched.c
> > +++ b/block/cfq-iosched.c
> > @@ -1142,6 +1142,9 @@ static void cfq_put_queue(struct cfq_queue *cfqq)
> > kmem_cache_free(cfq_pool, cfqq);
> > }
> >
> > +/*
> > + * Must always be called with the rcu_read_lock() held
> > + */
> > static void
> > __call_for_each_cic(struct io_context *ioc,
> > void (*func)(struct io_context *, struct cfq_io_context *))
> > @@ -1197,6 +1200,11 @@ static void cic_free_func(struct io_context *ioc, struct cfq_io_context *cic)
> > cfq_cic_free(cic);
> > }
> >
> > +/*
> > + * Must be called with rcu_read_lock() held or preemption otherwise disabled.
> > + * Only two callers of this - ->dtor() which is called with the rcu_read_lock(),
> > + * and ->trim() which is called with the task lock held
> > + */
> > static void cfq_free_io_context(struct io_context *ioc)
> > {
> > /*
> > @@ -1502,20 +1510,24 @@ static struct cfq_io_context *
> > cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
> > {
> > struct cfq_io_context *cic;
> > + unsigned long flags;
> > void *k;
> >
> > if (unlikely(!ioc))
> > return NULL;
> >
> > + rcu_read_lock();
> > +
> > /*
> > * we maintain a last-hit cache, to avoid browsing over the tree
> > */
> > cic = rcu_dereference(ioc->ioc_data);
> > - if (cic && cic->key == cfqd)
> > + if (cic && cic->key == cfqd) {
> > + rcu_read_unlock();
> > return cic;
> > + }
> >
> > do {
> > - rcu_read_lock();
> > cic = radix_tree_lookup(&ioc->radix_root, (unsigned long) cfqd);
> > rcu_read_unlock();
> > if (!cic)
> > @@ -1524,10 +1536,13 @@ cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
> > k = cic->key;
> > if (unlikely(!k)) {
> > cfq_drop_dead_cic(cfqd, ioc, cic);
> > + rcu_read_lock();
> > continue;
> > }
> >
> > + spin_lock_irqsave(&ioc->lock, flags);
> > rcu_assign_pointer(ioc->ioc_data, cic);
> > + spin_unlock_irqrestore(&ioc->lock, flags);
> > break;
> > } while (1);
> >
> > @@ -2134,6 +2149,10 @@ static void *cfq_init_queue(struct request_queue *q)
> >
> > static void cfq_slab_kill(void)
> > {
> > + /*
> > + * Caller already ensured that pending RCU callbacks are completed,
> > + * so we should have no busy allocations at this point.
> > + */
> > if (cfq_pool)
> > kmem_cache_destroy(cfq_pool);
> > if (cfq_ioc_pool)
> > @@ -2292,6 +2311,11 @@ static void __exit cfq_exit(void)
> > ioc_gone = &all_gone;
> > /* ioc_gone's update must be visible before reading ioc_count */
> > smp_wmb();
> > +
> > + /*
> > + * this also protects us from entering cfq_slab_kill() with
> > + * pending RCU callbacks
> > + */
> > if (elv_ioc_count_read(ioc_count))
> > wait_for_completion(ioc_gone);
> > cfq_slab_kill();
> >
> > --
> > Jens Axboe
> >
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-29 4:38 ` Paul E. McKenney
@ 2008-05-29 6:26 ` Jens Axboe
2008-05-29 6:42 ` Jens Axboe
0 siblings, 1 reply; 37+ messages in thread
From: Jens Axboe @ 2008-05-29 6:26 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Wed, May 28 2008, Paul E. McKenney wrote:
> On Wed, May 28, 2008 at 06:20:12AM -0700, Paul E. McKenney wrote:
> > On Wed, May 28, 2008 at 02:44:24PM +0200, Jens Axboe wrote:
> > > On Wed, May 28 2008, Paul E. McKenney wrote:
> > > > On Wed, May 28, 2008 at 12:07:21PM +0200, Jens Axboe wrote:
> > > > > On Tue, May 27 2008, Paul E. McKenney wrote:
> > > > > > On Tue, May 27, 2008 at 03:35:10PM +0200, Jens Axboe wrote:
> > > > > > > On Tue, May 27 2008, Alexey Dobriyan wrote:
> > > > > > > > On Sat, May 10, 2008 at 02:37:19PM +0400, Alexey Dobriyan wrote:
> > > > > > > > > > > > > @@ -41,8 +41,8 @@ int put_io_context(struct io_context *ioc)
> > > > > > > > > > > > > rcu_read_lock();
> > > > > > > > > > > > > if (ioc->aic && ioc->aic->dtor)
> > > > > > > > > > > > > ioc->aic->dtor(ioc->aic);
> > > > > > > > > > > > > - rcu_read_unlock();
> > > > > > > > > > > > > cfq_dtor(ioc);
> > > > > > > > > > > > > + rcu_read_unlock();
> > > > > > > > > > > > >
> > > > > > > > > > > > > kmem_cache_free(iocontext_cachep, ioc);
> > > > > > > > > > > > > return 1;
> > > > > > > > > > > >
> > > > > > > > > > > > This helps in sense that 3 times bulk cross-compiles finish to the end.
> > > > > > > > > > > > You'll hear me if another such oops will resurface.
> > > > > > > > > > >
> > > > > > > > > > > Still looking good?
> > > > > > > > > >
> > > > > > > > > > Yup!
> > > > > > > > >
> > > > > > > > > And this with patch in mainline, again with PREEMPT_RCU.
> > > > > > > >
> > > > > > > > Ping, this happened again with 2.6.26-rc4 and PREEMPT_RCU.
> > > > > > >
> > > > > > > Worrisome... Paul, would you mind taking a quick look at cfq
> > > > > > > and see if you can detect why breaks with preempt rcu? It's
> > > > > > > clearly a use-after-free symptom, but I don't see how it can
> > > > > > > happen.
> > > > > >
> > > > > > Some quick and probably off-the-mark questions...
> > > > >
> > > > > Thanks!
> > > >
> > > > Glad it actually was of help! ;-)
> > >
> > > Your reviews are ALWAYS greatly appreciated!
> >
> > :-)
> >
> > > > > > o What is the purpose of __call_for_each_cic()? When called
> > > > > > from call_for_each_cic(), it is under rcu_read_lock(), as
> > > > > > required, but it is also called from cfq_free_io_context(),
> > > > > > which is assigned to the ->dtor and ->exit members of the
> > > > > > cfq_io_context struct. What protects calls through these
> > > > > > members?
> > > > > >
> > > > > > (This is for the ->cic_list field of the cfq_io_context structure.
> > > > > > One possibility is that the io_context's ->lock member is held,
> > > > > > but I don't see this. Not that I looked all that hard...)
> > > > > >
> > > > > > My suggestion would be to simply change all invocations of
> > > > > > __call_for_each_cic() to instead invoke call_for_each_cic().
> > > > > > The rcu_read_lock()/rcu_read_unlock() pair is pretty
> > > > > > lightweight, even in CONFIG_PREEMPT_RCU.
> > > > >
> > > > > __call_for_each_cic() is always called under rcu_read_lock(), it merely
> > > > > exists to avoid a double rcu_read_lock(). Even if it is cheap. The
> > > > > convention follows the usual __lock_is_already_held() double under
> > > > > score, but I guess it could do with a comment! There are only two
> > > > > callers of the function, call_for_each_cic() which does the
> > > > > rcu_read_lock(), and cfq_free_io_context() which is called from ->dtor
> > > > > (and holds the rcu_read_lock() and ->trim which actually does not. That
> > > > > looks like it could be problematic, but it's only called when an io
> > > > > scheduler module is removed so not really critical. I'll add it, though!
> > > > > Actually, the task_lock() should be enough there. So no bug, but (again)
> > > > > it could do with a comment.
> > > >
> > > > Sounds good!
> > > >
> > > > > > o When calling cfq_slab_kill(), for example from cfq_exit(),
> > > > > > what ensures that all previous RCU callbacks have completed?
> > > > > >
> > > > > > I suspect that you need an rcu_barrier() at the beginning
> > > > > > of cfq_slab_kill(), but I could be missing something.
> > > > >
> > > > > So we have two callers of that, one is from the error path at init time
> > > > > and is obviously ok. The other does need rcu_barrier()! I'll add that.
> > > >
> > > > OK, that does make my brain hurt less. ;-)
> > >
> > > So that one was also OK, as Fabio pointed out. If you follow the
> > > ioc_gone and user tracking, the:
> > >
> > > if (elv_ioc_count_read(ioc_count))
> > > wait_for_completion(ioc_gone);
> > >
> > > also has the side effect of waiting for RCU callbacks calling
> > > kmem_cache_free() to have finished as well.
> >
> > I stand corrected.
>
> But one additional question...
>
> static void cfq_cic_free_rcu(struct rcu_head *head)
> {
> struct cfq_io_context *cic;
>
> cic = container_of(head, struct cfq_io_context, rcu_head);
>
> kmem_cache_free(cfq_ioc_pool, cic);
> elv_ioc_count_dec(ioc_count);
>
> if (ioc_gone && !elv_ioc_count_read(ioc_count))
> complete(ioc_gone);
> }
>
> Suppose that a pair of tasks both execute the elv_ioc_count_dec()
> at the same time, so that all counters are now zero. Both then
> find that there is still an ioc_gone, and that the count is
> now zero. One of the tasks invokes complete(ioc_gone). This
> awakens the corresponding cfq_exit(), which now returns, getting
> rid of its stack frame -- and corrupting the all_gone auto variable
> that ioc_gone references.
>
> Now the second task gets a big surprise when it tries to invoke
> complete(ioc_gone).
>
> Or is there something else that I am missing here?
No, I think that's a problem spot as well. To my knowledge, nobody has
ever hit that. The anticipatory scheduler has the same code.
What we want to avoid here is making cfq_cic_free_rcu() a lot more
expensive, which is why the elv_ioc_count_read() is behind that
ioc_gone check. I'll need to think a bit on how to handle that
better :-)
--
Jens Axboe
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-29 6:26 ` Jens Axboe
@ 2008-05-29 6:42 ` Jens Axboe
2008-05-29 9:17 ` Paul E. McKenney
0 siblings, 1 reply; 37+ messages in thread
From: Jens Axboe @ 2008-05-29 6:42 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Thu, May 29 2008, Jens Axboe wrote:
> > But one additional question...
> >
> > static void cfq_cic_free_rcu(struct rcu_head *head)
> > {
> > struct cfq_io_context *cic;
> >
> > cic = container_of(head, struct cfq_io_context, rcu_head);
> >
> > kmem_cache_free(cfq_ioc_pool, cic);
> > elv_ioc_count_dec(ioc_count);
> >
> > if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > complete(ioc_gone);
> > }
> >
> > Suppose that a pair of tasks both execute the elv_ioc_count_dec()
> > at the same time, so that all counters are now zero. Both then
> > find that there is still an ioc_gone, and that the count is
> > now zero. One of the tasks invokes complete(ioc_gone). This
> > awakens the corresponding cfq_exit(), which now returns, getting
> > rid of its stack frame -- and corrupting the all_gone auto variable
> > that ioc_gone references.
> >
> > Now the second task gets a big surprise when it tries to invoke
> > complete(ioc_gone).
> >
> > Or is there something else that I am missing here?
>
> No, I think that's a problem spot as well. To my knowledge, nobody has
> ever hit that. The anticipatory scheduler has the same code.
>
> What we want to avoid here is making cfq_cic_free_rcu() a lot more
> expensive, which is why the elv_ioc_count_read() is behind that
> ioc_gone check. I'll need to think a bit on how to handle that
> better :-)
So how about this? Add a spinlock for checking and clearing ioc_gone
back to NULL. It doesn't matter if we make the ioc_gone != NULL
case a little more expensive, as it will only happen on cfq-iosched
module unload. And it seems the clearest way of making this safe.
The last hunk should really not be necessary, as ioc_gone wont be
set back to NULL before wait_for_completion() is entered.
An identical patch is needed in AS as well.
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index d01b411..32aa367 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -48,6 +48,7 @@ static struct kmem_cache *cfq_ioc_pool;
static DEFINE_PER_CPU(unsigned long, ioc_count);
static struct completion *ioc_gone;
+static DEFINE_SPINLOCK(ioc_gone_lock);
#define CFQ_PRIO_LISTS IOPRIO_BE_NR
#define cfq_class_idle(cfqq) ((cfqq)->ioprio_class == IOPRIO_CLASS_IDLE)
@@ -1177,8 +1178,19 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
kmem_cache_free(cfq_ioc_pool, cic);
elv_ioc_count_dec(ioc_count);
- if (ioc_gone && !elv_ioc_count_read(ioc_count))
- complete(ioc_gone);
+ if (ioc_gone) {
+ /*
+ * CFQ scheduler is exiting, grab exit lock and check
+ * the pending io context count. If it hits zero,
+ * complete ioc_gone and set it back to NULL
+ */
+ spin_lock(&ioc_gone_lock);
+ if (ioc_gone && !elv_ioc_count_read(ioc_count)) {
+ complete(ioc_gone);
+ ioc_gone = NULL;
+ }
+ spin_unlock(&ioc_gone_lock);
+ }
}
static void cfq_cic_free(struct cfq_io_context *cic)
@@ -2317,7 +2329,7 @@ static void __exit cfq_exit(void)
* pending RCU callbacks
*/
if (elv_ioc_count_read(ioc_count))
- wait_for_completion(ioc_gone);
+ wait_for_completion(&all_gone);
cfq_slab_kill();
}
--
Jens Axboe
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-29 6:42 ` Jens Axboe
@ 2008-05-29 9:17 ` Paul E. McKenney
2008-05-29 10:13 ` Jens Axboe
0 siblings, 1 reply; 37+ messages in thread
From: Paul E. McKenney @ 2008-05-29 9:17 UTC (permalink / raw)
To: Jens Axboe; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Thu, May 29, 2008 at 08:42:02AM +0200, Jens Axboe wrote:
> On Thu, May 29 2008, Jens Axboe wrote:
> > > But one additional question...
> > >
> > > static void cfq_cic_free_rcu(struct rcu_head *head)
> > > {
> > > struct cfq_io_context *cic;
> > >
> > > cic = container_of(head, struct cfq_io_context, rcu_head);
> > >
> > > kmem_cache_free(cfq_ioc_pool, cic);
> > > elv_ioc_count_dec(ioc_count);
> > >
> > > if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > complete(ioc_gone);
> > > }
> > >
> > > Suppose that a pair of tasks both execute the elv_ioc_count_dec()
> > > at the same time, so that all counters are now zero. Both then
> > > find that there is still an ioc_gone, and that the count is
> > > now zero. One of the tasks invokes complete(ioc_gone). This
> > > awakens the corresponding cfq_exit(), which now returns, getting
> > > rid of its stack frame -- and corrupting the all_gone auto variable
> > > that ioc_gone references.
> > >
> > > Now the second task gets a big surprise when it tries to invoke
> > > complete(ioc_gone).
> > >
> > > Or is there something else that I am missing here?
> >
> > No, I think that's a problem spot as well. To my knowledge, nobody has
> > ever hit that. The anticipatory scheduler has the same code.
> >
> > What we want to avoid here is making cfq_cic_free_rcu() a lot more
> > expensive, which is why the elv_ioc_count_read() is behind that
> > ioc_gone check. I'll need to think a bit on how to handle that
> > better :-)
>
> So how about this? Add a spinlock for checking and clearing ioc_gone
> back to NULL. It doesn't matter if we make the ioc_gone != NULL
> case a little more expensive, as it will only happen on cfq-iosched
> module unload. And it seems the clearest way of making this safe.
> The last hunk should really not be necessary, as ioc_gone wont be
> set back to NULL before wait_for_completion() is entered.
Looks better! I do have one scenario that seems troublesome, but
it should be easy to fix, see below. (Assuming it really is a
problem, that is...)
Thanx, Paul
> An identical patch is needed in AS as well.
>
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> index d01b411..32aa367 100644
> --- a/block/cfq-iosched.c
> +++ b/block/cfq-iosched.c
> @@ -48,6 +48,7 @@ static struct kmem_cache *cfq_ioc_pool;
>
> static DEFINE_PER_CPU(unsigned long, ioc_count);
> static struct completion *ioc_gone;
> +static DEFINE_SPINLOCK(ioc_gone_lock);
>
> #define CFQ_PRIO_LISTS IOPRIO_BE_NR
> #define cfq_class_idle(cfqq) ((cfqq)->ioprio_class == IOPRIO_CLASS_IDLE)
> @@ -1177,8 +1178,19 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> kmem_cache_free(cfq_ioc_pool, cic);
> elv_ioc_count_dec(ioc_count);
>
> - if (ioc_gone && !elv_ioc_count_read(ioc_count))
> - complete(ioc_gone);
> + if (ioc_gone) {
> + /*
> + * CFQ scheduler is exiting, grab exit lock and check
> + * the pending io context count. If it hits zero,
> + * complete ioc_gone and set it back to NULL
> + */
Suppose that at this point some other CPU does the last complete().
They have set ioc_gone to NULL, so everything is fine. But suppose
that in the meantime, some other CPU sets up a cfq and then starts
tearing it down. Then ioc_gone would be non-NULL, and we would cause
this new teardown to end prematurely.
If this is a real problem, one way to get around it is to have a
generation number. We capture this before doing the elv_ioc_count_dec()
(alas, with a memory barrier between the capture and the elv_ioc_count_dec()),
and then check it under the lock. If it has changed, we know someone else
has already done the awakening for us. Increment the generation number
in the same place that ioc_gone is set to NULL.
Seem reasonable?
> + spin_lock(&ioc_gone_lock);
> + if (ioc_gone && !elv_ioc_count_read(ioc_count)) {
> + complete(ioc_gone);
> + ioc_gone = NULL;
> + }
> + spin_unlock(&ioc_gone_lock);
> + }
> }
>
> static void cfq_cic_free(struct cfq_io_context *cic)
> @@ -2317,7 +2329,7 @@ static void __exit cfq_exit(void)
> * pending RCU callbacks
> */
> if (elv_ioc_count_read(ioc_count))
> - wait_for_completion(ioc_gone);
> + wait_for_completion(&all_gone);
> cfq_slab_kill();
> }
>
>
> --
> Jens Axboe
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-29 9:17 ` Paul E. McKenney
@ 2008-05-29 10:13 ` Jens Axboe
2008-05-29 11:25 ` Paul E. McKenney
0 siblings, 1 reply; 37+ messages in thread
From: Jens Axboe @ 2008-05-29 10:13 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Thu, May 29 2008, Paul E. McKenney wrote:
> On Thu, May 29, 2008 at 08:42:02AM +0200, Jens Axboe wrote:
> > On Thu, May 29 2008, Jens Axboe wrote:
> > > > But one additional question...
> > > >
> > > > static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > {
> > > > struct cfq_io_context *cic;
> > > >
> > > > cic = container_of(head, struct cfq_io_context, rcu_head);
> > > >
> > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > elv_ioc_count_dec(ioc_count);
> > > >
> > > > if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > complete(ioc_gone);
> > > > }
> > > >
> > > > Suppose that a pair of tasks both execute the elv_ioc_count_dec()
> > > > at the same time, so that all counters are now zero. Both then
> > > > find that there is still an ioc_gone, and that the count is
> > > > now zero. One of the tasks invokes complete(ioc_gone). This
> > > > awakens the corresponding cfq_exit(), which now returns, getting
> > > > rid of its stack frame -- and corrupting the all_gone auto variable
> > > > that ioc_gone references.
> > > >
> > > > Now the second task gets a big surprise when it tries to invoke
> > > > complete(ioc_gone).
> > > >
> > > > Or is there something else that I am missing here?
> > >
> > > No, I think that's a problem spot as well. To my knowledge, nobody has
> > > ever hit that. The anticipatory scheduler has the same code.
> > >
> > > What we want to avoid here is making cfq_cic_free_rcu() a lot more
> > > expensive, which is why the elv_ioc_count_read() is behind that
> > > ioc_gone check. I'll need to think a bit on how to handle that
> > > better :-)
> >
> > So how about this? Add a spinlock for checking and clearing ioc_gone
> > back to NULL. It doesn't matter if we make the ioc_gone != NULL
> > case a little more expensive, as it will only happen on cfq-iosched
> > module unload. And it seems the clearest way of making this safe.
> > The last hunk should really not be necessary, as ioc_gone wont be
> > set back to NULL before wait_for_completion() is entered.
>
> Looks better! I do have one scenario that seems troublesome, but
> it should be easy to fix, see below. (Assuming it really is a
> problem, that is...)
>
> Thanx, Paul
>
> > An identical patch is needed in AS as well.
> >
> > diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> > index d01b411..32aa367 100644
> > --- a/block/cfq-iosched.c
> > +++ b/block/cfq-iosched.c
> > @@ -48,6 +48,7 @@ static struct kmem_cache *cfq_ioc_pool;
> >
> > static DEFINE_PER_CPU(unsigned long, ioc_count);
> > static struct completion *ioc_gone;
> > +static DEFINE_SPINLOCK(ioc_gone_lock);
> >
> > #define CFQ_PRIO_LISTS IOPRIO_BE_NR
> > #define cfq_class_idle(cfqq) ((cfqq)->ioprio_class == IOPRIO_CLASS_IDLE)
> > @@ -1177,8 +1178,19 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> > kmem_cache_free(cfq_ioc_pool, cic);
> > elv_ioc_count_dec(ioc_count);
> >
> > - if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > - complete(ioc_gone);
> > + if (ioc_gone) {
> > + /*
> > + * CFQ scheduler is exiting, grab exit lock and check
> > + * the pending io context count. If it hits zero,
> > + * complete ioc_gone and set it back to NULL
> > + */
>
> Suppose that at this point some other CPU does the last complete().
> They have set ioc_gone to NULL, so everything is fine. But suppose
> that in the meantime, some other CPU sets up a cfq and then starts
> tearing it down. Then ioc_gone would be non-NULL, and we would cause
> this new teardown to end prematurely.
>
> If this is a real problem, one way to get around it is to have a
> generation number. We capture this before doing the elv_ioc_count_dec()
> (alas, with a memory barrier between the capture and the elv_ioc_count_dec()),
> and then check it under the lock. If it has changed, we know someone else
> has already done the awakening for us. Increment the generation number
> in the same place that ioc_gone is set to NULL.
>
> Seem reasonable?
This isn't a problem, since cfq_exit() cannot be called before
all block queues in the system have been detached from CFQ.
cfq_exit() calls elv_unregister() before setting ioc_gone, so
when elv_unregister() has returned, CFQ is in its own little world.
Do we need an smp_wmb() between elv_unregister() and the ioc_gone
assignment to ensure this ordering as well? IIRC, the spin_lock
and spin_unlock in elv_unregister() isn't enough to guarentee this.
We are really down to splitting hairs now, but better safe than
sorry :-)
--
Jens Axboe
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-29 10:13 ` Jens Axboe
@ 2008-05-29 11:25 ` Paul E. McKenney
2008-05-29 11:44 ` Jens Axboe
0 siblings, 1 reply; 37+ messages in thread
From: Paul E. McKenney @ 2008-05-29 11:25 UTC (permalink / raw)
To: Jens Axboe; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Thu, May 29, 2008 at 12:13:54PM +0200, Jens Axboe wrote:
> On Thu, May 29 2008, Paul E. McKenney wrote:
> > On Thu, May 29, 2008 at 08:42:02AM +0200, Jens Axboe wrote:
> > > On Thu, May 29 2008, Jens Axboe wrote:
> > > > > But one additional question...
> > > > >
> > > > > static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > > {
> > > > > struct cfq_io_context *cic;
> > > > >
> > > > > cic = container_of(head, struct cfq_io_context, rcu_head);
> > > > >
> > > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > > elv_ioc_count_dec(ioc_count);
> > > > >
> > > > > if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > > complete(ioc_gone);
> > > > > }
> > > > >
> > > > > Suppose that a pair of tasks both execute the elv_ioc_count_dec()
> > > > > at the same time, so that all counters are now zero. Both then
> > > > > find that there is still an ioc_gone, and that the count is
> > > > > now zero. One of the tasks invokes complete(ioc_gone). This
> > > > > awakens the corresponding cfq_exit(), which now returns, getting
> > > > > rid of its stack frame -- and corrupting the all_gone auto variable
> > > > > that ioc_gone references.
> > > > >
> > > > > Now the second task gets a big surprise when it tries to invoke
> > > > > complete(ioc_gone).
> > > > >
> > > > > Or is there something else that I am missing here?
> > > >
> > > > No, I think that's a problem spot as well. To my knowledge, nobody has
> > > > ever hit that. The anticipatory scheduler has the same code.
> > > >
> > > > What we want to avoid here is making cfq_cic_free_rcu() a lot more
> > > > expensive, which is why the elv_ioc_count_read() is behind that
> > > > ioc_gone check. I'll need to think a bit on how to handle that
> > > > better :-)
> > >
> > > So how about this? Add a spinlock for checking and clearing ioc_gone
> > > back to NULL. It doesn't matter if we make the ioc_gone != NULL
> > > case a little more expensive, as it will only happen on cfq-iosched
> > > module unload. And it seems the clearest way of making this safe.
> > > The last hunk should really not be necessary, as ioc_gone wont be
> > > set back to NULL before wait_for_completion() is entered.
> >
> > Looks better! I do have one scenario that seems troublesome, but
> > it should be easy to fix, see below. (Assuming it really is a
> > problem, that is...)
> >
> > Thanx, Paul
> >
> > > An identical patch is needed in AS as well.
> > >
> > > diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> > > index d01b411..32aa367 100644
> > > --- a/block/cfq-iosched.c
> > > +++ b/block/cfq-iosched.c
> > > @@ -48,6 +48,7 @@ static struct kmem_cache *cfq_ioc_pool;
> > >
> > > static DEFINE_PER_CPU(unsigned long, ioc_count);
> > > static struct completion *ioc_gone;
> > > +static DEFINE_SPINLOCK(ioc_gone_lock);
> > >
> > > #define CFQ_PRIO_LISTS IOPRIO_BE_NR
> > > #define cfq_class_idle(cfqq) ((cfqq)->ioprio_class == IOPRIO_CLASS_IDLE)
> > > @@ -1177,8 +1178,19 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> > > kmem_cache_free(cfq_ioc_pool, cic);
> > > elv_ioc_count_dec(ioc_count);
> > >
> > > - if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > - complete(ioc_gone);
> > > + if (ioc_gone) {
> > > + /*
> > > + * CFQ scheduler is exiting, grab exit lock and check
> > > + * the pending io context count. If it hits zero,
> > > + * complete ioc_gone and set it back to NULL
> > > + */
> >
> > Suppose that at this point some other CPU does the last complete().
> > They have set ioc_gone to NULL, so everything is fine. But suppose
> > that in the meantime, some other CPU sets up a cfq and then starts
> > tearing it down. Then ioc_gone would be non-NULL, and we would cause
> > this new teardown to end prematurely.
> >
> > If this is a real problem, one way to get around it is to have a
> > generation number. We capture this before doing the elv_ioc_count_dec()
> > (alas, with a memory barrier between the capture and the elv_ioc_count_dec()),
> > and then check it under the lock. If it has changed, we know someone else
> > has already done the awakening for us. Increment the generation number
> > in the same place that ioc_gone is set to NULL.
> >
> > Seem reasonable?
>
> This isn't a problem, since cfq_exit() cannot be called before
> all block queues in the system have been detached from CFQ.
And once all block queues have been detached, no future block queues
can ever be attached again? Or perhaps a better way of putting it,
once CFQ has been shut down, can it be restarted without rebooting
the system? If it can be restarted without reboot, then I do not
yet see how the scenario above is avoided.
> cfq_exit() calls elv_unregister() before setting ioc_gone, so
> when elv_unregister() has returned, CFQ is in its own little world.
> Do we need an smp_wmb() between elv_unregister() and the ioc_gone
> assignment to ensure this ordering as well? IIRC, the spin_lock
> and spin_unlock in elv_unregister() isn't enough to guarentee this.
> We are really down to splitting hairs now, but better safe than
> sorry :-)
I believe that the spinlock takes care of that ordering issue. I am
instead worried about a "Rip Van Winkle" effect where a given task
is delayed at a crucial point. By the time it wakes back up, CFQ
has been not only restarted, but is now in the process of being torn
down again. (Assuming that it can in fact be restarted without
a reboot.)
Thanx, Paul
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-29 11:25 ` Paul E. McKenney
@ 2008-05-29 11:44 ` Jens Axboe
2008-05-29 12:11 ` Paul E. McKenney
0 siblings, 1 reply; 37+ messages in thread
From: Jens Axboe @ 2008-05-29 11:44 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Thu, May 29 2008, Paul E. McKenney wrote:
> On Thu, May 29, 2008 at 12:13:54PM +0200, Jens Axboe wrote:
> > On Thu, May 29 2008, Paul E. McKenney wrote:
> > > On Thu, May 29, 2008 at 08:42:02AM +0200, Jens Axboe wrote:
> > > > On Thu, May 29 2008, Jens Axboe wrote:
> > > > > > But one additional question...
> > > > > >
> > > > > > static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > > > {
> > > > > > struct cfq_io_context *cic;
> > > > > >
> > > > > > cic = container_of(head, struct cfq_io_context, rcu_head);
> > > > > >
> > > > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > > > elv_ioc_count_dec(ioc_count);
> > > > > >
> > > > > > if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > > > complete(ioc_gone);
> > > > > > }
> > > > > >
> > > > > > Suppose that a pair of tasks both execute the elv_ioc_count_dec()
> > > > > > at the same time, so that all counters are now zero. Both then
> > > > > > find that there is still an ioc_gone, and that the count is
> > > > > > now zero. One of the tasks invokes complete(ioc_gone). This
> > > > > > awakens the corresponding cfq_exit(), which now returns, getting
> > > > > > rid of its stack frame -- and corrupting the all_gone auto variable
> > > > > > that ioc_gone references.
> > > > > >
> > > > > > Now the second task gets a big surprise when it tries to invoke
> > > > > > complete(ioc_gone).
> > > > > >
> > > > > > Or is there something else that I am missing here?
> > > > >
> > > > > No, I think that's a problem spot as well. To my knowledge, nobody has
> > > > > ever hit that. The anticipatory scheduler has the same code.
> > > > >
> > > > > What we want to avoid here is making cfq_cic_free_rcu() a lot more
> > > > > expensive, which is why the elv_ioc_count_read() is behind that
> > > > > ioc_gone check. I'll need to think a bit on how to handle that
> > > > > better :-)
> > > >
> > > > So how about this? Add a spinlock for checking and clearing ioc_gone
> > > > back to NULL. It doesn't matter if we make the ioc_gone != NULL
> > > > case a little more expensive, as it will only happen on cfq-iosched
> > > > module unload. And it seems the clearest way of making this safe.
> > > > The last hunk should really not be necessary, as ioc_gone wont be
> > > > set back to NULL before wait_for_completion() is entered.
> > >
> > > Looks better! I do have one scenario that seems troublesome, but
> > > it should be easy to fix, see below. (Assuming it really is a
> > > problem, that is...)
> > >
> > > Thanx, Paul
> > >
> > > > An identical patch is needed in AS as well.
> > > >
> > > > diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> > > > index d01b411..32aa367 100644
> > > > --- a/block/cfq-iosched.c
> > > > +++ b/block/cfq-iosched.c
> > > > @@ -48,6 +48,7 @@ static struct kmem_cache *cfq_ioc_pool;
> > > >
> > > > static DEFINE_PER_CPU(unsigned long, ioc_count);
> > > > static struct completion *ioc_gone;
> > > > +static DEFINE_SPINLOCK(ioc_gone_lock);
> > > >
> > > > #define CFQ_PRIO_LISTS IOPRIO_BE_NR
> > > > #define cfq_class_idle(cfqq) ((cfqq)->ioprio_class == IOPRIO_CLASS_IDLE)
> > > > @@ -1177,8 +1178,19 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > elv_ioc_count_dec(ioc_count);
> > > >
> > > > - if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > - complete(ioc_gone);
> > > > + if (ioc_gone) {
> > > > + /*
> > > > + * CFQ scheduler is exiting, grab exit lock and check
> > > > + * the pending io context count. If it hits zero,
> > > > + * complete ioc_gone and set it back to NULL
> > > > + */
> > >
> > > Suppose that at this point some other CPU does the last complete().
> > > They have set ioc_gone to NULL, so everything is fine. But suppose
> > > that in the meantime, some other CPU sets up a cfq and then starts
> > > tearing it down. Then ioc_gone would be non-NULL, and we would cause
> > > this new teardown to end prematurely.
> > >
> > > If this is a real problem, one way to get around it is to have a
> > > generation number. We capture this before doing the elv_ioc_count_dec()
> > > (alas, with a memory barrier between the capture and the elv_ioc_count_dec()),
> > > and then check it under the lock. If it has changed, we know someone else
> > > has already done the awakening for us. Increment the generation number
> > > in the same place that ioc_gone is set to NULL.
> > >
> > > Seem reasonable?
> >
> > This isn't a problem, since cfq_exit() cannot be called before
> > all block queues in the system have been detached from CFQ.
>
> And once all block queues have been detached, no future block queues
> can ever be attached again? Or perhaps a better way of putting it,
> once CFQ has been shut down, can it be restarted without rebooting
> the system? If it can be restarted without reboot, then I do not
> yet see how the scenario above is avoided.
Certainly, you can modprobe cfq-iosched and switch any device to cfq
again. But I really don't see the problem in this case. Once CFQ
has exited, all tasks are detached from CFQ. They have to be, or
we could not unload the module.
> > cfq_exit() calls elv_unregister() before setting ioc_gone, so
> > when elv_unregister() has returned, CFQ is in its own little world.
> > Do we need an smp_wmb() between elv_unregister() and the ioc_gone
> > assignment to ensure this ordering as well? IIRC, the spin_lock
> > and spin_unlock in elv_unregister() isn't enough to guarentee this.
> > We are really down to splitting hairs now, but better safe than
> > sorry :-)
>
> I believe that the spinlock takes care of that ordering issue. I am
> instead worried about a "Rip Van Winkle" effect where a given task
> is delayed at a crucial point. By the time it wakes back up, CFQ
> has been not only restarted, but is now in the process of being torn
> down again. (Assuming that it can in fact be restarted without
> a reboot.)
Why would we have done the ioc dec and complete() before that task
had finished?
--
Jens Axboe
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-29 11:44 ` Jens Axboe
@ 2008-05-29 12:11 ` Paul E. McKenney
2008-05-29 12:13 ` Jens Axboe
0 siblings, 1 reply; 37+ messages in thread
From: Paul E. McKenney @ 2008-05-29 12:11 UTC (permalink / raw)
To: Jens Axboe; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Thu, May 29, 2008 at 01:44:31PM +0200, Jens Axboe wrote:
> On Thu, May 29 2008, Paul E. McKenney wrote:
> > On Thu, May 29, 2008 at 12:13:54PM +0200, Jens Axboe wrote:
> > > On Thu, May 29 2008, Paul E. McKenney wrote:
> > > > On Thu, May 29, 2008 at 08:42:02AM +0200, Jens Axboe wrote:
> > > > > On Thu, May 29 2008, Jens Axboe wrote:
> > > > > > > But one additional question...
> > > > > > >
> > > > > > > static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > > > > {
> > > > > > > struct cfq_io_context *cic;
> > > > > > >
> > > > > > > cic = container_of(head, struct cfq_io_context, rcu_head);
> > > > > > >
> > > > > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > > > > elv_ioc_count_dec(ioc_count);
> > > > > > >
> > > > > > > if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > > > > complete(ioc_gone);
> > > > > > > }
> > > > > > >
> > > > > > > Suppose that a pair of tasks both execute the elv_ioc_count_dec()
> > > > > > > at the same time, so that all counters are now zero. Both then
> > > > > > > find that there is still an ioc_gone, and that the count is
> > > > > > > now zero. One of the tasks invokes complete(ioc_gone). This
> > > > > > > awakens the corresponding cfq_exit(), which now returns, getting
> > > > > > > rid of its stack frame -- and corrupting the all_gone auto variable
> > > > > > > that ioc_gone references.
> > > > > > >
> > > > > > > Now the second task gets a big surprise when it tries to invoke
> > > > > > > complete(ioc_gone).
> > > > > > >
> > > > > > > Or is there something else that I am missing here?
> > > > > >
> > > > > > No, I think that's a problem spot as well. To my knowledge, nobody has
> > > > > > ever hit that. The anticipatory scheduler has the same code.
> > > > > >
> > > > > > What we want to avoid here is making cfq_cic_free_rcu() a lot more
> > > > > > expensive, which is why the elv_ioc_count_read() is behind that
> > > > > > ioc_gone check. I'll need to think a bit on how to handle that
> > > > > > better :-)
> > > > >
> > > > > So how about this? Add a spinlock for checking and clearing ioc_gone
> > > > > back to NULL. It doesn't matter if we make the ioc_gone != NULL
> > > > > case a little more expensive, as it will only happen on cfq-iosched
> > > > > module unload. And it seems the clearest way of making this safe.
> > > > > The last hunk should really not be necessary, as ioc_gone wont be
> > > > > set back to NULL before wait_for_completion() is entered.
> > > >
> > > > Looks better! I do have one scenario that seems troublesome, but
> > > > it should be easy to fix, see below. (Assuming it really is a
> > > > problem, that is...)
> > > >
> > > > Thanx, Paul
> > > >
> > > > > An identical patch is needed in AS as well.
> > > > >
> > > > > diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> > > > > index d01b411..32aa367 100644
> > > > > --- a/block/cfq-iosched.c
> > > > > +++ b/block/cfq-iosched.c
> > > > > @@ -48,6 +48,7 @@ static struct kmem_cache *cfq_ioc_pool;
> > > > >
> > > > > static DEFINE_PER_CPU(unsigned long, ioc_count);
> > > > > static struct completion *ioc_gone;
> > > > > +static DEFINE_SPINLOCK(ioc_gone_lock);
> > > > >
> > > > > #define CFQ_PRIO_LISTS IOPRIO_BE_NR
> > > > > #define cfq_class_idle(cfqq) ((cfqq)->ioprio_class == IOPRIO_CLASS_IDLE)
> > > > > @@ -1177,8 +1178,19 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > > elv_ioc_count_dec(ioc_count);
> > > > >
> > > > > - if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > > - complete(ioc_gone);
> > > > > + if (ioc_gone) {
> > > > > + /*
> > > > > + * CFQ scheduler is exiting, grab exit lock and check
> > > > > + * the pending io context count. If it hits zero,
> > > > > + * complete ioc_gone and set it back to NULL
> > > > > + */
> > > >
> > > > Suppose that at this point some other CPU does the last complete().
> > > > They have set ioc_gone to NULL, so everything is fine. But suppose
> > > > that in the meantime, some other CPU sets up a cfq and then starts
> > > > tearing it down. Then ioc_gone would be non-NULL, and we would cause
> > > > this new teardown to end prematurely.
> > > >
> > > > If this is a real problem, one way to get around it is to have a
> > > > generation number. We capture this before doing the elv_ioc_count_dec()
> > > > (alas, with a memory barrier between the capture and the elv_ioc_count_dec()),
> > > > and then check it under the lock. If it has changed, we know someone else
> > > > has already done the awakening for us. Increment the generation number
> > > > in the same place that ioc_gone is set to NULL.
> > > >
> > > > Seem reasonable?
> > >
> > > This isn't a problem, since cfq_exit() cannot be called before
> > > all block queues in the system have been detached from CFQ.
> >
> > And once all block queues have been detached, no future block queues
> > can ever be attached again? Or perhaps a better way of putting it,
> > once CFQ has been shut down, can it be restarted without rebooting
> > the system? If it can be restarted without reboot, then I do not
> > yet see how the scenario above is avoided.
>
> Certainly, you can modprobe cfq-iosched and switch any device to cfq
> again. But I really don't see the problem in this case. Once CFQ
> has exited, all tasks are detached from CFQ. They have to be, or
> we could not unload the module.
Ah, OK -- so we cannot reload the module until the earlier instantiation
has been unloaded, which cannot happen until everyone has returned from
any code in the CFQ module. If it is linked directly into the kernel,
it can only be "unloaded" by bringing the system down.
Sorry for my confusion!
This approach looks good to me.
Thanx, Paul
> > > cfq_exit() calls elv_unregister() before setting ioc_gone, so
> > > when elv_unregister() has returned, CFQ is in its own little world.
> > > Do we need an smp_wmb() between elv_unregister() and the ioc_gone
> > > assignment to ensure this ordering as well? IIRC, the spin_lock
> > > and spin_unlock in elv_unregister() isn't enough to guarentee this.
> > > We are really down to splitting hairs now, but better safe than
> > > sorry :-)
> >
> > I believe that the spinlock takes care of that ordering issue. I am
> > instead worried about a "Rip Van Winkle" effect where a given task
> > is delayed at a crucial point. By the time it wakes back up, CFQ
> > has been not only restarted, but is now in the process of being torn
> > down again. (Assuming that it can in fact be restarted without
> > a reboot.)
>
> Why would we have done the ioc dec and complete() before that task
> had finished?
>
> --
> Jens Axboe
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-29 12:11 ` Paul E. McKenney
@ 2008-05-29 12:13 ` Jens Axboe
2008-05-30 11:04 ` Paul E. McKenney
0 siblings, 1 reply; 37+ messages in thread
From: Jens Axboe @ 2008-05-29 12:13 UTC (permalink / raw)
To: Paul E. McKenney; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Thu, May 29 2008, Paul E. McKenney wrote:
> On Thu, May 29, 2008 at 01:44:31PM +0200, Jens Axboe wrote:
> > On Thu, May 29 2008, Paul E. McKenney wrote:
> > > On Thu, May 29, 2008 at 12:13:54PM +0200, Jens Axboe wrote:
> > > > On Thu, May 29 2008, Paul E. McKenney wrote:
> > > > > On Thu, May 29, 2008 at 08:42:02AM +0200, Jens Axboe wrote:
> > > > > > On Thu, May 29 2008, Jens Axboe wrote:
> > > > > > > > But one additional question...
> > > > > > > >
> > > > > > > > static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > > > > > {
> > > > > > > > struct cfq_io_context *cic;
> > > > > > > >
> > > > > > > > cic = container_of(head, struct cfq_io_context, rcu_head);
> > > > > > > >
> > > > > > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > > > > > elv_ioc_count_dec(ioc_count);
> > > > > > > >
> > > > > > > > if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > > > > > complete(ioc_gone);
> > > > > > > > }
> > > > > > > >
> > > > > > > > Suppose that a pair of tasks both execute the elv_ioc_count_dec()
> > > > > > > > at the same time, so that all counters are now zero. Both then
> > > > > > > > find that there is still an ioc_gone, and that the count is
> > > > > > > > now zero. One of the tasks invokes complete(ioc_gone). This
> > > > > > > > awakens the corresponding cfq_exit(), which now returns, getting
> > > > > > > > rid of its stack frame -- and corrupting the all_gone auto variable
> > > > > > > > that ioc_gone references.
> > > > > > > >
> > > > > > > > Now the second task gets a big surprise when it tries to invoke
> > > > > > > > complete(ioc_gone).
> > > > > > > >
> > > > > > > > Or is there something else that I am missing here?
> > > > > > >
> > > > > > > No, I think that's a problem spot as well. To my knowledge, nobody has
> > > > > > > ever hit that. The anticipatory scheduler has the same code.
> > > > > > >
> > > > > > > What we want to avoid here is making cfq_cic_free_rcu() a lot more
> > > > > > > expensive, which is why the elv_ioc_count_read() is behind that
> > > > > > > ioc_gone check. I'll need to think a bit on how to handle that
> > > > > > > better :-)
> > > > > >
> > > > > > So how about this? Add a spinlock for checking and clearing ioc_gone
> > > > > > back to NULL. It doesn't matter if we make the ioc_gone != NULL
> > > > > > case a little more expensive, as it will only happen on cfq-iosched
> > > > > > module unload. And it seems the clearest way of making this safe.
> > > > > > The last hunk should really not be necessary, as ioc_gone wont be
> > > > > > set back to NULL before wait_for_completion() is entered.
> > > > >
> > > > > Looks better! I do have one scenario that seems troublesome, but
> > > > > it should be easy to fix, see below. (Assuming it really is a
> > > > > problem, that is...)
> > > > >
> > > > > Thanx, Paul
> > > > >
> > > > > > An identical patch is needed in AS as well.
> > > > > >
> > > > > > diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> > > > > > index d01b411..32aa367 100644
> > > > > > --- a/block/cfq-iosched.c
> > > > > > +++ b/block/cfq-iosched.c
> > > > > > @@ -48,6 +48,7 @@ static struct kmem_cache *cfq_ioc_pool;
> > > > > >
> > > > > > static DEFINE_PER_CPU(unsigned long, ioc_count);
> > > > > > static struct completion *ioc_gone;
> > > > > > +static DEFINE_SPINLOCK(ioc_gone_lock);
> > > > > >
> > > > > > #define CFQ_PRIO_LISTS IOPRIO_BE_NR
> > > > > > #define cfq_class_idle(cfqq) ((cfqq)->ioprio_class == IOPRIO_CLASS_IDLE)
> > > > > > @@ -1177,8 +1178,19 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > > > elv_ioc_count_dec(ioc_count);
> > > > > >
> > > > > > - if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > > > - complete(ioc_gone);
> > > > > > + if (ioc_gone) {
> > > > > > + /*
> > > > > > + * CFQ scheduler is exiting, grab exit lock and check
> > > > > > + * the pending io context count. If it hits zero,
> > > > > > + * complete ioc_gone and set it back to NULL
> > > > > > + */
> > > > >
> > > > > Suppose that at this point some other CPU does the last complete().
> > > > > They have set ioc_gone to NULL, so everything is fine. But suppose
> > > > > that in the meantime, some other CPU sets up a cfq and then starts
> > > > > tearing it down. Then ioc_gone would be non-NULL, and we would cause
> > > > > this new teardown to end prematurely.
> > > > >
> > > > > If this is a real problem, one way to get around it is to have a
> > > > > generation number. We capture this before doing the elv_ioc_count_dec()
> > > > > (alas, with a memory barrier between the capture and the elv_ioc_count_dec()),
> > > > > and then check it under the lock. If it has changed, we know someone else
> > > > > has already done the awakening for us. Increment the generation number
> > > > > in the same place that ioc_gone is set to NULL.
> > > > >
> > > > > Seem reasonable?
> > > >
> > > > This isn't a problem, since cfq_exit() cannot be called before
> > > > all block queues in the system have been detached from CFQ.
> > >
> > > And once all block queues have been detached, no future block queues
> > > can ever be attached again? Or perhaps a better way of putting it,
> > > once CFQ has been shut down, can it be restarted without rebooting
> > > the system? If it can be restarted without reboot, then I do not
> > > yet see how the scenario above is avoided.
> >
> > Certainly, you can modprobe cfq-iosched and switch any device to cfq
> > again. But I really don't see the problem in this case. Once CFQ
> > has exited, all tasks are detached from CFQ. They have to be, or
> > we could not unload the module.
>
> Ah, OK -- so we cannot reload the module until the earlier
> instantiation has been unloaded, which cannot happen until everyone
> has returned from any code in the CFQ module. If it is linked
> directly into the kernel, it can only be "unloaded" by bringing the
> system down.
Right!
> Sorry for my confusion!
>
> This approach looks good to me.
Pheew good, I was starting to have a brain melt down as well :-)
--
Jens Axboe
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-29 12:13 ` Jens Axboe
@ 2008-05-30 11:04 ` Paul E. McKenney
2008-05-30 13:16 ` Paul E. McKenney
0 siblings, 1 reply; 37+ messages in thread
From: Paul E. McKenney @ 2008-05-30 11:04 UTC (permalink / raw)
To: Jens Axboe; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Thu, May 29, 2008 at 02:13:53PM +0200, Jens Axboe wrote:
> On Thu, May 29 2008, Paul E. McKenney wrote:
> > On Thu, May 29, 2008 at 01:44:31PM +0200, Jens Axboe wrote:
> > > On Thu, May 29 2008, Paul E. McKenney wrote:
> > > > On Thu, May 29, 2008 at 12:13:54PM +0200, Jens Axboe wrote:
> > > > > On Thu, May 29 2008, Paul E. McKenney wrote:
> > > > > > On Thu, May 29, 2008 at 08:42:02AM +0200, Jens Axboe wrote:
> > > > > > > On Thu, May 29 2008, Jens Axboe wrote:
> > > > > > > > > But one additional question...
> > > > > > > > >
> > > > > > > > > static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > > > > > > {
> > > > > > > > > struct cfq_io_context *cic;
> > > > > > > > >
> > > > > > > > > cic = container_of(head, struct cfq_io_context, rcu_head);
> > > > > > > > >
> > > > > > > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > > > > > > elv_ioc_count_dec(ioc_count);
> > > > > > > > >
> > > > > > > > > if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > > > > > > complete(ioc_gone);
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > > Suppose that a pair of tasks both execute the elv_ioc_count_dec()
> > > > > > > > > at the same time, so that all counters are now zero. Both then
> > > > > > > > > find that there is still an ioc_gone, and that the count is
> > > > > > > > > now zero. One of the tasks invokes complete(ioc_gone). This
> > > > > > > > > awakens the corresponding cfq_exit(), which now returns, getting
> > > > > > > > > rid of its stack frame -- and corrupting the all_gone auto variable
> > > > > > > > > that ioc_gone references.
> > > > > > > > >
> > > > > > > > > Now the second task gets a big surprise when it tries to invoke
> > > > > > > > > complete(ioc_gone).
> > > > > > > > >
> > > > > > > > > Or is there something else that I am missing here?
> > > > > > > >
> > > > > > > > No, I think that's a problem spot as well. To my knowledge, nobody has
> > > > > > > > ever hit that. The anticipatory scheduler has the same code.
> > > > > > > >
> > > > > > > > What we want to avoid here is making cfq_cic_free_rcu() a lot more
> > > > > > > > expensive, which is why the elv_ioc_count_read() is behind that
> > > > > > > > ioc_gone check. I'll need to think a bit on how to handle that
> > > > > > > > better :-)
> > > > > > >
> > > > > > > So how about this? Add a spinlock for checking and clearing ioc_gone
> > > > > > > back to NULL. It doesn't matter if we make the ioc_gone != NULL
> > > > > > > case a little more expensive, as it will only happen on cfq-iosched
> > > > > > > module unload. And it seems the clearest way of making this safe.
> > > > > > > The last hunk should really not be necessary, as ioc_gone wont be
> > > > > > > set back to NULL before wait_for_completion() is entered.
> > > > > >
> > > > > > Looks better! I do have one scenario that seems troublesome, but
> > > > > > it should be easy to fix, see below. (Assuming it really is a
> > > > > > problem, that is...)
> > > > > >
> > > > > > Thanx, Paul
> > > > > >
> > > > > > > An identical patch is needed in AS as well.
> > > > > > >
> > > > > > > diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> > > > > > > index d01b411..32aa367 100644
> > > > > > > --- a/block/cfq-iosched.c
> > > > > > > +++ b/block/cfq-iosched.c
> > > > > > > @@ -48,6 +48,7 @@ static struct kmem_cache *cfq_ioc_pool;
> > > > > > >
> > > > > > > static DEFINE_PER_CPU(unsigned long, ioc_count);
> > > > > > > static struct completion *ioc_gone;
> > > > > > > +static DEFINE_SPINLOCK(ioc_gone_lock);
> > > > > > >
> > > > > > > #define CFQ_PRIO_LISTS IOPRIO_BE_NR
> > > > > > > #define cfq_class_idle(cfqq) ((cfqq)->ioprio_class == IOPRIO_CLASS_IDLE)
> > > > > > > @@ -1177,8 +1178,19 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > > > > elv_ioc_count_dec(ioc_count);
> > > > > > >
> > > > > > > - if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > > > > - complete(ioc_gone);
> > > > > > > + if (ioc_gone) {
> > > > > > > + /*
> > > > > > > + * CFQ scheduler is exiting, grab exit lock and check
> > > > > > > + * the pending io context count. If it hits zero,
> > > > > > > + * complete ioc_gone and set it back to NULL
> > > > > > > + */
> > > > > >
> > > > > > Suppose that at this point some other CPU does the last complete().
> > > > > > They have set ioc_gone to NULL, so everything is fine. But suppose
> > > > > > that in the meantime, some other CPU sets up a cfq and then starts
> > > > > > tearing it down. Then ioc_gone would be non-NULL, and we would cause
> > > > > > this new teardown to end prematurely.
> > > > > >
> > > > > > If this is a real problem, one way to get around it is to have a
> > > > > > generation number. We capture this before doing the elv_ioc_count_dec()
> > > > > > (alas, with a memory barrier between the capture and the elv_ioc_count_dec()),
> > > > > > and then check it under the lock. If it has changed, we know someone else
> > > > > > has already done the awakening for us. Increment the generation number
> > > > > > in the same place that ioc_gone is set to NULL.
> > > > > >
> > > > > > Seem reasonable?
> > > > >
> > > > > This isn't a problem, since cfq_exit() cannot be called before
> > > > > all block queues in the system have been detached from CFQ.
> > > >
> > > > And once all block queues have been detached, no future block queues
> > > > can ever be attached again? Or perhaps a better way of putting it,
> > > > once CFQ has been shut down, can it be restarted without rebooting
> > > > the system? If it can be restarted without reboot, then I do not
> > > > yet see how the scenario above is avoided.
> > >
> > > Certainly, you can modprobe cfq-iosched and switch any device to cfq
> > > again. But I really don't see the problem in this case. Once CFQ
> > > has exited, all tasks are detached from CFQ. They have to be, or
> > > we could not unload the module.
> >
> > Ah, OK -- so we cannot reload the module until the earlier
> > instantiation has been unloaded, which cannot happen until everyone
> > has returned from any code in the CFQ module. If it is linked
> > directly into the kernel, it can only be "unloaded" by bringing the
> > system down.
>
> Right!
>
> > Sorry for my confusion!
> >
> > This approach looks good to me.
>
> Pheew good, I was starting to have a brain melt down as well :-)
OK, my apologies in advance, but one more thing to consider...
> @@ -1177,8 +1178,19 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> kmem_cache_free(cfq_ioc_pool, cic);
> elv_ioc_count_dec(ioc_count);
Need an smp_mb() here, and ...
>
> - if (ioc_gone && !elv_ioc_count_read(ioc_count))
> - complete(ioc_gone);
> + if (ioc_gone) {
> + /*
> + * CFQ scheduler is exiting, grab exit lock and check
> + * the pending io context count. If it hits zero,
> + * complete ioc_gone and set it back to NULL
> + */
> + spin_lock(&ioc_gone_lock);
> + if (ioc_gone && !elv_ioc_count_read(ioc_count)) {
> + complete(ioc_gone);
> + ioc_gone = NULL;
> + }
> + spin_unlock(&ioc_gone_lock);
> + }
> }
>
> static void cfq_cic_free(struct cfq_io_context *cic)
> static void __exit cfq_exit(void)
> {
> DECLARE_COMPLETION_ONSTACK(all_gone);
> elv_unregister(&iosched_cfq);
> ioc_gone = &all_gone;
> /* ioc_gone's update must be visible before reading ioc_count */
> smp_wmb();
This needs to be upgraded to an smp_mb() because we have a write followed
by a read that we need to keep ordered.
> if (elv_ioc_count_read(ioc_count))
> wait_for_completion(ioc_gone);
> cfq_slab_kill();
> }
In absence of these two changes, either compiler or CPU could reorder
the accesses to ioc_gone on the one hand and to ioc_count on the other,
which could result in a hang, for example, cfq_exit() might see the
reader's old count, but the reader might fail to see the assignment
to ioc_gone.
Thanx, Paul
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-30 11:04 ` Paul E. McKenney
@ 2008-05-30 13:16 ` Paul E. McKenney
0 siblings, 0 replies; 37+ messages in thread
From: Paul E. McKenney @ 2008-05-30 13:16 UTC (permalink / raw)
To: Jens Axboe; +Cc: Alexey Dobriyan, torvalds, Andrew Morton, linux-kernel
On Fri, May 30, 2008 at 04:04:10AM -0700, Paul E. McKenney wrote:
> On Thu, May 29, 2008 at 02:13:53PM +0200, Jens Axboe wrote:
> > On Thu, May 29 2008, Paul E. McKenney wrote:
> > > On Thu, May 29, 2008 at 01:44:31PM +0200, Jens Axboe wrote:
> > > > On Thu, May 29 2008, Paul E. McKenney wrote:
> > > > > On Thu, May 29, 2008 at 12:13:54PM +0200, Jens Axboe wrote:
> > > > > > On Thu, May 29 2008, Paul E. McKenney wrote:
> > > > > > > On Thu, May 29, 2008 at 08:42:02AM +0200, Jens Axboe wrote:
> > > > > > > > On Thu, May 29 2008, Jens Axboe wrote:
> > > > > > > > > > But one additional question...
> > > > > > > > > >
> > > > > > > > > > static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > > > > > > > {
> > > > > > > > > > struct cfq_io_context *cic;
> > > > > > > > > >
> > > > > > > > > > cic = container_of(head, struct cfq_io_context, rcu_head);
> > > > > > > > > >
> > > > > > > > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > > > > > > > elv_ioc_count_dec(ioc_count);
> > > > > > > > > >
> > > > > > > > > > if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > > > > > > > complete(ioc_gone);
> > > > > > > > > > }
> > > > > > > > > >
> > > > > > > > > > Suppose that a pair of tasks both execute the elv_ioc_count_dec()
> > > > > > > > > > at the same time, so that all counters are now zero. Both then
> > > > > > > > > > find that there is still an ioc_gone, and that the count is
> > > > > > > > > > now zero. One of the tasks invokes complete(ioc_gone). This
> > > > > > > > > > awakens the corresponding cfq_exit(), which now returns, getting
> > > > > > > > > > rid of its stack frame -- and corrupting the all_gone auto variable
> > > > > > > > > > that ioc_gone references.
> > > > > > > > > >
> > > > > > > > > > Now the second task gets a big surprise when it tries to invoke
> > > > > > > > > > complete(ioc_gone).
> > > > > > > > > >
> > > > > > > > > > Or is there something else that I am missing here?
> > > > > > > > >
> > > > > > > > > No, I think that's a problem spot as well. To my knowledge, nobody has
> > > > > > > > > ever hit that. The anticipatory scheduler has the same code.
> > > > > > > > >
> > > > > > > > > What we want to avoid here is making cfq_cic_free_rcu() a lot more
> > > > > > > > > expensive, which is why the elv_ioc_count_read() is behind that
> > > > > > > > > ioc_gone check. I'll need to think a bit on how to handle that
> > > > > > > > > better :-)
> > > > > > > >
> > > > > > > > So how about this? Add a spinlock for checking and clearing ioc_gone
> > > > > > > > back to NULL. It doesn't matter if we make the ioc_gone != NULL
> > > > > > > > case a little more expensive, as it will only happen on cfq-iosched
> > > > > > > > module unload. And it seems the clearest way of making this safe.
> > > > > > > > The last hunk should really not be necessary, as ioc_gone wont be
> > > > > > > > set back to NULL before wait_for_completion() is entered.
> > > > > > >
> > > > > > > Looks better! I do have one scenario that seems troublesome, but
> > > > > > > it should be easy to fix, see below. (Assuming it really is a
> > > > > > > problem, that is...)
> > > > > > >
> > > > > > > Thanx, Paul
> > > > > > >
> > > > > > > > An identical patch is needed in AS as well.
> > > > > > > >
> > > > > > > > diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> > > > > > > > index d01b411..32aa367 100644
> > > > > > > > --- a/block/cfq-iosched.c
> > > > > > > > +++ b/block/cfq-iosched.c
> > > > > > > > @@ -48,6 +48,7 @@ static struct kmem_cache *cfq_ioc_pool;
> > > > > > > >
> > > > > > > > static DEFINE_PER_CPU(unsigned long, ioc_count);
> > > > > > > > static struct completion *ioc_gone;
> > > > > > > > +static DEFINE_SPINLOCK(ioc_gone_lock);
> > > > > > > >
> > > > > > > > #define CFQ_PRIO_LISTS IOPRIO_BE_NR
> > > > > > > > #define cfq_class_idle(cfqq) ((cfqq)->ioprio_class == IOPRIO_CLASS_IDLE)
> > > > > > > > @@ -1177,8 +1178,19 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> > > > > > > > kmem_cache_free(cfq_ioc_pool, cic);
> > > > > > > > elv_ioc_count_dec(ioc_count);
> > > > > > > >
> > > > > > > > - if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > > > > > > > - complete(ioc_gone);
> > > > > > > > + if (ioc_gone) {
> > > > > > > > + /*
> > > > > > > > + * CFQ scheduler is exiting, grab exit lock and check
> > > > > > > > + * the pending io context count. If it hits zero,
> > > > > > > > + * complete ioc_gone and set it back to NULL
> > > > > > > > + */
> > > > > > >
> > > > > > > Suppose that at this point some other CPU does the last complete().
> > > > > > > They have set ioc_gone to NULL, so everything is fine. But suppose
> > > > > > > that in the meantime, some other CPU sets up a cfq and then starts
> > > > > > > tearing it down. Then ioc_gone would be non-NULL, and we would cause
> > > > > > > this new teardown to end prematurely.
> > > > > > >
> > > > > > > If this is a real problem, one way to get around it is to have a
> > > > > > > generation number. We capture this before doing the elv_ioc_count_dec()
> > > > > > > (alas, with a memory barrier between the capture and the elv_ioc_count_dec()),
> > > > > > > and then check it under the lock. If it has changed, we know someone else
> > > > > > > has already done the awakening for us. Increment the generation number
> > > > > > > in the same place that ioc_gone is set to NULL.
> > > > > > >
> > > > > > > Seem reasonable?
> > > > > >
> > > > > > This isn't a problem, since cfq_exit() cannot be called before
> > > > > > all block queues in the system have been detached from CFQ.
> > > > >
> > > > > And once all block queues have been detached, no future block queues
> > > > > can ever be attached again? Or perhaps a better way of putting it,
> > > > > once CFQ has been shut down, can it be restarted without rebooting
> > > > > the system? If it can be restarted without reboot, then I do not
> > > > > yet see how the scenario above is avoided.
> > > >
> > > > Certainly, you can modprobe cfq-iosched and switch any device to cfq
> > > > again. But I really don't see the problem in this case. Once CFQ
> > > > has exited, all tasks are detached from CFQ. They have to be, or
> > > > we could not unload the module.
> > >
> > > Ah, OK -- so we cannot reload the module until the earlier
> > > instantiation has been unloaded, which cannot happen until everyone
> > > has returned from any code in the CFQ module. If it is linked
> > > directly into the kernel, it can only be "unloaded" by bringing the
> > > system down.
> >
> > Right!
> >
> > > Sorry for my confusion!
> > >
> > > This approach looks good to me.
> >
> > Pheew good, I was starting to have a brain melt down as well :-)
>
> OK, my apologies in advance, but one more thing to consider...
>
> > @@ -1177,8 +1178,19 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> > kmem_cache_free(cfq_ioc_pool, cic);
> > elv_ioc_count_dec(ioc_count);
>
> Need an smp_mb() here, and ...
>
> >
> > - if (ioc_gone && !elv_ioc_count_read(ioc_count))
> > - complete(ioc_gone);
> > + if (ioc_gone) {
> > + /*
> > + * CFQ scheduler is exiting, grab exit lock and check
> > + * the pending io context count. If it hits zero,
> > + * complete ioc_gone and set it back to NULL
> > + */
> > + spin_lock(&ioc_gone_lock);
> > + if (ioc_gone && !elv_ioc_count_read(ioc_count)) {
> > + complete(ioc_gone);
> > + ioc_gone = NULL;
> > + }
> > + spin_unlock(&ioc_gone_lock);
> > + }
> > }
> >
> > static void cfq_cic_free(struct cfq_io_context *cic)
>
> > static void __exit cfq_exit(void)
> > {
> > DECLARE_COMPLETION_ONSTACK(all_gone);
> > elv_unregister(&iosched_cfq);
> > ioc_gone = &all_gone;
> > /* ioc_gone's update must be visible before reading ioc_count */
> > smp_wmb();
>
> This needs to be upgraded to an smp_mb() because we have a write followed
> by a read that we need to keep ordered.
>
> > if (elv_ioc_count_read(ioc_count))
> > wait_for_completion(ioc_gone);
> > cfq_slab_kill();
> > }
>
> In absence of these two changes, either compiler or CPU could reorder
> the accesses to ioc_gone on the one hand and to ioc_count on the other,
> which could result in a hang, for example, cfq_exit() might see the
> reader's old count, but the reader might fail to see the assignment
> to ioc_gone.
OK, make that two more things to consider...
Why the smp_read_barrier_depends() in cfq_get_io_context()? It appears
to be guarding the dereferencing of ioc. But we would have obtained
ioc from get_io_context(), which already dereferenced this pointer when
it bumped the ->refcount. So any barriers would seem to need to be
inside of get_io_context().
So what am I missing this time?
Thanx, Paul
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-28 12:44 ` Jens Axboe
2008-05-28 13:20 ` Paul E. McKenney
@ 2008-05-30 18:34 ` Alexey Dobriyan
2008-06-04 3:31 ` Paul E. McKenney
1 sibling, 1 reply; 37+ messages in thread
From: Alexey Dobriyan @ 2008-05-30 18:34 UTC (permalink / raw)
To: Jens Axboe; +Cc: Paul E. McKenney, torvalds, Andrew Morton, linux-kernel
[test patch]
Ha-ha! Posting full dmesg just in case something important is missed.
I checked that .version and number in uname match, timestamps between
vmlinux and vmlinuz are within 20 seconds, there is call to
__rcu_read_lock at the beginning of cfq_cic_lookup(), MD5 of vmlinuz and
vmliniz after reinstalling match so I'm pretty sure I booted right kernel.
Kernel is 2.6.24 with the following patch (one fix for my NIC, one
debugging for Oleg, one for you:
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index b399c62..8cdb821 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -1142,6 +1142,9 @@ static void cfq_put_queue(struct cfq_queue *cfqq)
kmem_cache_free(cfq_pool, cfqq);
}
+/*
+ * Must always be called with the rcu_read_lock() held
+ */
static void
__call_for_each_cic(struct io_context *ioc,
void (*func)(struct io_context *, struct cfq_io_context *))
@@ -1197,6 +1200,11 @@ static void cic_free_func(struct io_context *ioc, struct cfq_io_context *cic)
cfq_cic_free(cic);
}
+/*
+ * Must be called with rcu_read_lock() held or preemption otherwise disabled.
+ * Only two callers of this - ->dtor() which is called with the rcu_read_lock(),
+ * and ->trim() which is called with the task lock held
+ */
static void cfq_free_io_context(struct io_context *ioc)
{
/*
@@ -1502,20 +1510,24 @@ static struct cfq_io_context *
cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
{
struct cfq_io_context *cic;
+ unsigned long flags;
void *k;
if (unlikely(!ioc))
return NULL;
+ rcu_read_lock();
+
/*
* we maintain a last-hit cache, to avoid browsing over the tree
*/
cic = rcu_dereference(ioc->ioc_data);
- if (cic && cic->key == cfqd)
+ if (cic && cic->key == cfqd) {
+ rcu_read_unlock();
return cic;
+ }
do {
- rcu_read_lock();
cic = radix_tree_lookup(&ioc->radix_root, (unsigned long) cfqd);
rcu_read_unlock();
if (!cic)
@@ -1524,10 +1536,13 @@ cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
k = cic->key;
if (unlikely(!k)) {
cfq_drop_dead_cic(cfqd, ioc, cic);
+ rcu_read_lock();
continue;
}
+ spin_lock_irqsave(&ioc->lock, flags);
rcu_assign_pointer(ioc->ioc_data, cic);
+ spin_unlock_irqrestore(&ioc->lock, flags);
break;
} while (1);
@@ -2134,6 +2149,10 @@ static void *cfq_init_queue(struct request_queue *q)
static void cfq_slab_kill(void)
{
+ /*
+ * Caller already ensured that pending RCU callbacks are completed,
+ * so we should have no busy allocations at this point.
+ */
if (cfq_pool)
kmem_cache_destroy(cfq_pool);
if (cfq_ioc_pool)
@@ -2292,6 +2311,11 @@ static void __exit cfq_exit(void)
ioc_gone = &all_gone;
/* ioc_gone's update must be visible before reading ioc_count */
smp_wmb();
+
+ /*
+ * this also protects us from entering cfq_slab_kill() with
+ * pending RCU callbacks
+ */
if (elv_ioc_count_read(ioc_count))
wait_for_completion(ioc_gone);
cfq_slab_kill();
diff --git a/drivers/net/atlx/atl1.c b/drivers/net/atlx/atl1.c
index 6e4c80d..6ddc911 100644
--- a/drivers/net/atlx/atl1.c
+++ b/drivers/net/atlx/atl1.c
@@ -2023,6 +2023,7 @@ rrd_ok:
/* Good Receive */
pci_unmap_page(adapter->pdev, buffer_info->dma,
buffer_info->length, PCI_DMA_FROMDEVICE);
+ buffer_info->dma = 0;
skb = buffer_info->skb;
length = le16_to_cpu(rrd->xsz.xsum_sz.pkt_size);
diff --git a/kernel/pid.c b/kernel/pid.c
index 20d59fa..a9d6010 100644
--- a/kernel/pid.c
+++ b/kernel/pid.c
@@ -209,6 +209,15 @@ void put_pid(struct pid *pid)
ns = pid->numbers[pid->level].ns;
if ((atomic_read(&pid->count) == 1) ||
atomic_dec_and_test(&pid->count)) {
+ int type = PIDTYPE_MAX;
+ while (--type >= 0) {
+ if (!hlist_empty(&pid->tasks[type])) {
+ printk("%s: not empty pid = %p, type = %d\n",
+ __func__, pid, type);
+ WARN_ON(1);
+ return;
+ }
+ }
kmem_cache_free(ns->pid_cachep, pid);
put_pid_ns(ns);
}
[ 0.000000] Linux version 2.6.26-rc4 (ad@martell) (gcc version 4.1.2 (Gentoo 4.1.2 p1.0.2)) #5 SMP PREEMPT Wed May 28 22:22:37 MSD 2008
[ 0.000000] Command line: root=/dev/sda2 netconsole=@192.168.0.1/eth0,9353@192.168.0.42/00:1b:38:af:22:49 ignore_loglevel
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
[ 0.000000] BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
[ 0.000000] BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
[ 0.000000] BIOS-e820: 0000000000100000 - 000000007ff90000 (usable)
[ 0.000000] BIOS-e820: 000000007ff90000 - 000000007ff9e000 (ACPI data)
[ 0.000000] BIOS-e820: 000000007ff9e000 - 000000007ffe0000 (ACPI NVS)
[ 0.000000] BIOS-e820: 000000007ffe0000 - 0000000080000000 (reserved)
[ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
[ 0.000000] BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
[ 0.000000] BIOS-e820: 0000000100000000 - 0000000180000000 (usable)
[ 0.000000] debug: ignoring loglevel setting.
[ 0.000000] Entering add_active_range(0, 0, 159) 0 entries of 256 used
[ 0.000000] Entering add_active_range(0, 256, 524176) 1 entries of 256 used
[ 0.000000] Entering add_active_range(0, 1048576, 1572864) 2 entries of 256 used
[ 0.000000] max_pfn_mapped = 1572864
[ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
[ 0.000000] init_memory_mapping
[ 0.000000] DMI 2.4 present.
[ 0.000000] ACPI: RSDP 000FA980, 0024 (r2 ACPIAM)
[ 0.000000] ACPI: XSDT 7FF90100, 0054 (r1 KOZIRO FRONTIER 2000707 MSFT 97)
[ 0.000000] ACPI: FACP 7FF90290, 00F4 (r3 MSTEST OEMFACP 2000707 MSFT 97)
[ 0.000000] ACPI: DSDT 7FF905C0, 8FA9 (r1 A0637 A0637000 0 INTL 20060113)
[ 0.000000] ACPI: FACS 7FF9E000, 0040
[ 0.000000] ACPI: APIC 7FF90390, 006C (r1 MSTEST OEMAPIC 2000707 MSFT 97)
[ 0.000000] ACPI: MCFG 7FF90400, 003C (r1 MSTEST OEMMCFG 2000707 MSFT 97)
[ 0.000000] ACPI: SLIC 7FF90440, 0176 (r1 KOZIRO FRONTIER 2000707 MSFT 97)
[ 0.000000] ACPI: OEMB 7FF9E040, 007B (r1 MSTEST AMI_OEM 2000707 MSFT 97)
[ 0.000000] ACPI: HPET 7FF99570, 0038 (r1 MSTEST OEMHPET 2000707 MSFT 97)
[ 0.000000] Entering add_active_range(0, 0, 159) 0 entries of 256 used
[ 0.000000] Entering add_active_range(0, 256, 524176) 1 entries of 256 used
[ 0.000000] Entering add_active_range(0, 1048576, 1572864) 2 entries of 256 used
[ 0.000000] early res: 0 [0-fff] BIOS data page
[ 0.000000] early res: 1 [6000-7fff] TRAMPOLINE
[ 0.000000] early res: 2 [200000-af2207] TEXT DATA BSS
[ 0.000000] early res: 3 [9fc00-fffff] BIOS reserved
[ 0.000000] early res: 4 [8000-dfff] PGTABLE
[ 0.000000] [ffffe20000000000-ffffe200053fffff] PMD -> [ffff81000c200000-ffff8100115fffff] on node 0
[ 0.000000] Zone PFN ranges:
[ 0.000000] DMA 0 -> 4096
[ 0.000000] DMA32 4096 -> 1048576
[ 0.000000] Normal 1048576 -> 1572864
[ 0.000000] Movable zone start PFN for each node
[ 0.000000] early_node_map[3] active PFN ranges
[ 0.000000] 0: 0 -> 159
[ 0.000000] 0: 256 -> 524176
[ 0.000000] 0: 1048576 -> 1572864
[ 0.000000] On node 0 totalpages: 1048367
[ 0.000000] DMA zone: 56 pages used for memmap
[ 0.000000] DMA zone: 2397 pages reserved
[ 0.000000] DMA zone: 1546 pages, LIFO batch:0
[ 0.000000] DMA32 zone: 14280 pages used for memmap
[ 0.000000] DMA32 zone: 505800 pages, LIFO batch:31
[ 0.000000] Normal zone: 7168 pages used for memmap
[ 0.000000] Normal zone: 517120 pages, LIFO batch:31
[ 0.000000] Movable zone: 0 pages used for memmap
[ 0.000000] ACPI: PM-Timer IO Port: 0x808
[ 0.000000] ACPI: Local APIC address 0xfee00000
[ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x82] disabled)
[ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x83] disabled)
[ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
[ 0.000000] IOAPIC[0]: apic_id 2, version 0, address 0xfec00000, GSI 0-23
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.000000] ACPI: IRQ0 used by override.
[ 0.000000] ACPI: IRQ2 used by override.
[ 0.000000] ACPI: IRQ9 used by override.
[ 0.000000] Setting APIC routing to flat
[ 0.000000] ACPI: HPET id: 0x8086a202 base: 0xfed00000
[ 0.000000] Using ACPI (MADT) for SMP configuration information
[ 0.000000] Allocating PCI resources starting at 88000000 (gap: 80000000:7ee00000)
[ 0.000000] PERCPU: Allocating 33732 bytes of per cpu data
[ 0.000000] NR_CPUS: 2, nr_cpu_ids: 2
[4294014.506571] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 1024466
[4294014.506571] Kernel command line: root=/dev/sda2 netconsole=@192.168.0.1/eth0,9353@192.168.0.42/00:1b:38:af:22:49 ignore_loglevel
[4294014.506571] Initializing CPU#0
[4294014.506571] Preemptible RCU implementation.
[4294014.506571] PID hash table entries: 4096 (order: 12, 32768 bytes)
[4294014.506571] Extended CMOS year: 2000
[4294014.506571] TSC calibrated against PM_TIMER
[4294014.506571] time.c: Detected 2135.039 MHz processor.
[4294014.506571] Console: colour VGA+ 80x25
[4294014.506571] console [tty0] enabled
[4294014.506571] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[4294014.506571] ... MAX_LOCKDEP_SUBCLASSES: 8
[4294014.506571] ... MAX_LOCK_DEPTH: 48
[4294014.506571] ... MAX_LOCKDEP_KEYS: 2048
[4294014.506571] ... CLASSHASH_SIZE: 1024
[4294014.506571] ... MAX_LOCKDEP_ENTRIES: 8192
[4294014.506571] ... MAX_LOCKDEP_CHAINS: 16384
[4294014.506571] ... CHAINHASH_SIZE: 8192
[4294014.506571] memory used by lock dependency info: 1648 kB
[4294014.506571] per task-struct memory footprint: 2688 bytes
[4294014.506571] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[4294014.506571] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[4294014.506571] Checking aperture...
[4294014.506571] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[4294014.506571] Placing software IO TLB between 0x161d000 - 0x561d000
[4294014.506571] Memory: 4025184k/6291456k available (2445k kernel code, 167852k reserved, 1415k data, 240k init)
[4294014.506571] CPA: page pool initialized 64 of 64 pages preallocated
[4294014.506571] SLUB: Genslabs=12, HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[4294014.506571] hpet clockevent registered
[4294014.566487] Calibrating delay using timer specific routine.. 4273.61 BogoMIPS (lpj=2136806)
[4294014.566561] Mount-cache hash table entries: 256
[4294014.567488] CPU: L1 I cache: 32K, L1 D cache: 32K
[4294014.567561] CPU: L2 cache: 2048K
[4294014.567561] CPU: Physical Processor ID: 0
[4294014.567561] CPU: Processor Core ID: 0
[4294014.567561] CPU0: Thermal monitoring enabled (TM2)
[4294014.567561] using mwait in idle threads.
[4294014.567561] debug: unmapping init memory ffffffff805df000..ffffffff805e4000
[4294014.567561] ACPI: Core revision 20080321
[4294014.607912] CPU0: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz stepping 02
[4294014.608114] Using local APIC timer interrupts.
[4294014.609555] APIC timer calibration result 16679992
[4294014.609555] Detected 16.679 MHz APIC timer.
[4294014.609555] lockdep: fixing up alternatives.
[4294014.609555] Booting processor 1/1 ip 6000
[4294014.619023] Initializing CPU#1
[4294014.619023] Calibrating delay using timer specific routine.. 4269.89 BogoMIPS (lpj=2134946)
[4294014.619023] CPU: L1 I cache: 32K, L1 D cache: 32K
[4294014.619023] CPU: L2 cache: 2048K
[4294014.619023] CPU: Physical Processor ID: 0
[4294014.619023] CPU: Processor Core ID: 1
[4294014.619023] CPU1: Thermal monitoring enabled (TM2)
[4294014.619023] x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
[4294014.681144] CPU1: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz stepping 02
[4294014.682127] checking TSC synchronization [CPU#0 -> CPU#1]: passed.
[4294014.682650] Brought up 2 CPUs
[4294014.682733] Total of 2 processors activated (8543.50 BogoMIPS).
[4294014.684543] net_namespace: 1040 bytes
[4294014.684543] NET: Registered protocol family 16
[4294014.685543] No dock devices found.
[4294014.685592] ACPI: bus type pci registered
[4294014.685683] PCI: Using configuration type 1 for base access
[4294014.692161] ACPI: EC: Look up EC in DSDT
[4294014.745403] ACPI: Interpreter enabled
[4294014.745489] ACPI: (supports S0 S5)
[4294014.745671] ACPI: Using IOAPIC for interrupt routing
[4294014.783888] ACPI: PCI Root Bridge [PCI0] (0000:00)
[4294014.785059] pci 0000:00:1f.0: quirk: region 0800-087f claimed by ICH6 ACPI/GPIO/TCO
[4294014.785224] pci 0000:00:1f.0: quirk: region 0480-04bf claimed by ICH6 GPIO
[4294014.786302] PCI: Transparent bridge - 0000:00:1e.0
[4294014.786438] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[4294014.789050] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P2._PRT]
[4294014.790048] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
[4294014.791455] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P4._PRT]
[4294014.792085] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P7._PRT]
[4294014.792699] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P8._PRT]
[4294014.851947] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 *11 12 14 15)
[4294014.852880] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 7 10 11 12 14 15)
[4294014.853755] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 10 11 12 14 *15)
[4294014.854612] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 *10 11 12 14 15)
[4294014.855025] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
[4294014.855955] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
[4294014.856884] ACPI: PCI Interrupt Link [LNKG] (IRQs *3 4 5 6 7 10 11 12 14 15)
[4294014.859254] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 10 11 12 *14 15)
[4294014.861041] ACPI Warning (tbutils-0217): Incorrect checksum in table [OEMB] - 08, should be 03 [20080321]
[4294014.861041] Linux Plug and Play Support v0.97 (c) Adam Belay
[4294014.861041] pnp: PnP ACPI init
[4294014.861041] ACPI: bus type pnp registered
[4294014.876545] pnp: PnP ACPI: found 15 devices
[4294014.876616] ACPI: ACPI bus type pnp unregistered
[4294014.877616] SCSI subsystem initialized
[4294014.877616] libata version 3.00 loaded.
[4294014.878616] PCI: Using ACPI for IRQ routing
[4294014.885252] PCI-GART: No AMD northbridge found.
[4294014.897373] system 00:01: iomem range 0xfed14000-0xfed19fff has been reserved
[4294014.897532] system 00:07: ioport range 0x290-0x297 has been reserved
[4294014.897657] system 00:08: ioport range 0x4d0-0x4d1 has been reserved
[4294014.897755] system 00:08: ioport range 0x800-0x87f has been reserved
[4294014.898307] system 00:08: ioport range 0x480-0x4bf has been reserved
[4294014.898356] system 00:08: iomem range 0xffafe000-0xffb0cbff could not be reserved
[4294014.898356] system 00:08: iomem range 0xffb00000-0xffbfffff could not be reserved
[4294014.898356] system 00:08: iomem range 0xfed1c000-0xfed1ffff has been reserved
[4294014.898493] system 00:08: iomem range 0xfed20000-0xfed8ffff has been reserved
[4294014.898628] system 00:08: iomem range 0xfff00000-0xfffffffe could not be reserved
[4294014.898762] system 00:08: iomem range 0xfebfe000-0xfebfec00 has been reserved
[4294014.899305] system 00:0b: iomem range 0xfec00000-0xfec00fff has been reserved
[4294014.899440] system 00:0b: iomem range 0xfee00000-0xfee00fff could not be reserved
[4294014.899585] system 00:0d: iomem range 0xe0000000-0xefffffff has been reserved
[4294014.899729] system 00:0e: iomem range 0x0-0x9ffff could not be reserved
[4294014.899811] system 00:0e: iomem range 0xc0000-0xcffff has been reserved
[4294014.899915] system 00:0e: iomem range 0xe0000-0xfffff could not be reserved
[4294014.900017] system 00:0e: iomem range 0x100000-0x7fffffff could not be reserved
[4294014.900152] system 00:0e: iomem range 0x0-0xffffffffffffffff could not be reserved
[4294014.901942] PCI: Bridge: 0000:00:01.0
[4294014.902028] IO window: 9000-9fff
[4294014.902113] MEM window: 0xf8700000-0xfe7fffff
[4294014.902202] PREFETCH window: 0x00000000bfe00000-0x00000000dfdfffff
[4294014.902730] PCI: Bridge: 0000:00:1c.0
[4294014.902814] IO window: disabled.
[4294014.902900] MEM window: disabled.
[4294014.902985] PREFETCH window: 0x00000000dfe00000-0x00000000dfefffff
[4294014.903085] PCI: Bridge: 0000:00:1c.3
[4294014.903168] IO window: disabled.
[4294014.903192] MEM window: 0xfe900000-0xfe9fffff
[4294014.903192] PREFETCH window: disabled.
[4294014.903286] PCI: Bridge: 0000:00:1c.4
[4294014.903370] IO window: a000-afff
[4294014.903456] MEM window: 0xfe800000-0xfe8fffff
[4294014.903546] PREFETCH window: disabled.
[4294014.903637] PCI: Bridge: 0000:00:1e.0
[4294014.903728] IO window: b000-bfff
[4294014.903814] MEM window: 0xfea00000-0xfeafffff
[4294014.903903] PREFETCH window: 0x0000000088000000-0x00000000880fffff
[4294014.904021] ACPI: PCI Interrupt 0000:00:01.0[A] -> GSI 16 (level, low) -> IRQ 16
[4294014.904191] PCI: Setting latency timer of device 0000:00:01.0 to 64
[4294014.904311] ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 16 (level, low) -> IRQ 16
[4294014.904480] PCI: Setting latency timer of device 0000:00:1c.0 to 64
[4294014.904725] ACPI: PCI Interrupt 0000:00:1c.3[D] -> GSI 19 (level, low) -> IRQ 19
[4294014.904895] PCI: Setting latency timer of device 0000:00:1c.3 to 64
[4294014.905010] ACPI: PCI Interrupt 0000:00:1c.4[A] -> GSI 16 (level, low) -> IRQ 16
[4294014.905180] PCI: Setting latency timer of device 0000:00:1c.4 to 64
[4294014.905284] PCI: Setting latency timer of device 0000:00:1e.0 to 64
[4294014.905423] NET: Registered protocol family 2
[4294014.939148] IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
[4294014.939721] TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
[4294014.941766] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes)
[4294014.946760] TCP: Hash tables configured (established 262144 bind 65536)
[4294014.946894] TCP reno registered
[4294014.958451] NET: Registered protocol family 1
[4294014.976123] msgmni has been set to 7862 for ipc namespace ffffffff80572380
[4294014.976123] io scheduler noop registered
[4294014.977031] io scheduler cfq registered (default)
[4294014.977031] pci 0000:01:00.0: Boot video device
[4294014.986266] Real Time Clock Driver v1.12ac
[4294014.986266] Linux agpgart interface v0.103
[4294014.986470] ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 19 (level, low) -> IRQ 19
[4294014.986470] PCI: Setting latency timer of device 0000:03:00.0 to 64
[4294014.986470] atl1 0000:03:00.0: version 2.1.3
[4294015.012034] Switched to high resolution mode on CPU 1
[4294015.012878] Switched to high resolution mode on CPU 0
[4294015.023188] 8139too Fast Ethernet driver 0.9.28
[4294015.023188] ACPI: PCI Interrupt 0000:05:02.0[A] -> GSI 23 (level, low) -> IRQ 23
[4294015.024187] eth1: RealTek RTL8139 at 0xb800, 00:80:48:2e:06:2e, IRQ 23
[4294015.024187] eth1: Identified 8139 chip type 'RTL-8100B/8139D'
[4294015.024187] netconsole: local port 6665
[4294015.024273] netconsole: local IP 192.168.0.1
[4294015.024273] netconsole: interface eth0
[4294015.024273] netconsole: remote port 9353
[4294015.024276] netconsole: remote IP 192.168.0.42
[4294015.025888] netconsole: remote ethernet address 00:1b:38:af:22:49
[4294015.025992] netconsole: device eth0 not up yet, forcing it
[4294017.099102] atl1 0000:03:00.0: eth0 link is up 1000 Mbps full duplex
[4294017.111214] console [netcon0] enabled
[4294017.139758] netconsole: network logging started
[4294017.140754] Driver 'sd' needs updating - please use bus_type methods
[4294017.141147] ahci 0000:02:00.0: version 3.0
[4294017.141147] ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
[4294018.140956] ahci 0000:02:00.0: AHCI 0001.0000 32 slots 2 ports 3 Gbps 0x3 impl SATA mode
[4294018.141102] ahci 0000:02:00.0: flags: 64bit ncq pm led clo pmp pio slum part
[4294018.141249] PCI: Setting latency timer of device 0000:02:00.0 to 64
[4294018.142241] scsi0 : ahci
[4294018.143147] scsi1 : ahci
[4294018.143147] ata1: SATA max UDMA/133 abar m8192@0xfe8fe000 port 0xfe8fe100 irq 16
[4294018.143345] ata2: SATA max UDMA/133 abar m8192@0xfe8fe000 port 0xfe8fe180 irq 16
[4294018.448110] ata1: SATA link down (SStatus 0 SControl 300)
[4294018.753109] ata2: SATA link down (SStatus 0 SControl 300)
[4294018.754706] ata_piix 0000:00:1f.2: version 2.12
[4294018.754817] ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 19
[4294018.754995] ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
[4294018.755342] PCI: Setting latency timer of device 0000:00:1f.2 to 64
[4294018.755342] scsi2 : ata_piix
[4294018.756173] scsi3 : ata_piix
[4294018.770890] ata3: SATA max UDMA/133 cmd 0xec00 ctl 0xe880 bmdma 0xe400 irq 19
[4294018.771029] ata4: SATA max UDMA/133 cmd 0xe800 ctl 0xe480 bmdma 0xe408 irq 19
[4294019.226141] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[4294019.232666] ata3.00: ATA-8: ST3750330AS, SD15, max UDMA/133
[4294019.232767] ata3.00: 1465149168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[4294019.266037] ata3.01: ATA-7: ST3160811AS, 3.AAE, max UDMA/133
[4294019.266037] ata3.01: 312581808 sectors, multi 16: LBA48 NCQ (depth 0/32)
[4294019.272668] ata3.00: configured for UDMA/133
[4294019.324047] ata3.01: configured for UDMA/133
[4294019.780119] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
[4294019.816037] ata4.00: ATA-7: ST3250620AS, 3.AAE, max UDMA/133
[4294019.816037] ata4.00: 488397168 sectors, multi 16: LBA48 NCQ (depth 0/32)
[4294019.891046] ata4.00: configured for UDMA/133
[4294019.891046] scsi 2:0:0:0: Direct-Access ATA ST3750330AS SD15 PQ: 0 ANSI: 5
[4294019.893307] sd 2:0:0:0: [sda] 1465149168 512-byte hardware sectors (750156 MB)
[4294019.893459] sd 2:0:0:0: [sda] Write Protect is off
[4294019.893459] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[4294019.893513] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[4294019.894080] sd 2:0:0:0: [sda] 1465149168 512-byte hardware sectors (750156 MB)
[4294019.894305] sd 2:0:0:0: [sda] Write Protect is off
[4294019.894305] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[4294019.894362] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[4294019.894527] sda: sda1 sda2
[4294019.931316] sd 2:0:0:0: [sda] Attached SCSI disk
[4294019.931316] scsi 2:0:1:0: Direct-Access ATA ST3160811AS 3.AA PQ: 0 ANSI: 5
[4294019.932060] sd 2:0:1:0: [sdb] 312581808 512-byte hardware sectors (160042 MB)
[4294019.932234] sd 2:0:1:0: [sdb] Write Protect is off
[4294019.932307] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[4294019.932307] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[4294019.932307] sd 2:0:1:0: [sdb] 312581808 512-byte hardware sectors (160042 MB)
[4294019.932487] sd 2:0:1:0: [sdb] Write Protect is off
[4294019.932581] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[4294019.932749] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[4294019.932896] sdb: unknown partition table
[4294019.955303] sd 2:0:1:0: [sdb] Attached SCSI disk
[4294019.957296] scsi 3:0:0:0: Direct-Access ATA ST3250620AS 3.AA PQ: 0 ANSI: 5
[4294019.957296] sd 3:0:0:0: [sdc] 488397168 512-byte hardware sectors (250059 MB)
[4294019.957481] sd 3:0:0:0: [sdc] Write Protect is off
[4294019.957575] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[4294019.957750] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[4294019.958022] sd 3:0:0:0: [sdc] 488397168 512-byte hardware sectors (250059 MB)
[4294019.958202] sd 3:0:0:0: [sdc] Write Protect is off
[4294019.958300] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[4294019.958479] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[4294019.958625] sdc: sdc1
[4294019.974297] sd 3:0:0:0: [sdc] Attached SCSI disk
[4294019.974297] ACPI: PCI Interrupt 0000:00:1f.5[B] -> GSI 19 (level, low) -> IRQ 19
[4294019.974297] ata_piix 0000:00:1f.5: MAP [ P0 -- P1 -- ]
[4294019.974519] PCI: Setting latency timer of device 0000:00:1f.5 to 64
[4294019.975300] scsi4 : ata_piix
[4294019.975301] scsi5 : ata_piix
[4294019.984287] ata5: SATA max UDMA/133 cmd 0xd400 ctl 0xd080 bmdma 0xc880 irq 19
[4294019.984287] ata6: SATA max UDMA/133 cmd 0xd000 ctl 0xcc00 bmdma 0xc888 irq 19
[4294020.298796] ata5: SATA link down (SStatus 0 SControl 300)
[4294020.613795] ata6: SATA link down (SStatus 0 SControl 300)
[4294020.614953] ACPI: PCI Interrupt 0000:02:00.1[B] -> GSI 17 (level, low) -> IRQ 17
[4294020.615172] PCI: Setting latency timer of device 0000:02:00.1 to 64
[4294020.615408] scsi6 : pata_jmicron
[4294020.615408] scsi7 : pata_jmicron
[4294020.619409] ata7: PATA max UDMA/100 cmd 0xac00 ctl 0xa880 bmdma 0xa400 irq 17
[4294020.619409] ata8: PATA max UDMA/100 cmd 0xa800 ctl 0xa480 bmdma 0xa408 irq 17
[4294020.924351] ata7.01: ATAPI: _NEC DV-5800C, D9S2, max UDMA/33
[4294020.924889] ata7.01: configured for UDMA/33
[4294021.233785] scsi 6:0:1:0: CD-ROM _NEC DV-5800C D9S2 PQ: 0 ANSI: 5
[4294021.235400] PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
[4294021.235400] PNP: PS/2 appears to have AUX port disabled, if this is incorrect please boot with i8042.nopnp
[4294021.236404] serio: i8042 KBD port at 0x60,0x64 irq 1
[4294021.236404] mice: PS/2 mouse device common for all mice
[4294021.237401] Advanced Linux Sound Architecture Driver Version 1.0.16.
[4294021.238436] ACPI: PCI Interrupt 0000:00:1b.0[A] -> GSI 22 (level, low) -> IRQ 22
[4294021.240156] PCI: Setting latency timer of device 0000:00:1b.0 to 64
[4294021.253726] input: AT Translated Set 2 keyboard as /class/input/input0
[4294021.623063] ALSA device list:
[4294021.623063] #0: HDA Intel at 0xfebf8000 irq 22
[4294021.623179] TCP cubic registered
[4294021.660774] kjournald starting. Commit interval 5 seconds
[4294021.656982] EXT3-fs: mounted filesystem with ordered data mode.
[4294021.656982] VFS: Mounted root (ext3 filesystem) readonly.
[4294021.656982] debug: unmapping init memory ffffffff805e4000..ffffffff80620000
[4294021.661096] Write protecting the kernel read-only data: 3412k
[4294021.661314] Testing CPA: undo ffffffff8046e000-ffffffff8055e000
[4294021.661445] Testing CPA: again
[4294023.534235] Driver 'sr' needs updating - please use bus_type methods
[4294023.546361] sr0: scsi3-mmc drive: 48x/48x cd/rw xa/form2 cdda tray
[4294023.546473] Uniform CD-ROM driver Revision: 3.20
[4294023.546816] sr 6:0:1:0: Attached scsi CD-ROM sr0
[4294023.570726] usbcore: registered new interface driver usbfs
[4294023.570898] usbcore: registered new interface driver hub
[4294023.576263] usbcore: registered new device driver usb
[4294023.579267] USB Universal Host Controller Interface driver v3.0
[4294023.579998] ACPI: PCI Interrupt 0000:00:1a.0[A] -> GSI 16 (level, low) -> IRQ 16
[4294023.580193] PCI: Setting latency timer of device 0000:00:1a.0 to 64
[4294023.580300] uhci_hcd 0000:00:1a.0: UHCI Host Controller
[4294023.580725] uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
[4294023.580905] uhci_hcd 0000:00:1a.0: irq 16, io base 0x0000dc00
[4294023.581598] usb usb1: configuration #1 chosen from 1 choice
[4294023.581862] hub 1-0:1.0: USB hub found
[4294023.582035] hub 1-0:1.0: 2 ports detected
[4294023.683559] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
[4294023.683670] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[4294023.683809] usb usb1: Product: UHCI Host Controller
[4294023.683809] usb usb1: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
[4294023.683809] usb usb1: SerialNumber: 0000:00:1a.0
[4294023.683809] ACPI: PCI Interrupt 0000:00:1a.1[B] -> GSI 17 (level, low) -> IRQ 17
[4294023.684033] PCI: Setting latency timer of device 0000:00:1a.1 to 64
[4294023.684134] uhci_hcd 0000:00:1a.1: UHCI Host Controller
[4294023.684594] uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 2
[4294023.684765] uhci_hcd 0000:00:1a.1: irq 17, io base 0x0000e000
[4294023.685097] usb usb2: configuration #1 chosen from 1 choice
[4294023.685295] hub 2-0:1.0: USB hub found
[4294023.685400] hub 2-0:1.0: 2 ports detected
[4294023.786278] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
[4294023.786389] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[4294023.786593] usb usb2: Product: UHCI Host Controller
[4294023.786691] usb usb2: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
[4294023.786787] usb usb2: SerialNumber: 0000:00:1a.1
[4294023.786967] ACPI: PCI Interrupt 0000:00:1a.7[C] -> GSI 18 (level, low) -> IRQ 18
[4294023.787162] PCI: Setting latency timer of device 0000:00:1a.7 to 64
[4294023.787287] ehci_hcd 0000:00:1a.7: EHCI Host Controller
[4294023.787593] ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 3
[4294023.791704] ehci_hcd 0000:00:1a.7: debug port 1
[4294023.791816] PCI: cache line size of 32 is not supported by device 0000:00:1a.7
[4294023.791972] ehci_hcd 0000:00:1a.7: irq 18, io mem 0xfebffc00
[4294023.801337] ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[4294023.801820] usb usb3: configuration #1 chosen from 1 choice
[4294023.802010] hub 3-0:1.0: USB hub found
[4294023.802189] hub 3-0:1.0: 4 ports detected
[4294023.903972] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002
[4294023.904439] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[4294023.904591] usb usb3: Product: EHCI Host Controller
[4294023.904685] usb usb3: Manufacturer: Linux 2.6.26-rc4 ehci_hcd
[4294023.904787] usb usb3: SerialNumber: 0000:00:1a.7
[4294023.904931] ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 23
[4294023.905127] PCI: Setting latency timer of device 0000:00:1d.0 to 64
[4294023.905236] uhci_hcd 0000:00:1d.0: UHCI Host Controller
[4294023.905436] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 4
[4294023.905625] uhci_hcd 0000:00:1d.0: irq 23, io base 0x0000d480
[4294023.906151] usb usb4: configuration #1 chosen from 1 choice
[4294023.906369] hub 4-0:1.0: USB hub found
[4294023.906481] hub 4-0:1.0: 2 ports detected
[4294024.007354] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
[4294024.007875] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[4294024.008024] usb usb4: Product: UHCI Host Controller
[4294024.008122] usb usb4: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
[4294024.008220] usb usb4: SerialNumber: 0000:00:1d.0
[4294024.008395] ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 23
[4294024.008581] PCI: Setting latency timer of device 0000:00:1d.7 to 64
[4294024.008704] ehci_hcd 0000:00:1d.7: EHCI Host Controller
[4294024.008872] ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 5
[4294024.013011] ehci_hcd 0000:00:1d.7: debug port 1
[4294024.013126] PCI: cache line size of 32 is not supported by device 0000:00:1d.7
[4294024.013272] ehci_hcd 0000:00:1d.7: irq 23, io mem 0xfebff800
[4294024.022630] ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
[4294024.023099] usb usb5: configuration #1 chosen from 1 choice
[4294024.023303] hub 5-0:1.0: USB hub found
[4294024.023419] hub 5-0:1.0: 6 ports detected
[4294024.123981] usb usb5: New USB device found, idVendor=1d6b, idProduct=0002
[4294024.124094] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[4294024.124242] usb usb5: Product: EHCI Host Controller
[4294024.124340] usb usb5: Manufacturer: Linux 2.6.26-rc4 ehci_hcd
[4294024.124436] usb usb5: SerialNumber: 0000:00:1d.7
[4294024.124567] ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 19
[4294024.124759] PCI: Setting latency timer of device 0000:00:1d.1 to 64
[4294024.124867] uhci_hcd 0000:00:1d.1: UHCI Host Controller
[4294024.125862] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 6
[4294024.125862] uhci_hcd 0000:00:1d.1: irq 19, io base 0x0000d800
[4294024.125862] usb usb6: configuration #1 chosen from 1 choice
[4294024.126053] hub 6-0:1.0: USB hub found
[4294024.126158] hub 6-0:1.0: 2 ports detected
[4294024.227783] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001
[4294024.227783] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[4294024.227783] usb usb6: Product: UHCI Host Controller
[4294024.227783] usb usb6: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
[4294024.227783] usb usb6: SerialNumber: 0000:00:1d.1
[4294024.227783] ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 18
[4294024.227783] PCI: Setting latency timer of device 0000:00:1d.2 to 64
[4294024.227888] uhci_hcd 0000:00:1d.2: UHCI Host Controller
[4294024.234787] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 7
[4294024.234958] uhci_hcd 0000:00:1d.2: irq 18, io base 0x0000d880
[4294024.235390] usb usb7: configuration #1 chosen from 1 choice
[4294024.235641] hub 7-0:1.0: USB hub found
[4294024.235783] hub 7-0:1.0: 2 ports detected
[4294024.336777] usb usb7: New USB device found, idVendor=1d6b, idProduct=0001
[4294024.337631] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[4294024.337631] usb usb7: Product: UHCI Host Controller
[4294024.337631] usb usb7: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
[4294024.337631] usb usb7: SerialNumber: 0000:00:1d.2
[4294025.275801] EXT3 FS on sda2, internal journal
[4294025.576923] usbcore: registered new interface driver usblp
[4294025.699206] Adding 9775512k swap on /dev/sda1. Priority:-1 extents:1 across:9775512k
[4294034.064415] ip_tables: (C) 2000-2006 Netfilter Core Team
[4294034.072599] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[4294036.120084] eth1: link up, 100Mbps, full-duplex, lpa 0x45E1
[4294045.749739] CPA self-test:
[4294045.752254] 4k 16384 large 2528 gb 0 x 0[0-0] miss 262144
[4294045.761417] 4k 187392 large 2194 gb 0 x 0[0-0] miss 262144
[4294045.768378] 4k 187392 large 2194 gb 0 x 0[0-0] miss 262144
[4294045.768608] ok.
[4294376.517131] warning: `capget01' uses 32-bit capabilities (legacy support in use)
[4294665.514678] Adding 65528k swap on ./swapfile01. Priority:-2 extents:27 across:91612k
[4294667.501075] Adding 65528k swap on ./swapfile01. Priority:-3 extents:40 across:157332k
[4294669.395350] Adding 65528k swap on ./swapfile01. Priority:-4 extents:21 across:108832k
[4294669.504418] Unable to find swap-space signature
[4294669.548523] Adding 32k swap on alreadyused. Priority:-5 extents:1 across:32k
[4294669.594260] Adding 32k swap on swapfile02. Priority:-6 extents:3 across:56k
[4294669.657350] Adding 32k swap on swapfile03. Priority:-7 extents:1 across:32k
[4294669.697470] Adding 32k swap on swapfile04. Priority:-8 extents:2 across:76k
[4294669.711497] Adding 32k swap on swapfile05. Priority:-9 extents:1 across:32k
[4294669.757754] Adding 32k swap on swapfile06. Priority:-10 extents:2 across:48k
[4294669.772201] Adding 32k swap on swapfile07. Priority:-11 extents:1 across:32k
[4294669.792286] Adding 32k swap on swapfile08. Priority:-12 extents:2 across:1820k
[4294669.817947] Adding 32k swap on swapfile09. Priority:-13 extents:1 across:32k
[4294669.839671] Adding 32k swap on swapfile10. Priority:-14 extents:1 across:32k
[4294669.856414] Adding 32k swap on swapfile11. Priority:-15 extents:2 across:36k
[4294669.898118] Adding 32k swap on swapfile12. Priority:-16 extents:1 across:32k
[4294669.918366] Adding 32k swap on swapfile13. Priority:-17 extents:1 across:32k
[4294669.932981] Adding 32k swap on swapfile14. Priority:-18 extents:1 across:32k
[4294669.981569] Adding 32k swap on swapfile15. Priority:-19 extents:1 across:32k
[4294670.016775] Adding 32k swap on swapfile16. Priority:-20 extents:1 across:32k
[4294670.031717] Adding 32k swap on swapfile17. Priority:-21 extents:1 across:32k
[4294670.080750] Adding 32k swap on swapfile18. Priority:-22 extents:1 across:32k
[4294670.095724] Adding 32k swap on swapfile19. Priority:-23 extents:1 across:32k
[4294670.141188] Adding 32k swap on swapfile20. Priority:-24 extents:1 across:32k
[4294670.174754] Adding 32k swap on swapfile21. Priority:-25 extents:1 across:32k
[4294670.228986] Adding 32k swap on swapfile22. Priority:-26 extents:1 across:32k
[4294670.277409] Adding 32k swap on swapfile23. Priority:-27 extents:1 across:32k
[4294670.331498] Adding 32k swap on swapfile24. Priority:-28 extents:2 across:32k
[4294670.364858] Adding 32k swap on swapfile25. Priority:-29 extents:1 across:32k
[4294670.401739] Adding 32k swap on swapfile26. Priority:-30 extents:1 across:32k
[4294670.416923] Adding 32k swap on swapfile27. Priority:-31 extents:1 across:32k
[4294670.470795] Adding 32k swap on swapfile28. Priority:-32 extents:1 across:32k
[4294670.524872] Adding 32k swap on swapfile29. Priority:-33 extents:1 across:32k
[4294670.538548] Adding 32k swap on swapfile30. Priority:-34 extents:1 across:32k
[4294670.678744] Adding 32k swap on firstswapfile. Priority:-35 extents:1 across:32k
[4294670.679127] Adding 32k swap on secondswapfile. Priority:-36 extents:1 across:32k
[4294671.175248] warning: process `sysctl01' used the deprecated sysctl system call with 1.1.
[4294671.175336] warning: process `sysctl01' used the deprecated sysctl system call with 1.2.
[4294671.199132] warning: process `sysctl03' used the deprecated sysctl system call with 1.1.
[4294671.200735] warning: process `sysctl03' used the deprecated sysctl system call with 1.1.
[4294671.237127] warning: process `sysctl04' used the deprecated sysctl system call with
[4295759.981152] eth1: link down
[4300003.945097] general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
[4300003.945187] CPU 0
[4300003.945984] Modules linked in: ext2 nf_conntrack_irc xt_state iptable_filter ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables x_tables usblp ehci_hcd uhci_hcd usbcore sr_mod cdrom
[4300003.945984] Pid: 5929, comm: cc1 Not tainted 2.6.26-rc4 #5
[4300003.945984] RIP: 0010:[<ffffffff80304b10>] [<ffffffff80304b10>] __call_for_each_cic+0x20/0x50
[4300003.945984] RSP: 0018:ffff810074d77e58 EFLAGS: 00010202
[4300003.945984] RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
[4300003.945984] RDX: 0000000000005f5e RSI: 0000000000000000 RDI: ffff81005a1cc500
[4300003.945984] RBP: ffff810074d77e78 R08: 0000000000000000 R09: 0000000000000000
[4300003.945984] R10: 0000000000000000 R11: 0000000000000001 R12: ffff81011058e0c0
[4300003.945984] R13: ffffffff80304b80 R14: ffff81017fcc0000 R15: ffff81005a1cc668
[4300003.946943] FS: 00002b1c6a49f6f0(0000) GS:ffffffff805c6000(0000) knlGS:0000000000000000
[4300003.946943] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[4300003.946943] CR2: 00002aeeefeb5000 CR3: 000000005a8a6000 CR4: 00000000000006e0
[4300003.946943] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[4300003.946943] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[4300003.946943] Process cc1 (pid: 5929, threadinfo ffff810074d76000, task ffff81005a1cc500)
[4300003.946943] Stack: ffff810074d77e88 ffff81011058e0c0 0000000000000010 ffff81005a1cc600
[4300003.946943] ffff810074d77e88 ffffffff80304b50 ffff810074d77ea8 ffffffff802fffdb
[4300003.946943] ffff81005a1cc600 ffff81011058e0c0 ffff810074d77ec8 ffffffff803000af
[4300003.946943] Call Trace:
[4300003.947037] [<ffffffff80304b50>] cfq_free_io_context+0x10/0x20
[4300003.947037] [<ffffffff802fffdb>] put_io_context+0x7b/0x90
[4300003.947037] [<ffffffff803000af>] exit_io_context+0x8f/0xb0
[4300003.947037] [<ffffffff80235f28>] do_exit+0x558/0x770
[4300003.947037] [<ffffffff80236181>] do_group_exit+0x41/0xb0
[4300003.947037] [<ffffffff80236202>] sys_exit_group+0x12/0x20
[4300003.947037] [<ffffffff8020b68b>] system_call_after_swapgs+0x7b/0x80
[4300003.947037]
[4300003.947037]
[4300003.947037] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 48 8b 47 68 48 85 c0 74 1f 48 89 c3 90 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
[4300003.948942] RIP [<ffffffff80304b10>] __call_for_each_cic+0x20/0x50
[4300003.948942] RSP <ffff810074d77e58>
[4300004.868942] ---[ end trace 823782cee637f8c8 ]---
[4300004.869038] Fixing recursive fault but reboot is needed!
[4303375.170511] general protection fault: 0000 [2] PREEMPT SMP DEBUG_PAGEALLOC
[4303375.170600] CPU 1
[4303375.170649] Modules linked in: ext2 nf_conntrack_irc xt_state iptable_filter ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables x_tables usblp ehci_hcd uhci_hcd usbcore sr_mod cdrom
[4303375.170915] Pid: 4564, comm: cc1 Tainted: G D 2.6.26-rc4 #5
[4303375.170982] RIP: 0010:[<ffffffff802a38dc>] [<ffffffff802a38dc>] __d_lookup+0x8c/0x160
[4303375.171090] RSP: 0018:ffff81005a24bbc8 EFLAGS: 00010202
[4303375.171153] RAX: ffff81006361eee0 RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000013
[4303375.171255] RDX: ffff81000100c120 RSI: 0187320a548baac5 RDI: ffff81012873b4e0
[4303375.171348] RBP: ffff81005a24bc18 R08: 0000000953ed9e00 R09: ffff81000b59000e
[4303375.171348] R10: ffffffff802a3901 R11: 0000000000000001 R12: 6b6b6b6b6b6b6b2b
[4303375.171348] R13: 0000000053ed9e00 R14: ffff81012873b4e0 R15: ffff81005a24bcc8
[4303375.171348] FS: 00002b005628b6f0(0000) GS:ffff81017fc44320(0000) knlGS:0000000000000000
[4303375.171348] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[4303375.171348] CR2: 00002ac106d23000 CR3: 000000012db71000 CR4: 00000000000006e0
[4303375.171348] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[4303375.171348] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[4303375.171348] Process cc1 (pid: 4564, threadinfo ffff81005a24a000, task ffff81007f100000)
[4303375.171348] Stack: 000000005a24bbe8 ffff81012873b4e8 0000000000000007 0000000700000002
[4303375.171348] ffff81000b59000e ffff81000b590015 ffff81005a24bcc8 ffff81005a24be48
[4303375.171348] ffff81005a24bcc8 ffff81005a24bcd8 ffff81005a24bc68 ffffffff802990c5
[4303375.171348] Call Trace:
[4303375.171348] [<ffffffff802990c5>] do_lookup+0x35/0x200
[4303375.171348] [<ffffffff8029aa17>] __link_path_walk+0x8b7/0xea0
[4303375.171348] [<ffffffff8028a8d3>] ? check_object+0x223/0x250
[4303375.171348] [<ffffffff8029b062>] path_walk+0x62/0xd0
[4303375.171348] [<ffffffff8029b2fd>] do_path_lookup+0x8d/0x1d0
[4303375.171348] [<ffffffff8029c237>] __path_lookup_intent_open+0x67/0xc0
[4303375.171348] [<ffffffff8029c31c>] path_lookup_open+0xc/0x10
[4303375.171348] [<ffffffff8029c638>] do_filp_open+0x98/0x8d0
[4303375.171348] [<ffffffff80289fff>] ? init_object+0x4f/0x90
[4303375.171348] [<ffffffff8028b2e6>] ? __slab_alloc+0xa6/0x660
[4303375.171348] [<ffffffff8028e663>] ? get_unused_fd_flags+0x103/0x130
[4303375.171348] [<ffffffff8028e6e8>] do_sys_open+0x58/0xb0
[4303375.171348] [<ffffffff8028e76b>] sys_open+0x1b/0x20
[4303375.171348] [<ffffffff8020b68b>] system_call_after_swapgs+0x7b/0x80
[4303375.171348]
[4303375.171348]
[4303375.171348] Code: 51 33 00 e8 27 16 fc ff 48 8b 03 48 89 c3 8b 45 cc 48 85 db 48 89 45 c0 75 0d eb 4c 0f 1f 00 48 8b 1b 48 85 db 74 41 4c 8d 63 c0 <48> 8b 03 45 3b 6c 24 58 0f 18 08 75 e7 4d 3b 74 24 50 75 e0 49
[4303375.171348] RIP [<ffffffff802a38dc>] __d_lookup+0x8c/0x160
[4303375.171348] RSP <ffff81005a24bbc8>
[4303375.537819] ---[ end trace 823782cee637f8c8 ]---
[4315843.874439] Adding 65528k swap on ./swapfile01. Priority:-37 extents:25 across:78684k
[4315896.606972] Adding 65528k swap on ./swapfile01. Priority:-38 extents:39 across:110892k
[4315945.905293] Adding 65528k swap on ./swapfile01. Priority:-39 extents:27 across:120716k
[4315946.328798] Unable to find swap-space signature
[4315947.971412] Adding 32k swap on alreadyused. Priority:-40 extents:1 across:32k
[4315949.363720] Adding 32k swap on swapfile02. Priority:-41 extents:2 across:32k
[4315950.569903] Adding 32k swap on swapfile03. Priority:-42 extents:2 across:64k
[4315951.765296] Adding 32k swap on swapfile04. Priority:-43 extents:4 across:108k
[4315952.875703] Adding 32k swap on swapfile05. Priority:-44 extents:2 across:1816k
[4315954.078312] Adding 32k swap on swapfile06. Priority:-45 extents:1 across:32k
[4315955.321904] Adding 32k swap on swapfile07. Priority:-46 extents:2 across:96k
[4315956.603960] Adding 32k swap on swapfile08. Priority:-47 extents:1 across:32k
[4315957.703750] Adding 32k swap on swapfile09. Priority:-48 extents:2 across:44k
[4315958.868482] Adding 32k swap on swapfile10. Priority:-49 extents:1 across:32k
[4315960.118592] Adding 32k swap on swapfile11. Priority:-50 extents:1 across:32k
[4315961.177483] Adding 32k swap on swapfile12. Priority:-51 extents:1 across:32k
[4315962.383246] Adding 32k swap on swapfile13. Priority:-52 extents:1 across:32k
[4315963.502439] Adding 32k swap on swapfile14. Priority:-53 extents:1 across:32k
[4315964.841896] Adding 32k swap on swapfile15. Priority:-54 extents:2 across:32k
[4315966.076074] Adding 32k swap on swapfile16. Priority:-55 extents:1 across:32k
[4315967.312097] Adding 32k swap on swapfile17. Priority:-56 extents:1 across:32k
[4315968.520294] Adding 32k swap on swapfile18. Priority:-57 extents:2 across:44k
[4315970.568662] Adding 32k swap on swapfile19. Priority:-58 extents:2 across:32k
[4315971.745911] Adding 32k swap on swapfile20. Priority:-59 extents:2 across:168k
[4315972.889984] Adding 32k swap on swapfile21. Priority:-60 extents:2 across:1972k
[4315974.133342] Adding 32k swap on swapfile22. Priority:-61 extents:3 across:80k
[4315975.349761] Adding 32k swap on swapfile23. Priority:-62 extents:2 across:128k
[4315976.507355] Adding 32k swap on swapfile24. Priority:-63 extents:1 across:32k
[4315977.705342] Adding 32k swap on swapfile25. Priority:-64 extents:1 across:32k
[4315978.734944] Adding 32k swap on swapfile26. Priority:-65 extents:2 across:44k
[4315979.838154] Adding 32k swap on swapfile27. Priority:-66 extents:2 across:40k
[4315981.418786] Adding 32k swap on swapfile28. Priority:-67 extents:1 across:32k
[4315982.727733] Adding 32k swap on swapfile29. Priority:-68 extents:2 across:76k
[4315983.849257] Adding 32k swap on swapfile30. Priority:-69 extents:1 across:32k
[4315987.520587] Adding 32k swap on firstswapfile. Priority:-70 extents:1 across:32k
[4315987.520978] Adding 32k swap on secondswapfile. Priority:-71 extents:1 across:32k
[4323489.250419] Adding 65528k swap on ./swapfile01. Priority:-72 extents:33 across:91720k
[4323499.912624] Adding 65528k swap on ./swapfile01. Priority:-73 extents:21 across:108164k
[4323509.465260] Adding 65528k swap on ./swapfile01. Priority:-74 extents:22 across:135344k
[4323509.902961] Unable to find swap-space signature
[4323510.351682] Adding 32k swap on alreadyused. Priority:-75 extents:3 across:56k
[4323510.543041] Adding 32k swap on swapfile02. Priority:-76 extents:2 across:48k
[4323510.707738] Adding 32k swap on swapfile03. Priority:-77 extents:2 across:32k
[4323510.972613] Adding 32k swap on swapfile04. Priority:-78 extents:1 across:32k
[4323511.245119] Adding 32k swap on swapfile05. Priority:-79 extents:2 across:32k
[4323511.461175] Adding 32k swap on swapfile06. Priority:-80 extents:1 across:32k
[4323511.674829] Adding 32k swap on swapfile07. Priority:-81 extents:1 across:32k
[4323511.936764] Adding 32k swap on swapfile08. Priority:-82 extents:1 across:32k
[4323512.117014] Adding 32k swap on swapfile09. Priority:-83 extents:1 across:32k
[4323512.312200] Adding 32k swap on swapfile10. Priority:-84 extents:1 across:32k
[4323512.508573] Adding 32k swap on swapfile11. Priority:-85 extents:2 across:32k
[4323512.710991] Adding 32k swap on swapfile12. Priority:-86 extents:1 across:32k
[4323512.971457] Adding 32k swap on swapfile13. Priority:-87 extents:1 across:32k
[4323513.157480] Adding 32k swap on swapfile14. Priority:-88 extents:2 across:32k
[4323513.415857] Adding 32k swap on swapfile15. Priority:-89 extents:1 across:32k
[4323513.667773] Adding 32k swap on swapfile16. Priority:-90 extents:3 across:56k
[4323513.862025] Adding 32k swap on swapfile17. Priority:-91 extents:1 across:32k
[4323514.117461] Adding 32k swap on swapfile18. Priority:-92 extents:1 across:32k
[4323514.359461] Adding 32k swap on swapfile19. Priority:-93 extents:1 across:32k
[4323514.591493] Adding 32k swap on swapfile20. Priority:-94 extents:1 across:32k
[4323514.828623] Adding 32k swap on swapfile21. Priority:-95 extents:1 across:32k
[4323515.062932] Adding 32k swap on swapfile22. Priority:-96 extents:2 across:1816k
[4323515.207700] Adding 32k swap on swapfile23. Priority:-97 extents:2 across:32k
[4323515.471581] Adding 32k swap on swapfile24. Priority:-98 extents:1 across:32k
[4323515.703335] Adding 32k swap on swapfile25. Priority:-99 extents:1 across:32k
[4323515.969844] Adding 32k swap on swapfile26. Priority:-100 extents:1 across:32k
[4323516.237519] Adding 32k swap on swapfile27. Priority:-101 extents:1 across:32k
[4323516.448421] Adding 32k swap on swapfile28. Priority:-102 extents:1 across:32k
[4323516.662276] Adding 32k swap on swapfile29. Priority:-103 extents:1 across:32k
[4323517.011638] Adding 32k swap on swapfile30. Priority:-104 extents:1 across:32k
[4323517.691915] Adding 32k swap on firstswapfile. Priority:-105 extents:2 across:36k
[4323517.692095] Adding 32k swap on secondswapfile. Priority:-106 extents:1 across:32k
[4327950.912409] Adding 65528k swap on ./swapfile01. Priority:-107 extents:25 across:84528k
[4327957.079710] Adding 65528k swap on ./swapfile01. Priority:-108 extents:31 across:157044k
[4327963.890758] Adding 65528k swap on ./swapfile01. Priority:-109 extents:34 across:108204k
[4327964.096327] Unable to find swap-space signature
[4327964.398296] Adding 32k swap on alreadyused. Priority:-110 extents:2 across:36k
[4327964.736523] Adding 32k swap on swapfile02. Priority:-111 extents:1 across:32k
[4327964.886182] Adding 32k swap on swapfile03. Priority:-112 extents:2 across:136k
[4327965.024481] Adding 32k swap on swapfile04. Priority:-113 extents:2 across:1816k
[4327965.176555] Adding 32k swap on swapfile05. Priority:-114 extents:1 across:32k
[4327965.288691] Adding 32k swap on swapfile06. Priority:-115 extents:1 across:32k
[4327965.422738] Adding 32k swap on swapfile07. Priority:-116 extents:2 across:44k
[4327965.558104] Adding 32k swap on swapfile08. Priority:-117 extents:1 across:32k
[4327965.722730] Adding 32k swap on swapfile09. Priority:-118 extents:1 across:32k
[4327965.908405] Adding 32k swap on swapfile10. Priority:-119 extents:2 across:60k
[4327966.064049] Adding 32k swap on swapfile11. Priority:-120 extents:1 across:32k
[4327966.182639] Adding 32k swap on swapfile12. Priority:-121 extents:1 across:32k
[4327966.308008] Adding 32k swap on swapfile13. Priority:-122 extents:1 across:32k
[4327966.441889] Adding 32k swap on swapfile14. Priority:-123 extents:2 across:36k
[4327966.545185] Adding 32k swap on swapfile15. Priority:-124 extents:1 across:32k
[4327966.731191] Adding 32k swap on swapfile16. Priority:-125 extents:2 across:40k
[4327966.908109] Adding 32k swap on swapfile17. Priority:-126 extents:1 across:32k
[4327967.067012] Adding 32k swap on swapfile18. Priority:-127 extents:1 across:32k
[4327967.270205] Adding 32k swap on swapfile19. Priority:-128 extents:1 across:32k
[4327967.448180] Adding 32k swap on swapfile20. Priority:-129 extents:1 across:32k
[4327967.640517] Adding 32k swap on swapfile21. Priority:-130 extents:1 across:32k
[4327967.780410] Adding 32k swap on swapfile22. Priority:-131 extents:1 across:32k
[4327967.923915] Adding 32k swap on swapfile23. Priority:-132 extents:1 across:32k
[4327968.076929] Adding 32k swap on swapfile24. Priority:-133 extents:1 across:32k
[4327968.224444] Adding 32k swap on swapfile25. Priority:-134 extents:1 across:32k
[4327968.386681] Adding 32k swap on swapfile26. Priority:-135 extents:1 across:32k
[4327968.635164] Adding 32k swap on swapfile27. Priority:-136 extents:1 across:32k
[4327968.820812] Adding 32k swap on swapfile28. Priority:-137 extents:1 across:32k
[4327969.067842] Adding 32k swap on swapfile29. Priority:-138 extents:1 across:32k
[4327969.207317] Adding 32k swap on swapfile30. Priority:-139 extents:1 across:32k
[4327969.683050] Adding 32k swap on firstswapfile. Priority:-140 extents:1 across:32k
[4327969.683433] Adding 32k swap on secondswapfile. Priority:-141 extents:1 across:32k
[4331997.664608] Adding 65528k swap on ./swapfile01. Priority:-142 extents:32 across:89684k
[4332003.732463] Adding 65528k swap on ./swapfile01. Priority:-143 extents:29 across:149088k
[4332010.354281] Adding 65528k swap on ./swapfile01. Priority:-144 extents:21 across:140068k
[4332010.563304] Unable to find swap-space signature
[4332010.812699] Adding 32k swap on alreadyused. Priority:-145 extents:1 across:32k
[4332011.114655] Adding 32k swap on swapfile02. Priority:-146 extents:1 across:32k
[4332011.230734] Adding 32k swap on swapfile03. Priority:-147 extents:2 across:1836k
[4332011.405992] Adding 32k swap on swapfile04. Priority:-148 extents:1 across:32k
[4332011.629046] Adding 32k swap on swapfile05. Priority:-149 extents:1 across:32k
[4332011.824376] Adding 32k swap on swapfile06. Priority:-150 extents:1 across:32k
[4332011.956780] Adding 32k swap on swapfile07. Priority:-151 extents:2 across:44k
[4332012.084015] Adding 32k swap on swapfile08. Priority:-152 extents:1 across:32k
[4332012.382590] Adding 32k swap on swapfile09. Priority:-153 extents:1 across:32k
[4332012.600051] Adding 32k swap on swapfile10. Priority:-154 extents:2 across:32k
[4332012.739848] Adding 32k swap on swapfile11. Priority:-155 extents:1 across:32k
[4332012.951230] Adding 32k swap on swapfile12. Priority:-156 extents:2 across:36k
[4332013.027178] Adding 32k swap on swapfile13. Priority:-157 extents:2 across:32k
[4332013.157294] Adding 32k swap on swapfile14. Priority:-158 extents:1 across:32k
[4332013.324306] Adding 32k swap on swapfile15. Priority:-159 extents:1 across:32k
[4332013.478009] Adding 32k swap on swapfile16. Priority:-160 extents:1 across:32k
[4332013.691345] Adding 32k swap on swapfile17. Priority:-161 extents:2 across:32k
[4332013.758417] Adding 32k swap on swapfile18. Priority:-162 extents:1 across:32k
[4332013.856159] Adding 32k swap on swapfile19. Priority:-163 extents:1 across:32k
[4332013.988999] Adding 32k swap on swapfile20. Priority:-164 extents:1 across:32k
[4332014.108171] Adding 32k swap on swapfile21. Priority:-165 extents:1 across:32k
[4332014.191106] Adding 32k swap on swapfile22. Priority:-166 extents:1 across:32k
[4332014.304206] Adding 32k swap on swapfile23. Priority:-167 extents:1 across:32k
[4332014.502539] Adding 32k swap on swapfile24. Priority:-168 extents:1 across:32k
[4332014.684369] Adding 32k swap on swapfile25. Priority:-169 extents:1 across:32k
[4332014.853736] Adding 32k swap on swapfile26. Priority:-170 extents:2 across:36k
[4332014.973318] Adding 32k swap on swapfile27. Priority:-171 extents:1 across:32k
[4332015.091161] Adding 32k swap on swapfile28. Priority:-172 extents:1 across:32k
[4332015.201008] Adding 32k swap on swapfile29. Priority:-173 extents:2 across:36k
[4332015.324404] Adding 32k swap on swapfile30. Priority:-174 extents:1 across:32k
[4332015.752082] Adding 32k swap on firstswapfile. Priority:-175 extents:1 across:32k
[4332015.752424] Adding 32k swap on secondswapfile. Priority:-176 extents:1 across:32k
[4335684.679409] Adding 65528k swap on ./swapfile01. Priority:-177 extents:31 across:91752k
[4335691.057909] Adding 65528k swap on ./swapfile01. Priority:-178 extents:22 across:114500k
[4335696.822115] Adding 65528k swap on ./swapfile01. Priority:-179 extents:24 across:148172k
[4335696.993393] Unable to find swap-space signature
[4335697.283173] Adding 32k swap on alreadyused. Priority:-180 extents:1 across:32k
[4335697.422398] Adding 32k swap on swapfile02. Priority:-181 extents:1 across:32k
[4335697.527172] Adding 32k swap on swapfile03. Priority:-182 extents:1 across:32k
[4335697.658520] Adding 32k swap on swapfile04. Priority:-183 extents:1 across:32k
[4335697.745190] Adding 32k swap on swapfile05. Priority:-184 extents:2 across:32k
[4335697.846964] Adding 32k swap on swapfile06. Priority:-185 extents:1 across:32k
[4335697.957268] Adding 32k swap on swapfile07. Priority:-186 extents:1 across:32k
[4335698.126923] Adding 32k swap on swapfile08. Priority:-187 extents:1 across:32k
[4335698.337237] Adding 32k swap on swapfile09. Priority:-188 extents:1 across:32k
[4335698.450537] Adding 32k swap on swapfile10. Priority:-189 extents:1 across:32k
[4335698.611428] Adding 32k swap on swapfile11. Priority:-190 extents:1 across:32k
[4335698.767599] Adding 32k swap on swapfile12. Priority:-191 extents:1 across:32k
[4335698.898005] Adding 32k swap on swapfile13. Priority:-192 extents:1 across:32k
[4335699.030925] Adding 32k swap on swapfile14. Priority:-193 extents:2 across:44k
[4335699.150021] Adding 32k swap on swapfile15. Priority:-194 extents:1 across:32k
[4335699.322725] Adding 32k swap on swapfile16. Priority:-195 extents:1 across:32k
[4335699.500086] Adding 32k swap on swapfile17. Priority:-196 extents:1 across:32k
[4335699.574431] Adding 32k swap on swapfile18. Priority:-197 extents:1 across:32k
[4335699.750988] Adding 32k swap on swapfile19. Priority:-198 extents:1 across:32k
[4335699.915901] Adding 32k swap on swapfile20. Priority:-199 extents:1 across:32k
[4335700.088262] Adding 32k swap on swapfile21. Priority:-200 extents:1 across:32k
[4335700.230728] Adding 32k swap on swapfile22. Priority:-201 extents:1 across:32k
[4335700.317745] Adding 32k swap on swapfile23. Priority:-202 extents:1 across:32k
[4335700.495246] Adding 32k swap on swapfile24. Priority:-203 extents:1 across:32k
[4335700.641937] Adding 32k swap on swapfile25. Priority:-204 extents:1 across:32k
[4335700.754225] Adding 32k swap on swapfile26. Priority:-205 extents:1 across:32k
[4335700.868226] Adding 32k swap on swapfile27. Priority:-206 extents:1 across:32k
[4335701.081225] Adding 32k swap on swapfile28. Priority:-207 extents:1 across:32k
[4335701.306849] Adding 32k swap on swapfile29. Priority:-208 extents:1 across:32k
[4335701.409447] Adding 32k swap on swapfile30. Priority:-209 extents:1 across:32k
[4335701.792262] Adding 32k swap on firstswapfile. Priority:-210 extents:1 across:32k
[4335701.792621] Adding 32k swap on secondswapfile. Priority:-211 extents:1 across:32k
[4337797.781556] eth1: link up, 100Mbps, full-duplex, lpa 0x45E1
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-05-30 18:34 ` Alexey Dobriyan
@ 2008-06-04 3:31 ` Paul E. McKenney
2008-06-04 18:32 ` Linus Torvalds
2008-06-06 14:49 ` Paul E. McKenney
0 siblings, 2 replies; 37+ messages in thread
From: Paul E. McKenney @ 2008-06-04 3:31 UTC (permalink / raw)
To: Alexey Dobriyan; +Cc: Jens Axboe, torvalds, Andrew Morton, linux-kernel
On Fri, May 30, 2008 at 10:34:28PM +0400, Alexey Dobriyan wrote:
> [test patch]
>
> Ha-ha! Posting full dmesg just in case something important is missed.
>
> I checked that .version and number in uname match, timestamps between
> vmlinux and vmlinuz are within 20 seconds, there is call to
> __rcu_read_lock at the beginning of cfq_cic_lookup(), MD5 of vmlinuz and
> vmliniz after reinstalling match so I'm pretty sure I booted right kernel.
>
> Kernel is 2.6.24 with the following patch (one fix for my NIC, one
> debugging for Oleg, one for you:
Interesting...
> [4300003.945984] RIP: 0010:[<ffffffff80304b10>] [<ffffffff80304b10>] __call_for_each_cic+0x20/0x50
> [4300003.946943] Call Trace:
> [4300003.947037] [<ffffffff80304b50>] cfq_free_io_context+0x10/0x20
> [4300003.947037] [<ffffffff802fffdb>] put_io_context+0x7b/0x90
> [4300003.947037] [<ffffffff803000af>] exit_io_context+0x8f/0xb0
> [4300003.947037] [<ffffffff80235f28>] do_exit+0x558/0x770
> [4300003.947037] [<ffffffff80236181>] do_group_exit+0x41/0xb0
> [4300003.947037] [<ffffffff80236202>] sys_exit_group+0x12/0x20
> [4300003.947037] [<ffffffff8020b68b>] system_call_after_swapgs+0x7b/0x80
The fact that put_io_context() was called from exit_io_context() means
that this is the last thread of a process exiting. The fact that
cfq_free_io_context() was called (via cfq_dtor()) from put_io_context()
means that this was the last reference to the io_context. Yet when
we traverse the cic_list, part of it is corrupted -- ascii "k"s through
RAX and RBX.
In other news, I have been hammering preemptable RCU with rcutorture, and
will be making rcutorture more aggressive. When I get back home, I will
try reproducing this test workload.
Thanx, Paul
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> index b399c62..8cdb821 100644
> --- a/block/cfq-iosched.c
> +++ b/block/cfq-iosched.c
> @@ -1142,6 +1142,9 @@ static void cfq_put_queue(struct cfq_queue *cfqq)
> kmem_cache_free(cfq_pool, cfqq);
> }
>
> +/*
> + * Must always be called with the rcu_read_lock() held
> + */
> static void
> __call_for_each_cic(struct io_context *ioc,
> void (*func)(struct io_context *, struct cfq_io_context *))
> @@ -1197,6 +1200,11 @@ static void cic_free_func(struct io_context *ioc, struct cfq_io_context *cic)
> cfq_cic_free(cic);
> }
>
> +/*
> + * Must be called with rcu_read_lock() held or preemption otherwise disabled.
> + * Only two callers of this - ->dtor() which is called with the rcu_read_lock(),
> + * and ->trim() which is called with the task lock held
> + */
> static void cfq_free_io_context(struct io_context *ioc)
> {
> /*
> @@ -1502,20 +1510,24 @@ static struct cfq_io_context *
> cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
> {
> struct cfq_io_context *cic;
> + unsigned long flags;
> void *k;
>
> if (unlikely(!ioc))
> return NULL;
>
> + rcu_read_lock();
> +
> /*
> * we maintain a last-hit cache, to avoid browsing over the tree
> */
> cic = rcu_dereference(ioc->ioc_data);
> - if (cic && cic->key == cfqd)
> + if (cic && cic->key == cfqd) {
> + rcu_read_unlock();
> return cic;
> + }
>
> do {
> - rcu_read_lock();
> cic = radix_tree_lookup(&ioc->radix_root, (unsigned long) cfqd);
> rcu_read_unlock();
> if (!cic)
> @@ -1524,10 +1536,13 @@ cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
> k = cic->key;
> if (unlikely(!k)) {
> cfq_drop_dead_cic(cfqd, ioc, cic);
> + rcu_read_lock();
> continue;
> }
>
> + spin_lock_irqsave(&ioc->lock, flags);
> rcu_assign_pointer(ioc->ioc_data, cic);
> + spin_unlock_irqrestore(&ioc->lock, flags);
> break;
> } while (1);
>
> @@ -2134,6 +2149,10 @@ static void *cfq_init_queue(struct request_queue *q)
>
> static void cfq_slab_kill(void)
> {
> + /*
> + * Caller already ensured that pending RCU callbacks are completed,
> + * so we should have no busy allocations at this point.
> + */
> if (cfq_pool)
> kmem_cache_destroy(cfq_pool);
> if (cfq_ioc_pool)
> @@ -2292,6 +2311,11 @@ static void __exit cfq_exit(void)
> ioc_gone = &all_gone;
> /* ioc_gone's update must be visible before reading ioc_count */
> smp_wmb();
> +
> + /*
> + * this also protects us from entering cfq_slab_kill() with
> + * pending RCU callbacks
> + */
> if (elv_ioc_count_read(ioc_count))
> wait_for_completion(ioc_gone);
> cfq_slab_kill();
> diff --git a/drivers/net/atlx/atl1.c b/drivers/net/atlx/atl1.c
> index 6e4c80d..6ddc911 100644
> --- a/drivers/net/atlx/atl1.c
> +++ b/drivers/net/atlx/atl1.c
> @@ -2023,6 +2023,7 @@ rrd_ok:
> /* Good Receive */
> pci_unmap_page(adapter->pdev, buffer_info->dma,
> buffer_info->length, PCI_DMA_FROMDEVICE);
> + buffer_info->dma = 0;
> skb = buffer_info->skb;
> length = le16_to_cpu(rrd->xsz.xsum_sz.pkt_size);
>
> diff --git a/kernel/pid.c b/kernel/pid.c
> index 20d59fa..a9d6010 100644
> --- a/kernel/pid.c
> +++ b/kernel/pid.c
> @@ -209,6 +209,15 @@ void put_pid(struct pid *pid)
> ns = pid->numbers[pid->level].ns;
> if ((atomic_read(&pid->count) == 1) ||
> atomic_dec_and_test(&pid->count)) {
> + int type = PIDTYPE_MAX;
> + while (--type >= 0) {
> + if (!hlist_empty(&pid->tasks[type])) {
> + printk("%s: not empty pid = %p, type = %d\n",
> + __func__, pid, type);
> + WARN_ON(1);
> + return;
> + }
> + }
> kmem_cache_free(ns->pid_cachep, pid);
> put_pid_ns(ns);
> }
>
>
> [ 0.000000] Linux version 2.6.26-rc4 (ad@martell) (gcc version 4.1.2 (Gentoo 4.1.2 p1.0.2)) #5 SMP PREEMPT Wed May 28 22:22:37 MSD 2008
> [ 0.000000] Command line: root=/dev/sda2 netconsole=@192.168.0.1/eth0,9353@192.168.0.42/00:1b:38:af:22:49 ignore_loglevel
> [ 0.000000] BIOS-provided physical RAM map:
> [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
> [ 0.000000] BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
> [ 0.000000] BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
> [ 0.000000] BIOS-e820: 0000000000100000 - 000000007ff90000 (usable)
> [ 0.000000] BIOS-e820: 000000007ff90000 - 000000007ff9e000 (ACPI data)
> [ 0.000000] BIOS-e820: 000000007ff9e000 - 000000007ffe0000 (ACPI NVS)
> [ 0.000000] BIOS-e820: 000000007ffe0000 - 0000000080000000 (reserved)
> [ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
> [ 0.000000] BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
> [ 0.000000] BIOS-e820: 0000000100000000 - 0000000180000000 (usable)
> [ 0.000000] debug: ignoring loglevel setting.
> [ 0.000000] Entering add_active_range(0, 0, 159) 0 entries of 256 used
> [ 0.000000] Entering add_active_range(0, 256, 524176) 1 entries of 256 used
> [ 0.000000] Entering add_active_range(0, 1048576, 1572864) 2 entries of 256 used
> [ 0.000000] max_pfn_mapped = 1572864
> [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
> [ 0.000000] init_memory_mapping
> [ 0.000000] DMI 2.4 present.
> [ 0.000000] ACPI: RSDP 000FA980, 0024 (r2 ACPIAM)
> [ 0.000000] ACPI: XSDT 7FF90100, 0054 (r1 KOZIRO FRONTIER 2000707 MSFT 97)
> [ 0.000000] ACPI: FACP 7FF90290, 00F4 (r3 MSTEST OEMFACP 2000707 MSFT 97)
> [ 0.000000] ACPI: DSDT 7FF905C0, 8FA9 (r1 A0637 A0637000 0 INTL 20060113)
> [ 0.000000] ACPI: FACS 7FF9E000, 0040
> [ 0.000000] ACPI: APIC 7FF90390, 006C (r1 MSTEST OEMAPIC 2000707 MSFT 97)
> [ 0.000000] ACPI: MCFG 7FF90400, 003C (r1 MSTEST OEMMCFG 2000707 MSFT 97)
> [ 0.000000] ACPI: SLIC 7FF90440, 0176 (r1 KOZIRO FRONTIER 2000707 MSFT 97)
> [ 0.000000] ACPI: OEMB 7FF9E040, 007B (r1 MSTEST AMI_OEM 2000707 MSFT 97)
> [ 0.000000] ACPI: HPET 7FF99570, 0038 (r1 MSTEST OEMHPET 2000707 MSFT 97)
> [ 0.000000] Entering add_active_range(0, 0, 159) 0 entries of 256 used
> [ 0.000000] Entering add_active_range(0, 256, 524176) 1 entries of 256 used
> [ 0.000000] Entering add_active_range(0, 1048576, 1572864) 2 entries of 256 used
> [ 0.000000] early res: 0 [0-fff] BIOS data page
> [ 0.000000] early res: 1 [6000-7fff] TRAMPOLINE
> [ 0.000000] early res: 2 [200000-af2207] TEXT DATA BSS
> [ 0.000000] early res: 3 [9fc00-fffff] BIOS reserved
> [ 0.000000] early res: 4 [8000-dfff] PGTABLE
> [ 0.000000] [ffffe20000000000-ffffe200053fffff] PMD -> [ffff81000c200000-ffff8100115fffff] on node 0
> [ 0.000000] Zone PFN ranges:
> [ 0.000000] DMA 0 -> 4096
> [ 0.000000] DMA32 4096 -> 1048576
> [ 0.000000] Normal 1048576 -> 1572864
> [ 0.000000] Movable zone start PFN for each node
> [ 0.000000] early_node_map[3] active PFN ranges
> [ 0.000000] 0: 0 -> 159
> [ 0.000000] 0: 256 -> 524176
> [ 0.000000] 0: 1048576 -> 1572864
> [ 0.000000] On node 0 totalpages: 1048367
> [ 0.000000] DMA zone: 56 pages used for memmap
> [ 0.000000] DMA zone: 2397 pages reserved
> [ 0.000000] DMA zone: 1546 pages, LIFO batch:0
> [ 0.000000] DMA32 zone: 14280 pages used for memmap
> [ 0.000000] DMA32 zone: 505800 pages, LIFO batch:31
> [ 0.000000] Normal zone: 7168 pages used for memmap
> [ 0.000000] Normal zone: 517120 pages, LIFO batch:31
> [ 0.000000] Movable zone: 0 pages used for memmap
> [ 0.000000] ACPI: PM-Timer IO Port: 0x808
> [ 0.000000] ACPI: Local APIC address 0xfee00000
> [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
> [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
> [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x82] disabled)
> [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x83] disabled)
> [ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
> [ 0.000000] IOAPIC[0]: apic_id 2, version 0, address 0xfec00000, GSI 0-23
> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> [ 0.000000] ACPI: IRQ0 used by override.
> [ 0.000000] ACPI: IRQ2 used by override.
> [ 0.000000] ACPI: IRQ9 used by override.
> [ 0.000000] Setting APIC routing to flat
> [ 0.000000] ACPI: HPET id: 0x8086a202 base: 0xfed00000
> [ 0.000000] Using ACPI (MADT) for SMP configuration information
> [ 0.000000] Allocating PCI resources starting at 88000000 (gap: 80000000:7ee00000)
> [ 0.000000] PERCPU: Allocating 33732 bytes of per cpu data
> [ 0.000000] NR_CPUS: 2, nr_cpu_ids: 2
> [4294014.506571] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 1024466
> [4294014.506571] Kernel command line: root=/dev/sda2 netconsole=@192.168.0.1/eth0,9353@192.168.0.42/00:1b:38:af:22:49 ignore_loglevel
> [4294014.506571] Initializing CPU#0
> [4294014.506571] Preemptible RCU implementation.
> [4294014.506571] PID hash table entries: 4096 (order: 12, 32768 bytes)
> [4294014.506571] Extended CMOS year: 2000
> [4294014.506571] TSC calibrated against PM_TIMER
> [4294014.506571] time.c: Detected 2135.039 MHz processor.
> [4294014.506571] Console: colour VGA+ 80x25
> [4294014.506571] console [tty0] enabled
> [4294014.506571] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
> [4294014.506571] ... MAX_LOCKDEP_SUBCLASSES: 8
> [4294014.506571] ... MAX_LOCK_DEPTH: 48
> [4294014.506571] ... MAX_LOCKDEP_KEYS: 2048
> [4294014.506571] ... CLASSHASH_SIZE: 1024
> [4294014.506571] ... MAX_LOCKDEP_ENTRIES: 8192
> [4294014.506571] ... MAX_LOCKDEP_CHAINS: 16384
> [4294014.506571] ... CHAINHASH_SIZE: 8192
> [4294014.506571] memory used by lock dependency info: 1648 kB
> [4294014.506571] per task-struct memory footprint: 2688 bytes
> [4294014.506571] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
> [4294014.506571] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
> [4294014.506571] Checking aperture...
> [4294014.506571] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> [4294014.506571] Placing software IO TLB between 0x161d000 - 0x561d000
> [4294014.506571] Memory: 4025184k/6291456k available (2445k kernel code, 167852k reserved, 1415k data, 240k init)
> [4294014.506571] CPA: page pool initialized 64 of 64 pages preallocated
> [4294014.506571] SLUB: Genslabs=12, HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
> [4294014.506571] hpet clockevent registered
> [4294014.566487] Calibrating delay using timer specific routine.. 4273.61 BogoMIPS (lpj=2136806)
> [4294014.566561] Mount-cache hash table entries: 256
> [4294014.567488] CPU: L1 I cache: 32K, L1 D cache: 32K
> [4294014.567561] CPU: L2 cache: 2048K
> [4294014.567561] CPU: Physical Processor ID: 0
> [4294014.567561] CPU: Processor Core ID: 0
> [4294014.567561] CPU0: Thermal monitoring enabled (TM2)
> [4294014.567561] using mwait in idle threads.
> [4294014.567561] debug: unmapping init memory ffffffff805df000..ffffffff805e4000
> [4294014.567561] ACPI: Core revision 20080321
> [4294014.607912] CPU0: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz stepping 02
> [4294014.608114] Using local APIC timer interrupts.
> [4294014.609555] APIC timer calibration result 16679992
> [4294014.609555] Detected 16.679 MHz APIC timer.
> [4294014.609555] lockdep: fixing up alternatives.
> [4294014.609555] Booting processor 1/1 ip 6000
> [4294014.619023] Initializing CPU#1
> [4294014.619023] Calibrating delay using timer specific routine.. 4269.89 BogoMIPS (lpj=2134946)
> [4294014.619023] CPU: L1 I cache: 32K, L1 D cache: 32K
> [4294014.619023] CPU: L2 cache: 2048K
> [4294014.619023] CPU: Physical Processor ID: 0
> [4294014.619023] CPU: Processor Core ID: 1
> [4294014.619023] CPU1: Thermal monitoring enabled (TM2)
> [4294014.619023] x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
> [4294014.681144] CPU1: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz stepping 02
> [4294014.682127] checking TSC synchronization [CPU#0 -> CPU#1]: passed.
> [4294014.682650] Brought up 2 CPUs
> [4294014.682733] Total of 2 processors activated (8543.50 BogoMIPS).
> [4294014.684543] net_namespace: 1040 bytes
> [4294014.684543] NET: Registered protocol family 16
> [4294014.685543] No dock devices found.
> [4294014.685592] ACPI: bus type pci registered
> [4294014.685683] PCI: Using configuration type 1 for base access
> [4294014.692161] ACPI: EC: Look up EC in DSDT
> [4294014.745403] ACPI: Interpreter enabled
> [4294014.745489] ACPI: (supports S0 S5)
> [4294014.745671] ACPI: Using IOAPIC for interrupt routing
> [4294014.783888] ACPI: PCI Root Bridge [PCI0] (0000:00)
> [4294014.785059] pci 0000:00:1f.0: quirk: region 0800-087f claimed by ICH6 ACPI/GPIO/TCO
> [4294014.785224] pci 0000:00:1f.0: quirk: region 0480-04bf claimed by ICH6 GPIO
> [4294014.786302] PCI: Transparent bridge - 0000:00:1e.0
> [4294014.786438] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
> [4294014.789050] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P2._PRT]
> [4294014.790048] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
> [4294014.791455] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P4._PRT]
> [4294014.792085] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P7._PRT]
> [4294014.792699] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P8._PRT]
> [4294014.851947] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 *11 12 14 15)
> [4294014.852880] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 7 10 11 12 14 15)
> [4294014.853755] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 10 11 12 14 *15)
> [4294014.854612] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 *10 11 12 14 15)
> [4294014.855025] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
> [4294014.855955] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
> [4294014.856884] ACPI: PCI Interrupt Link [LNKG] (IRQs *3 4 5 6 7 10 11 12 14 15)
> [4294014.859254] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 10 11 12 *14 15)
> [4294014.861041] ACPI Warning (tbutils-0217): Incorrect checksum in table [OEMB] - 08, should be 03 [20080321]
> [4294014.861041] Linux Plug and Play Support v0.97 (c) Adam Belay
> [4294014.861041] pnp: PnP ACPI init
> [4294014.861041] ACPI: bus type pnp registered
> [4294014.876545] pnp: PnP ACPI: found 15 devices
> [4294014.876616] ACPI: ACPI bus type pnp unregistered
> [4294014.877616] SCSI subsystem initialized
> [4294014.877616] libata version 3.00 loaded.
> [4294014.878616] PCI: Using ACPI for IRQ routing
> [4294014.885252] PCI-GART: No AMD northbridge found.
> [4294014.897373] system 00:01: iomem range 0xfed14000-0xfed19fff has been reserved
> [4294014.897532] system 00:07: ioport range 0x290-0x297 has been reserved
> [4294014.897657] system 00:08: ioport range 0x4d0-0x4d1 has been reserved
> [4294014.897755] system 00:08: ioport range 0x800-0x87f has been reserved
> [4294014.898307] system 00:08: ioport range 0x480-0x4bf has been reserved
> [4294014.898356] system 00:08: iomem range 0xffafe000-0xffb0cbff could not be reserved
> [4294014.898356] system 00:08: iomem range 0xffb00000-0xffbfffff could not be reserved
> [4294014.898356] system 00:08: iomem range 0xfed1c000-0xfed1ffff has been reserved
> [4294014.898493] system 00:08: iomem range 0xfed20000-0xfed8ffff has been reserved
> [4294014.898628] system 00:08: iomem range 0xfff00000-0xfffffffe could not be reserved
> [4294014.898762] system 00:08: iomem range 0xfebfe000-0xfebfec00 has been reserved
> [4294014.899305] system 00:0b: iomem range 0xfec00000-0xfec00fff has been reserved
> [4294014.899440] system 00:0b: iomem range 0xfee00000-0xfee00fff could not be reserved
> [4294014.899585] system 00:0d: iomem range 0xe0000000-0xefffffff has been reserved
> [4294014.899729] system 00:0e: iomem range 0x0-0x9ffff could not be reserved
> [4294014.899811] system 00:0e: iomem range 0xc0000-0xcffff has been reserved
> [4294014.899915] system 00:0e: iomem range 0xe0000-0xfffff could not be reserved
> [4294014.900017] system 00:0e: iomem range 0x100000-0x7fffffff could not be reserved
> [4294014.900152] system 00:0e: iomem range 0x0-0xffffffffffffffff could not be reserved
> [4294014.901942] PCI: Bridge: 0000:00:01.0
> [4294014.902028] IO window: 9000-9fff
> [4294014.902113] MEM window: 0xf8700000-0xfe7fffff
> [4294014.902202] PREFETCH window: 0x00000000bfe00000-0x00000000dfdfffff
> [4294014.902730] PCI: Bridge: 0000:00:1c.0
> [4294014.902814] IO window: disabled.
> [4294014.902900] MEM window: disabled.
> [4294014.902985] PREFETCH window: 0x00000000dfe00000-0x00000000dfefffff
> [4294014.903085] PCI: Bridge: 0000:00:1c.3
> [4294014.903168] IO window: disabled.
> [4294014.903192] MEM window: 0xfe900000-0xfe9fffff
> [4294014.903192] PREFETCH window: disabled.
> [4294014.903286] PCI: Bridge: 0000:00:1c.4
> [4294014.903370] IO window: a000-afff
> [4294014.903456] MEM window: 0xfe800000-0xfe8fffff
> [4294014.903546] PREFETCH window: disabled.
> [4294014.903637] PCI: Bridge: 0000:00:1e.0
> [4294014.903728] IO window: b000-bfff
> [4294014.903814] MEM window: 0xfea00000-0xfeafffff
> [4294014.903903] PREFETCH window: 0x0000000088000000-0x00000000880fffff
> [4294014.904021] ACPI: PCI Interrupt 0000:00:01.0[A] -> GSI 16 (level, low) -> IRQ 16
> [4294014.904191] PCI: Setting latency timer of device 0000:00:01.0 to 64
> [4294014.904311] ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 16 (level, low) -> IRQ 16
> [4294014.904480] PCI: Setting latency timer of device 0000:00:1c.0 to 64
> [4294014.904725] ACPI: PCI Interrupt 0000:00:1c.3[D] -> GSI 19 (level, low) -> IRQ 19
> [4294014.904895] PCI: Setting latency timer of device 0000:00:1c.3 to 64
> [4294014.905010] ACPI: PCI Interrupt 0000:00:1c.4[A] -> GSI 16 (level, low) -> IRQ 16
> [4294014.905180] PCI: Setting latency timer of device 0000:00:1c.4 to 64
> [4294014.905284] PCI: Setting latency timer of device 0000:00:1e.0 to 64
> [4294014.905423] NET: Registered protocol family 2
> [4294014.939148] IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
> [4294014.939721] TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
> [4294014.941766] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes)
> [4294014.946760] TCP: Hash tables configured (established 262144 bind 65536)
> [4294014.946894] TCP reno registered
> [4294014.958451] NET: Registered protocol family 1
> [4294014.976123] msgmni has been set to 7862 for ipc namespace ffffffff80572380
> [4294014.976123] io scheduler noop registered
> [4294014.977031] io scheduler cfq registered (default)
> [4294014.977031] pci 0000:01:00.0: Boot video device
> [4294014.986266] Real Time Clock Driver v1.12ac
> [4294014.986266] Linux agpgart interface v0.103
> [4294014.986470] ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 19 (level, low) -> IRQ 19
> [4294014.986470] PCI: Setting latency timer of device 0000:03:00.0 to 64
> [4294014.986470] atl1 0000:03:00.0: version 2.1.3
> [4294015.012034] Switched to high resolution mode on CPU 1
> [4294015.012878] Switched to high resolution mode on CPU 0
> [4294015.023188] 8139too Fast Ethernet driver 0.9.28
> [4294015.023188] ACPI: PCI Interrupt 0000:05:02.0[A] -> GSI 23 (level, low) -> IRQ 23
> [4294015.024187] eth1: RealTek RTL8139 at 0xb800, 00:80:48:2e:06:2e, IRQ 23
> [4294015.024187] eth1: Identified 8139 chip type 'RTL-8100B/8139D'
> [4294015.024187] netconsole: local port 6665
> [4294015.024273] netconsole: local IP 192.168.0.1
> [4294015.024273] netconsole: interface eth0
> [4294015.024273] netconsole: remote port 9353
> [4294015.024276] netconsole: remote IP 192.168.0.42
> [4294015.025888] netconsole: remote ethernet address 00:1b:38:af:22:49
> [4294015.025992] netconsole: device eth0 not up yet, forcing it
> [4294017.099102] atl1 0000:03:00.0: eth0 link is up 1000 Mbps full duplex
> [4294017.111214] console [netcon0] enabled
> [4294017.139758] netconsole: network logging started
> [4294017.140754] Driver 'sd' needs updating - please use bus_type methods
> [4294017.141147] ahci 0000:02:00.0: version 3.0
> [4294017.141147] ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
> [4294018.140956] ahci 0000:02:00.0: AHCI 0001.0000 32 slots 2 ports 3 Gbps 0x3 impl SATA mode
> [4294018.141102] ahci 0000:02:00.0: flags: 64bit ncq pm led clo pmp pio slum part
> [4294018.141249] PCI: Setting latency timer of device 0000:02:00.0 to 64
> [4294018.142241] scsi0 : ahci
> [4294018.143147] scsi1 : ahci
> [4294018.143147] ata1: SATA max UDMA/133 abar m8192@0xfe8fe000 port 0xfe8fe100 irq 16
> [4294018.143345] ata2: SATA max UDMA/133 abar m8192@0xfe8fe000 port 0xfe8fe180 irq 16
> [4294018.448110] ata1: SATA link down (SStatus 0 SControl 300)
> [4294018.753109] ata2: SATA link down (SStatus 0 SControl 300)
> [4294018.754706] ata_piix 0000:00:1f.2: version 2.12
> [4294018.754817] ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 19
> [4294018.754995] ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
> [4294018.755342] PCI: Setting latency timer of device 0000:00:1f.2 to 64
> [4294018.755342] scsi2 : ata_piix
> [4294018.756173] scsi3 : ata_piix
> [4294018.770890] ata3: SATA max UDMA/133 cmd 0xec00 ctl 0xe880 bmdma 0xe400 irq 19
> [4294018.771029] ata4: SATA max UDMA/133 cmd 0xe800 ctl 0xe480 bmdma 0xe408 irq 19
> [4294019.226141] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
> [4294019.232666] ata3.00: ATA-8: ST3750330AS, SD15, max UDMA/133
> [4294019.232767] ata3.00: 1465149168 sectors, multi 16: LBA48 NCQ (depth 0/32)
> [4294019.266037] ata3.01: ATA-7: ST3160811AS, 3.AAE, max UDMA/133
> [4294019.266037] ata3.01: 312581808 sectors, multi 16: LBA48 NCQ (depth 0/32)
> [4294019.272668] ata3.00: configured for UDMA/133
> [4294019.324047] ata3.01: configured for UDMA/133
> [4294019.780119] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
> [4294019.816037] ata4.00: ATA-7: ST3250620AS, 3.AAE, max UDMA/133
> [4294019.816037] ata4.00: 488397168 sectors, multi 16: LBA48 NCQ (depth 0/32)
> [4294019.891046] ata4.00: configured for UDMA/133
> [4294019.891046] scsi 2:0:0:0: Direct-Access ATA ST3750330AS SD15 PQ: 0 ANSI: 5
> [4294019.893307] sd 2:0:0:0: [sda] 1465149168 512-byte hardware sectors (750156 MB)
> [4294019.893459] sd 2:0:0:0: [sda] Write Protect is off
> [4294019.893459] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
> [4294019.893513] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [4294019.894080] sd 2:0:0:0: [sda] 1465149168 512-byte hardware sectors (750156 MB)
> [4294019.894305] sd 2:0:0:0: [sda] Write Protect is off
> [4294019.894305] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
> [4294019.894362] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [4294019.894527] sda: sda1 sda2
> [4294019.931316] sd 2:0:0:0: [sda] Attached SCSI disk
> [4294019.931316] scsi 2:0:1:0: Direct-Access ATA ST3160811AS 3.AA PQ: 0 ANSI: 5
> [4294019.932060] sd 2:0:1:0: [sdb] 312581808 512-byte hardware sectors (160042 MB)
> [4294019.932234] sd 2:0:1:0: [sdb] Write Protect is off
> [4294019.932307] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
> [4294019.932307] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [4294019.932307] sd 2:0:1:0: [sdb] 312581808 512-byte hardware sectors (160042 MB)
> [4294019.932487] sd 2:0:1:0: [sdb] Write Protect is off
> [4294019.932581] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
> [4294019.932749] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [4294019.932896] sdb: unknown partition table
> [4294019.955303] sd 2:0:1:0: [sdb] Attached SCSI disk
> [4294019.957296] scsi 3:0:0:0: Direct-Access ATA ST3250620AS 3.AA PQ: 0 ANSI: 5
> [4294019.957296] sd 3:0:0:0: [sdc] 488397168 512-byte hardware sectors (250059 MB)
> [4294019.957481] sd 3:0:0:0: [sdc] Write Protect is off
> [4294019.957575] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
> [4294019.957750] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [4294019.958022] sd 3:0:0:0: [sdc] 488397168 512-byte hardware sectors (250059 MB)
> [4294019.958202] sd 3:0:0:0: [sdc] Write Protect is off
> [4294019.958300] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
> [4294019.958479] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> [4294019.958625] sdc: sdc1
> [4294019.974297] sd 3:0:0:0: [sdc] Attached SCSI disk
> [4294019.974297] ACPI: PCI Interrupt 0000:00:1f.5[B] -> GSI 19 (level, low) -> IRQ 19
> [4294019.974297] ata_piix 0000:00:1f.5: MAP [ P0 -- P1 -- ]
> [4294019.974519] PCI: Setting latency timer of device 0000:00:1f.5 to 64
> [4294019.975300] scsi4 : ata_piix
> [4294019.975301] scsi5 : ata_piix
> [4294019.984287] ata5: SATA max UDMA/133 cmd 0xd400 ctl 0xd080 bmdma 0xc880 irq 19
> [4294019.984287] ata6: SATA max UDMA/133 cmd 0xd000 ctl 0xcc00 bmdma 0xc888 irq 19
> [4294020.298796] ata5: SATA link down (SStatus 0 SControl 300)
> [4294020.613795] ata6: SATA link down (SStatus 0 SControl 300)
> [4294020.614953] ACPI: PCI Interrupt 0000:02:00.1[B] -> GSI 17 (level, low) -> IRQ 17
> [4294020.615172] PCI: Setting latency timer of device 0000:02:00.1 to 64
> [4294020.615408] scsi6 : pata_jmicron
> [4294020.615408] scsi7 : pata_jmicron
> [4294020.619409] ata7: PATA max UDMA/100 cmd 0xac00 ctl 0xa880 bmdma 0xa400 irq 17
> [4294020.619409] ata8: PATA max UDMA/100 cmd 0xa800 ctl 0xa480 bmdma 0xa408 irq 17
> [4294020.924351] ata7.01: ATAPI: _NEC DV-5800C, D9S2, max UDMA/33
> [4294020.924889] ata7.01: configured for UDMA/33
> [4294021.233785] scsi 6:0:1:0: CD-ROM _NEC DV-5800C D9S2 PQ: 0 ANSI: 5
> [4294021.235400] PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
> [4294021.235400] PNP: PS/2 appears to have AUX port disabled, if this is incorrect please boot with i8042.nopnp
> [4294021.236404] serio: i8042 KBD port at 0x60,0x64 irq 1
> [4294021.236404] mice: PS/2 mouse device common for all mice
> [4294021.237401] Advanced Linux Sound Architecture Driver Version 1.0.16.
> [4294021.238436] ACPI: PCI Interrupt 0000:00:1b.0[A] -> GSI 22 (level, low) -> IRQ 22
> [4294021.240156] PCI: Setting latency timer of device 0000:00:1b.0 to 64
> [4294021.253726] input: AT Translated Set 2 keyboard as /class/input/input0
> [4294021.623063] ALSA device list:
> [4294021.623063] #0: HDA Intel at 0xfebf8000 irq 22
> [4294021.623179] TCP cubic registered
> [4294021.660774] kjournald starting. Commit interval 5 seconds
> [4294021.656982] EXT3-fs: mounted filesystem with ordered data mode.
> [4294021.656982] VFS: Mounted root (ext3 filesystem) readonly.
> [4294021.656982] debug: unmapping init memory ffffffff805e4000..ffffffff80620000
> [4294021.661096] Write protecting the kernel read-only data: 3412k
> [4294021.661314] Testing CPA: undo ffffffff8046e000-ffffffff8055e000
> [4294021.661445] Testing CPA: again
> [4294023.534235] Driver 'sr' needs updating - please use bus_type methods
> [4294023.546361] sr0: scsi3-mmc drive: 48x/48x cd/rw xa/form2 cdda tray
> [4294023.546473] Uniform CD-ROM driver Revision: 3.20
> [4294023.546816] sr 6:0:1:0: Attached scsi CD-ROM sr0
> [4294023.570726] usbcore: registered new interface driver usbfs
> [4294023.570898] usbcore: registered new interface driver hub
> [4294023.576263] usbcore: registered new device driver usb
> [4294023.579267] USB Universal Host Controller Interface driver v3.0
> [4294023.579998] ACPI: PCI Interrupt 0000:00:1a.0[A] -> GSI 16 (level, low) -> IRQ 16
> [4294023.580193] PCI: Setting latency timer of device 0000:00:1a.0 to 64
> [4294023.580300] uhci_hcd 0000:00:1a.0: UHCI Host Controller
> [4294023.580725] uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
> [4294023.580905] uhci_hcd 0000:00:1a.0: irq 16, io base 0x0000dc00
> [4294023.581598] usb usb1: configuration #1 chosen from 1 choice
> [4294023.581862] hub 1-0:1.0: USB hub found
> [4294023.582035] hub 1-0:1.0: 2 ports detected
> [4294023.683559] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
> [4294023.683670] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [4294023.683809] usb usb1: Product: UHCI Host Controller
> [4294023.683809] usb usb1: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
> [4294023.683809] usb usb1: SerialNumber: 0000:00:1a.0
> [4294023.683809] ACPI: PCI Interrupt 0000:00:1a.1[B] -> GSI 17 (level, low) -> IRQ 17
> [4294023.684033] PCI: Setting latency timer of device 0000:00:1a.1 to 64
> [4294023.684134] uhci_hcd 0000:00:1a.1: UHCI Host Controller
> [4294023.684594] uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 2
> [4294023.684765] uhci_hcd 0000:00:1a.1: irq 17, io base 0x0000e000
> [4294023.685097] usb usb2: configuration #1 chosen from 1 choice
> [4294023.685295] hub 2-0:1.0: USB hub found
> [4294023.685400] hub 2-0:1.0: 2 ports detected
> [4294023.786278] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
> [4294023.786389] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [4294023.786593] usb usb2: Product: UHCI Host Controller
> [4294023.786691] usb usb2: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
> [4294023.786787] usb usb2: SerialNumber: 0000:00:1a.1
> [4294023.786967] ACPI: PCI Interrupt 0000:00:1a.7[C] -> GSI 18 (level, low) -> IRQ 18
> [4294023.787162] PCI: Setting latency timer of device 0000:00:1a.7 to 64
> [4294023.787287] ehci_hcd 0000:00:1a.7: EHCI Host Controller
> [4294023.787593] ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 3
> [4294023.791704] ehci_hcd 0000:00:1a.7: debug port 1
> [4294023.791816] PCI: cache line size of 32 is not supported by device 0000:00:1a.7
> [4294023.791972] ehci_hcd 0000:00:1a.7: irq 18, io mem 0xfebffc00
> [4294023.801337] ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
> [4294023.801820] usb usb3: configuration #1 chosen from 1 choice
> [4294023.802010] hub 3-0:1.0: USB hub found
> [4294023.802189] hub 3-0:1.0: 4 ports detected
> [4294023.903972] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002
> [4294023.904439] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [4294023.904591] usb usb3: Product: EHCI Host Controller
> [4294023.904685] usb usb3: Manufacturer: Linux 2.6.26-rc4 ehci_hcd
> [4294023.904787] usb usb3: SerialNumber: 0000:00:1a.7
> [4294023.904931] ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 23
> [4294023.905127] PCI: Setting latency timer of device 0000:00:1d.0 to 64
> [4294023.905236] uhci_hcd 0000:00:1d.0: UHCI Host Controller
> [4294023.905436] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 4
> [4294023.905625] uhci_hcd 0000:00:1d.0: irq 23, io base 0x0000d480
> [4294023.906151] usb usb4: configuration #1 chosen from 1 choice
> [4294023.906369] hub 4-0:1.0: USB hub found
> [4294023.906481] hub 4-0:1.0: 2 ports detected
> [4294024.007354] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
> [4294024.007875] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [4294024.008024] usb usb4: Product: UHCI Host Controller
> [4294024.008122] usb usb4: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
> [4294024.008220] usb usb4: SerialNumber: 0000:00:1d.0
> [4294024.008395] ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 23
> [4294024.008581] PCI: Setting latency timer of device 0000:00:1d.7 to 64
> [4294024.008704] ehci_hcd 0000:00:1d.7: EHCI Host Controller
> [4294024.008872] ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 5
> [4294024.013011] ehci_hcd 0000:00:1d.7: debug port 1
> [4294024.013126] PCI: cache line size of 32 is not supported by device 0000:00:1d.7
> [4294024.013272] ehci_hcd 0000:00:1d.7: irq 23, io mem 0xfebff800
> [4294024.022630] ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
> [4294024.023099] usb usb5: configuration #1 chosen from 1 choice
> [4294024.023303] hub 5-0:1.0: USB hub found
> [4294024.023419] hub 5-0:1.0: 6 ports detected
> [4294024.123981] usb usb5: New USB device found, idVendor=1d6b, idProduct=0002
> [4294024.124094] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [4294024.124242] usb usb5: Product: EHCI Host Controller
> [4294024.124340] usb usb5: Manufacturer: Linux 2.6.26-rc4 ehci_hcd
> [4294024.124436] usb usb5: SerialNumber: 0000:00:1d.7
> [4294024.124567] ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 19
> [4294024.124759] PCI: Setting latency timer of device 0000:00:1d.1 to 64
> [4294024.124867] uhci_hcd 0000:00:1d.1: UHCI Host Controller
> [4294024.125862] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 6
> [4294024.125862] uhci_hcd 0000:00:1d.1: irq 19, io base 0x0000d800
> [4294024.125862] usb usb6: configuration #1 chosen from 1 choice
> [4294024.126053] hub 6-0:1.0: USB hub found
> [4294024.126158] hub 6-0:1.0: 2 ports detected
> [4294024.227783] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001
> [4294024.227783] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [4294024.227783] usb usb6: Product: UHCI Host Controller
> [4294024.227783] usb usb6: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
> [4294024.227783] usb usb6: SerialNumber: 0000:00:1d.1
> [4294024.227783] ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 18
> [4294024.227783] PCI: Setting latency timer of device 0000:00:1d.2 to 64
> [4294024.227888] uhci_hcd 0000:00:1d.2: UHCI Host Controller
> [4294024.234787] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 7
> [4294024.234958] uhci_hcd 0000:00:1d.2: irq 18, io base 0x0000d880
> [4294024.235390] usb usb7: configuration #1 chosen from 1 choice
> [4294024.235641] hub 7-0:1.0: USB hub found
> [4294024.235783] hub 7-0:1.0: 2 ports detected
> [4294024.336777] usb usb7: New USB device found, idVendor=1d6b, idProduct=0001
> [4294024.337631] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> [4294024.337631] usb usb7: Product: UHCI Host Controller
> [4294024.337631] usb usb7: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
> [4294024.337631] usb usb7: SerialNumber: 0000:00:1d.2
> [4294025.275801] EXT3 FS on sda2, internal journal
> [4294025.576923] usbcore: registered new interface driver usblp
> [4294025.699206] Adding 9775512k swap on /dev/sda1. Priority:-1 extents:1 across:9775512k
> [4294034.064415] ip_tables: (C) 2000-2006 Netfilter Core Team
> [4294034.072599] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
> [4294036.120084] eth1: link up, 100Mbps, full-duplex, lpa 0x45E1
> [4294045.749739] CPA self-test:
> [4294045.752254] 4k 16384 large 2528 gb 0 x 0[0-0] miss 262144
> [4294045.761417] 4k 187392 large 2194 gb 0 x 0[0-0] miss 262144
> [4294045.768378] 4k 187392 large 2194 gb 0 x 0[0-0] miss 262144
> [4294045.768608] ok.
> [4294376.517131] warning: `capget01' uses 32-bit capabilities (legacy support in use)
> [4294665.514678] Adding 65528k swap on ./swapfile01. Priority:-2 extents:27 across:91612k
> [4294667.501075] Adding 65528k swap on ./swapfile01. Priority:-3 extents:40 across:157332k
> [4294669.395350] Adding 65528k swap on ./swapfile01. Priority:-4 extents:21 across:108832k
> [4294669.504418] Unable to find swap-space signature
> [4294669.548523] Adding 32k swap on alreadyused. Priority:-5 extents:1 across:32k
> [4294669.594260] Adding 32k swap on swapfile02. Priority:-6 extents:3 across:56k
> [4294669.657350] Adding 32k swap on swapfile03. Priority:-7 extents:1 across:32k
> [4294669.697470] Adding 32k swap on swapfile04. Priority:-8 extents:2 across:76k
> [4294669.711497] Adding 32k swap on swapfile05. Priority:-9 extents:1 across:32k
> [4294669.757754] Adding 32k swap on swapfile06. Priority:-10 extents:2 across:48k
> [4294669.772201] Adding 32k swap on swapfile07. Priority:-11 extents:1 across:32k
> [4294669.792286] Adding 32k swap on swapfile08. Priority:-12 extents:2 across:1820k
> [4294669.817947] Adding 32k swap on swapfile09. Priority:-13 extents:1 across:32k
> [4294669.839671] Adding 32k swap on swapfile10. Priority:-14 extents:1 across:32k
> [4294669.856414] Adding 32k swap on swapfile11. Priority:-15 extents:2 across:36k
> [4294669.898118] Adding 32k swap on swapfile12. Priority:-16 extents:1 across:32k
> [4294669.918366] Adding 32k swap on swapfile13. Priority:-17 extents:1 across:32k
> [4294669.932981] Adding 32k swap on swapfile14. Priority:-18 extents:1 across:32k
> [4294669.981569] Adding 32k swap on swapfile15. Priority:-19 extents:1 across:32k
> [4294670.016775] Adding 32k swap on swapfile16. Priority:-20 extents:1 across:32k
> [4294670.031717] Adding 32k swap on swapfile17. Priority:-21 extents:1 across:32k
> [4294670.080750] Adding 32k swap on swapfile18. Priority:-22 extents:1 across:32k
> [4294670.095724] Adding 32k swap on swapfile19. Priority:-23 extents:1 across:32k
> [4294670.141188] Adding 32k swap on swapfile20. Priority:-24 extents:1 across:32k
> [4294670.174754] Adding 32k swap on swapfile21. Priority:-25 extents:1 across:32k
> [4294670.228986] Adding 32k swap on swapfile22. Priority:-26 extents:1 across:32k
> [4294670.277409] Adding 32k swap on swapfile23. Priority:-27 extents:1 across:32k
> [4294670.331498] Adding 32k swap on swapfile24. Priority:-28 extents:2 across:32k
> [4294670.364858] Adding 32k swap on swapfile25. Priority:-29 extents:1 across:32k
> [4294670.401739] Adding 32k swap on swapfile26. Priority:-30 extents:1 across:32k
> [4294670.416923] Adding 32k swap on swapfile27. Priority:-31 extents:1 across:32k
> [4294670.470795] Adding 32k swap on swapfile28. Priority:-32 extents:1 across:32k
> [4294670.524872] Adding 32k swap on swapfile29. Priority:-33 extents:1 across:32k
> [4294670.538548] Adding 32k swap on swapfile30. Priority:-34 extents:1 across:32k
> [4294670.678744] Adding 32k swap on firstswapfile. Priority:-35 extents:1 across:32k
> [4294670.679127] Adding 32k swap on secondswapfile. Priority:-36 extents:1 across:32k
> [4294671.175248] warning: process `sysctl01' used the deprecated sysctl system call with 1.1.
> [4294671.175336] warning: process `sysctl01' used the deprecated sysctl system call with 1.2.
> [4294671.199132] warning: process `sysctl03' used the deprecated sysctl system call with 1.1.
> [4294671.200735] warning: process `sysctl03' used the deprecated sysctl system call with 1.1.
> [4294671.237127] warning: process `sysctl04' used the deprecated sysctl system call with
> [4295759.981152] eth1: link down
> [4300003.945097] general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> [4300003.945187] CPU 0
> [4300003.945984] Modules linked in: ext2 nf_conntrack_irc xt_state iptable_filter ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables x_tables usblp ehci_hcd uhci_hcd usbcore sr_mod cdrom
> [4300003.945984] Pid: 5929, comm: cc1 Not tainted 2.6.26-rc4 #5
> [4300003.945984] RIP: 0010:[<ffffffff80304b10>] [<ffffffff80304b10>] __call_for_each_cic+0x20/0x50
> [4300003.945984] RSP: 0018:ffff810074d77e58 EFLAGS: 00010202
> [4300003.945984] RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> [4300003.945984] RDX: 0000000000005f5e RSI: 0000000000000000 RDI: ffff81005a1cc500
> [4300003.945984] RBP: ffff810074d77e78 R08: 0000000000000000 R09: 0000000000000000
> [4300003.945984] R10: 0000000000000000 R11: 0000000000000001 R12: ffff81011058e0c0
> [4300003.945984] R13: ffffffff80304b80 R14: ffff81017fcc0000 R15: ffff81005a1cc668
> [4300003.946943] FS: 00002b1c6a49f6f0(0000) GS:ffffffff805c6000(0000) knlGS:0000000000000000
> [4300003.946943] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [4300003.946943] CR2: 00002aeeefeb5000 CR3: 000000005a8a6000 CR4: 00000000000006e0
> [4300003.946943] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [4300003.946943] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [4300003.946943] Process cc1 (pid: 5929, threadinfo ffff810074d76000, task ffff81005a1cc500)
> [4300003.946943] Stack: ffff810074d77e88 ffff81011058e0c0 0000000000000010 ffff81005a1cc600
> [4300003.946943] ffff810074d77e88 ffffffff80304b50 ffff810074d77ea8 ffffffff802fffdb
> [4300003.946943] ffff81005a1cc600 ffff81011058e0c0 ffff810074d77ec8 ffffffff803000af
> [4300003.946943] Call Trace:
> [4300003.947037] [<ffffffff80304b50>] cfq_free_io_context+0x10/0x20
> [4300003.947037] [<ffffffff802fffdb>] put_io_context+0x7b/0x90
> [4300003.947037] [<ffffffff803000af>] exit_io_context+0x8f/0xb0
> [4300003.947037] [<ffffffff80235f28>] do_exit+0x558/0x770
> [4300003.947037] [<ffffffff80236181>] do_group_exit+0x41/0xb0
> [4300003.947037] [<ffffffff80236202>] sys_exit_group+0x12/0x20
> [4300003.947037] [<ffffffff8020b68b>] system_call_after_swapgs+0x7b/0x80
> [4300003.947037]
> [4300003.947037]
> [4300003.947037] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 48 8b 47 68 48 85 c0 74 1f 48 89 c3 90 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> [4300003.948942] RIP [<ffffffff80304b10>] __call_for_each_cic+0x20/0x50
> [4300003.948942] RSP <ffff810074d77e58>
> [4300004.868942] ---[ end trace 823782cee637f8c8 ]---
> [4300004.869038] Fixing recursive fault but reboot is needed!
> [4303375.170511] general protection fault: 0000 [2] PREEMPT SMP DEBUG_PAGEALLOC
> [4303375.170600] CPU 1
> [4303375.170649] Modules linked in: ext2 nf_conntrack_irc xt_state iptable_filter ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables x_tables usblp ehci_hcd uhci_hcd usbcore sr_mod cdrom
> [4303375.170915] Pid: 4564, comm: cc1 Tainted: G D 2.6.26-rc4 #5
> [4303375.170982] RIP: 0010:[<ffffffff802a38dc>] [<ffffffff802a38dc>] __d_lookup+0x8c/0x160
> [4303375.171090] RSP: 0018:ffff81005a24bbc8 EFLAGS: 00010202
> [4303375.171153] RAX: ffff81006361eee0 RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000013
> [4303375.171255] RDX: ffff81000100c120 RSI: 0187320a548baac5 RDI: ffff81012873b4e0
> [4303375.171348] RBP: ffff81005a24bc18 R08: 0000000953ed9e00 R09: ffff81000b59000e
> [4303375.171348] R10: ffffffff802a3901 R11: 0000000000000001 R12: 6b6b6b6b6b6b6b2b
> [4303375.171348] R13: 0000000053ed9e00 R14: ffff81012873b4e0 R15: ffff81005a24bcc8
> [4303375.171348] FS: 00002b005628b6f0(0000) GS:ffff81017fc44320(0000) knlGS:0000000000000000
> [4303375.171348] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [4303375.171348] CR2: 00002ac106d23000 CR3: 000000012db71000 CR4: 00000000000006e0
> [4303375.171348] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [4303375.171348] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [4303375.171348] Process cc1 (pid: 4564, threadinfo ffff81005a24a000, task ffff81007f100000)
> [4303375.171348] Stack: 000000005a24bbe8 ffff81012873b4e8 0000000000000007 0000000700000002
> [4303375.171348] ffff81000b59000e ffff81000b590015 ffff81005a24bcc8 ffff81005a24be48
> [4303375.171348] ffff81005a24bcc8 ffff81005a24bcd8 ffff81005a24bc68 ffffffff802990c5
> [4303375.171348] Call Trace:
> [4303375.171348] [<ffffffff802990c5>] do_lookup+0x35/0x200
> [4303375.171348] [<ffffffff8029aa17>] __link_path_walk+0x8b7/0xea0
> [4303375.171348] [<ffffffff8028a8d3>] ? check_object+0x223/0x250
> [4303375.171348] [<ffffffff8029b062>] path_walk+0x62/0xd0
> [4303375.171348] [<ffffffff8029b2fd>] do_path_lookup+0x8d/0x1d0
> [4303375.171348] [<ffffffff8029c237>] __path_lookup_intent_open+0x67/0xc0
> [4303375.171348] [<ffffffff8029c31c>] path_lookup_open+0xc/0x10
> [4303375.171348] [<ffffffff8029c638>] do_filp_open+0x98/0x8d0
> [4303375.171348] [<ffffffff80289fff>] ? init_object+0x4f/0x90
> [4303375.171348] [<ffffffff8028b2e6>] ? __slab_alloc+0xa6/0x660
> [4303375.171348] [<ffffffff8028e663>] ? get_unused_fd_flags+0x103/0x130
> [4303375.171348] [<ffffffff8028e6e8>] do_sys_open+0x58/0xb0
> [4303375.171348] [<ffffffff8028e76b>] sys_open+0x1b/0x20
> [4303375.171348] [<ffffffff8020b68b>] system_call_after_swapgs+0x7b/0x80
> [4303375.171348]
> [4303375.171348]
> [4303375.171348] Code: 51 33 00 e8 27 16 fc ff 48 8b 03 48 89 c3 8b 45 cc 48 85 db 48 89 45 c0 75 0d eb 4c 0f 1f 00 48 8b 1b 48 85 db 74 41 4c 8d 63 c0 <48> 8b 03 45 3b 6c 24 58 0f 18 08 75 e7 4d 3b 74 24 50 75 e0 49
> [4303375.171348] RIP [<ffffffff802a38dc>] __d_lookup+0x8c/0x160
> [4303375.171348] RSP <ffff81005a24bbc8>
> [4303375.537819] ---[ end trace 823782cee637f8c8 ]---
> [4315843.874439] Adding 65528k swap on ./swapfile01. Priority:-37 extents:25 across:78684k
> [4315896.606972] Adding 65528k swap on ./swapfile01. Priority:-38 extents:39 across:110892k
> [4315945.905293] Adding 65528k swap on ./swapfile01. Priority:-39 extents:27 across:120716k
> [4315946.328798] Unable to find swap-space signature
> [4315947.971412] Adding 32k swap on alreadyused. Priority:-40 extents:1 across:32k
> [4315949.363720] Adding 32k swap on swapfile02. Priority:-41 extents:2 across:32k
> [4315950.569903] Adding 32k swap on swapfile03. Priority:-42 extents:2 across:64k
> [4315951.765296] Adding 32k swap on swapfile04. Priority:-43 extents:4 across:108k
> [4315952.875703] Adding 32k swap on swapfile05. Priority:-44 extents:2 across:1816k
> [4315954.078312] Adding 32k swap on swapfile06. Priority:-45 extents:1 across:32k
> [4315955.321904] Adding 32k swap on swapfile07. Priority:-46 extents:2 across:96k
> [4315956.603960] Adding 32k swap on swapfile08. Priority:-47 extents:1 across:32k
> [4315957.703750] Adding 32k swap on swapfile09. Priority:-48 extents:2 across:44k
> [4315958.868482] Adding 32k swap on swapfile10. Priority:-49 extents:1 across:32k
> [4315960.118592] Adding 32k swap on swapfile11. Priority:-50 extents:1 across:32k
> [4315961.177483] Adding 32k swap on swapfile12. Priority:-51 extents:1 across:32k
> [4315962.383246] Adding 32k swap on swapfile13. Priority:-52 extents:1 across:32k
> [4315963.502439] Adding 32k swap on swapfile14. Priority:-53 extents:1 across:32k
> [4315964.841896] Adding 32k swap on swapfile15. Priority:-54 extents:2 across:32k
> [4315966.076074] Adding 32k swap on swapfile16. Priority:-55 extents:1 across:32k
> [4315967.312097] Adding 32k swap on swapfile17. Priority:-56 extents:1 across:32k
> [4315968.520294] Adding 32k swap on swapfile18. Priority:-57 extents:2 across:44k
> [4315970.568662] Adding 32k swap on swapfile19. Priority:-58 extents:2 across:32k
> [4315971.745911] Adding 32k swap on swapfile20. Priority:-59 extents:2 across:168k
> [4315972.889984] Adding 32k swap on swapfile21. Priority:-60 extents:2 across:1972k
> [4315974.133342] Adding 32k swap on swapfile22. Priority:-61 extents:3 across:80k
> [4315975.349761] Adding 32k swap on swapfile23. Priority:-62 extents:2 across:128k
> [4315976.507355] Adding 32k swap on swapfile24. Priority:-63 extents:1 across:32k
> [4315977.705342] Adding 32k swap on swapfile25. Priority:-64 extents:1 across:32k
> [4315978.734944] Adding 32k swap on swapfile26. Priority:-65 extents:2 across:44k
> [4315979.838154] Adding 32k swap on swapfile27. Priority:-66 extents:2 across:40k
> [4315981.418786] Adding 32k swap on swapfile28. Priority:-67 extents:1 across:32k
> [4315982.727733] Adding 32k swap on swapfile29. Priority:-68 extents:2 across:76k
> [4315983.849257] Adding 32k swap on swapfile30. Priority:-69 extents:1 across:32k
> [4315987.520587] Adding 32k swap on firstswapfile. Priority:-70 extents:1 across:32k
> [4315987.520978] Adding 32k swap on secondswapfile. Priority:-71 extents:1 across:32k
> [4323489.250419] Adding 65528k swap on ./swapfile01. Priority:-72 extents:33 across:91720k
> [4323499.912624] Adding 65528k swap on ./swapfile01. Priority:-73 extents:21 across:108164k
> [4323509.465260] Adding 65528k swap on ./swapfile01. Priority:-74 extents:22 across:135344k
> [4323509.902961] Unable to find swap-space signature
> [4323510.351682] Adding 32k swap on alreadyused. Priority:-75 extents:3 across:56k
> [4323510.543041] Adding 32k swap on swapfile02. Priority:-76 extents:2 across:48k
> [4323510.707738] Adding 32k swap on swapfile03. Priority:-77 extents:2 across:32k
> [4323510.972613] Adding 32k swap on swapfile04. Priority:-78 extents:1 across:32k
> [4323511.245119] Adding 32k swap on swapfile05. Priority:-79 extents:2 across:32k
> [4323511.461175] Adding 32k swap on swapfile06. Priority:-80 extents:1 across:32k
> [4323511.674829] Adding 32k swap on swapfile07. Priority:-81 extents:1 across:32k
> [4323511.936764] Adding 32k swap on swapfile08. Priority:-82 extents:1 across:32k
> [4323512.117014] Adding 32k swap on swapfile09. Priority:-83 extents:1 across:32k
> [4323512.312200] Adding 32k swap on swapfile10. Priority:-84 extents:1 across:32k
> [4323512.508573] Adding 32k swap on swapfile11. Priority:-85 extents:2 across:32k
> [4323512.710991] Adding 32k swap on swapfile12. Priority:-86 extents:1 across:32k
> [4323512.971457] Adding 32k swap on swapfile13. Priority:-87 extents:1 across:32k
> [4323513.157480] Adding 32k swap on swapfile14. Priority:-88 extents:2 across:32k
> [4323513.415857] Adding 32k swap on swapfile15. Priority:-89 extents:1 across:32k
> [4323513.667773] Adding 32k swap on swapfile16. Priority:-90 extents:3 across:56k
> [4323513.862025] Adding 32k swap on swapfile17. Priority:-91 extents:1 across:32k
> [4323514.117461] Adding 32k swap on swapfile18. Priority:-92 extents:1 across:32k
> [4323514.359461] Adding 32k swap on swapfile19. Priority:-93 extents:1 across:32k
> [4323514.591493] Adding 32k swap on swapfile20. Priority:-94 extents:1 across:32k
> [4323514.828623] Adding 32k swap on swapfile21. Priority:-95 extents:1 across:32k
> [4323515.062932] Adding 32k swap on swapfile22. Priority:-96 extents:2 across:1816k
> [4323515.207700] Adding 32k swap on swapfile23. Priority:-97 extents:2 across:32k
> [4323515.471581] Adding 32k swap on swapfile24. Priority:-98 extents:1 across:32k
> [4323515.703335] Adding 32k swap on swapfile25. Priority:-99 extents:1 across:32k
> [4323515.969844] Adding 32k swap on swapfile26. Priority:-100 extents:1 across:32k
> [4323516.237519] Adding 32k swap on swapfile27. Priority:-101 extents:1 across:32k
> [4323516.448421] Adding 32k swap on swapfile28. Priority:-102 extents:1 across:32k
> [4323516.662276] Adding 32k swap on swapfile29. Priority:-103 extents:1 across:32k
> [4323517.011638] Adding 32k swap on swapfile30. Priority:-104 extents:1 across:32k
> [4323517.691915] Adding 32k swap on firstswapfile. Priority:-105 extents:2 across:36k
> [4323517.692095] Adding 32k swap on secondswapfile. Priority:-106 extents:1 across:32k
> [4327950.912409] Adding 65528k swap on ./swapfile01. Priority:-107 extents:25 across:84528k
> [4327957.079710] Adding 65528k swap on ./swapfile01. Priority:-108 extents:31 across:157044k
> [4327963.890758] Adding 65528k swap on ./swapfile01. Priority:-109 extents:34 across:108204k
> [4327964.096327] Unable to find swap-space signature
> [4327964.398296] Adding 32k swap on alreadyused. Priority:-110 extents:2 across:36k
> [4327964.736523] Adding 32k swap on swapfile02. Priority:-111 extents:1 across:32k
> [4327964.886182] Adding 32k swap on swapfile03. Priority:-112 extents:2 across:136k
> [4327965.024481] Adding 32k swap on swapfile04. Priority:-113 extents:2 across:1816k
> [4327965.176555] Adding 32k swap on swapfile05. Priority:-114 extents:1 across:32k
> [4327965.288691] Adding 32k swap on swapfile06. Priority:-115 extents:1 across:32k
> [4327965.422738] Adding 32k swap on swapfile07. Priority:-116 extents:2 across:44k
> [4327965.558104] Adding 32k swap on swapfile08. Priority:-117 extents:1 across:32k
> [4327965.722730] Adding 32k swap on swapfile09. Priority:-118 extents:1 across:32k
> [4327965.908405] Adding 32k swap on swapfile10. Priority:-119 extents:2 across:60k
> [4327966.064049] Adding 32k swap on swapfile11. Priority:-120 extents:1 across:32k
> [4327966.182639] Adding 32k swap on swapfile12. Priority:-121 extents:1 across:32k
> [4327966.308008] Adding 32k swap on swapfile13. Priority:-122 extents:1 across:32k
> [4327966.441889] Adding 32k swap on swapfile14. Priority:-123 extents:2 across:36k
> [4327966.545185] Adding 32k swap on swapfile15. Priority:-124 extents:1 across:32k
> [4327966.731191] Adding 32k swap on swapfile16. Priority:-125 extents:2 across:40k
> [4327966.908109] Adding 32k swap on swapfile17. Priority:-126 extents:1 across:32k
> [4327967.067012] Adding 32k swap on swapfile18. Priority:-127 extents:1 across:32k
> [4327967.270205] Adding 32k swap on swapfile19. Priority:-128 extents:1 across:32k
> [4327967.448180] Adding 32k swap on swapfile20. Priority:-129 extents:1 across:32k
> [4327967.640517] Adding 32k swap on swapfile21. Priority:-130 extents:1 across:32k
> [4327967.780410] Adding 32k swap on swapfile22. Priority:-131 extents:1 across:32k
> [4327967.923915] Adding 32k swap on swapfile23. Priority:-132 extents:1 across:32k
> [4327968.076929] Adding 32k swap on swapfile24. Priority:-133 extents:1 across:32k
> [4327968.224444] Adding 32k swap on swapfile25. Priority:-134 extents:1 across:32k
> [4327968.386681] Adding 32k swap on swapfile26. Priority:-135 extents:1 across:32k
> [4327968.635164] Adding 32k swap on swapfile27. Priority:-136 extents:1 across:32k
> [4327968.820812] Adding 32k swap on swapfile28. Priority:-137 extents:1 across:32k
> [4327969.067842] Adding 32k swap on swapfile29. Priority:-138 extents:1 across:32k
> [4327969.207317] Adding 32k swap on swapfile30. Priority:-139 extents:1 across:32k
> [4327969.683050] Adding 32k swap on firstswapfile. Priority:-140 extents:1 across:32k
> [4327969.683433] Adding 32k swap on secondswapfile. Priority:-141 extents:1 across:32k
> [4331997.664608] Adding 65528k swap on ./swapfile01. Priority:-142 extents:32 across:89684k
> [4332003.732463] Adding 65528k swap on ./swapfile01. Priority:-143 extents:29 across:149088k
> [4332010.354281] Adding 65528k swap on ./swapfile01. Priority:-144 extents:21 across:140068k
> [4332010.563304] Unable to find swap-space signature
> [4332010.812699] Adding 32k swap on alreadyused. Priority:-145 extents:1 across:32k
> [4332011.114655] Adding 32k swap on swapfile02. Priority:-146 extents:1 across:32k
> [4332011.230734] Adding 32k swap on swapfile03. Priority:-147 extents:2 across:1836k
> [4332011.405992] Adding 32k swap on swapfile04. Priority:-148 extents:1 across:32k
> [4332011.629046] Adding 32k swap on swapfile05. Priority:-149 extents:1 across:32k
> [4332011.824376] Adding 32k swap on swapfile06. Priority:-150 extents:1 across:32k
> [4332011.956780] Adding 32k swap on swapfile07. Priority:-151 extents:2 across:44k
> [4332012.084015] Adding 32k swap on swapfile08. Priority:-152 extents:1 across:32k
> [4332012.382590] Adding 32k swap on swapfile09. Priority:-153 extents:1 across:32k
> [4332012.600051] Adding 32k swap on swapfile10. Priority:-154 extents:2 across:32k
> [4332012.739848] Adding 32k swap on swapfile11. Priority:-155 extents:1 across:32k
> [4332012.951230] Adding 32k swap on swapfile12. Priority:-156 extents:2 across:36k
> [4332013.027178] Adding 32k swap on swapfile13. Priority:-157 extents:2 across:32k
> [4332013.157294] Adding 32k swap on swapfile14. Priority:-158 extents:1 across:32k
> [4332013.324306] Adding 32k swap on swapfile15. Priority:-159 extents:1 across:32k
> [4332013.478009] Adding 32k swap on swapfile16. Priority:-160 extents:1 across:32k
> [4332013.691345] Adding 32k swap on swapfile17. Priority:-161 extents:2 across:32k
> [4332013.758417] Adding 32k swap on swapfile18. Priority:-162 extents:1 across:32k
> [4332013.856159] Adding 32k swap on swapfile19. Priority:-163 extents:1 across:32k
> [4332013.988999] Adding 32k swap on swapfile20. Priority:-164 extents:1 across:32k
> [4332014.108171] Adding 32k swap on swapfile21. Priority:-165 extents:1 across:32k
> [4332014.191106] Adding 32k swap on swapfile22. Priority:-166 extents:1 across:32k
> [4332014.304206] Adding 32k swap on swapfile23. Priority:-167 extents:1 across:32k
> [4332014.502539] Adding 32k swap on swapfile24. Priority:-168 extents:1 across:32k
> [4332014.684369] Adding 32k swap on swapfile25. Priority:-169 extents:1 across:32k
> [4332014.853736] Adding 32k swap on swapfile26. Priority:-170 extents:2 across:36k
> [4332014.973318] Adding 32k swap on swapfile27. Priority:-171 extents:1 across:32k
> [4332015.091161] Adding 32k swap on swapfile28. Priority:-172 extents:1 across:32k
> [4332015.201008] Adding 32k swap on swapfile29. Priority:-173 extents:2 across:36k
> [4332015.324404] Adding 32k swap on swapfile30. Priority:-174 extents:1 across:32k
> [4332015.752082] Adding 32k swap on firstswapfile. Priority:-175 extents:1 across:32k
> [4332015.752424] Adding 32k swap on secondswapfile. Priority:-176 extents:1 across:32k
> [4335684.679409] Adding 65528k swap on ./swapfile01. Priority:-177 extents:31 across:91752k
> [4335691.057909] Adding 65528k swap on ./swapfile01. Priority:-178 extents:22 across:114500k
> [4335696.822115] Adding 65528k swap on ./swapfile01. Priority:-179 extents:24 across:148172k
> [4335696.993393] Unable to find swap-space signature
> [4335697.283173] Adding 32k swap on alreadyused. Priority:-180 extents:1 across:32k
> [4335697.422398] Adding 32k swap on swapfile02. Priority:-181 extents:1 across:32k
> [4335697.527172] Adding 32k swap on swapfile03. Priority:-182 extents:1 across:32k
> [4335697.658520] Adding 32k swap on swapfile04. Priority:-183 extents:1 across:32k
> [4335697.745190] Adding 32k swap on swapfile05. Priority:-184 extents:2 across:32k
> [4335697.846964] Adding 32k swap on swapfile06. Priority:-185 extents:1 across:32k
> [4335697.957268] Adding 32k swap on swapfile07. Priority:-186 extents:1 across:32k
> [4335698.126923] Adding 32k swap on swapfile08. Priority:-187 extents:1 across:32k
> [4335698.337237] Adding 32k swap on swapfile09. Priority:-188 extents:1 across:32k
> [4335698.450537] Adding 32k swap on swapfile10. Priority:-189 extents:1 across:32k
> [4335698.611428] Adding 32k swap on swapfile11. Priority:-190 extents:1 across:32k
> [4335698.767599] Adding 32k swap on swapfile12. Priority:-191 extents:1 across:32k
> [4335698.898005] Adding 32k swap on swapfile13. Priority:-192 extents:1 across:32k
> [4335699.030925] Adding 32k swap on swapfile14. Priority:-193 extents:2 across:44k
> [4335699.150021] Adding 32k swap on swapfile15. Priority:-194 extents:1 across:32k
> [4335699.322725] Adding 32k swap on swapfile16. Priority:-195 extents:1 across:32k
> [4335699.500086] Adding 32k swap on swapfile17. Priority:-196 extents:1 across:32k
> [4335699.574431] Adding 32k swap on swapfile18. Priority:-197 extents:1 across:32k
> [4335699.750988] Adding 32k swap on swapfile19. Priority:-198 extents:1 across:32k
> [4335699.915901] Adding 32k swap on swapfile20. Priority:-199 extents:1 across:32k
> [4335700.088262] Adding 32k swap on swapfile21. Priority:-200 extents:1 across:32k
> [4335700.230728] Adding 32k swap on swapfile22. Priority:-201 extents:1 across:32k
> [4335700.317745] Adding 32k swap on swapfile23. Priority:-202 extents:1 across:32k
> [4335700.495246] Adding 32k swap on swapfile24. Priority:-203 extents:1 across:32k
> [4335700.641937] Adding 32k swap on swapfile25. Priority:-204 extents:1 across:32k
> [4335700.754225] Adding 32k swap on swapfile26. Priority:-205 extents:1 across:32k
> [4335700.868226] Adding 32k swap on swapfile27. Priority:-206 extents:1 across:32k
> [4335701.081225] Adding 32k swap on swapfile28. Priority:-207 extents:1 across:32k
> [4335701.306849] Adding 32k swap on swapfile29. Priority:-208 extents:1 across:32k
> [4335701.409447] Adding 32k swap on swapfile30. Priority:-209 extents:1 across:32k
> [4335701.792262] Adding 32k swap on firstswapfile. Priority:-210 extents:1 across:32k
> [4335701.792621] Adding 32k swap on secondswapfile. Priority:-211 extents:1 across:32k
> [4337797.781556] eth1: link up, 100Mbps, full-duplex, lpa 0x45E1
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-06-04 3:31 ` Paul E. McKenney
@ 2008-06-04 18:32 ` Linus Torvalds
2008-06-05 4:23 ` Paul E. McKenney
2008-06-06 14:49 ` Paul E. McKenney
1 sibling, 1 reply; 37+ messages in thread
From: Linus Torvalds @ 2008-06-04 18:32 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Alexey Dobriyan, Jens Axboe, torvalds, Andrew Morton,
linux-kernel
On Tue, 3 Jun 2008, Paul E. McKenney wrote:
>
> The fact that put_io_context() was called from exit_io_context() means
> that this is the last thread of a process exiting. The fact that
> cfq_free_io_context() was called (via cfq_dtor()) from put_io_context()
> means that this was the last reference to the io_context. Yet when
> we traverse the cic_list, part of it is corrupted -- ascii "k"s through
> RAX and RBX.
The "ascii 'k's" are just the slab POISON_FREE byte (0x6b).
IOW, something simply kfree'd the memory too early, causing the list
traversal to then break.
Linus
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-06-04 18:32 ` Linus Torvalds
@ 2008-06-05 4:23 ` Paul E. McKenney
0 siblings, 0 replies; 37+ messages in thread
From: Paul E. McKenney @ 2008-06-05 4:23 UTC (permalink / raw)
To: Linus Torvalds
Cc: Alexey Dobriyan, Jens Axboe, torvalds, Andrew Morton,
linux-kernel
On Wed, Jun 04, 2008 at 11:32:03AM -0700, Linus Torvalds wrote:
>
>
> On Tue, 3 Jun 2008, Paul E. McKenney wrote:
> >
> > The fact that put_io_context() was called from exit_io_context() means
> > that this is the last thread of a process exiting. The fact that
> > cfq_free_io_context() was called (via cfq_dtor()) from put_io_context()
> > means that this was the last reference to the io_context. Yet when
> > we traverse the cic_list, part of it is corrupted -- ascii "k"s through
> > RAX and RBX.
>
> The "ascii 'k's" are just the slab POISON_FREE byte (0x6b).
>
> IOW, something simply kfree'd the memory too early, causing the list
> traversal to then break.
Hence the CFQ code perhaps just an innocent bystander in the wrong place
at the wrong time. Fair enough!
Thanx, Paul
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: 2.6.26-rc4: RIP __call_for_each_cic+0x20/0x50
2008-06-04 3:31 ` Paul E. McKenney
2008-06-04 18:32 ` Linus Torvalds
@ 2008-06-06 14:49 ` Paul E. McKenney
1 sibling, 0 replies; 37+ messages in thread
From: Paul E. McKenney @ 2008-06-06 14:49 UTC (permalink / raw)
To: Alexey Dobriyan; +Cc: Jens Axboe, torvalds, Andrew Morton, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 64486 bytes --]
On Tue, Jun 03, 2008 at 08:31:18PM -0700, Paul E. McKenney wrote:
> On Fri, May 30, 2008 at 10:34:28PM +0400, Alexey Dobriyan wrote:
> > [test patch]
> >
> > Ha-ha! Posting full dmesg just in case something important is missed.
> >
> > I checked that .version and number in uname match, timestamps between
> > vmlinux and vmlinuz are within 20 seconds, there is call to
> > __rcu_read_lock at the beginning of cfq_cic_lookup(), MD5 of vmlinuz and
> > vmliniz after reinstalling match so I'm pretty sure I booted right kernel.
> >
> > Kernel is 2.6.24 with the following patch (one fix for my NIC, one
> > debugging for Oleg, one for you:
>
> Interesting...
>
> > [4300003.945984] RIP: 0010:[<ffffffff80304b10>] [<ffffffff80304b10>] __call_for_each_cic+0x20/0x50
> > [4300003.946943] Call Trace:
> > [4300003.947037] [<ffffffff80304b50>] cfq_free_io_context+0x10/0x20
> > [4300003.947037] [<ffffffff802fffdb>] put_io_context+0x7b/0x90
> > [4300003.947037] [<ffffffff803000af>] exit_io_context+0x8f/0xb0
> > [4300003.947037] [<ffffffff80235f28>] do_exit+0x558/0x770
> > [4300003.947037] [<ffffffff80236181>] do_group_exit+0x41/0xb0
> > [4300003.947037] [<ffffffff80236202>] sys_exit_group+0x12/0x20
> > [4300003.947037] [<ffffffff8020b68b>] system_call_after_swapgs+0x7b/0x80
>
> The fact that put_io_context() was called from exit_io_context() means
> that this is the last thread of a process exiting. The fact that
> cfq_free_io_context() was called (via cfq_dtor()) from put_io_context()
> means that this was the last reference to the io_context. Yet when
> we traverse the cic_list, part of it is corrupted -- ascii "k"s through
> RAX and RBX.
>
> In other news, I have been hammering preemptable RCU with rcutorture, and
> will be making rcutorture more aggressive. When I get back home, I will
> try reproducing this test workload.
I have been running the attached scripts, one for LTP and one for
the 170 parallel randconfig kernel builds. I have been using the
"ltp-full-20080131" version of LTP. Thus far, I have seen "task X
blocked for more than 120 seconds" messages and OOM killer messages.
I disabled the former with:
echo 0 > /proc/sys/kernel/hung_task_timeout_secs
I have not yet seen the types of failures you posted. So, are my scripts
doing the right thing? If so, how long should I expect to run before
seeing a failure? I see loads in excess of 10,000, for whatever that
is worth.
Thanx, Paul
> > diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> > index b399c62..8cdb821 100644
> > --- a/block/cfq-iosched.c
> > +++ b/block/cfq-iosched.c
> > @@ -1142,6 +1142,9 @@ static void cfq_put_queue(struct cfq_queue *cfqq)
> > kmem_cache_free(cfq_pool, cfqq);
> > }
> >
> > +/*
> > + * Must always be called with the rcu_read_lock() held
> > + */
> > static void
> > __call_for_each_cic(struct io_context *ioc,
> > void (*func)(struct io_context *, struct cfq_io_context *))
> > @@ -1197,6 +1200,11 @@ static void cic_free_func(struct io_context *ioc, struct cfq_io_context *cic)
> > cfq_cic_free(cic);
> > }
> >
> > +/*
> > + * Must be called with rcu_read_lock() held or preemption otherwise disabled.
> > + * Only two callers of this - ->dtor() which is called with the rcu_read_lock(),
> > + * and ->trim() which is called with the task lock held
> > + */
> > static void cfq_free_io_context(struct io_context *ioc)
> > {
> > /*
> > @@ -1502,20 +1510,24 @@ static struct cfq_io_context *
> > cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
> > {
> > struct cfq_io_context *cic;
> > + unsigned long flags;
> > void *k;
> >
> > if (unlikely(!ioc))
> > return NULL;
> >
> > + rcu_read_lock();
> > +
> > /*
> > * we maintain a last-hit cache, to avoid browsing over the tree
> > */
> > cic = rcu_dereference(ioc->ioc_data);
> > - if (cic && cic->key == cfqd)
> > + if (cic && cic->key == cfqd) {
> > + rcu_read_unlock();
> > return cic;
> > + }
> >
> > do {
> > - rcu_read_lock();
> > cic = radix_tree_lookup(&ioc->radix_root, (unsigned long) cfqd);
> > rcu_read_unlock();
> > if (!cic)
> > @@ -1524,10 +1536,13 @@ cfq_cic_lookup(struct cfq_data *cfqd, struct io_context *ioc)
> > k = cic->key;
> > if (unlikely(!k)) {
> > cfq_drop_dead_cic(cfqd, ioc, cic);
> > + rcu_read_lock();
> > continue;
> > }
> >
> > + spin_lock_irqsave(&ioc->lock, flags);
> > rcu_assign_pointer(ioc->ioc_data, cic);
> > + spin_unlock_irqrestore(&ioc->lock, flags);
> > break;
> > } while (1);
> >
> > @@ -2134,6 +2149,10 @@ static void *cfq_init_queue(struct request_queue *q)
> >
> > static void cfq_slab_kill(void)
> > {
> > + /*
> > + * Caller already ensured that pending RCU callbacks are completed,
> > + * so we should have no busy allocations at this point.
> > + */
> > if (cfq_pool)
> > kmem_cache_destroy(cfq_pool);
> > if (cfq_ioc_pool)
> > @@ -2292,6 +2311,11 @@ static void __exit cfq_exit(void)
> > ioc_gone = &all_gone;
> > /* ioc_gone's update must be visible before reading ioc_count */
> > smp_wmb();
> > +
> > + /*
> > + * this also protects us from entering cfq_slab_kill() with
> > + * pending RCU callbacks
> > + */
> > if (elv_ioc_count_read(ioc_count))
> > wait_for_completion(ioc_gone);
> > cfq_slab_kill();
> > diff --git a/drivers/net/atlx/atl1.c b/drivers/net/atlx/atl1.c
> > index 6e4c80d..6ddc911 100644
> > --- a/drivers/net/atlx/atl1.c
> > +++ b/drivers/net/atlx/atl1.c
> > @@ -2023,6 +2023,7 @@ rrd_ok:
> > /* Good Receive */
> > pci_unmap_page(adapter->pdev, buffer_info->dma,
> > buffer_info->length, PCI_DMA_FROMDEVICE);
> > + buffer_info->dma = 0;
> > skb = buffer_info->skb;
> > length = le16_to_cpu(rrd->xsz.xsum_sz.pkt_size);
> >
> > diff --git a/kernel/pid.c b/kernel/pid.c
> > index 20d59fa..a9d6010 100644
> > --- a/kernel/pid.c
> > +++ b/kernel/pid.c
> > @@ -209,6 +209,15 @@ void put_pid(struct pid *pid)
> > ns = pid->numbers[pid->level].ns;
> > if ((atomic_read(&pid->count) == 1) ||
> > atomic_dec_and_test(&pid->count)) {
> > + int type = PIDTYPE_MAX;
> > + while (--type >= 0) {
> > + if (!hlist_empty(&pid->tasks[type])) {
> > + printk("%s: not empty pid = %p, type = %d\n",
> > + __func__, pid, type);
> > + WARN_ON(1);
> > + return;
> > + }
> > + }
> > kmem_cache_free(ns->pid_cachep, pid);
> > put_pid_ns(ns);
> > }
> >
> >
> > [ 0.000000] Linux version 2.6.26-rc4 (ad@martell) (gcc version 4.1.2 (Gentoo 4.1.2 p1.0.2)) #5 SMP PREEMPT Wed May 28 22:22:37 MSD 2008
> > [ 0.000000] Command line: root=/dev/sda2 netconsole=@192.168.0.1/eth0,9353@192.168.0.42/00:1b:38:af:22:49 ignore_loglevel
> > [ 0.000000] BIOS-provided physical RAM map:
> > [ 0.000000] BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
> > [ 0.000000] BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
> > [ 0.000000] BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)
> > [ 0.000000] BIOS-e820: 0000000000100000 - 000000007ff90000 (usable)
> > [ 0.000000] BIOS-e820: 000000007ff90000 - 000000007ff9e000 (ACPI data)
> > [ 0.000000] BIOS-e820: 000000007ff9e000 - 000000007ffe0000 (ACPI NVS)
> > [ 0.000000] BIOS-e820: 000000007ffe0000 - 0000000080000000 (reserved)
> > [ 0.000000] BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
> > [ 0.000000] BIOS-e820: 00000000ffb00000 - 0000000100000000 (reserved)
> > [ 0.000000] BIOS-e820: 0000000100000000 - 0000000180000000 (usable)
> > [ 0.000000] debug: ignoring loglevel setting.
> > [ 0.000000] Entering add_active_range(0, 0, 159) 0 entries of 256 used
> > [ 0.000000] Entering add_active_range(0, 256, 524176) 1 entries of 256 used
> > [ 0.000000] Entering add_active_range(0, 1048576, 1572864) 2 entries of 256 used
> > [ 0.000000] max_pfn_mapped = 1572864
> > [ 0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
> > [ 0.000000] init_memory_mapping
> > [ 0.000000] DMI 2.4 present.
> > [ 0.000000] ACPI: RSDP 000FA980, 0024 (r2 ACPIAM)
> > [ 0.000000] ACPI: XSDT 7FF90100, 0054 (r1 KOZIRO FRONTIER 2000707 MSFT 97)
> > [ 0.000000] ACPI: FACP 7FF90290, 00F4 (r3 MSTEST OEMFACP 2000707 MSFT 97)
> > [ 0.000000] ACPI: DSDT 7FF905C0, 8FA9 (r1 A0637 A0637000 0 INTL 20060113)
> > [ 0.000000] ACPI: FACS 7FF9E000, 0040
> > [ 0.000000] ACPI: APIC 7FF90390, 006C (r1 MSTEST OEMAPIC 2000707 MSFT 97)
> > [ 0.000000] ACPI: MCFG 7FF90400, 003C (r1 MSTEST OEMMCFG 2000707 MSFT 97)
> > [ 0.000000] ACPI: SLIC 7FF90440, 0176 (r1 KOZIRO FRONTIER 2000707 MSFT 97)
> > [ 0.000000] ACPI: OEMB 7FF9E040, 007B (r1 MSTEST AMI_OEM 2000707 MSFT 97)
> > [ 0.000000] ACPI: HPET 7FF99570, 0038 (r1 MSTEST OEMHPET 2000707 MSFT 97)
> > [ 0.000000] Entering add_active_range(0, 0, 159) 0 entries of 256 used
> > [ 0.000000] Entering add_active_range(0, 256, 524176) 1 entries of 256 used
> > [ 0.000000] Entering add_active_range(0, 1048576, 1572864) 2 entries of 256 used
> > [ 0.000000] early res: 0 [0-fff] BIOS data page
> > [ 0.000000] early res: 1 [6000-7fff] TRAMPOLINE
> > [ 0.000000] early res: 2 [200000-af2207] TEXT DATA BSS
> > [ 0.000000] early res: 3 [9fc00-fffff] BIOS reserved
> > [ 0.000000] early res: 4 [8000-dfff] PGTABLE
> > [ 0.000000] [ffffe20000000000-ffffe200053fffff] PMD -> [ffff81000c200000-ffff8100115fffff] on node 0
> > [ 0.000000] Zone PFN ranges:
> > [ 0.000000] DMA 0 -> 4096
> > [ 0.000000] DMA32 4096 -> 1048576
> > [ 0.000000] Normal 1048576 -> 1572864
> > [ 0.000000] Movable zone start PFN for each node
> > [ 0.000000] early_node_map[3] active PFN ranges
> > [ 0.000000] 0: 0 -> 159
> > [ 0.000000] 0: 256 -> 524176
> > [ 0.000000] 0: 1048576 -> 1572864
> > [ 0.000000] On node 0 totalpages: 1048367
> > [ 0.000000] DMA zone: 56 pages used for memmap
> > [ 0.000000] DMA zone: 2397 pages reserved
> > [ 0.000000] DMA zone: 1546 pages, LIFO batch:0
> > [ 0.000000] DMA32 zone: 14280 pages used for memmap
> > [ 0.000000] DMA32 zone: 505800 pages, LIFO batch:31
> > [ 0.000000] Normal zone: 7168 pages used for memmap
> > [ 0.000000] Normal zone: 517120 pages, LIFO batch:31
> > [ 0.000000] Movable zone: 0 pages used for memmap
> > [ 0.000000] ACPI: PM-Timer IO Port: 0x808
> > [ 0.000000] ACPI: Local APIC address 0xfee00000
> > [ 0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
> > [ 0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
> > [ 0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x82] disabled)
> > [ 0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x83] disabled)
> > [ 0.000000] ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
> > [ 0.000000] IOAPIC[0]: apic_id 2, version 0, address 0xfec00000, GSI 0-23
> > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> > [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
> > [ 0.000000] ACPI: IRQ0 used by override.
> > [ 0.000000] ACPI: IRQ2 used by override.
> > [ 0.000000] ACPI: IRQ9 used by override.
> > [ 0.000000] Setting APIC routing to flat
> > [ 0.000000] ACPI: HPET id: 0x8086a202 base: 0xfed00000
> > [ 0.000000] Using ACPI (MADT) for SMP configuration information
> > [ 0.000000] Allocating PCI resources starting at 88000000 (gap: 80000000:7ee00000)
> > [ 0.000000] PERCPU: Allocating 33732 bytes of per cpu data
> > [ 0.000000] NR_CPUS: 2, nr_cpu_ids: 2
> > [4294014.506571] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 1024466
> > [4294014.506571] Kernel command line: root=/dev/sda2 netconsole=@192.168.0.1/eth0,9353@192.168.0.42/00:1b:38:af:22:49 ignore_loglevel
> > [4294014.506571] Initializing CPU#0
> > [4294014.506571] Preemptible RCU implementation.
> > [4294014.506571] PID hash table entries: 4096 (order: 12, 32768 bytes)
> > [4294014.506571] Extended CMOS year: 2000
> > [4294014.506571] TSC calibrated against PM_TIMER
> > [4294014.506571] time.c: Detected 2135.039 MHz processor.
> > [4294014.506571] Console: colour VGA+ 80x25
> > [4294014.506571] console [tty0] enabled
> > [4294014.506571] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
> > [4294014.506571] ... MAX_LOCKDEP_SUBCLASSES: 8
> > [4294014.506571] ... MAX_LOCK_DEPTH: 48
> > [4294014.506571] ... MAX_LOCKDEP_KEYS: 2048
> > [4294014.506571] ... CLASSHASH_SIZE: 1024
> > [4294014.506571] ... MAX_LOCKDEP_ENTRIES: 8192
> > [4294014.506571] ... MAX_LOCKDEP_CHAINS: 16384
> > [4294014.506571] ... CHAINHASH_SIZE: 8192
> > [4294014.506571] memory used by lock dependency info: 1648 kB
> > [4294014.506571] per task-struct memory footprint: 2688 bytes
> > [4294014.506571] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
> > [4294014.506571] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
> > [4294014.506571] Checking aperture...
> > [4294014.506571] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> > [4294014.506571] Placing software IO TLB between 0x161d000 - 0x561d000
> > [4294014.506571] Memory: 4025184k/6291456k available (2445k kernel code, 167852k reserved, 1415k data, 240k init)
> > [4294014.506571] CPA: page pool initialized 64 of 64 pages preallocated
> > [4294014.506571] SLUB: Genslabs=12, HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
> > [4294014.506571] hpet clockevent registered
> > [4294014.566487] Calibrating delay using timer specific routine.. 4273.61 BogoMIPS (lpj=2136806)
> > [4294014.566561] Mount-cache hash table entries: 256
> > [4294014.567488] CPU: L1 I cache: 32K, L1 D cache: 32K
> > [4294014.567561] CPU: L2 cache: 2048K
> > [4294014.567561] CPU: Physical Processor ID: 0
> > [4294014.567561] CPU: Processor Core ID: 0
> > [4294014.567561] CPU0: Thermal monitoring enabled (TM2)
> > [4294014.567561] using mwait in idle threads.
> > [4294014.567561] debug: unmapping init memory ffffffff805df000..ffffffff805e4000
> > [4294014.567561] ACPI: Core revision 20080321
> > [4294014.607912] CPU0: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz stepping 02
> > [4294014.608114] Using local APIC timer interrupts.
> > [4294014.609555] APIC timer calibration result 16679992
> > [4294014.609555] Detected 16.679 MHz APIC timer.
> > [4294014.609555] lockdep: fixing up alternatives.
> > [4294014.609555] Booting processor 1/1 ip 6000
> > [4294014.619023] Initializing CPU#1
> > [4294014.619023] Calibrating delay using timer specific routine.. 4269.89 BogoMIPS (lpj=2134946)
> > [4294014.619023] CPU: L1 I cache: 32K, L1 D cache: 32K
> > [4294014.619023] CPU: L2 cache: 2048K
> > [4294014.619023] CPU: Physical Processor ID: 0
> > [4294014.619023] CPU: Processor Core ID: 1
> > [4294014.619023] CPU1: Thermal monitoring enabled (TM2)
> > [4294014.619023] x86 PAT enabled: cpu 1, old 0x7040600070406, new 0x7010600070106
> > [4294014.681144] CPU1: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz stepping 02
> > [4294014.682127] checking TSC synchronization [CPU#0 -> CPU#1]: passed.
> > [4294014.682650] Brought up 2 CPUs
> > [4294014.682733] Total of 2 processors activated (8543.50 BogoMIPS).
> > [4294014.684543] net_namespace: 1040 bytes
> > [4294014.684543] NET: Registered protocol family 16
> > [4294014.685543] No dock devices found.
> > [4294014.685592] ACPI: bus type pci registered
> > [4294014.685683] PCI: Using configuration type 1 for base access
> > [4294014.692161] ACPI: EC: Look up EC in DSDT
> > [4294014.745403] ACPI: Interpreter enabled
> > [4294014.745489] ACPI: (supports S0 S5)
> > [4294014.745671] ACPI: Using IOAPIC for interrupt routing
> > [4294014.783888] ACPI: PCI Root Bridge [PCI0] (0000:00)
> > [4294014.785059] pci 0000:00:1f.0: quirk: region 0800-087f claimed by ICH6 ACPI/GPIO/TCO
> > [4294014.785224] pci 0000:00:1f.0: quirk: region 0480-04bf claimed by ICH6 GPIO
> > [4294014.786302] PCI: Transparent bridge - 0000:00:1e.0
> > [4294014.786438] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
> > [4294014.789050] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P2._PRT]
> > [4294014.790048] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P1._PRT]
> > [4294014.791455] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P4._PRT]
> > [4294014.792085] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P7._PRT]
> > [4294014.792699] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0P8._PRT]
> > [4294014.851947] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 10 *11 12 14 15)
> > [4294014.852880] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 7 10 11 12 14 15)
> > [4294014.853755] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 10 11 12 14 *15)
> > [4294014.854612] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 *10 11 12 14 15)
> > [4294014.855025] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
> > [4294014.855955] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 10 11 12 14 15) *0, disabled.
> > [4294014.856884] ACPI: PCI Interrupt Link [LNKG] (IRQs *3 4 5 6 7 10 11 12 14 15)
> > [4294014.859254] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 10 11 12 *14 15)
> > [4294014.861041] ACPI Warning (tbutils-0217): Incorrect checksum in table [OEMB] - 08, should be 03 [20080321]
> > [4294014.861041] Linux Plug and Play Support v0.97 (c) Adam Belay
> > [4294014.861041] pnp: PnP ACPI init
> > [4294014.861041] ACPI: bus type pnp registered
> > [4294014.876545] pnp: PnP ACPI: found 15 devices
> > [4294014.876616] ACPI: ACPI bus type pnp unregistered
> > [4294014.877616] SCSI subsystem initialized
> > [4294014.877616] libata version 3.00 loaded.
> > [4294014.878616] PCI: Using ACPI for IRQ routing
> > [4294014.885252] PCI-GART: No AMD northbridge found.
> > [4294014.897373] system 00:01: iomem range 0xfed14000-0xfed19fff has been reserved
> > [4294014.897532] system 00:07: ioport range 0x290-0x297 has been reserved
> > [4294014.897657] system 00:08: ioport range 0x4d0-0x4d1 has been reserved
> > [4294014.897755] system 00:08: ioport range 0x800-0x87f has been reserved
> > [4294014.898307] system 00:08: ioport range 0x480-0x4bf has been reserved
> > [4294014.898356] system 00:08: iomem range 0xffafe000-0xffb0cbff could not be reserved
> > [4294014.898356] system 00:08: iomem range 0xffb00000-0xffbfffff could not be reserved
> > [4294014.898356] system 00:08: iomem range 0xfed1c000-0xfed1ffff has been reserved
> > [4294014.898493] system 00:08: iomem range 0xfed20000-0xfed8ffff has been reserved
> > [4294014.898628] system 00:08: iomem range 0xfff00000-0xfffffffe could not be reserved
> > [4294014.898762] system 00:08: iomem range 0xfebfe000-0xfebfec00 has been reserved
> > [4294014.899305] system 00:0b: iomem range 0xfec00000-0xfec00fff has been reserved
> > [4294014.899440] system 00:0b: iomem range 0xfee00000-0xfee00fff could not be reserved
> > [4294014.899585] system 00:0d: iomem range 0xe0000000-0xefffffff has been reserved
> > [4294014.899729] system 00:0e: iomem range 0x0-0x9ffff could not be reserved
> > [4294014.899811] system 00:0e: iomem range 0xc0000-0xcffff has been reserved
> > [4294014.899915] system 00:0e: iomem range 0xe0000-0xfffff could not be reserved
> > [4294014.900017] system 00:0e: iomem range 0x100000-0x7fffffff could not be reserved
> > [4294014.900152] system 00:0e: iomem range 0x0-0xffffffffffffffff could not be reserved
> > [4294014.901942] PCI: Bridge: 0000:00:01.0
> > [4294014.902028] IO window: 9000-9fff
> > [4294014.902113] MEM window: 0xf8700000-0xfe7fffff
> > [4294014.902202] PREFETCH window: 0x00000000bfe00000-0x00000000dfdfffff
> > [4294014.902730] PCI: Bridge: 0000:00:1c.0
> > [4294014.902814] IO window: disabled.
> > [4294014.902900] MEM window: disabled.
> > [4294014.902985] PREFETCH window: 0x00000000dfe00000-0x00000000dfefffff
> > [4294014.903085] PCI: Bridge: 0000:00:1c.3
> > [4294014.903168] IO window: disabled.
> > [4294014.903192] MEM window: 0xfe900000-0xfe9fffff
> > [4294014.903192] PREFETCH window: disabled.
> > [4294014.903286] PCI: Bridge: 0000:00:1c.4
> > [4294014.903370] IO window: a000-afff
> > [4294014.903456] MEM window: 0xfe800000-0xfe8fffff
> > [4294014.903546] PREFETCH window: disabled.
> > [4294014.903637] PCI: Bridge: 0000:00:1e.0
> > [4294014.903728] IO window: b000-bfff
> > [4294014.903814] MEM window: 0xfea00000-0xfeafffff
> > [4294014.903903] PREFETCH window: 0x0000000088000000-0x00000000880fffff
> > [4294014.904021] ACPI: PCI Interrupt 0000:00:01.0[A] -> GSI 16 (level, low) -> IRQ 16
> > [4294014.904191] PCI: Setting latency timer of device 0000:00:01.0 to 64
> > [4294014.904311] ACPI: PCI Interrupt 0000:00:1c.0[A] -> GSI 16 (level, low) -> IRQ 16
> > [4294014.904480] PCI: Setting latency timer of device 0000:00:1c.0 to 64
> > [4294014.904725] ACPI: PCI Interrupt 0000:00:1c.3[D] -> GSI 19 (level, low) -> IRQ 19
> > [4294014.904895] PCI: Setting latency timer of device 0000:00:1c.3 to 64
> > [4294014.905010] ACPI: PCI Interrupt 0000:00:1c.4[A] -> GSI 16 (level, low) -> IRQ 16
> > [4294014.905180] PCI: Setting latency timer of device 0000:00:1c.4 to 64
> > [4294014.905284] PCI: Setting latency timer of device 0000:00:1e.0 to 64
> > [4294014.905423] NET: Registered protocol family 2
> > [4294014.939148] IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
> > [4294014.939721] TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
> > [4294014.941766] TCP bind hash table entries: 65536 (order: 9, 3670016 bytes)
> > [4294014.946760] TCP: Hash tables configured (established 262144 bind 65536)
> > [4294014.946894] TCP reno registered
> > [4294014.958451] NET: Registered protocol family 1
> > [4294014.976123] msgmni has been set to 7862 for ipc namespace ffffffff80572380
> > [4294014.976123] io scheduler noop registered
> > [4294014.977031] io scheduler cfq registered (default)
> > [4294014.977031] pci 0000:01:00.0: Boot video device
> > [4294014.986266] Real Time Clock Driver v1.12ac
> > [4294014.986266] Linux agpgart interface v0.103
> > [4294014.986470] ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 19 (level, low) -> IRQ 19
> > [4294014.986470] PCI: Setting latency timer of device 0000:03:00.0 to 64
> > [4294014.986470] atl1 0000:03:00.0: version 2.1.3
> > [4294015.012034] Switched to high resolution mode on CPU 1
> > [4294015.012878] Switched to high resolution mode on CPU 0
> > [4294015.023188] 8139too Fast Ethernet driver 0.9.28
> > [4294015.023188] ACPI: PCI Interrupt 0000:05:02.0[A] -> GSI 23 (level, low) -> IRQ 23
> > [4294015.024187] eth1: RealTek RTL8139 at 0xb800, 00:80:48:2e:06:2e, IRQ 23
> > [4294015.024187] eth1: Identified 8139 chip type 'RTL-8100B/8139D'
> > [4294015.024187] netconsole: local port 6665
> > [4294015.024273] netconsole: local IP 192.168.0.1
> > [4294015.024273] netconsole: interface eth0
> > [4294015.024273] netconsole: remote port 9353
> > [4294015.024276] netconsole: remote IP 192.168.0.42
> > [4294015.025888] netconsole: remote ethernet address 00:1b:38:af:22:49
> > [4294015.025992] netconsole: device eth0 not up yet, forcing it
> > [4294017.099102] atl1 0000:03:00.0: eth0 link is up 1000 Mbps full duplex
> > [4294017.111214] console [netcon0] enabled
> > [4294017.139758] netconsole: network logging started
> > [4294017.140754] Driver 'sd' needs updating - please use bus_type methods
> > [4294017.141147] ahci 0000:02:00.0: version 3.0
> > [4294017.141147] ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
> > [4294018.140956] ahci 0000:02:00.0: AHCI 0001.0000 32 slots 2 ports 3 Gbps 0x3 impl SATA mode
> > [4294018.141102] ahci 0000:02:00.0: flags: 64bit ncq pm led clo pmp pio slum part
> > [4294018.141249] PCI: Setting latency timer of device 0000:02:00.0 to 64
> > [4294018.142241] scsi0 : ahci
> > [4294018.143147] scsi1 : ahci
> > [4294018.143147] ata1: SATA max UDMA/133 abar m8192@0xfe8fe000 port 0xfe8fe100 irq 16
> > [4294018.143345] ata2: SATA max UDMA/133 abar m8192@0xfe8fe000 port 0xfe8fe180 irq 16
> > [4294018.448110] ata1: SATA link down (SStatus 0 SControl 300)
> > [4294018.753109] ata2: SATA link down (SStatus 0 SControl 300)
> > [4294018.754706] ata_piix 0000:00:1f.2: version 2.12
> > [4294018.754817] ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 19
> > [4294018.754995] ata_piix 0000:00:1f.2: MAP [ P0 P2 P1 P3 ]
> > [4294018.755342] PCI: Setting latency timer of device 0000:00:1f.2 to 64
> > [4294018.755342] scsi2 : ata_piix
> > [4294018.756173] scsi3 : ata_piix
> > [4294018.770890] ata3: SATA max UDMA/133 cmd 0xec00 ctl 0xe880 bmdma 0xe400 irq 19
> > [4294018.771029] ata4: SATA max UDMA/133 cmd 0xe800 ctl 0xe480 bmdma 0xe408 irq 19
> > [4294019.226141] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
> > [4294019.232666] ata3.00: ATA-8: ST3750330AS, SD15, max UDMA/133
> > [4294019.232767] ata3.00: 1465149168 sectors, multi 16: LBA48 NCQ (depth 0/32)
> > [4294019.266037] ata3.01: ATA-7: ST3160811AS, 3.AAE, max UDMA/133
> > [4294019.266037] ata3.01: 312581808 sectors, multi 16: LBA48 NCQ (depth 0/32)
> > [4294019.272668] ata3.00: configured for UDMA/133
> > [4294019.324047] ata3.01: configured for UDMA/133
> > [4294019.780119] ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
> > [4294019.816037] ata4.00: ATA-7: ST3250620AS, 3.AAE, max UDMA/133
> > [4294019.816037] ata4.00: 488397168 sectors, multi 16: LBA48 NCQ (depth 0/32)
> > [4294019.891046] ata4.00: configured for UDMA/133
> > [4294019.891046] scsi 2:0:0:0: Direct-Access ATA ST3750330AS SD15 PQ: 0 ANSI: 5
> > [4294019.893307] sd 2:0:0:0: [sda] 1465149168 512-byte hardware sectors (750156 MB)
> > [4294019.893459] sd 2:0:0:0: [sda] Write Protect is off
> > [4294019.893459] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
> > [4294019.893513] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> > [4294019.894080] sd 2:0:0:0: [sda] 1465149168 512-byte hardware sectors (750156 MB)
> > [4294019.894305] sd 2:0:0:0: [sda] Write Protect is off
> > [4294019.894305] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
> > [4294019.894362] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> > [4294019.894527] sda: sda1 sda2
> > [4294019.931316] sd 2:0:0:0: [sda] Attached SCSI disk
> > [4294019.931316] scsi 2:0:1:0: Direct-Access ATA ST3160811AS 3.AA PQ: 0 ANSI: 5
> > [4294019.932060] sd 2:0:1:0: [sdb] 312581808 512-byte hardware sectors (160042 MB)
> > [4294019.932234] sd 2:0:1:0: [sdb] Write Protect is off
> > [4294019.932307] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
> > [4294019.932307] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> > [4294019.932307] sd 2:0:1:0: [sdb] 312581808 512-byte hardware sectors (160042 MB)
> > [4294019.932487] sd 2:0:1:0: [sdb] Write Protect is off
> > [4294019.932581] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
> > [4294019.932749] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> > [4294019.932896] sdb: unknown partition table
> > [4294019.955303] sd 2:0:1:0: [sdb] Attached SCSI disk
> > [4294019.957296] scsi 3:0:0:0: Direct-Access ATA ST3250620AS 3.AA PQ: 0 ANSI: 5
> > [4294019.957296] sd 3:0:0:0: [sdc] 488397168 512-byte hardware sectors (250059 MB)
> > [4294019.957481] sd 3:0:0:0: [sdc] Write Protect is off
> > [4294019.957575] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
> > [4294019.957750] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> > [4294019.958022] sd 3:0:0:0: [sdc] 488397168 512-byte hardware sectors (250059 MB)
> > [4294019.958202] sd 3:0:0:0: [sdc] Write Protect is off
> > [4294019.958300] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
> > [4294019.958479] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
> > [4294019.958625] sdc: sdc1
> > [4294019.974297] sd 3:0:0:0: [sdc] Attached SCSI disk
> > [4294019.974297] ACPI: PCI Interrupt 0000:00:1f.5[B] -> GSI 19 (level, low) -> IRQ 19
> > [4294019.974297] ata_piix 0000:00:1f.5: MAP [ P0 -- P1 -- ]
> > [4294019.974519] PCI: Setting latency timer of device 0000:00:1f.5 to 64
> > [4294019.975300] scsi4 : ata_piix
> > [4294019.975301] scsi5 : ata_piix
> > [4294019.984287] ata5: SATA max UDMA/133 cmd 0xd400 ctl 0xd080 bmdma 0xc880 irq 19
> > [4294019.984287] ata6: SATA max UDMA/133 cmd 0xd000 ctl 0xcc00 bmdma 0xc888 irq 19
> > [4294020.298796] ata5: SATA link down (SStatus 0 SControl 300)
> > [4294020.613795] ata6: SATA link down (SStatus 0 SControl 300)
> > [4294020.614953] ACPI: PCI Interrupt 0000:02:00.1[B] -> GSI 17 (level, low) -> IRQ 17
> > [4294020.615172] PCI: Setting latency timer of device 0000:02:00.1 to 64
> > [4294020.615408] scsi6 : pata_jmicron
> > [4294020.615408] scsi7 : pata_jmicron
> > [4294020.619409] ata7: PATA max UDMA/100 cmd 0xac00 ctl 0xa880 bmdma 0xa400 irq 17
> > [4294020.619409] ata8: PATA max UDMA/100 cmd 0xa800 ctl 0xa480 bmdma 0xa408 irq 17
> > [4294020.924351] ata7.01: ATAPI: _NEC DV-5800C, D9S2, max UDMA/33
> > [4294020.924889] ata7.01: configured for UDMA/33
> > [4294021.233785] scsi 6:0:1:0: CD-ROM _NEC DV-5800C D9S2 PQ: 0 ANSI: 5
> > [4294021.235400] PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
> > [4294021.235400] PNP: PS/2 appears to have AUX port disabled, if this is incorrect please boot with i8042.nopnp
> > [4294021.236404] serio: i8042 KBD port at 0x60,0x64 irq 1
> > [4294021.236404] mice: PS/2 mouse device common for all mice
> > [4294021.237401] Advanced Linux Sound Architecture Driver Version 1.0.16.
> > [4294021.238436] ACPI: PCI Interrupt 0000:00:1b.0[A] -> GSI 22 (level, low) -> IRQ 22
> > [4294021.240156] PCI: Setting latency timer of device 0000:00:1b.0 to 64
> > [4294021.253726] input: AT Translated Set 2 keyboard as /class/input/input0
> > [4294021.623063] ALSA device list:
> > [4294021.623063] #0: HDA Intel at 0xfebf8000 irq 22
> > [4294021.623179] TCP cubic registered
> > [4294021.660774] kjournald starting. Commit interval 5 seconds
> > [4294021.656982] EXT3-fs: mounted filesystem with ordered data mode.
> > [4294021.656982] VFS: Mounted root (ext3 filesystem) readonly.
> > [4294021.656982] debug: unmapping init memory ffffffff805e4000..ffffffff80620000
> > [4294021.661096] Write protecting the kernel read-only data: 3412k
> > [4294021.661314] Testing CPA: undo ffffffff8046e000-ffffffff8055e000
> > [4294021.661445] Testing CPA: again
> > [4294023.534235] Driver 'sr' needs updating - please use bus_type methods
> > [4294023.546361] sr0: scsi3-mmc drive: 48x/48x cd/rw xa/form2 cdda tray
> > [4294023.546473] Uniform CD-ROM driver Revision: 3.20
> > [4294023.546816] sr 6:0:1:0: Attached scsi CD-ROM sr0
> > [4294023.570726] usbcore: registered new interface driver usbfs
> > [4294023.570898] usbcore: registered new interface driver hub
> > [4294023.576263] usbcore: registered new device driver usb
> > [4294023.579267] USB Universal Host Controller Interface driver v3.0
> > [4294023.579998] ACPI: PCI Interrupt 0000:00:1a.0[A] -> GSI 16 (level, low) -> IRQ 16
> > [4294023.580193] PCI: Setting latency timer of device 0000:00:1a.0 to 64
> > [4294023.580300] uhci_hcd 0000:00:1a.0: UHCI Host Controller
> > [4294023.580725] uhci_hcd 0000:00:1a.0: new USB bus registered, assigned bus number 1
> > [4294023.580905] uhci_hcd 0000:00:1a.0: irq 16, io base 0x0000dc00
> > [4294023.581598] usb usb1: configuration #1 chosen from 1 choice
> > [4294023.581862] hub 1-0:1.0: USB hub found
> > [4294023.582035] hub 1-0:1.0: 2 ports detected
> > [4294023.683559] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
> > [4294023.683670] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> > [4294023.683809] usb usb1: Product: UHCI Host Controller
> > [4294023.683809] usb usb1: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
> > [4294023.683809] usb usb1: SerialNumber: 0000:00:1a.0
> > [4294023.683809] ACPI: PCI Interrupt 0000:00:1a.1[B] -> GSI 17 (level, low) -> IRQ 17
> > [4294023.684033] PCI: Setting latency timer of device 0000:00:1a.1 to 64
> > [4294023.684134] uhci_hcd 0000:00:1a.1: UHCI Host Controller
> > [4294023.684594] uhci_hcd 0000:00:1a.1: new USB bus registered, assigned bus number 2
> > [4294023.684765] uhci_hcd 0000:00:1a.1: irq 17, io base 0x0000e000
> > [4294023.685097] usb usb2: configuration #1 chosen from 1 choice
> > [4294023.685295] hub 2-0:1.0: USB hub found
> > [4294023.685400] hub 2-0:1.0: 2 ports detected
> > [4294023.786278] usb usb2: New USB device found, idVendor=1d6b, idProduct=0001
> > [4294023.786389] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> > [4294023.786593] usb usb2: Product: UHCI Host Controller
> > [4294023.786691] usb usb2: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
> > [4294023.786787] usb usb2: SerialNumber: 0000:00:1a.1
> > [4294023.786967] ACPI: PCI Interrupt 0000:00:1a.7[C] -> GSI 18 (level, low) -> IRQ 18
> > [4294023.787162] PCI: Setting latency timer of device 0000:00:1a.7 to 64
> > [4294023.787287] ehci_hcd 0000:00:1a.7: EHCI Host Controller
> > [4294023.787593] ehci_hcd 0000:00:1a.7: new USB bus registered, assigned bus number 3
> > [4294023.791704] ehci_hcd 0000:00:1a.7: debug port 1
> > [4294023.791816] PCI: cache line size of 32 is not supported by device 0000:00:1a.7
> > [4294023.791972] ehci_hcd 0000:00:1a.7: irq 18, io mem 0xfebffc00
> > [4294023.801337] ehci_hcd 0000:00:1a.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
> > [4294023.801820] usb usb3: configuration #1 chosen from 1 choice
> > [4294023.802010] hub 3-0:1.0: USB hub found
> > [4294023.802189] hub 3-0:1.0: 4 ports detected
> > [4294023.903972] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002
> > [4294023.904439] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> > [4294023.904591] usb usb3: Product: EHCI Host Controller
> > [4294023.904685] usb usb3: Manufacturer: Linux 2.6.26-rc4 ehci_hcd
> > [4294023.904787] usb usb3: SerialNumber: 0000:00:1a.7
> > [4294023.904931] ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 23 (level, low) -> IRQ 23
> > [4294023.905127] PCI: Setting latency timer of device 0000:00:1d.0 to 64
> > [4294023.905236] uhci_hcd 0000:00:1d.0: UHCI Host Controller
> > [4294023.905436] uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 4
> > [4294023.905625] uhci_hcd 0000:00:1d.0: irq 23, io base 0x0000d480
> > [4294023.906151] usb usb4: configuration #1 chosen from 1 choice
> > [4294023.906369] hub 4-0:1.0: USB hub found
> > [4294023.906481] hub 4-0:1.0: 2 ports detected
> > [4294024.007354] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001
> > [4294024.007875] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> > [4294024.008024] usb usb4: Product: UHCI Host Controller
> > [4294024.008122] usb usb4: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
> > [4294024.008220] usb usb4: SerialNumber: 0000:00:1d.0
> > [4294024.008395] ACPI: PCI Interrupt 0000:00:1d.7[A] -> GSI 23 (level, low) -> IRQ 23
> > [4294024.008581] PCI: Setting latency timer of device 0000:00:1d.7 to 64
> > [4294024.008704] ehci_hcd 0000:00:1d.7: EHCI Host Controller
> > [4294024.008872] ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 5
> > [4294024.013011] ehci_hcd 0000:00:1d.7: debug port 1
> > [4294024.013126] PCI: cache line size of 32 is not supported by device 0000:00:1d.7
> > [4294024.013272] ehci_hcd 0000:00:1d.7: irq 23, io mem 0xfebff800
> > [4294024.022630] ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
> > [4294024.023099] usb usb5: configuration #1 chosen from 1 choice
> > [4294024.023303] hub 5-0:1.0: USB hub found
> > [4294024.023419] hub 5-0:1.0: 6 ports detected
> > [4294024.123981] usb usb5: New USB device found, idVendor=1d6b, idProduct=0002
> > [4294024.124094] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> > [4294024.124242] usb usb5: Product: EHCI Host Controller
> > [4294024.124340] usb usb5: Manufacturer: Linux 2.6.26-rc4 ehci_hcd
> > [4294024.124436] usb usb5: SerialNumber: 0000:00:1d.7
> > [4294024.124567] ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 19
> > [4294024.124759] PCI: Setting latency timer of device 0000:00:1d.1 to 64
> > [4294024.124867] uhci_hcd 0000:00:1d.1: UHCI Host Controller
> > [4294024.125862] uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 6
> > [4294024.125862] uhci_hcd 0000:00:1d.1: irq 19, io base 0x0000d800
> > [4294024.125862] usb usb6: configuration #1 chosen from 1 choice
> > [4294024.126053] hub 6-0:1.0: USB hub found
> > [4294024.126158] hub 6-0:1.0: 2 ports detected
> > [4294024.227783] usb usb6: New USB device found, idVendor=1d6b, idProduct=0001
> > [4294024.227783] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> > [4294024.227783] usb usb6: Product: UHCI Host Controller
> > [4294024.227783] usb usb6: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
> > [4294024.227783] usb usb6: SerialNumber: 0000:00:1d.1
> > [4294024.227783] ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 18
> > [4294024.227783] PCI: Setting latency timer of device 0000:00:1d.2 to 64
> > [4294024.227888] uhci_hcd 0000:00:1d.2: UHCI Host Controller
> > [4294024.234787] uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 7
> > [4294024.234958] uhci_hcd 0000:00:1d.2: irq 18, io base 0x0000d880
> > [4294024.235390] usb usb7: configuration #1 chosen from 1 choice
> > [4294024.235641] hub 7-0:1.0: USB hub found
> > [4294024.235783] hub 7-0:1.0: 2 ports detected
> > [4294024.336777] usb usb7: New USB device found, idVendor=1d6b, idProduct=0001
> > [4294024.337631] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
> > [4294024.337631] usb usb7: Product: UHCI Host Controller
> > [4294024.337631] usb usb7: Manufacturer: Linux 2.6.26-rc4 uhci_hcd
> > [4294024.337631] usb usb7: SerialNumber: 0000:00:1d.2
> > [4294025.275801] EXT3 FS on sda2, internal journal
> > [4294025.576923] usbcore: registered new interface driver usblp
> > [4294025.699206] Adding 9775512k swap on /dev/sda1. Priority:-1 extents:1 across:9775512k
> > [4294034.064415] ip_tables: (C) 2000-2006 Netfilter Core Team
> > [4294034.072599] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
> > [4294036.120084] eth1: link up, 100Mbps, full-duplex, lpa 0x45E1
> > [4294045.749739] CPA self-test:
> > [4294045.752254] 4k 16384 large 2528 gb 0 x 0[0-0] miss 262144
> > [4294045.761417] 4k 187392 large 2194 gb 0 x 0[0-0] miss 262144
> > [4294045.768378] 4k 187392 large 2194 gb 0 x 0[0-0] miss 262144
> > [4294045.768608] ok.
> > [4294376.517131] warning: `capget01' uses 32-bit capabilities (legacy support in use)
> > [4294665.514678] Adding 65528k swap on ./swapfile01. Priority:-2 extents:27 across:91612k
> > [4294667.501075] Adding 65528k swap on ./swapfile01. Priority:-3 extents:40 across:157332k
> > [4294669.395350] Adding 65528k swap on ./swapfile01. Priority:-4 extents:21 across:108832k
> > [4294669.504418] Unable to find swap-space signature
> > [4294669.548523] Adding 32k swap on alreadyused. Priority:-5 extents:1 across:32k
> > [4294669.594260] Adding 32k swap on swapfile02. Priority:-6 extents:3 across:56k
> > [4294669.657350] Adding 32k swap on swapfile03. Priority:-7 extents:1 across:32k
> > [4294669.697470] Adding 32k swap on swapfile04. Priority:-8 extents:2 across:76k
> > [4294669.711497] Adding 32k swap on swapfile05. Priority:-9 extents:1 across:32k
> > [4294669.757754] Adding 32k swap on swapfile06. Priority:-10 extents:2 across:48k
> > [4294669.772201] Adding 32k swap on swapfile07. Priority:-11 extents:1 across:32k
> > [4294669.792286] Adding 32k swap on swapfile08. Priority:-12 extents:2 across:1820k
> > [4294669.817947] Adding 32k swap on swapfile09. Priority:-13 extents:1 across:32k
> > [4294669.839671] Adding 32k swap on swapfile10. Priority:-14 extents:1 across:32k
> > [4294669.856414] Adding 32k swap on swapfile11. Priority:-15 extents:2 across:36k
> > [4294669.898118] Adding 32k swap on swapfile12. Priority:-16 extents:1 across:32k
> > [4294669.918366] Adding 32k swap on swapfile13. Priority:-17 extents:1 across:32k
> > [4294669.932981] Adding 32k swap on swapfile14. Priority:-18 extents:1 across:32k
> > [4294669.981569] Adding 32k swap on swapfile15. Priority:-19 extents:1 across:32k
> > [4294670.016775] Adding 32k swap on swapfile16. Priority:-20 extents:1 across:32k
> > [4294670.031717] Adding 32k swap on swapfile17. Priority:-21 extents:1 across:32k
> > [4294670.080750] Adding 32k swap on swapfile18. Priority:-22 extents:1 across:32k
> > [4294670.095724] Adding 32k swap on swapfile19. Priority:-23 extents:1 across:32k
> > [4294670.141188] Adding 32k swap on swapfile20. Priority:-24 extents:1 across:32k
> > [4294670.174754] Adding 32k swap on swapfile21. Priority:-25 extents:1 across:32k
> > [4294670.228986] Adding 32k swap on swapfile22. Priority:-26 extents:1 across:32k
> > [4294670.277409] Adding 32k swap on swapfile23. Priority:-27 extents:1 across:32k
> > [4294670.331498] Adding 32k swap on swapfile24. Priority:-28 extents:2 across:32k
> > [4294670.364858] Adding 32k swap on swapfile25. Priority:-29 extents:1 across:32k
> > [4294670.401739] Adding 32k swap on swapfile26. Priority:-30 extents:1 across:32k
> > [4294670.416923] Adding 32k swap on swapfile27. Priority:-31 extents:1 across:32k
> > [4294670.470795] Adding 32k swap on swapfile28. Priority:-32 extents:1 across:32k
> > [4294670.524872] Adding 32k swap on swapfile29. Priority:-33 extents:1 across:32k
> > [4294670.538548] Adding 32k swap on swapfile30. Priority:-34 extents:1 across:32k
> > [4294670.678744] Adding 32k swap on firstswapfile. Priority:-35 extents:1 across:32k
> > [4294670.679127] Adding 32k swap on secondswapfile. Priority:-36 extents:1 across:32k
> > [4294671.175248] warning: process `sysctl01' used the deprecated sysctl system call with 1.1.
> > [4294671.175336] warning: process `sysctl01' used the deprecated sysctl system call with 1.2.
> > [4294671.199132] warning: process `sysctl03' used the deprecated sysctl system call with 1.1.
> > [4294671.200735] warning: process `sysctl03' used the deprecated sysctl system call with 1.1.
> > [4294671.237127] warning: process `sysctl04' used the deprecated sysctl system call with
> > [4295759.981152] eth1: link down
> > [4300003.945097] general protection fault: 0000 [1] PREEMPT SMP DEBUG_PAGEALLOC
> > [4300003.945187] CPU 0
> > [4300003.945984] Modules linked in: ext2 nf_conntrack_irc xt_state iptable_filter ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables x_tables usblp ehci_hcd uhci_hcd usbcore sr_mod cdrom
> > [4300003.945984] Pid: 5929, comm: cc1 Not tainted 2.6.26-rc4 #5
> > [4300003.945984] RIP: 0010:[<ffffffff80304b10>] [<ffffffff80304b10>] __call_for_each_cic+0x20/0x50
> > [4300003.945984] RSP: 0018:ffff810074d77e58 EFLAGS: 00010202
> > [4300003.945984] RAX: 6b6b6b6b6b6b6b6b RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000000
> > [4300003.945984] RDX: 0000000000005f5e RSI: 0000000000000000 RDI: ffff81005a1cc500
> > [4300003.945984] RBP: ffff810074d77e78 R08: 0000000000000000 R09: 0000000000000000
> > [4300003.945984] R10: 0000000000000000 R11: 0000000000000001 R12: ffff81011058e0c0
> > [4300003.945984] R13: ffffffff80304b80 R14: ffff81017fcc0000 R15: ffff81005a1cc668
> > [4300003.946943] FS: 00002b1c6a49f6f0(0000) GS:ffffffff805c6000(0000) knlGS:0000000000000000
> > [4300003.946943] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [4300003.946943] CR2: 00002aeeefeb5000 CR3: 000000005a8a6000 CR4: 00000000000006e0
> > [4300003.946943] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [4300003.946943] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [4300003.946943] Process cc1 (pid: 5929, threadinfo ffff810074d76000, task ffff81005a1cc500)
> > [4300003.946943] Stack: ffff810074d77e88 ffff81011058e0c0 0000000000000010 ffff81005a1cc600
> > [4300003.946943] ffff810074d77e88 ffffffff80304b50 ffff810074d77ea8 ffffffff802fffdb
> > [4300003.946943] ffff81005a1cc600 ffff81011058e0c0 ffff810074d77ec8 ffffffff803000af
> > [4300003.946943] Call Trace:
> > [4300003.947037] [<ffffffff80304b50>] cfq_free_io_context+0x10/0x20
> > [4300003.947037] [<ffffffff802fffdb>] put_io_context+0x7b/0x90
> > [4300003.947037] [<ffffffff803000af>] exit_io_context+0x8f/0xb0
> > [4300003.947037] [<ffffffff80235f28>] do_exit+0x558/0x770
> > [4300003.947037] [<ffffffff80236181>] do_group_exit+0x41/0xb0
> > [4300003.947037] [<ffffffff80236202>] sys_exit_group+0x12/0x20
> > [4300003.947037] [<ffffffff8020b68b>] system_call_after_swapgs+0x7b/0x80
> > [4300003.947037]
> > [4300003.947037]
> > [4300003.947037] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 55 49 89 f5 41 54 49 89 fc 53 48 83 ec 08 48 8b 47 68 48 85 c0 74 1f 48 89 c3 90 <48> 8b 03 48 8d 73 88 4c 89 e7 0f 18 08 41 ff d5 48 8b 03 48 85
> > [4300003.948942] RIP [<ffffffff80304b10>] __call_for_each_cic+0x20/0x50
> > [4300003.948942] RSP <ffff810074d77e58>
> > [4300004.868942] ---[ end trace 823782cee637f8c8 ]---
> > [4300004.869038] Fixing recursive fault but reboot is needed!
> > [4303375.170511] general protection fault: 0000 [2] PREEMPT SMP DEBUG_PAGEALLOC
> > [4303375.170600] CPU 1
> > [4303375.170649] Modules linked in: ext2 nf_conntrack_irc xt_state iptable_filter ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_conntrack ip_tables x_tables usblp ehci_hcd uhci_hcd usbcore sr_mod cdrom
> > [4303375.170915] Pid: 4564, comm: cc1 Tainted: G D 2.6.26-rc4 #5
> > [4303375.170982] RIP: 0010:[<ffffffff802a38dc>] [<ffffffff802a38dc>] __d_lookup+0x8c/0x160
> > [4303375.171090] RSP: 0018:ffff81005a24bbc8 EFLAGS: 00010202
> > [4303375.171153] RAX: ffff81006361eee0 RBX: 6b6b6b6b6b6b6b6b RCX: 0000000000000013
> > [4303375.171255] RDX: ffff81000100c120 RSI: 0187320a548baac5 RDI: ffff81012873b4e0
> > [4303375.171348] RBP: ffff81005a24bc18 R08: 0000000953ed9e00 R09: ffff81000b59000e
> > [4303375.171348] R10: ffffffff802a3901 R11: 0000000000000001 R12: 6b6b6b6b6b6b6b2b
> > [4303375.171348] R13: 0000000053ed9e00 R14: ffff81012873b4e0 R15: ffff81005a24bcc8
> > [4303375.171348] FS: 00002b005628b6f0(0000) GS:ffff81017fc44320(0000) knlGS:0000000000000000
> > [4303375.171348] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [4303375.171348] CR2: 00002ac106d23000 CR3: 000000012db71000 CR4: 00000000000006e0
> > [4303375.171348] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [4303375.171348] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [4303375.171348] Process cc1 (pid: 4564, threadinfo ffff81005a24a000, task ffff81007f100000)
> > [4303375.171348] Stack: 000000005a24bbe8 ffff81012873b4e8 0000000000000007 0000000700000002
> > [4303375.171348] ffff81000b59000e ffff81000b590015 ffff81005a24bcc8 ffff81005a24be48
> > [4303375.171348] ffff81005a24bcc8 ffff81005a24bcd8 ffff81005a24bc68 ffffffff802990c5
> > [4303375.171348] Call Trace:
> > [4303375.171348] [<ffffffff802990c5>] do_lookup+0x35/0x200
> > [4303375.171348] [<ffffffff8029aa17>] __link_path_walk+0x8b7/0xea0
> > [4303375.171348] [<ffffffff8028a8d3>] ? check_object+0x223/0x250
> > [4303375.171348] [<ffffffff8029b062>] path_walk+0x62/0xd0
> > [4303375.171348] [<ffffffff8029b2fd>] do_path_lookup+0x8d/0x1d0
> > [4303375.171348] [<ffffffff8029c237>] __path_lookup_intent_open+0x67/0xc0
> > [4303375.171348] [<ffffffff8029c31c>] path_lookup_open+0xc/0x10
> > [4303375.171348] [<ffffffff8029c638>] do_filp_open+0x98/0x8d0
> > [4303375.171348] [<ffffffff80289fff>] ? init_object+0x4f/0x90
> > [4303375.171348] [<ffffffff8028b2e6>] ? __slab_alloc+0xa6/0x660
> > [4303375.171348] [<ffffffff8028e663>] ? get_unused_fd_flags+0x103/0x130
> > [4303375.171348] [<ffffffff8028e6e8>] do_sys_open+0x58/0xb0
> > [4303375.171348] [<ffffffff8028e76b>] sys_open+0x1b/0x20
> > [4303375.171348] [<ffffffff8020b68b>] system_call_after_swapgs+0x7b/0x80
> > [4303375.171348]
> > [4303375.171348]
> > [4303375.171348] Code: 51 33 00 e8 27 16 fc ff 48 8b 03 48 89 c3 8b 45 cc 48 85 db 48 89 45 c0 75 0d eb 4c 0f 1f 00 48 8b 1b 48 85 db 74 41 4c 8d 63 c0 <48> 8b 03 45 3b 6c 24 58 0f 18 08 75 e7 4d 3b 74 24 50 75 e0 49
> > [4303375.171348] RIP [<ffffffff802a38dc>] __d_lookup+0x8c/0x160
> > [4303375.171348] RSP <ffff81005a24bbc8>
> > [4303375.537819] ---[ end trace 823782cee637f8c8 ]---
> > [4315843.874439] Adding 65528k swap on ./swapfile01. Priority:-37 extents:25 across:78684k
> > [4315896.606972] Adding 65528k swap on ./swapfile01. Priority:-38 extents:39 across:110892k
> > [4315945.905293] Adding 65528k swap on ./swapfile01. Priority:-39 extents:27 across:120716k
> > [4315946.328798] Unable to find swap-space signature
> > [4315947.971412] Adding 32k swap on alreadyused. Priority:-40 extents:1 across:32k
> > [4315949.363720] Adding 32k swap on swapfile02. Priority:-41 extents:2 across:32k
> > [4315950.569903] Adding 32k swap on swapfile03. Priority:-42 extents:2 across:64k
> > [4315951.765296] Adding 32k swap on swapfile04. Priority:-43 extents:4 across:108k
> > [4315952.875703] Adding 32k swap on swapfile05. Priority:-44 extents:2 across:1816k
> > [4315954.078312] Adding 32k swap on swapfile06. Priority:-45 extents:1 across:32k
> > [4315955.321904] Adding 32k swap on swapfile07. Priority:-46 extents:2 across:96k
> > [4315956.603960] Adding 32k swap on swapfile08. Priority:-47 extents:1 across:32k
> > [4315957.703750] Adding 32k swap on swapfile09. Priority:-48 extents:2 across:44k
> > [4315958.868482] Adding 32k swap on swapfile10. Priority:-49 extents:1 across:32k
> > [4315960.118592] Adding 32k swap on swapfile11. Priority:-50 extents:1 across:32k
> > [4315961.177483] Adding 32k swap on swapfile12. Priority:-51 extents:1 across:32k
> > [4315962.383246] Adding 32k swap on swapfile13. Priority:-52 extents:1 across:32k
> > [4315963.502439] Adding 32k swap on swapfile14. Priority:-53 extents:1 across:32k
> > [4315964.841896] Adding 32k swap on swapfile15. Priority:-54 extents:2 across:32k
> > [4315966.076074] Adding 32k swap on swapfile16. Priority:-55 extents:1 across:32k
> > [4315967.312097] Adding 32k swap on swapfile17. Priority:-56 extents:1 across:32k
> > [4315968.520294] Adding 32k swap on swapfile18. Priority:-57 extents:2 across:44k
> > [4315970.568662] Adding 32k swap on swapfile19. Priority:-58 extents:2 across:32k
> > [4315971.745911] Adding 32k swap on swapfile20. Priority:-59 extents:2 across:168k
> > [4315972.889984] Adding 32k swap on swapfile21. Priority:-60 extents:2 across:1972k
> > [4315974.133342] Adding 32k swap on swapfile22. Priority:-61 extents:3 across:80k
> > [4315975.349761] Adding 32k swap on swapfile23. Priority:-62 extents:2 across:128k
> > [4315976.507355] Adding 32k swap on swapfile24. Priority:-63 extents:1 across:32k
> > [4315977.705342] Adding 32k swap on swapfile25. Priority:-64 extents:1 across:32k
> > [4315978.734944] Adding 32k swap on swapfile26. Priority:-65 extents:2 across:44k
> > [4315979.838154] Adding 32k swap on swapfile27. Priority:-66 extents:2 across:40k
> > [4315981.418786] Adding 32k swap on swapfile28. Priority:-67 extents:1 across:32k
> > [4315982.727733] Adding 32k swap on swapfile29. Priority:-68 extents:2 across:76k
> > [4315983.849257] Adding 32k swap on swapfile30. Priority:-69 extents:1 across:32k
> > [4315987.520587] Adding 32k swap on firstswapfile. Priority:-70 extents:1 across:32k
> > [4315987.520978] Adding 32k swap on secondswapfile. Priority:-71 extents:1 across:32k
> > [4323489.250419] Adding 65528k swap on ./swapfile01. Priority:-72 extents:33 across:91720k
> > [4323499.912624] Adding 65528k swap on ./swapfile01. Priority:-73 extents:21 across:108164k
> > [4323509.465260] Adding 65528k swap on ./swapfile01. Priority:-74 extents:22 across:135344k
> > [4323509.902961] Unable to find swap-space signature
> > [4323510.351682] Adding 32k swap on alreadyused. Priority:-75 extents:3 across:56k
> > [4323510.543041] Adding 32k swap on swapfile02. Priority:-76 extents:2 across:48k
> > [4323510.707738] Adding 32k swap on swapfile03. Priority:-77 extents:2 across:32k
> > [4323510.972613] Adding 32k swap on swapfile04. Priority:-78 extents:1 across:32k
> > [4323511.245119] Adding 32k swap on swapfile05. Priority:-79 extents:2 across:32k
> > [4323511.461175] Adding 32k swap on swapfile06. Priority:-80 extents:1 across:32k
> > [4323511.674829] Adding 32k swap on swapfile07. Priority:-81 extents:1 across:32k
> > [4323511.936764] Adding 32k swap on swapfile08. Priority:-82 extents:1 across:32k
> > [4323512.117014] Adding 32k swap on swapfile09. Priority:-83 extents:1 across:32k
> > [4323512.312200] Adding 32k swap on swapfile10. Priority:-84 extents:1 across:32k
> > [4323512.508573] Adding 32k swap on swapfile11. Priority:-85 extents:2 across:32k
> > [4323512.710991] Adding 32k swap on swapfile12. Priority:-86 extents:1 across:32k
> > [4323512.971457] Adding 32k swap on swapfile13. Priority:-87 extents:1 across:32k
> > [4323513.157480] Adding 32k swap on swapfile14. Priority:-88 extents:2 across:32k
> > [4323513.415857] Adding 32k swap on swapfile15. Priority:-89 extents:1 across:32k
> > [4323513.667773] Adding 32k swap on swapfile16. Priority:-90 extents:3 across:56k
> > [4323513.862025] Adding 32k swap on swapfile17. Priority:-91 extents:1 across:32k
> > [4323514.117461] Adding 32k swap on swapfile18. Priority:-92 extents:1 across:32k
> > [4323514.359461] Adding 32k swap on swapfile19. Priority:-93 extents:1 across:32k
> > [4323514.591493] Adding 32k swap on swapfile20. Priority:-94 extents:1 across:32k
> > [4323514.828623] Adding 32k swap on swapfile21. Priority:-95 extents:1 across:32k
> > [4323515.062932] Adding 32k swap on swapfile22. Priority:-96 extents:2 across:1816k
> > [4323515.207700] Adding 32k swap on swapfile23. Priority:-97 extents:2 across:32k
> > [4323515.471581] Adding 32k swap on swapfile24. Priority:-98 extents:1 across:32k
> > [4323515.703335] Adding 32k swap on swapfile25. Priority:-99 extents:1 across:32k
> > [4323515.969844] Adding 32k swap on swapfile26. Priority:-100 extents:1 across:32k
> > [4323516.237519] Adding 32k swap on swapfile27. Priority:-101 extents:1 across:32k
> > [4323516.448421] Adding 32k swap on swapfile28. Priority:-102 extents:1 across:32k
> > [4323516.662276] Adding 32k swap on swapfile29. Priority:-103 extents:1 across:32k
> > [4323517.011638] Adding 32k swap on swapfile30. Priority:-104 extents:1 across:32k
> > [4323517.691915] Adding 32k swap on firstswapfile. Priority:-105 extents:2 across:36k
> > [4323517.692095] Adding 32k swap on secondswapfile. Priority:-106 extents:1 across:32k
> > [4327950.912409] Adding 65528k swap on ./swapfile01. Priority:-107 extents:25 across:84528k
> > [4327957.079710] Adding 65528k swap on ./swapfile01. Priority:-108 extents:31 across:157044k
> > [4327963.890758] Adding 65528k swap on ./swapfile01. Priority:-109 extents:34 across:108204k
> > [4327964.096327] Unable to find swap-space signature
> > [4327964.398296] Adding 32k swap on alreadyused. Priority:-110 extents:2 across:36k
> > [4327964.736523] Adding 32k swap on swapfile02. Priority:-111 extents:1 across:32k
> > [4327964.886182] Adding 32k swap on swapfile03. Priority:-112 extents:2 across:136k
> > [4327965.024481] Adding 32k swap on swapfile04. Priority:-113 extents:2 across:1816k
> > [4327965.176555] Adding 32k swap on swapfile05. Priority:-114 extents:1 across:32k
> > [4327965.288691] Adding 32k swap on swapfile06. Priority:-115 extents:1 across:32k
> > [4327965.422738] Adding 32k swap on swapfile07. Priority:-116 extents:2 across:44k
> > [4327965.558104] Adding 32k swap on swapfile08. Priority:-117 extents:1 across:32k
> > [4327965.722730] Adding 32k swap on swapfile09. Priority:-118 extents:1 across:32k
> > [4327965.908405] Adding 32k swap on swapfile10. Priority:-119 extents:2 across:60k
> > [4327966.064049] Adding 32k swap on swapfile11. Priority:-120 extents:1 across:32k
> > [4327966.182639] Adding 32k swap on swapfile12. Priority:-121 extents:1 across:32k
> > [4327966.308008] Adding 32k swap on swapfile13. Priority:-122 extents:1 across:32k
> > [4327966.441889] Adding 32k swap on swapfile14. Priority:-123 extents:2 across:36k
> > [4327966.545185] Adding 32k swap on swapfile15. Priority:-124 extents:1 across:32k
> > [4327966.731191] Adding 32k swap on swapfile16. Priority:-125 extents:2 across:40k
> > [4327966.908109] Adding 32k swap on swapfile17. Priority:-126 extents:1 across:32k
> > [4327967.067012] Adding 32k swap on swapfile18. Priority:-127 extents:1 across:32k
> > [4327967.270205] Adding 32k swap on swapfile19. Priority:-128 extents:1 across:32k
> > [4327967.448180] Adding 32k swap on swapfile20. Priority:-129 extents:1 across:32k
> > [4327967.640517] Adding 32k swap on swapfile21. Priority:-130 extents:1 across:32k
> > [4327967.780410] Adding 32k swap on swapfile22. Priority:-131 extents:1 across:32k
> > [4327967.923915] Adding 32k swap on swapfile23. Priority:-132 extents:1 across:32k
> > [4327968.076929] Adding 32k swap on swapfile24. Priority:-133 extents:1 across:32k
> > [4327968.224444] Adding 32k swap on swapfile25. Priority:-134 extents:1 across:32k
> > [4327968.386681] Adding 32k swap on swapfile26. Priority:-135 extents:1 across:32k
> > [4327968.635164] Adding 32k swap on swapfile27. Priority:-136 extents:1 across:32k
> > [4327968.820812] Adding 32k swap on swapfile28. Priority:-137 extents:1 across:32k
> > [4327969.067842] Adding 32k swap on swapfile29. Priority:-138 extents:1 across:32k
> > [4327969.207317] Adding 32k swap on swapfile30. Priority:-139 extents:1 across:32k
> > [4327969.683050] Adding 32k swap on firstswapfile. Priority:-140 extents:1 across:32k
> > [4327969.683433] Adding 32k swap on secondswapfile. Priority:-141 extents:1 across:32k
> > [4331997.664608] Adding 65528k swap on ./swapfile01. Priority:-142 extents:32 across:89684k
> > [4332003.732463] Adding 65528k swap on ./swapfile01. Priority:-143 extents:29 across:149088k
> > [4332010.354281] Adding 65528k swap on ./swapfile01. Priority:-144 extents:21 across:140068k
> > [4332010.563304] Unable to find swap-space signature
> > [4332010.812699] Adding 32k swap on alreadyused. Priority:-145 extents:1 across:32k
> > [4332011.114655] Adding 32k swap on swapfile02. Priority:-146 extents:1 across:32k
> > [4332011.230734] Adding 32k swap on swapfile03. Priority:-147 extents:2 across:1836k
> > [4332011.405992] Adding 32k swap on swapfile04. Priority:-148 extents:1 across:32k
> > [4332011.629046] Adding 32k swap on swapfile05. Priority:-149 extents:1 across:32k
> > [4332011.824376] Adding 32k swap on swapfile06. Priority:-150 extents:1 across:32k
> > [4332011.956780] Adding 32k swap on swapfile07. Priority:-151 extents:2 across:44k
> > [4332012.084015] Adding 32k swap on swapfile08. Priority:-152 extents:1 across:32k
> > [4332012.382590] Adding 32k swap on swapfile09. Priority:-153 extents:1 across:32k
> > [4332012.600051] Adding 32k swap on swapfile10. Priority:-154 extents:2 across:32k
> > [4332012.739848] Adding 32k swap on swapfile11. Priority:-155 extents:1 across:32k
> > [4332012.951230] Adding 32k swap on swapfile12. Priority:-156 extents:2 across:36k
> > [4332013.027178] Adding 32k swap on swapfile13. Priority:-157 extents:2 across:32k
> > [4332013.157294] Adding 32k swap on swapfile14. Priority:-158 extents:1 across:32k
> > [4332013.324306] Adding 32k swap on swapfile15. Priority:-159 extents:1 across:32k
> > [4332013.478009] Adding 32k swap on swapfile16. Priority:-160 extents:1 across:32k
> > [4332013.691345] Adding 32k swap on swapfile17. Priority:-161 extents:2 across:32k
> > [4332013.758417] Adding 32k swap on swapfile18. Priority:-162 extents:1 across:32k
> > [4332013.856159] Adding 32k swap on swapfile19. Priority:-163 extents:1 across:32k
> > [4332013.988999] Adding 32k swap on swapfile20. Priority:-164 extents:1 across:32k
> > [4332014.108171] Adding 32k swap on swapfile21. Priority:-165 extents:1 across:32k
> > [4332014.191106] Adding 32k swap on swapfile22. Priority:-166 extents:1 across:32k
> > [4332014.304206] Adding 32k swap on swapfile23. Priority:-167 extents:1 across:32k
> > [4332014.502539] Adding 32k swap on swapfile24. Priority:-168 extents:1 across:32k
> > [4332014.684369] Adding 32k swap on swapfile25. Priority:-169 extents:1 across:32k
> > [4332014.853736] Adding 32k swap on swapfile26. Priority:-170 extents:2 across:36k
> > [4332014.973318] Adding 32k swap on swapfile27. Priority:-171 extents:1 across:32k
> > [4332015.091161] Adding 32k swap on swapfile28. Priority:-172 extents:1 across:32k
> > [4332015.201008] Adding 32k swap on swapfile29. Priority:-173 extents:2 across:36k
> > [4332015.324404] Adding 32k swap on swapfile30. Priority:-174 extents:1 across:32k
> > [4332015.752082] Adding 32k swap on firstswapfile. Priority:-175 extents:1 across:32k
> > [4332015.752424] Adding 32k swap on secondswapfile. Priority:-176 extents:1 across:32k
> > [4335684.679409] Adding 65528k swap on ./swapfile01. Priority:-177 extents:31 across:91752k
> > [4335691.057909] Adding 65528k swap on ./swapfile01. Priority:-178 extents:22 across:114500k
> > [4335696.822115] Adding 65528k swap on ./swapfile01. Priority:-179 extents:24 across:148172k
> > [4335696.993393] Unable to find swap-space signature
> > [4335697.283173] Adding 32k swap on alreadyused. Priority:-180 extents:1 across:32k
> > [4335697.422398] Adding 32k swap on swapfile02. Priority:-181 extents:1 across:32k
> > [4335697.527172] Adding 32k swap on swapfile03. Priority:-182 extents:1 across:32k
> > [4335697.658520] Adding 32k swap on swapfile04. Priority:-183 extents:1 across:32k
> > [4335697.745190] Adding 32k swap on swapfile05. Priority:-184 extents:2 across:32k
> > [4335697.846964] Adding 32k swap on swapfile06. Priority:-185 extents:1 across:32k
> > [4335697.957268] Adding 32k swap on swapfile07. Priority:-186 extents:1 across:32k
> > [4335698.126923] Adding 32k swap on swapfile08. Priority:-187 extents:1 across:32k
> > [4335698.337237] Adding 32k swap on swapfile09. Priority:-188 extents:1 across:32k
> > [4335698.450537] Adding 32k swap on swapfile10. Priority:-189 extents:1 across:32k
> > [4335698.611428] Adding 32k swap on swapfile11. Priority:-190 extents:1 across:32k
> > [4335698.767599] Adding 32k swap on swapfile12. Priority:-191 extents:1 across:32k
> > [4335698.898005] Adding 32k swap on swapfile13. Priority:-192 extents:1 across:32k
> > [4335699.030925] Adding 32k swap on swapfile14. Priority:-193 extents:2 across:44k
> > [4335699.150021] Adding 32k swap on swapfile15. Priority:-194 extents:1 across:32k
> > [4335699.322725] Adding 32k swap on swapfile16. Priority:-195 extents:1 across:32k
> > [4335699.500086] Adding 32k swap on swapfile17. Priority:-196 extents:1 across:32k
> > [4335699.574431] Adding 32k swap on swapfile18. Priority:-197 extents:1 across:32k
> > [4335699.750988] Adding 32k swap on swapfile19. Priority:-198 extents:1 across:32k
> > [4335699.915901] Adding 32k swap on swapfile20. Priority:-199 extents:1 across:32k
> > [4335700.088262] Adding 32k swap on swapfile21. Priority:-200 extents:1 across:32k
> > [4335700.230728] Adding 32k swap on swapfile22. Priority:-201 extents:1 across:32k
> > [4335700.317745] Adding 32k swap on swapfile23. Priority:-202 extents:1 across:32k
> > [4335700.495246] Adding 32k swap on swapfile24. Priority:-203 extents:1 across:32k
> > [4335700.641937] Adding 32k swap on swapfile25. Priority:-204 extents:1 across:32k
> > [4335700.754225] Adding 32k swap on swapfile26. Priority:-205 extents:1 across:32k
> > [4335700.868226] Adding 32k swap on swapfile27. Priority:-206 extents:1 across:32k
> > [4335701.081225] Adding 32k swap on swapfile28. Priority:-207 extents:1 across:32k
> > [4335701.306849] Adding 32k swap on swapfile29. Priority:-208 extents:1 across:32k
> > [4335701.409447] Adding 32k swap on swapfile30. Priority:-209 extents:1 across:32k
> > [4335701.792262] Adding 32k swap on firstswapfile. Priority:-210 extents:1 across:32k
> > [4335701.792621] Adding 32k swap on secondswapfile. Priority:-211 extents:1 across:32k
> > [4337797.781556] eth1: link up, 100Mbps, full-duplex, lpa 0x45E1
> >
[-- Attachment #2: kbtorture.sh --]
[-- Type: application/x-sh, Size: 517 bytes --]
[-- Attachment #3: ltptorture.sh --]
[-- Type: application/x-sh, Size: 111 bytes --]
^ permalink raw reply [flat|nested] 37+ messages in thread
end of thread, other threads:[~2008-06-06 14:50 UTC | newest]
Thread overview: 37+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-04-27 22:55 2.6.25-$sha1: RIP call_for_each_cic+0x25/0x50 Alexey Dobriyan
2008-04-28 12:01 ` Andrew Morton
2008-04-28 12:04 ` Jens Axboe
2008-04-28 19:55 ` Alexey Dobriyan
2008-04-29 6:21 ` Alexey Dobriyan
2008-04-29 9:06 ` Jens Axboe
2008-04-30 22:12 ` Alexey Dobriyan
2008-05-04 19:08 ` Jens Axboe
2008-05-04 20:15 ` Alexey Dobriyan
2008-05-04 19:25 ` Jens Axboe
2008-05-04 21:17 ` Alexey Dobriyan
2008-05-10 10:37 ` 2.6.25-$sha1: RIP __call_for_each_cic+0x20/0x50 Alexey Dobriyan
2008-05-27 5:27 ` 2.6.26-rc4: " Alexey Dobriyan
2008-05-27 13:35 ` Jens Axboe
2008-05-27 15:18 ` Paul E. McKenney
2008-05-28 10:07 ` Jens Axboe
2008-05-28 10:30 ` Paul E. McKenney
2008-05-28 12:44 ` Jens Axboe
2008-05-28 13:20 ` Paul E. McKenney
2008-05-29 4:38 ` Paul E. McKenney
2008-05-29 6:26 ` Jens Axboe
2008-05-29 6:42 ` Jens Axboe
2008-05-29 9:17 ` Paul E. McKenney
2008-05-29 10:13 ` Jens Axboe
2008-05-29 11:25 ` Paul E. McKenney
2008-05-29 11:44 ` Jens Axboe
2008-05-29 12:11 ` Paul E. McKenney
2008-05-29 12:13 ` Jens Axboe
2008-05-30 11:04 ` Paul E. McKenney
2008-05-30 13:16 ` Paul E. McKenney
2008-05-30 18:34 ` Alexey Dobriyan
2008-06-04 3:31 ` Paul E. McKenney
2008-06-04 18:32 ` Linus Torvalds
2008-06-05 4:23 ` Paul E. McKenney
2008-06-06 14:49 ` Paul E. McKenney
2008-05-28 11:52 ` Fabio Checconi
2008-05-28 11:58 ` Jens Axboe
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox