Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[David Miller]

From: "Marco Berizzi" <pupilla@hotmail.com>
Date: Mon, 28 Apr 2008 11:25:23 +0200

Netdev added to CC:, please include in all bug reports as
that's where the networking developers sit.

Thank you.

> I know, it is pretty useless this report,
> but I have nothing more than these bitmaps
> and the message in the subject :-(
> 
> http://80.204.235.230/1.jpg
> http://80.204.235.230/2.jpg
> http://80.204.235.230/2.jpg
> 
> 2.6.24.4 never crashed.
> This box is running only openswan.
> It is impossible for me to reproduce
> this bug :-(((
> It is happened after about 24 hours from
> 2.6.24.4 ==>> 2.6.25 upgrade.
> This linux box is an ipsec gateway and
> firewall.
> Here is .config and dmesg.
> 
> #
> # Automatically generated make config: don't edit
> # Linux kernel version: 2.6.25
> # Tue Apr 22 10:21:32 2008
> #
> # CONFIG_64BIT is not set
> CONFIG_X86_32=y
> # CONFIG_X86_64 is not set
> CONFIG_X86=y
> # CONFIG_GENERIC_LOCKBREAK is not set
> CONFIG_GENERIC_TIME=y
> CONFIG_GENERIC_CMOS_UPDATE=y
> CONFIG_CLOCKSOURCE_WATCHDOG=y
> CONFIG_GENERIC_CLOCKEVENTS=y
> CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
> CONFIG_LOCKDEP_SUPPORT=y
> CONFIG_STACKTRACE_SUPPORT=y
> CONFIG_HAVE_LATENCYTOP_SUPPORT=y
> CONFIG_SEMAPHORE_SLEEPERS=y
> CONFIG_FAST_CMPXCHG_LOCAL=y
> CONFIG_MMU=y
> CONFIG_ZONE_DMA=y
> CONFIG_GENERIC_ISA_DMA=y
> CONFIG_GENERIC_IOMAP=y
> CONFIG_GENERIC_BUG=y
> CONFIG_GENERIC_HWEIGHT=y
> # CONFIG_GENERIC_GPIO is not set
> CONFIG_ARCH_MAY_HAVE_PC_FDC=y
> CONFIG_DMI=y
> # CONFIG_RWSEM_GENERIC_SPINLOCK is not set
> CONFIG_RWSEM_XCHGADD_ALGORITHM=y
> # CONFIG_ARCH_HAS_ILOG2_U32 is not set
> # CONFIG_ARCH_HAS_ILOG2_U64 is not set
> CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
> CONFIG_GENERIC_CALIBRATE_DELAY=y
> # CONFIG_GENERIC_TIME_VSYSCALL is not set
> CONFIG_ARCH_HAS_CPU_RELAX=y
> # CONFIG_HAVE_SETUP_PER_CPU_AREA is not set
> CONFIG_ARCH_HIBERNATION_POSSIBLE=y
> CONFIG_ARCH_SUSPEND_POSSIBLE=y
> # CONFIG_ZONE_DMA32 is not set
> CONFIG_ARCH_POPULATES_NODE_MAP=y
> # CONFIG_AUDIT_ARCH is not set
> CONFIG_ARCH_SUPPORTS_AOUT=y
> CONFIG_GENERIC_HARDIRQS=y
> CONFIG_GENERIC_IRQ_PROBE=y
> CONFIG_X86_BIOS_REBOOT=y
> CONFIG_KTIME_SCALAR=y
> CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
> 
> #
> # General setup
> #
> # CONFIG_EXPERIMENTAL is not set
> CONFIG_BROKEN_ON_SMP=y
> CONFIG_INIT_ENV_ARG_LIMIT=32
> CONFIG_LOCALVERSION=""
> # CONFIG_LOCALVERSION_AUTO is not set
> CONFIG_SWAP=y
> CONFIG_SYSVIPC=y
> CONFIG_SYSVIPC_SYSCTL=y
> CONFIG_BSD_PROCESS_ACCT=y
> # CONFIG_BSD_PROCESS_ACCT_V3 is not set
> # CONFIG_TASKSTATS is not set
> # CONFIG_AUDIT is not set
> # CONFIG_IKCONFIG is not set
> CONFIG_LOG_BUF_SHIFT=14
> # CONFIG_CGROUPS is not set
> # CONFIG_GROUP_SCHED is not set
> # CONFIG_SYSFS_DEPRECATED_V2 is not set
> # CONFIG_RELAY is not set
> CONFIG_NAMESPACES=y
> # CONFIG_UTS_NS is not set
> # CONFIG_IPC_NS is not set
> # CONFIG_BLK_DEV_INITRD is not set
> CONFIG_SYSCTL=y
> # CONFIG_EMBEDDED is not set
> CONFIG_UID16=y
> CONFIG_SYSCTL_SYSCALL=y
> CONFIG_KALLSYMS=y
> # CONFIG_KALLSYMS_EXTRA_PASS is not set
> CONFIG_HOTPLUG=y
> CONFIG_PRINTK=y
> CONFIG_BUG=y
> CONFIG_ELF_CORE=y
> CONFIG_COMPAT_BRK=y
> CONFIG_BASE_FULL=y
> CONFIG_FUTEX=y
> CONFIG_ANON_INODES=y
> CONFIG_EPOLL=y
> CONFIG_SIGNALFD=y
> CONFIG_TIMERFD=y
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_VM_EVENT_COUNTERS=y
> CONFIG_SLUB_DEBUG=y
> # CONFIG_SLAB is not set
> CONFIG_SLUB=y
> # CONFIG_SLOB is not set
> # CONFIG_PROFILING is not set
> # CONFIG_MARKERS is not set
> CONFIG_HAVE_OPROFILE=y
> # CONFIG_KPROBES is not set
> CONFIG_HAVE_KPROBES=y
> CONFIG_HAVE_KRETPROBES=y
> CONFIG_PROC_PAGE_MONITOR=y
> CONFIG_SLABINFO=y
> CONFIG_RT_MUTEXES=y
> # CONFIG_TINY_SHMEM is not set
> CONFIG_BASE_SMALL=0
> CONFIG_MODULES=y
> CONFIG_MODULE_UNLOAD=y
> # CONFIG_MODVERSIONS is not set
> # CONFIG_MODULE_SRCVERSION_ALL is not set
> # CONFIG_KMOD is not set
> CONFIG_BLOCK=y
> # CONFIG_LBD is not set
> # CONFIG_BLK_DEV_IO_TRACE is not set
> # CONFIG_LSF is not set
> 
> #
> # IO Schedulers
> #
> CONFIG_IOSCHED_NOOP=y
> # CONFIG_IOSCHED_AS is not set
> CONFIG_IOSCHED_DEADLINE=y
> # CONFIG_IOSCHED_CFQ is not set
> # CONFIG_DEFAULT_AS is not set
> CONFIG_DEFAULT_DEADLINE=y
> # CONFIG_DEFAULT_CFQ is not set
> # CONFIG_DEFAULT_NOOP is not set
> CONFIG_DEFAULT_IOSCHED="deadline"
> CONFIG_CLASSIC_RCU=y
> 
> #
> # Processor type and features
> #
> # CONFIG_TICK_ONESHOT is not set
> # CONFIG_NO_HZ is not set
> # CONFIG_HIGH_RES_TIMERS is not set
> CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
> # CONFIG_SMP is not set
> CONFIG_X86_PC=y
> # CONFIG_X86_ELAN is not set
> # CONFIG_X86_VOYAGER is not set
> # CONFIG_X86_NUMAQ is not set
> # CONFIG_X86_SUMMIT is not set
> # CONFIG_X86_BIGSMP is not set
> # CONFIG_X86_VISWS is not set
> # CONFIG_X86_GENERICARCH is not set
> # CONFIG_X86_ES7000 is not set
> # CONFIG_X86_RDC321X is not set
> # CONFIG_X86_VSMP is not set
> # CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER is not set
> # CONFIG_PARAVIRT_GUEST is not set
> # CONFIG_M386 is not set
> # CONFIG_M486 is not set
> # CONFIG_M586 is not set
> # CONFIG_M586TSC is not set
> # CONFIG_M586MMX is not set
> # CONFIG_M686 is not set
> # CONFIG_MPENTIUMII is not set
> CONFIG_MPENTIUMIII=y
> # CONFIG_MPENTIUMM is not set
> # CONFIG_MPENTIUM4 is not set
> # CONFIG_MK6 is not set
> # CONFIG_MK7 is not set
> # CONFIG_MK8 is not set
> # CONFIG_MCRUSOE is not set
> # CONFIG_MEFFICEON is not set
> # CONFIG_MWINCHIPC6 is not set
> # CONFIG_MWINCHIP2 is not set
> # CONFIG_MWINCHIP3D is not set
> # CONFIG_MGEODEGX1 is not set
> # CONFIG_MGEODE_LX is not set
> # CONFIG_MCYRIXIII is not set
> # CONFIG_MVIAC3_2 is not set
> # CONFIG_MVIAC7 is not set
> # CONFIG_MPSC is not set
> # CONFIG_MCORE2 is not set
> # CONFIG_GENERIC_CPU is not set
> # CONFIG_X86_GENERIC is not set
> CONFIG_X86_CMPXCHG=y
> CONFIG_X86_L1_CACHE_SHIFT=5
> CONFIG_X86_XADD=y
> CONFIG_X86_WP_WORKS_OK=y
> CONFIG_X86_INVLPG=y
> CONFIG_X86_BSWAP=y
> CONFIG_X86_POPAD_OK=y
> CONFIG_X86_GOOD_APIC=y
> CONFIG_X86_INTEL_USERCOPY=y
> CONFIG_X86_USE_PPRO_CHECKSUM=y
> CONFIG_X86_P6_NOP=y
> CONFIG_X86_TSC=y
> CONFIG_X86_CMOV=y
> CONFIG_X86_MINIMUM_CPU_FAMILY=6
> CONFIG_X86_DEBUGCTLMSR=y
> # CONFIG_HPET_TIMER is not set
> # CONFIG_IOMMU_HELPER is not set
> CONFIG_PREEMPT_NONE=y
> # CONFIG_PREEMPT_VOLUNTARY is not set
> # CONFIG_PREEMPT is not set
> CONFIG_X86_UP_APIC=y
> CONFIG_X86_UP_IOAPIC=y
> CONFIG_X86_LOCAL_APIC=y
> CONFIG_X86_IO_APIC=y
> # CONFIG_X86_MCE is not set
> CONFIG_VM86=y
> # CONFIG_TOSHIBA is not set
> # CONFIG_I8K is not set
> # CONFIG_X86_REBOOTFIXUPS is not set
> # CONFIG_MICROCODE is not set
> # CONFIG_X86_MSR is not set
> # CONFIG_X86_CPUID is not set
> CONFIG_NOHIGHMEM=y
> # CONFIG_HIGHMEM4G is not set
> # CONFIG_HIGHMEM64G is not set
> CONFIG_PAGE_OFFSET=0xC0000000
> # CONFIG_X86_PAE is not set
> CONFIG_FLATMEM=y
> CONFIG_FLAT_NODE_MEM_MAP=y
> # CONFIG_SPARSEMEM_STATIC is not set
> # CONFIG_SPARSEMEM_VMEMMAP_ENABLE is not set
> CONFIG_SPLIT_PTLOCK_CPUS=4
> # CONFIG_RESOURCES_64BIT is not set
> CONFIG_ZONE_DMA_FLAG=1
> CONFIG_BOUNCE=y
> CONFIG_VIRT_TO_BUS=y
> # CONFIG_MATH_EMULATION is not set
> # CONFIG_MTRR is not set
> # CONFIG_EFI is not set
> CONFIG_SECCOMP=y
> # CONFIG_HZ_100 is not set
> CONFIG_HZ_250=y
> # CONFIG_HZ_300 is not set
> # CONFIG_HZ_1000 is not set
> CONFIG_HZ=250
> # CONFIG_SCHED_HRTICK is not set
> # CONFIG_KEXEC is not set
> CONFIG_PHYSICAL_START=0x100000
> CONFIG_PHYSICAL_ALIGN=0x100000
> # CONFIG_COMPAT_VDSO is not set
> 
> #
> # Power management options
> #
> CONFIG_PM=y
> # CONFIG_PM_LEGACY is not set
> # CONFIG_PM_DEBUG is not set
> # CONFIG_SUSPEND is not set
> # CONFIG_HIBERNATION is not set
> CONFIG_ACPI=y
> # CONFIG_ACPI_PROCFS is not set
> # CONFIG_ACPI_PROCFS_POWER is not set
> # CONFIG_ACPI_SYSFS_POWER is not set
> # CONFIG_ACPI_PROC_EVENT is not set
> # CONFIG_ACPI_AC is not set
> # CONFIG_ACPI_BATTERY is not set
> # CONFIG_ACPI_BUTTON is not set
> # CONFIG_ACPI_FAN is not set
> # CONFIG_ACPI_PROCESSOR is not set
> # CONFIG_ACPI_ASUS is not set
> # CONFIG_ACPI_TOSHIBA is not set
> # CONFIG_ACPI_CUSTOM_DSDT is not set
> CONFIG_ACPI_BLACKLIST_YEAR=0
> # CONFIG_ACPI_DEBUG is not set
> CONFIG_ACPI_EC=y
> CONFIG_ACPI_POWER=y
> CONFIG_ACPI_SYSTEM=y
> CONFIG_X86_PM_TIMER=y
> # CONFIG_ACPI_SBS is not set
> 
> #
> # CPU Frequency scaling
> #
> # CONFIG_CPU_FREQ is not set
> # CONFIG_CPU_IDLE is not set
> 
> #
> # Bus options (PCI etc.)
> #
> CONFIG_PCI=y
> # CONFIG_PCI_GOBIOS is not set
> # CONFIG_PCI_GOMMCONFIG is not set
> # CONFIG_PCI_GODIRECT is not set
> CONFIG_PCI_GOANY=y
> CONFIG_PCI_BIOS=y
> CONFIG_PCI_DIRECT=y
> CONFIG_PCI_MMCONFIG=y
> CONFIG_PCI_DOMAINS=y
> # CONFIG_PCIEPORTBUS is not set
> CONFIG_ARCH_SUPPORTS_MSI=y
> # CONFIG_PCI_MSI is not set
> # CONFIG_PCI_LEGACY is not set
> # CONFIG_HT_IRQ is not set
> CONFIG_ISA_DMA_API=y
> # CONFIG_ISA is not set
> # CONFIG_MCA is not set
> # CONFIG_SCx200 is not set
> # CONFIG_PCCARD is not set
> # CONFIG_HOTPLUG_PCI is not set
> 
> #
> # Executable file formats / Emulations
> #
> CONFIG_BINFMT_ELF=y
> # CONFIG_BINFMT_AOUT is not set
> # CONFIG_BINFMT_MISC is not set
> 
> #
> # Networking
> #
> CONFIG_NET=y
> 
> #
> # Networking options
> #
> CONFIG_PACKET=y
> CONFIG_PACKET_MMAP=y
> CONFIG_UNIX=y
> CONFIG_XFRM=y
> CONFIG_XFRM_USER=y
> CONFIG_NET_KEY=y
> CONFIG_INET=y
> # CONFIG_IP_MULTICAST is not set
> CONFIG_IP_ADVANCED_ROUTER=y
> CONFIG_ASK_IP_FIB_HASH=y
> # CONFIG_IP_FIB_TRIE is not set
> CONFIG_IP_FIB_HASH=y
> CONFIG_IP_MULTIPLE_TABLES=y
> CONFIG_IP_ROUTE_MULTIPATH=y
> CONFIG_IP_ROUTE_VERBOSE=y
> # CONFIG_IP_PNP is not set
> # CONFIG_NET_IPIP is not set
> # CONFIG_NET_IPGRE is not set
> CONFIG_SYN_COOKIES=y
> # CONFIG_INET_AH is not set
> CONFIG_INET_ESP=y
> CONFIG_INET_IPCOMP=y
> CONFIG_INET_XFRM_TUNNEL=y
> CONFIG_INET_TUNNEL=y
> CONFIG_INET_XFRM_MODE_TRANSPORT=y
> CONFIG_INET_XFRM_MODE_TUNNEL=y
> # CONFIG_INET_XFRM_MODE_BEET is not set
> # CONFIG_INET_LRO is not set
> CONFIG_INET_DIAG=y
> CONFIG_INET_TCP_DIAG=y
> # CONFIG_TCP_CONG_ADVANCED is not set
> CONFIG_TCP_CONG_CUBIC=y
> CONFIG_DEFAULT_TCP_CONG="cubic"
> # CONFIG_IP_VS is not set
> # CONFIG_IPV6 is not set
> # CONFIG_INET6_XFRM_TUNNEL is not set
> # CONFIG_INET6_TUNNEL is not set
> # CONFIG_NETWORK_SECMARK is not set
> CONFIG_NETFILTER=y
> # CONFIG_NETFILTER_DEBUG is not set
> CONFIG_NETFILTER_ADVANCED=y
> 
> #
> # Core Netfilter Configuration
> #
> CONFIG_NETFILTER_NETLINK=m
> CONFIG_NETFILTER_NETLINK_QUEUE=m
> CONFIG_NETFILTER_NETLINK_LOG=m
> CONFIG_NF_CONNTRACK=y
> CONFIG_NF_CT_ACCT=y
> CONFIG_NF_CONNTRACK_MARK=y
> # CONFIG_NF_CONNTRACK_EVENTS is not set
> CONFIG_NF_CT_PROTO_GRE=m
> CONFIG_NF_CT_PROTO_UDPLITE=m
> # CONFIG_NF_CONNTRACK_AMANDA is not set
> CONFIG_NF_CONNTRACK_FTP=m
> CONFIG_NF_CONNTRACK_H323=m
> # CONFIG_NF_CONNTRACK_IRC is not set
> CONFIG_NF_CONNTRACK_NETBIOS_NS=m
> CONFIG_NF_CONNTRACK_PPTP=m
> CONFIG_NF_CONNTRACK_SIP=m
> # CONFIG_NF_CONNTRACK_TFTP is not set
> # CONFIG_NF_CT_NETLINK is not set
> CONFIG_NETFILTER_XTABLES=y
> CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
> CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
> CONFIG_NETFILTER_XT_TARGET_DSCP=m
> CONFIG_NETFILTER_XT_TARGET_MARK=y
> CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
> CONFIG_NETFILTER_XT_TARGET_NFLOG=m
> CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
> CONFIG_NETFILTER_XT_TARGET_RATEEST=m
> CONFIG_NETFILTER_XT_TARGET_TRACE=m
> CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
> CONFIG_NETFILTER_XT_MATCH_COMMENT=m
> CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
> CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m
> CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
> CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
> CONFIG_NETFILTER_XT_MATCH_DCCP=m
> CONFIG_NETFILTER_XT_MATCH_DSCP=m
> CONFIG_NETFILTER_XT_MATCH_ESP=m
> CONFIG_NETFILTER_XT_MATCH_HELPER=y
> CONFIG_NETFILTER_XT_MATCH_IPRANGE=m
> CONFIG_NETFILTER_XT_MATCH_LENGTH=m
> CONFIG_NETFILTER_XT_MATCH_LIMIT=y
> CONFIG_NETFILTER_XT_MATCH_MAC=m
> CONFIG_NETFILTER_XT_MATCH_MARK=y
> CONFIG_NETFILTER_XT_MATCH_OWNER=m
> CONFIG_NETFILTER_XT_MATCH_POLICY=y
> CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
> CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
> CONFIG_NETFILTER_XT_MATCH_QUOTA=m
> CONFIG_NETFILTER_XT_MATCH_RATEEST=m
> CONFIG_NETFILTER_XT_MATCH_REALM=m
> CONFIG_NETFILTER_XT_MATCH_STATE=y
> CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
> CONFIG_NETFILTER_XT_MATCH_STRING=m
> CONFIG_NETFILTER_XT_MATCH_TCPMSS=y
> CONFIG_NETFILTER_XT_MATCH_TIME=m
> CONFIG_NETFILTER_XT_MATCH_U32=m
> CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m
> 
> #
> # IP: Netfilter Configuration
> #
> CONFIG_NF_CONNTRACK_IPV4=y
> # CONFIG_NF_CONNTRACK_PROC_COMPAT is not set
> # CONFIG_IP_NF_QUEUE is not set
> CONFIG_IP_NF_IPTABLES=y
> # CONFIG_IP_NF_MATCH_RECENT is not set
> CONFIG_IP_NF_MATCH_ECN=m
> # CONFIG_IP_NF_MATCH_AH is not set
> CONFIG_IP_NF_MATCH_TTL=m
> CONFIG_IP_NF_MATCH_ADDRTYPE=m
> CONFIG_IP_NF_FILTER=y
> CONFIG_IP_NF_TARGET_REJECT=y
> CONFIG_IP_NF_TARGET_LOG=m
> CONFIG_IP_NF_TARGET_ULOG=m
> CONFIG_NF_NAT=y
> CONFIG_NF_NAT_NEEDED=y
> CONFIG_IP_NF_TARGET_MASQUERADE=m
> CONFIG_IP_NF_TARGET_REDIRECT=m
> CONFIG_IP_NF_TARGET_NETMAP=m
> # CONFIG_NF_NAT_SNMP_BASIC is not set
> CONFIG_NF_NAT_PROTO_GRE=m
> CONFIG_NF_NAT_FTP=m
> # CONFIG_NF_NAT_IRC is not set
> # CONFIG_NF_NAT_TFTP is not set
> # CONFIG_NF_NAT_AMANDA is not set
> CONFIG_NF_NAT_PPTP=m
> CONFIG_NF_NAT_H323=m
> CONFIG_NF_NAT_SIP=m
> CONFIG_IP_NF_MANGLE=y
> CONFIG_IP_NF_TARGET_ECN=m
> CONFIG_IP_NF_TARGET_TTL=m
> CONFIG_IP_NF_RAW=m
> CONFIG_IP_NF_ARPTABLES=m
> CONFIG_IP_NF_ARPFILTER=m
> CONFIG_IP_NF_ARP_MANGLE=m
> # CONFIG_ATM is not set
> # CONFIG_BRIDGE is not set
> # CONFIG_VLAN_8021Q is not set
> # CONFIG_DECNET is not set
> # CONFIG_LLC2 is not set
> # CONFIG_IPX is not set
> # CONFIG_ATALK is not set
> CONFIG_NET_SCHED=y
> 
> #
> # Queueing/Scheduling
> #
> # CONFIG_NET_SCH_CBQ is not set
> CONFIG_NET_SCH_HTB=m
> CONFIG_NET_SCH_HFSC=m
> CONFIG_NET_SCH_PRIO=m
> CONFIG_NET_SCH_RR=m
> CONFIG_NET_SCH_RED=m
> CONFIG_NET_SCH_SFQ=m
> CONFIG_NET_SCH_TEQL=m
> CONFIG_NET_SCH_TBF=m
> CONFIG_NET_SCH_GRED=m
> CONFIG_NET_SCH_DSMARK=m
> CONFIG_NET_SCH_NETEM=m
> CONFIG_NET_SCH_INGRESS=m
> 
> #
> # Classification
> #
> CONFIG_NET_CLS=y
> CONFIG_NET_CLS_BASIC=m
> CONFIG_NET_CLS_TCINDEX=m
> CONFIG_NET_CLS_ROUTE4=m
> CONFIG_NET_CLS_ROUTE=y
> CONFIG_NET_CLS_FW=m
> CONFIG_NET_CLS_U32=m
> CONFIG_CLS_U32_PERF=y
> CONFIG_CLS_U32_MARK=y
> CONFIG_NET_CLS_RSVP=m
> # CONFIG_NET_CLS_RSVP6 is not set
> CONFIG_NET_CLS_FLOW=m
> CONFIG_NET_EMATCH=y
> CONFIG_NET_EMATCH_STACK=32
> CONFIG_NET_EMATCH_CMP=m
> CONFIG_NET_EMATCH_NBYTE=m
> CONFIG_NET_EMATCH_U32=m
> CONFIG_NET_EMATCH_META=m
> CONFIG_NET_EMATCH_TEXT=m
> CONFIG_NET_CLS_ACT=y
> CONFIG_NET_ACT_POLICE=m
> CONFIG_NET_ACT_GACT=m
> CONFIG_GACT_PROB=y
> CONFIG_NET_ACT_MIRRED=m
> CONFIG_NET_ACT_IPT=m
> CONFIG_NET_ACT_NAT=m
> CONFIG_NET_ACT_PEDIT=m
> # CONFIG_NET_ACT_SIMP is not set
> # CONFIG_NET_CLS_IND is not set
> CONFIG_NET_SCH_FIFO=y
> 
> #
> # Network testing
> #
> # CONFIG_NET_PKTGEN is not set
> # CONFIG_HAMRADIO is not set
> # CONFIG_CAN is not set
> # CONFIG_IRDA is not set
> # CONFIG_BT is not set
> CONFIG_FIB_RULES=y
> 
> #
> # Wireless
> #
> # CONFIG_CFG80211 is not set
> # CONFIG_WIRELESS_EXT is not set
> # CONFIG_MAC80211 is not set
> # CONFIG_IEEE80211 is not set
> # CONFIG_RFKILL is not set
> 
> #
> # Device Drivers
> #
> 
> #
> # Generic Driver Options
> #
> CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
> CONFIG_STANDALONE=y
> # CONFIG_PREVENT_FIRMWARE_BUILD is not set
> # CONFIG_FW_LOADER is not set
> # CONFIG_SYS_HYPERVISOR is not set
> # CONFIG_CONNECTOR is not set
> # CONFIG_MTD is not set
> # CONFIG_PARPORT is not set
> CONFIG_PNP=y
> # CONFIG_PNP_DEBUG is not set
> 
> #
> # Protocols
> #
> CONFIG_PNPACPI=y
> CONFIG_BLK_DEV=y
> CONFIG_BLK_DEV_FD=m
> # CONFIG_BLK_CPQ_DA is not set
> # CONFIG_BLK_CPQ_CISS_DA is not set
> # CONFIG_BLK_DEV_DAC960 is not set
> # CONFIG_BLK_DEV_COW_COMMON is not set
> # CONFIG_BLK_DEV_LOOP is not set
> # CONFIG_BLK_DEV_NBD is not set
> # CONFIG_BLK_DEV_SX8 is not set
> # CONFIG_BLK_DEV_RAM is not set
> # CONFIG_CDROM_PKTCDVD is not set
> # CONFIG_ATA_OVER_ETH is not set
> # CONFIG_MISC_DEVICES is not set
> CONFIG_HAVE_IDE=y
> CONFIG_IDE=y
> CONFIG_BLK_DEV_IDE=y
> 
> #
> # Please see Documentation/ide/ide.txt for help/info on IDE drives
> #
> # CONFIG_BLK_DEV_IDE_SATA is not set
> # CONFIG_BLK_DEV_HD_IDE is not set
> CONFIG_BLK_DEV_IDEDISK=y
> # CONFIG_IDEDISK_MULTI_MODE is not set
> CONFIG_BLK_DEV_IDECD=m
> CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y
> # CONFIG_BLK_DEV_IDETAPE is not set
> # CONFIG_BLK_DEV_IDEFLOPPY is not set
> # CONFIG_BLK_DEV_IDEACPI is not set
> # CONFIG_IDE_TASK_IOCTL is not set
> # CONFIG_IDE_PROC_FS is not set
> 
> #
> # IDE chipset support/bugfixes
> #
> # CONFIG_IDE_GENERIC is not set
> # CONFIG_BLK_DEV_PLATFORM is not set
> # CONFIG_BLK_DEV_CMD640 is not set
> # CONFIG_BLK_DEV_IDEPNP is not set
> CONFIG_BLK_DEV_IDEDMA_SFF=y
> 
> #
> # PCI IDE chipsets support
> #
> CONFIG_BLK_DEV_IDEPCI=y
> # CONFIG_IDEPCI_PCIBUS_ORDER is not set
> # CONFIG_BLK_DEV_GENERIC is not set
> # CONFIG_BLK_DEV_RZ1000 is not set
> CONFIG_BLK_DEV_IDEDMA_PCI=y
> # CONFIG_BLK_DEV_AEC62XX is not set
> # CONFIG_BLK_DEV_ALI15X3 is not set
> # CONFIG_BLK_DEV_AMD74XX is not set
> # CONFIG_BLK_DEV_ATIIXP is not set
> # CONFIG_BLK_DEV_CMD64X is not set
> # CONFIG_BLK_DEV_TRIFLEX is not set
> # CONFIG_BLK_DEV_CY82C693 is not set
> # CONFIG_BLK_DEV_CS5530 is not set
> # CONFIG_BLK_DEV_CS5535 is not set
> # CONFIG_BLK_DEV_HPT34X is not set
> # CONFIG_BLK_DEV_HPT366 is not set
> # CONFIG_BLK_DEV_JMICRON is not set
> # CONFIG_BLK_DEV_SC1200 is not set
> CONFIG_BLK_DEV_PIIX=y
> # CONFIG_BLK_DEV_IT8213 is not set
> # CONFIG_BLK_DEV_IT821X is not set
> # CONFIG_BLK_DEV_NS87415 is not set
> # CONFIG_BLK_DEV_PDC202XX_OLD is not set
> # CONFIG_BLK_DEV_PDC202XX_NEW is not set
> # CONFIG_BLK_DEV_SVWKS is not set
> # CONFIG_BLK_DEV_SIIMAGE is not set
> # CONFIG_BLK_DEV_SIS5513 is not set
> # CONFIG_BLK_DEV_SLC90E66 is not set
> # CONFIG_BLK_DEV_TRM290 is not set
> # CONFIG_BLK_DEV_VIA82CXXX is not set
> # CONFIG_BLK_DEV_TC86C001 is not set
> CONFIG_BLK_DEV_IDEDMA=y
> CONFIG_IDE_ARCH_OBSOLETE_INIT=y
> # CONFIG_BLK_DEV_HD is not set
> 
> #
> # SCSI device support
> #
> # CONFIG_RAID_ATTRS is not set
> # CONFIG_SCSI is not set
> # CONFIG_SCSI_DMA is not set
> # CONFIG_SCSI_NETLINK is not set
> # CONFIG_ATA is not set
> # CONFIG_MD is not set
> # CONFIG_FUSION is not set
> 
> #
> # IEEE 1394 (FireWire) support
> #
> 
> #
> # An alternative FireWire stack is available with EXPERIMENTAL=y
> #
> # CONFIG_IEEE1394 is not set
> # CONFIG_I2O is not set
> # CONFIG_MACINTOSH_DRIVERS is not set
> CONFIG_NETDEVICES=y
> # CONFIG_NETDEVICES_MULTIQUEUE is not set
> CONFIG_IFB=m
> # CONFIG_DUMMY is not set
> # CONFIG_BONDING is not set
> # CONFIG_EQUALIZER is not set
> CONFIG_TUN=m
> # CONFIG_VETH is not set
> # CONFIG_NET_SB1000 is not set
> # CONFIG_ARCNET is not set
> # CONFIG_PHYLIB is not set
> CONFIG_NET_ETHERNET=y
> CONFIG_MII=m
> # CONFIG_HAPPYMEAL is not set
> # CONFIG_SUNGEM is not set
> # CONFIG_CASSINI is not set
> CONFIG_NET_VENDOR_3COM=y
> CONFIG_VORTEX=m
> CONFIG_TYPHOON=m
> # CONFIG_NET_TULIP is not set
> # CONFIG_HP100 is not set
> # CONFIG_IBM_NEW_EMAC_ZMII is not set
> # CONFIG_IBM_NEW_EMAC_RGMII is not set
> # CONFIG_IBM_NEW_EMAC_TAH is not set
> # CONFIG_IBM_NEW_EMAC_EMAC4 is not set
> CONFIG_NET_PCI=y
> # CONFIG_PCNET32 is not set
> # CONFIG_AMD8111_ETH is not set
> # CONFIG_ADAPTEC_STARFIRE is not set
> # CONFIG_B44 is not set
> # CONFIG_FORCEDETH is not set
> # CONFIG_EEPRO100 is not set
> CONFIG_E100=m
> # CONFIG_FEALNX is not set
> # CONFIG_NATSEMI is not set
> # CONFIG_NE2K_PCI is not set
> CONFIG_8139TOO=m
> # CONFIG_8139TOO_PIO is not set
> # CONFIG_8139TOO_TUNE_TWISTER is not set
> # CONFIG_8139TOO_8129 is not set
> # CONFIG_8139_OLD_RX_RESET is not set
> # CONFIG_R6040 is not set
> CONFIG_SIS900=m
> # CONFIG_EPIC100 is not set
> # CONFIG_SUNDANCE is not set
> # CONFIG_TLAN is not set
> CONFIG_VIA_RHINE=m
> # CONFIG_VIA_RHINE_MMIO is not set
> # CONFIG_VIA_RHINE_NAPI is not set
> # CONFIG_NETDEV_1000 is not set
> # CONFIG_NETDEV_10000 is not set
> # CONFIG_TR is not set
> 
> #
> # Wireless LAN
> #
> # CONFIG_WLAN_PRE80211 is not set
> # CONFIG_WLAN_80211 is not set
> # CONFIG_WAN is not set
> # CONFIG_FDDI is not set
> # CONFIG_PPP is not set
> # CONFIG_SLIP is not set
> # CONFIG_NETPOLL is not set
> # CONFIG_NET_POLL_CONTROLLER is not set
> # CONFIG_ISDN is not set
> # CONFIG_PHONE is not set
> 
> #
> # Input device support
> #
> CONFIG_INPUT=y
> # CONFIG_INPUT_FF_MEMLESS is not set
> # CONFIG_INPUT_POLLDEV is not set
> 
> #
> # Userland interfaces
> #
> CONFIG_INPUT_MOUSEDEV=y
> # CONFIG_INPUT_MOUSEDEV_PSAUX is not set
> CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
> CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
> # CONFIG_INPUT_JOYDEV is not set
> # CONFIG_INPUT_EVDEV is not set
> # CONFIG_INPUT_EVBUG is not set
> 
> #
> # Input Device Drivers
> #
> CONFIG_INPUT_KEYBOARD=y
> CONFIG_KEYBOARD_ATKBD=y
> # CONFIG_KEYBOARD_SUNKBD is not set
> # CONFIG_KEYBOARD_LKKBD is not set
> # CONFIG_KEYBOARD_XTKBD is not set
> # CONFIG_KEYBOARD_NEWTON is not set
> # CONFIG_KEYBOARD_STOWAWAY is not set
> # CONFIG_INPUT_MOUSE is not set
> # CONFIG_INPUT_JOYSTICK is not set
> # CONFIG_INPUT_TABLET is not set
> # CONFIG_INPUT_TOUCHSCREEN is not set
> # CONFIG_INPUT_MISC is not set
> 
> #
> # Hardware I/O ports
> #
> CONFIG_SERIO=y
> CONFIG_SERIO_I8042=y
> # CONFIG_SERIO_SERPORT is not set
> # CONFIG_SERIO_CT82C710 is not set
> # CONFIG_SERIO_PCIPS2 is not set
> CONFIG_SERIO_LIBPS2=y
> # CONFIG_SERIO_RAW is not set
> # CONFIG_GAMEPORT is not set
> 
> #
> # Character devices
> #
> CONFIG_VT=y
> CONFIG_VT_CONSOLE=y
> CONFIG_HW_CONSOLE=y
> # CONFIG_VT_HW_CONSOLE_BINDING is not set
> # CONFIG_SERIAL_NONSTANDARD is not set
> 
> #
> # Serial drivers
> #
> # CONFIG_SERIAL_8250 is not set
> CONFIG_FIX_EARLYCON_MEM=y
> 
> #
> # Non-8250 serial port support
> #
> # CONFIG_SERIAL_JSM is not set
> CONFIG_UNIX98_PTYS=y
> # CONFIG_LEGACY_PTYS is not set
> # CONFIG_IPMI_HANDLER is not set
> # CONFIG_HW_RANDOM is not set
> # CONFIG_NVRAM is not set
> # CONFIG_RTC is not set
> # CONFIG_GEN_RTC is not set
> # CONFIG_R3964 is not set
> # CONFIG_APPLICOM is not set
> # CONFIG_MWAVE is not set
> # CONFIG_PC8736x_GPIO is not set
> # CONFIG_NSC_GPIO is not set
> # CONFIG_CS5535_GPIO is not set
> # CONFIG_RAW_DRIVER is not set
> # CONFIG_HPET is not set
> # CONFIG_HANGCHECK_TIMER is not set
> CONFIG_DEVPORT=y
> # CONFIG_I2C is not set
> 
> #
> # SPI support
> #
> # CONFIG_SPI is not set
> # CONFIG_SPI_MASTER is not set
> # CONFIG_W1 is not set
> # CONFIG_POWER_SUPPLY is not set
> # CONFIG_HWMON is not set
> # CONFIG_THERMAL is not set
> # CONFIG_WATCHDOG is not set
> 
> #
> # Sonics Silicon Backplane
> #
> CONFIG_SSB_POSSIBLE=y
> # CONFIG_SSB is not set
> 
> #
> # Multifunction device drivers
> #
> # CONFIG_MFD_SM501 is not set
> 
> #
> # Multimedia devices
> #
> # CONFIG_VIDEO_DEV is not set
> # CONFIG_DVB_CORE is not set
> # CONFIG_DAB is not set
> 
> #
> # Graphics support
> #
> # CONFIG_AGP is not set
> # CONFIG_DRM is not set
> # CONFIG_VGASTATE is not set
> # CONFIG_VIDEO_OUTPUT_CONTROL is not set
> # CONFIG_FB is not set
> # CONFIG_BACKLIGHT_LCD_SUPPORT is not set
> 
> #
> # Display device support
> #
> # CONFIG_DISPLAY_SUPPORT is not set
> 
> #
> # Console display driver support
> #
> CONFIG_VGA_CONSOLE=y
> # CONFIG_VGACON_SOFT_SCROLLBACK is not set
> # CONFIG_VIDEO_SELECT is not set
> CONFIG_DUMMY_CONSOLE=y
> 
> #
> # Sound
> #
> # CONFIG_SOUND is not set
> # CONFIG_HID_SUPPORT is not set
> # CONFIG_USB_SUPPORT is not set
> # CONFIG_MMC is not set
> # CONFIG_MEMSTICK is not set
> # CONFIG_NEW_LEDS is not set
> # CONFIG_INFINIBAND is not set
> # CONFIG_RTC_CLASS is not set
> # CONFIG_DMADEVICES is not set
> 
> #
> # Userspace I/O
> #
> # CONFIG_UIO is not set
> 
> #
> # Firmware Drivers
> #
> # CONFIG_EDD is not set
> # CONFIG_DELL_RBU is not set
> # CONFIG_DCDBAS is not set
> # CONFIG_DMIID is not set
> 
> #
> # File systems
> #
> CONFIG_EXT2_FS=m
> # CONFIG_EXT2_FS_XATTR is not set
> # CONFIG_EXT2_FS_XIP is not set
> # CONFIG_EXT3_FS is not set
> # CONFIG_REISERFS_FS is not set
> # CONFIG_JFS_FS is not set
> # CONFIG_FS_POSIX_ACL is not set
> CONFIG_XFS_FS=y
> # CONFIG_XFS_QUOTA is not set
> # CONFIG_XFS_SECURITY is not set
> # CONFIG_XFS_POSIX_ACL is not set
> # CONFIG_XFS_RT is not set
> # CONFIG_OCFS2_FS is not set
> # CONFIG_DNOTIFY is not set
> CONFIG_INOTIFY=y
> CONFIG_INOTIFY_USER=y
> # CONFIG_QUOTA is not set
> # CONFIG_AUTOFS_FS is not set
> # CONFIG_AUTOFS4_FS is not set
> # CONFIG_FUSE_FS is not set
> 
> #
> # CD-ROM/DVD Filesystems
> #
> CONFIG_ISO9660_FS=m
> # CONFIG_JOLIET is not set
> # CONFIG_ZISOFS is not set
> # CONFIG_UDF_FS is not set
> 
> #
> # DOS/FAT/NT Filesystems
> #
> # CONFIG_MSDOS_FS is not set
> # CONFIG_VFAT_FS is not set
> # CONFIG_NTFS_FS is not set
> 
> #
> # Pseudo filesystems
> #
> CONFIG_PROC_FS=y
> # CONFIG_PROC_KCORE is not set
> CONFIG_PROC_SYSCTL=y
> CONFIG_SYSFS=y
> # CONFIG_TMPFS is not set
> # CONFIG_HUGETLBFS is not set
> # CONFIG_HUGETLB_PAGE is not set
> # CONFIG_CONFIGFS_FS is not set
> 
> #
> # Miscellaneous filesystems
> #
> # CONFIG_HFSPLUS_FS is not set
> # CONFIG_CRAMFS is not set
> # CONFIG_VXFS_FS is not set
> # CONFIG_MINIX_FS is not set
> # CONFIG_HPFS_FS is not set
> # CONFIG_QNX4FS_FS is not set
> # CONFIG_ROMFS_FS is not set
> # CONFIG_SYSV_FS is not set
> # CONFIG_UFS_FS is not set
> # CONFIG_NETWORK_FILESYSTEMS is not set
> 
> #
> # Partition Types
> #
> # CONFIG_PARTITION_ADVANCED is not set
> CONFIG_MSDOS_PARTITION=y
> # CONFIG_NLS is not set
> 
> #
> # Kernel hacking
> #
> CONFIG_TRACE_IRQFLAGS_SUPPORT=y
> # CONFIG_PRINTK_TIME is not set
> # CONFIG_ENABLE_WARN_DEPRECATED is not set
> # CONFIG_ENABLE_MUST_CHECK is not set
> # CONFIG_MAGIC_SYSRQ is not set
> # CONFIG_UNUSED_SYMBOLS is not set
> # CONFIG_DEBUG_FS is not set
> # CONFIG_HEADERS_CHECK is not set
> # CONFIG_DEBUG_KERNEL is not set
> # CONFIG_SLUB_DEBUG_ON is not set
> # CONFIG_SLUB_STATS is not set
> CONFIG_DEBUG_BUGVERBOSE=y
> # CONFIG_LATENCYTOP is not set
> # CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
> # CONFIG_SAMPLES is not set
> CONFIG_EARLY_PRINTK=y
> CONFIG_X86_FIND_SMP_CONFIG=y
> CONFIG_X86_MPPARSE=y
> CONFIG_DOUBLEFAULT=y
> CONFIG_IO_DELAY_TYPE_0X80=0
> CONFIG_IO_DELAY_TYPE_0XED=1
> CONFIG_IO_DELAY_TYPE_UDELAY=2
> CONFIG_IO_DELAY_TYPE_NONE=3
> CONFIG_IO_DELAY_0X80=y
> # CONFIG_IO_DELAY_0XED is not set
> # CONFIG_IO_DELAY_UDELAY is not set
> # CONFIG_IO_DELAY_NONE is not set
> CONFIG_DEFAULT_IO_DELAY_TYPE=0
> 
> #
> # Security options
> #
> # CONFIG_KEYS is not set
> # CONFIG_SECURITY is not set
> CONFIG_CRYPTO=y
> CONFIG_CRYPTO_ALGAPI=y
> CONFIG_CRYPTO_AEAD=y
> CONFIG_CRYPTO_BLKCIPHER=y
> # CONFIG_CRYPTO_SEQIV is not set
> CONFIG_CRYPTO_HASH=y
> CONFIG_CRYPTO_MANAGER=y
> CONFIG_CRYPTO_HMAC=y
> # CONFIG_CRYPTO_NULL is not set
> # CONFIG_CRYPTO_MD4 is not set
> CONFIG_CRYPTO_MD5=y
> CONFIG_CRYPTO_SHA1=y
> CONFIG_CRYPTO_SHA256=y
> CONFIG_CRYPTO_SHA512=y
> # CONFIG_CRYPTO_WP512 is not set
> # CONFIG_CRYPTO_TGR192 is not set
> CONFIG_CRYPTO_ECB=m
> CONFIG_CRYPTO_CBC=y
> # CONFIG_CRYPTO_PCBC is not set
> # CONFIG_CRYPTO_CTR is not set
> # CONFIG_CRYPTO_GCM is not set
> # CONFIG_CRYPTO_CCM is not set
> # CONFIG_CRYPTO_CRYPTD is not set
> CONFIG_CRYPTO_DES=y
> # CONFIG_CRYPTO_FCRYPT is not set
> CONFIG_CRYPTO_BLOWFISH=m
> # CONFIG_CRYPTO_TWOFISH is not set
> CONFIG_CRYPTO_TWOFISH_COMMON=m
> CONFIG_CRYPTO_TWOFISH_586=m
> CONFIG_CRYPTO_SERPENT=m
> CONFIG_CRYPTO_AES=y
> CONFIG_CRYPTO_AES_586=y
> # CONFIG_CRYPTO_CAST5 is not set
> # CONFIG_CRYPTO_CAST6 is not set
> # CONFIG_CRYPTO_TEA is not set
> # CONFIG_CRYPTO_ARC4 is not set
> # CONFIG_CRYPTO_KHAZAD is not set
> # CONFIG_CRYPTO_ANUBIS is not set
> # CONFIG_CRYPTO_SEED is not set
> CONFIG_CRYPTO_DEFLATE=y
> # CONFIG_CRYPTO_MICHAEL_MIC is not set
> # CONFIG_CRYPTO_CRC32C is not set
> # CONFIG_CRYPTO_CAMELLIA is not set
> # CONFIG_CRYPTO_TEST is not set
> CONFIG_CRYPTO_AUTHENC=y
> CONFIG_CRYPTO_LZO=m
> # CONFIG_CRYPTO_HW is not set
> CONFIG_HAVE_KVM=y
> # CONFIG_VIRTUALIZATION is not set
> 
> #
> # Library routines
> #
> CONFIG_BITREVERSE=m
> CONFIG_CRC_CCITT=m
> CONFIG_CRC16=m
> # CONFIG_CRC_ITU_T is not set
> CONFIG_CRC32=m
> # CONFIG_CRC7 is not set
> CONFIG_LIBCRC32C=m
> CONFIG_ZLIB_INFLATE=y
> CONFIG_ZLIB_DEFLATE=y
> CONFIG_LZO_COMPRESS=m
> CONFIG_LZO_DECOMPRESS=m
> CONFIG_TEXTSEARCH=y
> CONFIG_TEXTSEARCH_KMP=m
> CONFIG_TEXTSEARCH_BM=m
> CONFIG_TEXTSEARCH_FSM=m
> CONFIG_PLIST=y
> CONFIG_HAS_IOMEM=y
> CONFIG_HAS_IOPORT=y
> CONFIG_HAS_DMA=y
> 
> Linux version 2.6.25 (root@Halley) (gcc version 3.3.6) #1 Tue Apr 22
> 11:11:01 CEST 2008
> BIOS-provided physical RAM map:
>  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
>  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
>  BIOS-e820: 00000000000f0400 - 0000000000100000 (reserved)
>  BIOS-e820: 0000000000100000 - 0000000007ffe400 (usable)
>  BIOS-e820: 0000000007ffe400 - 0000000007fffc00 (ACPI data)
>  BIOS-e820: 0000000007fffc00 - 0000000008000000 (ACPI NVS)
>  BIOS-e820: 00000000ffff0400 - 0000000100000000 (reserved)
> 127MB LOWMEM available.
> Scan SMP from c0000000 for 1024 bytes.
> Scan SMP from c009fc00 for 1024 bytes.
> Scan SMP from c00f0000 for 65536 bytes.
> Scan SMP from c009fc00 for 1024 bytes.
> Entering add_active_range(0, 0, 32766) 0 entries of 256 used
> Zone PFN ranges:
>   DMA             0 -    4096
>   Normal       4096 -   32766
> Movable zone start PFN for each node
> early_node_map[1] active PFN ranges
>     0:        0 -   32766
> On node 0 totalpages: 32766
>   DMA zone: 32 pages used for memmap
>   DMA zone: 0 pages reserved
>   DMA zone: 4064 pages, LIFO batch:0
>   Normal zone: 223 pages used for memmap
>   Normal zone: 28447 pages, LIFO batch:7
>   Movable zone: 0 pages used for memmap
> DMI not present or invalid.
> ACPI: RSDP 000F7010, 0014 (r0 PTLTD )
> ACPI: RSDT 07FFE567, 0028 (r1 PTLTD    RSDT          0 PTL   1000000)
> ACPI: FACP 07FFE58F, 0074 (r1 SAG    D1131/15        1         F4240)
> ACPI: DSDT 07FFE603, 15FD (r1 SAG    D1131/15        1 MSFT  100000B)
> ACPI: FACS 07FFFFC0, 0040
> ACPI: PM-Timer IO Port: 0xf0c8
> Allocating PCI resources starting at 10000000 (gap: 08000000:f7ff0400)
> Built 1 zonelists in Zone order, mobility grouping on.  Total pages:
> 32511
> Kernel command line: auto BOOT_IMAGE=Linux ro root=301 slub_debug
> Local APIC disabled by BIOS -- you can enable it with "lapic"
> mapped APIC to ffffb000 (01101000)
> Enabling fast FPU save and restore... done.
> Enabling unmasked SIMD FPU exception support... done.
> Initializing CPU#0
> PID hash table entries: 512 (order: 9, 2048 bytes)
> Detected 448.638 MHz processor.
> Console: colour VGA+ 80x25
> console [tty0] enabled
> Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
> Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
> Memory: 126348k/131064k available (2016k kernel code, 4188k reserved,
> 664k data, 184k init, 0k highmem)
> virtual kernel memory layout:
>     fixmap  : 0xfffb5000 - 0xfffff000   ( 296 kB)
>     vmalloc : 0xc8800000 - 0xfffb3000   ( 887 MB)
>     lowmem  : 0xc0000000 - 0xc7ffe000   ( 127 MB)
>       .init : 0xc03a2000 - 0xc03d0000   ( 184 kB)
>       .data : 0xc02f83cf - 0xc039e3dc   ( 664 kB)
>       .text : 0xc0100000 - 0xc02f83cf   (2016 kB)
> Checking if this processor honours the WP bit even in supervisor
> mode...Ok.
> CPA: page pool initialized 1 of 1 pages preallocated
> SLUB: Genslabs=12, HWalign=32, Order=0-1, MinObjects=4, CPUs=1, Nodes=1
> Calibrating delay using timer specific routine.. 898.14 BogoMIPS
> (lpj=1796296)
> Mount-cache hash table entries: 512
> CPU: L1 I cache: 16K, L1 D cache: 16K
> CPU: L2 cache: 512K
> Compat vDSO mapped to ffffe000.
> CPU: Intel Pentium III (Katmai) stepping 03
> Checking 'hlt' instruction... OK.
> Freeing SMP alternatives: 0k freed
> ACPI: Core revision 20070126
> ACPI: setting ELCR to 0200 (from 0e20)
> net_namespace: 444 bytes
> NET: Registered protocol family 16
> ACPI: bus type pci registered
> PCI: PCI BIOS revision 2.10 entry at 0xfd9c4, last bus=1
> PCI: Using configuration type 1
> Setting up standard PCI resources
> ACPI: EC: Look up EC in DSDT
> ACPI: Interpreter enabled
> ACPI: (supports S0 S5)
> ACPI: Using PIC for interrupt routing
> ACPI: Device [ECP] status [00000008]: functional but not present;
> setting present
> ACPI: PCI Root Bridge [PCI0] (0000:00)
> * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
> * this clock source is slow. Consider trying other clock sources
> pci 0000:00:07.3: quirk: region f0c0-f0ff claimed by PIIX4 ACPI
> pci 0000:00:07.3: quirk: region f0b0-f0bf claimed by PIIX4 SMB
> ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
> ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 14 15)
> ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 *10 11 12 14 15)
> ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 *5 6 7 9 10 11 12 14 15)
> ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 *9 10 11 12 14 15)
> Linux Plug and Play Support v0.97 (c) Adam Belay
> pnp: PnP ACPI init
> ACPI: bus type pnp registered
> pnp: PnP ACPI: found 12 devices
> ACPI: ACPI bus type pnp unregistered
> PCI: Using ACPI for IRQ routing
> PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a
> report
> system 00:01: ioport range 0x370-0x371 has been reserved
> system 00:01: ioport range 0x4d0-0x4d1 has been reserved
> system 00:01: ioport range 0xf0b0-0xf0bf has been reserved
> system 00:01: ioport range 0xf0c0-0xf0cf has been reserved
> system 00:01: ioport range 0xf0d0-0xf0ff has been reserved
> PCI: Bridge: 0000:00:01.0
>   IO window: d000-dfff
>   MEM window: 0xfea00000-0xfeafffff
>   PREFETCH window: 0x00000000de000000-0x00000000deffffff
> NET: Registered protocol family 2
> IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
> TCP established hash table entries: 4096 (order: 3, 32768 bytes)
> TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
> TCP: Hash tables configured (established 4096 bind 4096)
> TCP reno registered
> SGI XFS with no debug enabled
> io scheduler noop registered
> io scheduler deadline registered (default)
> pci 0000:00:00.0: Limiting direct PCI/PCI transfers
> pci 0000:00:0c.0: Firmware left e100 interrupts enabled; disabling
> pci 0000:00:0e.0: Firmware left e100 interrupts enabled; disabling
> pci 0000:01:00.0: Boot video device
> Uniform Multi-Platform E-IDE driver
> ide: Assuming 33MHz system bus speed for PIO modes; override with
> idebus=xx
> PIIX4: IDE controller (0x8086:0x7111 rev 0x01) at  PCI slot 0000:00:07.1
> PIIX4: not 100% native mode: will probe irqs later
>     ide0: BM-DMA at 0xfcf0-0xfcf7, BIOS settings: hda:DMA, hdb:PIO
>     ide1: BM-DMA at 0xfcf8-0xfcff, BIOS settings: hdc:DMA, hdd:PIO
> Probing IDE interface ide0...
> hda: ST320413A, ATA DISK drive
> hda: host max PIO4 wanted PIO255(auto-tune) selected PIO4
> hda: UDMA/33 mode selected
> Probing IDE interface ide1...
> hdc: LTN382, ATAPI CD/DVD-ROM drive
> hdc: host max PIO4 wanted PIO255(auto-tune) selected PIO4
> hdc: UDMA/33 mode selected
> ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
> ide1 at 0x170-0x177,0x376 on irq 15
> hda: max request size: 128KiB
> hda: 39102336 sectors (20020 MB) w/1024KiB Cache, CHS=38792/16/63
> hda: cache flushes not supported
>  hda: hda1 hda2 < hda5 hda6 hda7 hda8 hda9 >
> i8042 aux: probe of 00:07 failed with error -16
> PNP: PS/2 Controller [PNP0303:KEYB] at 0x60,0x64 irq 1
> PNP: PS/2 appears to have AUX port disabled, if this is incorrect please
> boot with i8042.nopnp
> serio: i8042 KBD port at 0x60,0x64 irq 1
> mice: PS/2 mouse device common for all mice
> nf_conntrack version 0.5.0 (2048 buckets, 8192 max)
> ip_tables: (C) 2000-2006 Netfilter Core Team
> TCP cubic registered
> Initializing XFRM netlink socket
> NET: Registered protocol family 1
> NET: Registered protocol family 17
> NET: Registered protocol family 15
> Using IPI Shortcut mode
> Filesystem "hda1": Disabling barriers, not supported by the underlying
> device
> XFS mounting filesystem hda1
> Starting XFS recovery on filesystem: hda1 (logdev: internal)
> Ending XFS recovery on filesystem: hda1 (logdev: internal)
> VFS: Mounted root (xfs filesystem) readonly.
> Freeing unused kernel memory: 184k freed
> Adding 674688k swap on /dev/hda9.  Priority:-1 extents:1 across:674688k
> Filesystem "hda1": Disabling barriers, not supported by the underlying
> device
> Filesystem "hda1": Disabling barriers, not supported by the underlying
> device
> e100: Intel(R) PRO/100 Network Driver, 3.5.23-k4-NAPI
> e100: Copyright(c) 1999-2006 Intel Corporation
> ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10
> PCI: setting IRQ 10 as level-triggered
> ACPI: PCI Interrupt 0000:00:0c.0[A] -Link [LNKB] -GSI 10 (level,
> low) -IRQ 10
> e100: eth0: e100_probe: addr 0xfecff000, irq 10, MAC addr
> 00:90:27:74:66:4c
> ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 5
> PCI: setting IRQ 5 as level-triggered
> ACPI: PCI Interrupt 0000:00:0e.0[A] -Link [LNKC] -GSI 5 (level,
> low) -IRQ 5
> e100: eth1: e100_probe: addr 0xfecfe000, irq 5, MAC addr
> 00:d0:b7:18:14:bc
> ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11
> PCI: setting IRQ 11 as level-triggered
> ACPI: PCI Interrupt 0000:00:0a.0[A] -Link [LNKA] -GSI 11 (level,
> low) -IRQ 11
> 3c59x: Donald Becker and others.
> 0000:00:0a.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001e800.
> Filesystem "hda5": Disabling barriers, not supported by the underlying
> device
> XFS mounting filesystem hda5
> Starting XFS recovery on filesystem: hda5 (logdev: internal)
> Ending XFS recovery on filesystem: hda5 (logdev: internal)
> Filesystem "hda6": Disabling barriers, not supported by the underlying
> device
> XFS mounting filesystem hda6
> Starting XFS recovery on filesystem: hda6 (logdev: internal)
> Ending XFS recovery on filesystem: hda6 (logdev: internal)
> Filesystem "hda7": Disabling barriers, not supported by the underlying
> device
> XFS mounting filesystem hda7
> Starting XFS recovery on filesystem: hda7 (logdev: internal)
> Ending XFS recovery on filesystem: hda7 (logdev: internal)
> Filesystem "hda8": Disabling barriers, not supported by the underlying
> device
> XFS mounting filesystem hda8
> Ending clean XFS mount for filesystem: hda8
> e100: eth0: e100_watchdog: link up, 100Mbps, full-duplex
> eth2:  setting full-duplex.
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

David Miller wrote:

> Netdev added to CC:, please include in all bug reports as
> that's where the networking developers sit.

ok.
Two different systems have crashed since
2.6.25 upgrade (I upgraded a total of 8
systems).

For now, the only two boxes that crashed
have these modules loaded (where the other
systems doesn't have these modules loaded):

sch_sfq 6088 3
sch_htb 15208 1
cls_fw 3848 1

Here is the script for setup htb (which classify
traffic inside the ipsec tunnels):

tc qdisc add dev eth0 root handle 1:0 htb default 11

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 1700kbit ceil
1700kbit burst 15k
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 200kbit ceil
400kbit burst 5k
tc class add dev eth0 parent 1:1 classid 1:11 htb rate 600kbit ceil
1600kbit burst 5k
tc class add dev eth0 parent 1:1 classid 1:12 htb rate 800kbit ceil
1700kbit burst 15k
tc qdisc add dev eth0 parent 1:10 handle 10:0 sfq perturb 10
tc qdisc add dev eth0 parent 1:11 handle 20:0 sfq perturb 5
tc qdisc add dev eth0 parent 1:12 handle 30:0 sfq perturb 10

FW="tc filter add dev eth0 protocol ip parent 1:0 prio 1"
$FW handle 110 fw flowid 1:10
$FW handle 111 fw flowid 1:11
$FW handle 112 fw flowid 1:12

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Just few minutes ago, another 2.6.25 crash
with this message:

EIP: [<c028000a>] netif_rx+0x6a/0xd0 SS:ESP 0068:c039f868

http://80.204.235.230/4.jpg

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Herbert Xu]

Marco Berizzi <pupilla@hotmail.com> wrote:
> Just few minutes ago, another 2.6.25 crash
> with this message:
> 
> EIP: [<c028000a>] netif_rx+0x6a/0xd0 SS:ESP 0068:c039f868
> 
> http://80.204.235.230/4.jpg

OK, the xfrm_output_resume trail revealed nothing.  Combined
with this crash however, it would appear that you've got a problem
with live skbs being freed.  Unfortunately such problems are
difficult to track down.

Perhaps you could enable SLAB debugging to see if we can get
closer to the culprit? Alternatively you could try a git bisection.

Unless we have a way of reproducing this there isn't a lot more
that we can do I'm afraid.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Herbert Xu wrote:

> Marco Berizzi <pupilla@hotmail.com> wrote:
>> Just few minutes ago, another 2.6.25 crash
>> with this message:
>> 
>> EIP: [<c028000a>] netif_rx+0x6a/0xd0 SS:ESP 0068:c039f868
>> 
>> http://80.204.235.230/4.jpg
> 
> OK, the xfrm_output_resume trail revealed nothing.  Combined
> with this crash however, it would appear that you've got a problem
> with live skbs being freed.  Unfortunately such problems are
> difficult to track down.
> 
> Perhaps you could enable SLAB debugging to see if we can get
> closer to the culprit?

Yes indeed, I will try next monday.

> Alternatively you could try a git bisection.

Yes, but it could take forever... :-((
As I said, I have removed the sfq and htb modules,
and I would like to wait one week to see if these
boxes will crash. I have 8 linux boxes running 2.6.25
with the same .config and only the two with htb/sfq
qdisc are crashing. Just few minutes ago I have seen
this message forwarded to netdev by David Miller:

Re: PROBLEM: kernel lockup while changing TC rules

it is talking about htb/sfq. Maybe it is related.
 
> Unless we have a way of reproducing this there isn't a lot more
> that we can do I'm afraid.

thanks anyway Herbert.

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Marco Berizziwrote:


> Herbert Xu wrote:
>
> > Marco Berizzi <pupilla@hotmail.com> wrote:
> >> Just few minutes ago, another 2.6.25 crash
> >> with this message:
> >>
> >> EIP: [<c028000a>] netif_rx+0x6a/0xd0 SS:ESP 0068:c039f868
> >>
> >> http://80.204.235.230/4.jpg
> >
> > OK, the xfrm_output_resume trail revealed nothing.  Combined
> > with this crash however, it would appear that you've got a problem
> > with live skbs being freed.  Unfortunately such problems are
> > difficult to track down.
> >
> > Perhaps you could enable SLAB debugging to see if we can get
> > closer to the culprit?
>
> Yes indeed, I will try next monday.
>
> > Alternatively you could try a git bisection.
>
> Yes, but it could take forever... :-((
> As I said, I have removed the sfq and htb modules,
> and I would like to wait one week to see if these
> boxes will crash. I have 8 linux boxes running 2.6.25
> with the same .config and only the two with htb/sfq
> qdisc are crashing. Just few minutes ago I have seen
> this message forwarded to netdev by David Miller:
>
> Re: PROBLEM: kernel lockup while changing TC rules
>
> it is talking about htb/sfq. Maybe it is related.
>
> > Unless we have a way of reproducing this there isn't a lot more
> > that we can do I'm afraid.
>
> thanks anyway Herbert.

ok, I can confirm after one week uptime: this problem
is not happening after removing sch_htb/sch_sfq and
cls_fw modules.
What should I do to help track down this problem?

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Marco Berizzi wrote:

> ok, I can confirm after one week uptime: this problem
> is not happening after removing sch_htb/sch_sfq and
> cls_fw modules.
> What should I do to help track down this problem?

I have rebooted the two boxes with slub_debug.
This is the output taken with network console.
Herbert does it help you?

BUG: unable to handle kernel paging request at 6b6b6b9f
IP: [<c02e3e84>] xfrm_output_resume+0x64/0x100
Oops: 0000 [#1]
Modules linked in: netconsole sch_sfq sch_htb cls_fw nf_nat_pptp
nf_nat_proto_gre nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_ftp
nf_conntrack_ftp 3c59x mii

Pid: 895, comm: squid Not tainted (2.6.25 #1)
EIP: 0060:[<c02e3e84>] EFLAGS: 00010246 CPU: 0
EIP is at xfrm_output_resume+0x64/0x100
EAX: 00000001 EBX: c3f3ed20 ECX: 00000001 EDX: 6b6b6b6b
ESI: 80000000 EDI: c3d9c280 EBP: 0000001c ESP: c91f5bd8
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process squid (pid: 895, ti=c91f4000 task=c918eb00 task.ti=c91f4000)
Stack: 00000001 c039ce60 c3f3ed20 80000000 c3f3ed20 80000000 c02dd147
00000000
       c9c4a0c0 c02dd0d0 80000000 c3f3ed20 c02a6709 c3f3ed20 c02a68e1
c9c4b960
       c9c4a0c0 c02a66f0 80000000 c3f3ed40 00000000 c3f3ed20 c4d446b0
c9c4b960
Call Trace:
 [<c02dd147>] xfrm4_output+0x47/0x90
 [<c02dd0d0>] xfrm4_output_finish+0x0/0x30
 [<c02a6709>] ip_forward_finish+0x19/0x30
 [<c02a68e1>] ip_forward+0x1c1/0x320
 [<c02a66f0>] ip_forward_finish+0x0/0x30
 [<c02a55ab>] ip_rcv_finish+0xeb/0x2b0
 [<c02a54c0>] ip_rcv_finish+0x0/0x2b0
 [<c02a58d7>] ip_rcv+0x167/0x260
 [<c02a54c0>] ip_rcv_finish+0x0/0x2b0
 [<c02804ca>] netif_receive_skb+0x28a/0x360
 [<c0280605>] process_backlog+0x65/0xc0
 [<c02807de>] net_rx_action+0x14e/0x170
 [<c011af04>] __do_softirq+0x74/0x90
 [<c011af46>] do_softirq+0x26/0x30
 [<c011ae12>] local_bh_enable+0x42/0x80
 [<c029a310>] __nf_conntrack_find+0xd0/0xe0
 [<c029a32f>] nf_conntrack_find_get+0xf/0x60
 [<c029ad27>] nf_conntrack_in+0x1b7/0x290
 [<c02990d2>] nf_iterate+0x72/0x90
 [<c02aa500>] dst_output+0x0/0x10
 [<c02aa500>] dst_output+0x0/0x10
 [<c0299156>] nf_hook_slow+0x66/0xe0
 [<c02aa500>] dst_output+0x0/0x10
 [<c02a7c23>] __ip_local_out+0xb3/0xc0
 [<c02aa500>] dst_output+0x0/0x10
 [<c02a7c38>] ip_local_out+0x8/0x20
 [<c02a8649>] ip_queue_xmit+0x2b9/0x370
 [<c0152fd6>] check_bytes_and_report+0x26/0xc0
 [<c02a66f0>] ip_forward_finish+0x0/0x30
 [<c01530d1>] check_pad_bytes+0x61/0x80
 [<c0153280>] check_object+0x80/0x1b0
 [<c02bdd93>] tcp_v4_send_check+0x53/0xf0
 [<c02b9769>] tcp_transmit_skb+0x2b9/0x4b0
 [<c02bbaeb>] tcp_send_fin+0x7b/0xc0
 [<c02bacf1>] tcp_write_xmit+0x151/0x270
 [<c02bae21>] __tcp_push_pending_frames+0x11/0x70
 [<c02b09d7>] tcp_close+0x2a7/0x320
 [<c02ca85e>] inet_release+0x4e/0x60
 [<c0274957>] sock_release+0x67/0x80
 [<c0275261>] sock_close+0x21/0x40
 [<c01599da>] __fput+0x14a/0x160
 [<c01580fa>] filp_close+0x3a/0x70
 [<c0158177>] sys_close+0x47/0x80
 [<c0102d6e>] syscall_call+0x7/0xb
 =======================
Code: 00 00 00 8b 43 70 85 c0 74 05 e8 48 59 f9 ff c7 43 70 00 00 00 00
8b 43 18 8b 50 40 89 d8 ff 52 24 83 f8 01 89 c2 75 3f 8b 53 18 <8b> 42
34 85 c0 74 6a 8b 42 40 8b 4a 0c 0f b7 30 8d 04 f5 04 00
EIP: [<c02e3e84>] xfrm_output_resume+0x64/0x100 SS:ESP 0068:c91f5bd8
Kernel panic - not syncing: Fatal exception in interrupt
Rebooting in 5 seconds..Linux version 2.6.25 (root@Mimosa) (gcc version
3.3.5) #1 Tue Apr 29 19:06:06 CEST 2008

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Herbert Xu]

On Fri, May 09, 2008 at 11:50:00AM +0200, Marco Berizzi wrote:
> 
> I have rebooted the two boxes with slub_debug.
> This is the output taken with network console.
> Herbert does it help you?
> 
> BUG: unable to handle kernel paging request at 6b6b6b9f

Unfortunately this just confirms that your skb has been freed
prematurely because 6b is the poison value.

However, it doesn't point us at the offender.

Although I wonder if there might be something in netfilter that's
freeing it since that's the only thing that could've run on your
machine between the previous access and the crash.  I'll take
another look.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Herbert Xu wrote:

> On Fri, May 09, 2008 at 11:50:00AM +0200, Marco Berizzi wrote:
> >
> > I have rebooted the two boxes with slub_debug.
> > This is the output taken with network console.
> > Herbert does it help you?
> >
> > BUG: unable to handle kernel paging request at 6b6b6b9f
>
> Unfortunately this just confirms that your skb has been freed
> prematurely because 6b is the poison value.
>
> However, it doesn't point us at the offender.

:-((

> Although I wonder if there might be something in netfilter that's
> freeing it since that's the only thing that could've run on your
> machine between the previous access and the crash.  I'll take
> another look.

Let me know if the only solution is git bisect.
The bug is spotted when sch_sfq/_htb and cls_fw
are loaded.

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Ingo Molnar]

* Marco Berizzi <pupilla@hotmail.com> wrote:

> Herbert Xu wrote:
> 
> > On Fri, May 09, 2008 at 11:50:00AM +0200, Marco Berizzi wrote:
> > >
> > > I have rebooted the two boxes with slub_debug.
> > > This is the output taken with network console.
> > > Herbert does it help you?
> > >
> > > BUG: unable to handle kernel paging request at 6b6b6b9f
> >
> > Unfortunately this just confirms that your skb has been freed
> > prematurely because 6b is the poison value.
> >
> > However, it doesn't point us at the offender.
> 
> :-((

you could try x86.git and enable CONFIG_KMEMCHECK, which catches all 
sorts of memory corruption bugs at its root:

  http://people.redhat.com/mingo/x86.git/README

it's for 32-bit currently, and depends on the following CONFIG details 
in your .config:

        depends on X86_32
        depends on !X86_USE_3DNOW
        depends on !CC_OPTIMIZE_FOR_SIZE
        depends on !DEBUG_PAGEALLOC && SLUB

note that CONFIG_DEBUG_PAGEALLOC=y will catch a few types of corruption 
too. Note that KMEMCHECK and DEBUG_PAGEALLOC are exclusive.

	Ingo

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Herbert Xu wrote:

> On Fri, May 09, 2008 at 11:50:00AM +0200, Marco Berizzi wrote:
> >
> > I have rebooted the two boxes with slub_debug.
> > This is the output taken with network console.
> > Herbert does it help you?
> >
> > BUG: unable to handle kernel paging request at 6b6b6b9f
>
> Unfortunately this just confirms that your skb has been freed
> prematurely because 6b is the poison value.
>
> However, it doesn't point us at the offender.
>
> Although I wonder if there might be something in netfilter that's
> freeing it since that's the only thing that could've run on your
> machine between the previous access and the crash.  I'll take
> another look.

Here is the results of the first git bisect.
The error message is different from the message
subject, so I hope will be useful for you.
Now, this box is crashing immediately after the
network stack is up.

PS: linux version shown by dmesg is 2.6.24 but it
is the results of 'git bisect good v2.6.24' and
'git bisect bad v2.6.25'

Restarting system.
machine restart
Linux version 2.6.24 (root@Calimero) (gcc version 4.2.3) #1 Fri May 9
14:39:54 CEST 2008
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
 BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000dc000 - 00000000000e0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000000a000000 (usable)
 BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
160MB LOWMEM available.
Scan SMP from c0000000 for 1024 bytes.
Scan SMP from c009fc00 for 1024 bytes.
Scan SMP from c00f0000 for 65536 bytes.
Scan SMP from c009f800 for 1024 bytes.
Entering add_active_range(0, 0, 40960) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->    40960
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0:        0 ->    40960
On node 0 totalpages: 40960
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 288 pages used for memmap
  Normal zone: 36576 pages, LIFO batch:7
  Movable zone: 0 pages used for memmap
DMI 2.1 present.
ACPI: DMI detected: Hewlett-Packard
Allocating PCI resources starting at 10000000 (gap: 0a000000:f5ff0000)
Built 1 zonelists in Zone order, mobility grouping on.  Total pages:
40640
Kernel command line: auto BOOT_IMAGE=Linux ro root=301 slub_debug
Local APIC disabled by BIOS -- you can enable it with "lapic"
mapped APIC to ffffb000 (01141000)
Enabling fast FPU save and restore... done.
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 4096 bytes)
Detected 267.285 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 158944k/163840k available (2005k kernel code, 4464k reserved,
603k data, 176k init, 0k highmem)
virtual kernel memory layout:
    fixmap  : 0xfffb5000 - 0xfffff000   ( 296 kB)
    vmalloc : 0xca800000 - 0xfffb3000   ( 855 MB)
    lowmem  : 0xc0000000 - 0xca000000   ( 160 MB)
      .init : 0xc0390000 - 0xc03bc000   ( 176 kB)
      .data : 0xc02f557a - 0xc038c2c0   ( 603 kB)
      .text : 0xc0100000 - 0xc02f557a   (2005 kB)
Checking if this processor honours the WP bit even in supervisor
mode...Ok.
SLUB: Genslabs=11, HWalign=32, Order=0-1, MinObjects=4, CPUs=1, Nodes=1
Calibrating delay using timer specific routine.. 535.25 BogoMIPS
(lpj=1070512)
Mount-cache hash table entries: 512
CPU: L1 I cache: 16K, L1 D cache: 16K
Compat vDSO mapped to ffffe000.
CPU: Intel Celeron (Covington) stepping 00
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 0k freed
ACPI: Core revision 20070126
ACPI Exception (tbxface-0629): AE_NO_ACPI_TABLES, While loading
namespace from ACPI tables [20070126]
ACPI: Unable to load the System Description Tables
net_namespace: 448 bytes
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfda61, last bus=1
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
* Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
* this clock source is slow. Consider trying other clock sources
PCI quirk: region 6100-613f claimed by PIIX4 ACPI
PCI quirk: region 5f00-5f0f claimed by PIIX4 SMB
PCI: Using IRQ router PIIX/ICH [8086/7110] at 0000:00:07.0
Time: tsc clocksource has been installed.
PCI: Bridge: 0000:00:01.0
  IO window: b000-bfff
  MEM window: efe00000-efefffff
  PREFETCH window: e5c00000-e7cfffff
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
SGI XFS with no debug enabled
io scheduler noop registered (default)
Limiting direct PCI/PCI transfers.
Boot video device is 0000:01:00.0
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
PIIX4: IDE controller (0x8086:0x7111 rev 0x01) at  PCI slot 0000:00:07.1
PIIX4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio
Probing IDE interface ide0...
hda: QUANTUM FIREBALL EX3.2A, ATA DISK drive
hda: host max PIO4 wanted PIO255(auto-tune) selected PIO4
hda: UDMA/33 mode selected
Probing IDE interface ide1...
hdc: CRD-8160B, ATAPI CD/DVD-ROM drive
hdc: host max PIO4 wanted PIO255(auto-tune) selected PIO4
hdc: MWDMA2 mode selected
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 6306048 sectors (3228 MB) w/418KiB Cache, CHS=6256/16/63
hda: cache flushes not supported
 hda: hda1 hda2 < hda5 hda6 hda7 hda8 hda9 >
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
nf_conntrack version 0.5.0 (3072 buckets, 12288 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 15
Using IPI Shortcut mode
input: AT Translated Set 2 keyboard as
/devices/platform/i8042/serio0/input/input0
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda1
Ending clean XFS mount for filesystem: hda1
VFS: Mounted root (xfs filesystem) readonly.
Freeing unused kernel memory: 176k freed
Adding 330584k swap on /dev/hda9.  Priority:-1 extents:1 across:330584k
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
PCI: setting IRQ 10 as level-triggered
PCI: Found IRQ 10 for device 0000:00:09.0
3c59x: Donald Becker and others.
0000:00:09.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001dc00.
PCI: setting IRQ 11 as level-triggered
PCI: Found IRQ 11 for device 0000:00:0a.0
0000:00:0a.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001da00.
PCI: setting IRQ 9 as level-triggered
PCI: Found IRQ 9 for device 0000:00:0b.0
PCI: Sharing IRQ 9 with 0000:00:07.2
0000:00:0b.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001d800.
netconsole: local port 6665
netconsole: local IP 85.32.35.30
netconsole: interface eth0
netconsole: remote port 6666
netconsole: remote IP 80.204.235.230
netconsole: remote ethernet address 00:1b:d4:0a:43:d0
netconsole: device eth0 not up yet, forcing it
eth0:  setting full-duplex.
netconsole: carrier detect appears untrustworthy, waiting 4 seconds
console [netcon0] enabled
netconsole: network logging started
Filesystem "hda5": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda5
Ending clean XFS mount for filesystem: hda5
Filesystem "hda6": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda6
Ending clean XFS mount for filesystem: hda6
Filesystem "hda7": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda7
Ending clean XFS mount for filesystem: hda7
Filesystem "hda8": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda8
Ending clean XFS mount for filesystem: hda8
BUG: unable to handle kernel NULL pointer dereference at 00000224
IP: [<c02a07d3>] ip_route_input+0x33/0xe00
Oops: 0000 [#1]
Modules linked in: netconsole sch_sfq sch_htb cls_fw nf_nat_pptp
nf_nat_proto_gre nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_ftp
nf_conntrack_ftp 3c59x mii

Pid: 676, comm: ldconfig Not tainted (2.6.24 #1)
EIP: 0060:[<c02a07d3>] EFLAGS: 00010202 CPU: 0
EIP is at ip_route_input+0x33/0xe00
EAX: 00000000 EBX: c50112ac ECX: 00000000 EDX: 00000040
ESI: 00000002 EDI: c907ed80 EBP: 120110ac ESP: c90c3bd4
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process ldconfig (pid: 676, ti=c90c2000 task=c906f580 task.ti=c90c2000)
Stack: 00000000 c9e111e0 c9d18840 c90c3cac c02a010d 00000002 00000000
c90c3c14
       00014001 c90c3ce8 c03de660 c907eea0 c9077b60 00000000 00000001
00000001
       00000002 00000001 00000000 120110ac 1e232055 00000000 00000000
00000000
Call Trace:
 [<c02a010d>] __ip_route_output_key+0x4ad/0x840
 [<c02a04bc>] ip_route_output_flow+0x1c/0x200
 [<c02d38db>] ip_route_me_harder+0xab/0x290
 [<c02d87e2>] reject_tg+0x232/0x400
 [<c02d85b0>] reject_tg+0x0/0x400
 [<c02d793c>] ipt_do_table+0x2bc/0x4b0
 [<c02d7bc0>] ipt_hook+0x0/0x20
 [<c02968c7>] nf_iterate+0x57/0x80
 [<c02969c8>] nf_hook_slow+0x98/0xd0
 [<c02a4440>] ip_forward_finish+0x0/0x30
 [<c02a471a>] ip_forward+0x2aa/0x320
 [<c02a4440>] ip_forward_finish+0x0/0x30
 [<c02a307b>] ip_rcv_finish+0xeb/0x310
 [<c02a2f90>] ip_rcv_finish+0x0/0x310
 [<c02a3691>] ip_rcv+0x191/0x250
 [<c02a2f90>] ip_rcv_finish+0x0/0x310
 [<c02a3500>] ip_rcv+0x0/0x250
 [<c027dfb1>] netif_receive_skb+0x1f1/0x370
 [<c02806db>] process_backlog+0x5b/0xb0
 [<c0280224>] net_rx_action+0xb4/0x140
 [<c011a642>] __do_softirq+0x42/0x90
 [<c011a6b6>] do_softirq+0x26/0x30
 [<c010560a>] do_IRQ+0x4a/0x80
 [<c0151412>] check_object+0xc2/0x210
 [<c01039a7>] common_interrupt+0x23/0x28
 [<c0152480>] kmem_cache_alloc+0x30/0xa0
 [<c015d588>] getname+0x28/0xd0
 [<c015d588>] getname+0x28/0xd0
 [<c01541ee>] do_sys_open+0x1e/0xe0
 [<c01542ec>] sys_open+0x1c/0x20
 [<c0102ffe>] syscall_call+0x7/0xb
 [<c02f0000>] rest_init+0x20/0x40
 =======================
Code: 81 ec b8 00 00 00 89 44 24 30 8b 84 24 d0 00 00 8b 54 24 30 0f b6
8c 24 cc 00 00 00 8b 70 4c 8b 42 14 80 e1 1c 89 f2 c1 e2 05 <8b> b8 24
02 00 00 89 d8 31 ea 88 4c 24 53 e8 7a dd ff ff 8b 15
EIP: [<c02a07d3>] ip_route_input+0x33/0xe00 SS:ESP 0068:c90c3bd4
Kernel panic - not syncing: Fatal exception in interrupt
Linux version 2.6.24 (root@Calimero) (gcc version 4.2.3) #1 Fri May 9
14:39:54 CEST 2008
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
 BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000dc000 - 00000000000e0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000000a000000 (usable)
 BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
160MB LOWMEM available.
Scan SMP from c0000000 for 1024 bytes.
Scan SMP from c009fc00 for 1024 bytes.
Scan SMP from c00f0000 for 65536 bytes.
Scan SMP from c009f800 for 1024 bytes.
Entering add_active_range(0, 0, 40960) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->    40960
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0:        0 ->    40960
On node 0 totalpages: 40960
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 288 pages used for memmap
  Normal zone: 36576 pages, LIFO batch:7
  Movable zone: 0 pages used for memmap
DMI 2.1 present.
ACPI: DMI detected: Hewlett-Packard
Allocating PCI resources starting at 10000000 (gap: 0a000000:f5ff0000)
Built 1 zonelists in Zone order, mobility grouping on.  Total pages:
40640
Kernel command line: auto BOOT_IMAGE=Linux ro root=301 slub_debug
Local APIC disabled by BIOS -- you can enable it with "lapic"
mapped APIC to ffffb000 (01141000)
Enabling fast FPU save and restore... done.
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 4096 bytes)
Detected 267.289 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 158944k/163840k available (2005k kernel code, 4464k reserved,
603k data, 176k init, 0k highmem)
virtual kernel memory layout:
    fixmap  : 0xfffb5000 - 0xfffff000   ( 296 kB)
    vmalloc : 0xca800000 - 0xfffb3000   ( 855 MB)
    lowmem  : 0xc0000000 - 0xca000000   ( 160 MB)
      .init : 0xc0390000 - 0xc03bc000   ( 176 kB)
      .data : 0xc02f557a - 0xc038c2c0   ( 603 kB)
      .text : 0xc0100000 - 0xc02f557a   (2005 kB)
Checking if this processor honours the WP bit even in supervisor
mode...Ok.
SLUB: Genslabs=11, HWalign=32, Order=0-1, MinObjects=4, CPUs=1, Nodes=1
Calibrating delay using timer specific routine.. 535.28 BogoMIPS
(lpj=1070571)
Mount-cache hash table entries: 512
CPU: L1 I cache: 16K, L1 D cache: 16K
Compat vDSO mapped to ffffe000.
CPU: Intel Celeron (Covington) stepping 00
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 0k freed
ACPI: Core revision 20070126
ACPI Exception (tbxface-0629): AE_NO_ACPI_TABLES, While loading
namespace from ACPI tables [20070126]
ACPI: Unable to load the System Description Tables
net_namespace: 448 bytes
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfda61, last bus=1
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
* Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
* this clock source is slow. Consider trying other clock sources
PCI quirk: region 6100-613f claimed by PIIX4 ACPI
PCI quirk: region 5f00-5f0f claimed by PIIX4 SMB
PCI: Using IRQ router PIIX/ICH [8086/7110] at 0000:00:07.0
Time: tsc clocksource has been installed.
PCI: Bridge: 0000:00:01.0
  IO window: b000-bfff
  MEM window: efe00000-efefffff
  PREFETCH window: e5c00000-e7cfffff
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
SGI XFS with no debug enabled
io scheduler noop registered (default)
Limiting direct PCI/PCI transfers.
Boot video device is 0000:01:00.0
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
PIIX4: IDE controller (0x8086:0x7111 rev 0x01) at  PCI slot 0000:00:07.1
PIIX4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio
Probing IDE interface ide0...
hda: QUANTUM FIREBALL EX3.2A, ATA DISK drive
hda: host max PIO4 wanted PIO255(auto-tune) selected PIO4
hda: UDMA/33 mode selected
Probing IDE interface ide1...
hdc: CRD-8160B, ATAPI CD/DVD-ROM drive
hdc: host max PIO4 wanted PIO255(auto-tune) selected PIO4
hdc: MWDMA2 mode selected
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 6306048 sectors (3228 MB) w/418KiB Cache, CHS=6256/16/63
hda: cache flushes not supported
 hda: hda1 hda2 < hda5 hda6 hda7 hda8 hda9 >
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
nf_conntrack version 0.5.0 (3072 buckets, 12288 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 15
Using IPI Shortcut mode
input: AT Translated Set 2 keyboard as
/devices/platform/i8042/serio0/input/input0
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda1
Starting XFS recovery on filesystem: hda1 (logdev: internal)
Ending XFS recovery on filesystem: hda1 (logdev: internal)
VFS: Mounted root (xfs filesystem) readonly.
Freeing unused kernel memory: 176k freed
Adding 330584k swap on /dev/hda9.  Priority:-1 extents:1 across:330584k
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
PCI: setting IRQ 10 as level-triggered
PCI: Found IRQ 10 for device 0000:00:09.0
3c59x: Donald Becker and others.
0000:00:09.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001dc00.
PCI: setting IRQ 11 as level-triggered
PCI: Found IRQ 11 for device 0000:00:0a.0
0000:00:0a.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001da00.
PCI: setting IRQ 9 as level-triggered
PCI: Found IRQ 9 for device 0000:00:0b.0
PCI: Sharing IRQ 9 with 0000:00:07.2
0000:00:0b.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001d800.
netconsole: local port 6665
netconsole: local IP 85.32.35.30
netconsole: interface eth0
netconsole: remote port 6666
netconsole: remote IP 80.204.235.230
netconsole: remote ethernet address 00:1b:d4:0a:43:d0
netconsole: device eth0 not up yet, forcing it
eth0:  setting full-duplex.
netconsole: carrier detect appears untrustworthy, waiting 4 seconds
console [netcon0] enabled
netconsole: network logging started
Filesystem "hda5": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda5
Ending clean XFS mount for filesystem: hda5
Filesystem "hda6": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda6
Starting XFS recovery on filesystem: hda6 (logdev: internal)
Ending XFS recovery on filesystem: hda6 (logdev: internal)
Filesystem "hda7": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda7
Ending clean XFS mount for filesystem: hda7
Filesystem "hda8": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda8
Ending clean XFS mount for filesystem: hda8
BUG: unable to handle kernel NULL pointer dereference at 00000224
IP: [<c02a07d3>] ip_route_input+0x33/0xe00
Oops: 0000 [#1]
Modules linked in: netconsole sch_sfq sch_htb nf_nat_pptp
nf_nat_proto_gre nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_ftp
nf_conntrack_ftp 3c59x mii

Pid: 0, comm: swapper Not tainted (2.6.24 #1)
EIP: 0060:[<c02a07d3>] EFLAGS: 00010202 CPU: 0
EIP is at ip_route_input+0x33/0xe00
EAX: 00000000 EBX: 4b0112ac ECX: 00000000 EDX: 00000040
ESI: 00000002 EDI: c90807e0 EBP: f4ee46c3 ESP: c038fc4c
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=c038e000 task=c03672e0 task.ti=c038e000)
Stack: 00000000 c9e151e0 c9d08840 c038fd24 c02a010d 00000000 00000000
c038fc8c
       00000000 c038fd60 c03de660 c9080360 c9f7da80 00000000 00000001
00000001
       00000002 00000001 00000000 f4ee46c3 1e232055 00000000 00000000
00000000
Call Trace:
 [<c02a010d>] __ip_route_output_key+0x4ad/0x840
 [<c02a04bc>] ip_route_output_flow+0x1c/0x200
 [<c02d38db>] ip_route_me_harder+0xab/0x290
 [<c02d87e2>] reject_tg+0x232/0x400
 [<c02d85b0>] reject_tg+0x0/0x400
 [<c02d793c>] ipt_do_table+0x2bc/0x4b0
 [<c02d7bc0>] ipt_hook+0x0/0x20
 [<c02968c7>] nf_iterate+0x57/0x80
 [<c02969c8>] nf_hook_slow+0x98/0xd0
 [<c02a4440>] ip_forward_finish+0x0/0x30
 [<c02a471a>] ip_forward+0x2aa/0x320
 [<c02a4440>] ip_forward_finish+0x0/0x30
 [<c02a307b>] ip_rcv_finish+0xeb/0x310
 [<c02a2f90>] ip_rcv_finish+0x0/0x310
 [<c02a3691>] ip_rcv+0x191/0x250
 [<c02a2f90>] ip_rcv_finish+0x0/0x310
 [<c02a3500>] ip_rcv+0x0/0x250
 [<c027dfb1>] netif_receive_skb+0x1f1/0x370
 [<c02806db>] process_backlog+0x5b/0xb0
 [<c0280224>] net_rx_action+0xb4/0x140
 [<c011a642>] __do_softirq+0x42/0x90
 [<c011a6b6>] do_softirq+0x26/0x30
 [<c010560a>] do_IRQ+0x4a/0x80
 [<c01039a7>] common_interrupt+0x23/0x28
 [<c012007b>] do_notify_parent+0x9b/0x170
 [<c0101c4d>] default_idle+0x4d/0x80
 [<c0101c00>] default_idle+0x0/0x80
 [<c0101a66>] cpu_idle+0x26/0x80
 =======================
Code: 81 ec b8 00 00 00 89 44 24 30 8b 84 24 d0 00 00 00 8b 54 24 30 0f
b6 8c 24 cc 00 00 00 8b 70 4c 8b 42 14 80 e1 1c 89 f2 c1 e2 05 <8b> b8
24 02 00 00 89 d8 31 ea 88 4c 24 53 e8 7a dd ff ff 8b 15
EIP: [<c02a07d3>] ip_route_input+0x33/0xe00 SS:ESP 0068:c038fc4c
Kernel panic - not syncing: Fatal exception in interrupt
atkbd.c: Unknown key released (translated set 2, code 0xe0 on
isa0060/serio0).
atkbd.c: Use 'setkeycodes e060 <keycode>' to make it known.
atkbd.c: Unknown key released (translated set 2, code 0xe0 on
isa0060/serio0).
atkbd.c: Use 'setkeycodes e060 <keycode>' to make it known.
Linux version 2.6.24 (root@Calimero) (gcc version 4.2.3) #1 Fri May 9
14:39:54 CEST 2008
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
 BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000dc000 - 00000000000e0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000000a000000 (usable)
 BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
160MB LOWMEM available.
Scan SMP from c0000000 for 1024 bytes.
Scan SMP from c009fc00 for 1024 bytes.
Scan SMP from c00f0000 for 65536 bytes.
Scan SMP from c009f800 for 1024 bytes.
Entering add_active_range(0, 0, 40960) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->    40960
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0:        0 ->    40960
On node 0 totalpages: 40960
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 288 pages used for memmap
  Normal zone: 36576 pages, LIFO batch:7
  Movable zone: 0 pages used for memmap
DMI 2.1 present.
ACPI: DMI detected: Hewlett-Packard
Allocating PCI resources starting at 10000000 (gap: 0a000000:f5ff0000)
Built 1 zonelists in Zone order, mobility grouping on.  Total pages:
40640
Kernel command line: auto BOOT_IMAGE=Linux ro root=301 slub_debug
Local APIC disabled by BIOS -- you can enable it with "lapic"
mapped APIC to ffffb000 (01141000)
Enabling fast FPU save and restore... done.
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 4096 bytes)
Detected 267.291 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 158944k/163840k available (2005k kernel code, 4464k reserved,
603k data, 176k init, 0k highmem)
virtual kernel memory layout:
    fixmap  : 0xfffb5000 - 0xfffff000   ( 296 kB)
    vmalloc : 0xca800000 - 0xfffb3000   ( 855 MB)
    lowmem  : 0xc0000000 - 0xca000000   ( 160 MB)
      .init : 0xc0390000 - 0xc03bc000   ( 176 kB)
      .data : 0xc02f557a - 0xc038c2c0   ( 603 kB)
      .text : 0xc0100000 - 0xc02f557a   (2005 kB)
Checking if this processor honours the WP bit even in supervisor
mode...Ok.
SLUB: Genslabs=11, HWalign=32, Order=0-1, MinObjects=4, CPUs=1, Nodes=1
Calibrating delay using timer specific routine.. 535.26 BogoMIPS
(lpj=1070531)
Mount-cache hash table entries: 512
CPU: L1 I cache: 16K, L1 D cache: 16K
Compat vDSO mapped to ffffe000.
CPU: Intel Celeron (Covington) stepping 00
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 0k freed
ACPI: Core revision 20070126
ACPI Exception (tbxface-0629): AE_NO_ACPI_TABLES, While loading
namespace from ACPI tables [20070126]
ACPI: Unable to load the System Description Tables
net_namespace: 448 bytes
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfda61, last bus=1
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
* Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
* this clock source is slow. Consider trying other clock sources
PCI quirk: region 6100-613f claimed by PIIX4 ACPI
PCI quirk: region 5f00-5f0f claimed by PIIX4 SMB
PCI: Using IRQ router PIIX/ICH [8086/7110] at 0000:00:07.0
Time: tsc clocksource has been installed.
PCI: Bridge: 0000:00:01.0
  IO window: b000-bfff
  MEM window: efe00000-efefffff
  PREFETCH window: e5c00000-e7cfffff
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
SGI XFS with no debug enabled
io scheduler noop registered (default)
Limiting direct PCI/PCI transfers.
Boot video device is 0000:01:00.0
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
PIIX4: IDE controller (0x8086:0x7111 rev 0x01) at  PCI slot 0000:00:07.1
PIIX4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio
Probing IDE interface ide0...
hda: QUANTUM FIREBALL EX3.2A, ATA DISK drive
hda: host max PIO4 wanted PIO255(auto-tune) selected PIO4
hda: UDMA/33 mode selected
Probing IDE interface ide1...
hdc: CRD-8160B, ATAPI CD/DVD-ROM drive
hdc: host max PIO4 wanted PIO255(auto-tune) selected PIO4
hdc: MWDMA2 mode selected
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 6306048 sectors (3228 MB) w/418KiB Cache, CHS=6256/16/63
hda: cache flushes not supported
 hda: hda1 hda2 < hda5 hda6 hda7 hda8 hda9 >
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
nf_conntrack version 0.5.0 (3072 buckets, 12288 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 15
Using IPI Shortcut mode
input: AT Raw Set 2 keyboard as
/devices/platform/i8042/serio0/input/input0
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda1
Starting XFS recovery on filesystem: hda1 (logdev: internal)
Ending XFS recovery on filesystem: hda1 (logdev: internal)
VFS: Mounted root (xfs filesystem) readonly.
Freeing unused kernel memory: 176k freed
Adding 330584k swap on /dev/hda9.  Priority:-1 extents:1 across:330584k
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
PCI: setting IRQ 10 as level-triggered
PCI: Found IRQ 10 for device 0000:00:09.0
3c59x: Donald Becker and others.
0000:00:09.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001dc00.
PCI: setting IRQ 11 as level-triggered
PCI: Found IRQ 11 for device 0000:00:0a.0
0000:00:0a.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001da00.
PCI: setting IRQ 9 as level-triggered
PCI: Found IRQ 9 for device 0000:00:0b.0
PCI: Sharing IRQ 9 with 0000:00:07.2
0000:00:0b.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001d800.
netconsole: local port 6665
netconsole: local IP 85.32.35.30
netconsole: interface eth0
netconsole: remote port 6666
netconsole: remote IP 80.204.235.230
netconsole: remote ethernet address 00:1b:d4:0a:43:d0
netconsole: device eth0 not up yet, forcing it
eth0:  setting full-duplex.
netconsole: carrier detect appears untrustworthy, waiting 4 seconds
console [netcon0] enabled
netconsole: network logging started
Filesystem "hda5": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda5
Ending clean XFS mount for filesystem: hda5
Filesystem "hda6": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda6
Starting XFS recovery on filesystem: hda6 (logdev: internal)
Ending XFS recovery on filesystem: hda6 (logdev: internal)
Filesystem "hda7": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda7
Ending clean XFS mount for filesystem: hda7
Filesystem "hda8": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda8
Ending clean XFS mount for filesystem: hda8
BUG: unable to handle kernel NULL pointer dereference at 00000224
IP: [<c02a07d3>] ip_route_input+0x33/0xe00
Oops: 0000 [#1]
Modules linked in: netconsole sch_sfq sch_htb cls_fw nf_nat_pptp
nf_nat_proto_gre nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_ftp
nf_conntrack_ftp 3c59x mii

Pid: 667, comm: rc.inetd Not tainted (2.6.24 #1)
EIP: 0060:[<c02a07d3>] EFLAGS: 00010202 CPU: 0
EIP is at ip_route_input+0x33/0xe00
EAX: 00000000 EBX: c50112ac ECX: 00000000 EDX: 00000040
ESI: 00000002 EDI: c9f4f120 EBP: 120110ac ESP: c9079bd4
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process rc.inetd (pid: 667, ti=c9078000 task=c9009580 task.ti=c9078000)
Stack: 00000000 c9e101e0 c9d14840 c9079cac c02a010d c03770b4 00000000
c9079c14
       b7f1a000 c9079ce8 c03de660 c9f4f5a0 c904ad20 00000060 00000001
00000001
       00000002 00000001 00000000 120110ac 1e232055 00000000 00000000
00000000
Call Trace:
 [<c02a010d>] __ip_route_output_key+0x4ad/0x840
 [<c02a04bc>] ip_route_output_flow+0x1c/0x200
 [<c02d38db>] ip_route_me_harder+0xab/0x290
 [<c02d87e2>] reject_tg+0x232/0x400
 [<c02d85b0>] reject_tg+0x0/0x400
 [<c02d793c>] ipt_do_table+0x2bc/0x4b0
 [<c02d7bc0>] ipt_hook+0x0/0x20
 [<c02968c7>] nf_iterate+0x57/0x80
 [<c02969c8>] nf_hook_slow+0x98/0xd0
 [<c02a4440>] ip_forward_finish+0x0/0x30
 [<c02a471a>] ip_forward+0x2aa/0x320
 [<c02a4440>] ip_forward_finish+0x0/0x30
 [<c02a307b>] ip_rcv_finish+0xeb/0x310
 [<c02a2f90>] ip_rcv_finish+0x0/0x310
 [<c02a3691>] ip_rcv+0x191/0x250
 [<c02a2f90>] ip_rcv_finish+0x0/0x310
 [<c02a3500>] ip_rcv+0x0/0x250
 [<c027dfb1>] netif_receive_skb+0x1f1/0x370
 [<c02806db>] process_backlog+0x5b/0xb0
 [<c0280224>] net_rx_action+0xb4/0x140
 [<c011a642>] __do_softirq+0x42/0x90
 [<c011a6b6>] do_softirq+0x26/0x30
 [<c010560a>] do_IRQ+0x4a/0x80
 [<c0151412>] check_object+0xc2/0x210
 [<c01039a7>] common_interrupt+0x23/0x28
 [<c0152480>] kmem_cache_alloc+0x30/0xa0
 [<c015d588>] getname+0x28/0xd0
 [<c015d588>] getname+0x28/0xd0
 [<c01541ee>] do_sys_open+0x1e/0xe0
 [<c01542ec>] sys_open+0x1c/0x20
 [<c0102ffe>] syscall_call+0x7/0xb
 [<c02f0000>] rest_init+0x20/0x40
 =======================
Code: 81 ec b8 00 00 00 89 44 24 30 8b 84 24 d0 00 00 00 8b 54 24 30 0f
b6 8c 24 cc 00 00 00 8b 70 4c 8b 42 14 80 e1 1c 89 f2 c1 e2 05 <8b> b8
24 02 00 00 89 d8 31 ea 88 4c 24 53 e8 7a dd ff ff 8b 15
EIP: [<c02a07d3>] ip_route_input+0x33/0xe00 SS:ESP 0068:c9079bd4
Kernel panic - not syncing: Fatal exception in interrupt
Linux version 2.6.24 (root@Calimero) (gcc version 4.2.3) #1 Fri May 9
14:39:54 CEST 2008
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
 BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000dc000 - 00000000000e0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000000a000000 (usable)
 BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
160MB LOWMEM available.
Scan SMP from c0000000 for 1024 bytes.
Scan SMP from c009fc00 for 1024 bytes.
Scan SMP from c00f0000 for 65536 bytes.
Scan SMP from c009f800 for 1024 bytes.
Entering add_active_range(0, 0, 40960) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->    40960
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0:        0 ->    40960
On node 0 totalpages: 40960
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 288 pages used for memmap
  Normal zone: 36576 pages, LIFO batch:7
  Movable zone: 0 pages used for memmap
DMI 2.1 present.
ACPI: DMI detected: Hewlett-Packard
Allocating PCI resources starting at 10000000 (gap: 0a000000:f5ff0000)
Built 1 zonelists in Zone order, mobility grouping on.  Total pages:
40640
Kernel command line: auto BOOT_IMAGE=Linux ro root=301 slub_debug
Local APIC disabled by BIOS -- you can enable it with "lapic"
mapped APIC to ffffb000 (01141000)
Enabling fast FPU save and restore... done.
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 4096 bytes)
Detected 267.276 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 158944k/163840k available (2005k kernel code, 4464k reserved,
603k data, 176k init, 0k highmem)
virtual kernel memory layout:
    fixmap  : 0xfffb5000 - 0xfffff000   ( 296 kB)
    vmalloc : 0xca800000 - 0xfffb3000   ( 855 MB)
    lowmem  : 0xc0000000 - 0xca000000   ( 160 MB)
      .init : 0xc0390000 - 0xc03bc000   ( 176 kB)
      .data : 0xc02f557a - 0xc038c2c0   ( 603 kB)
      .text : 0xc0100000 - 0xc02f557a   (2005 kB)
Checking if this processor honours the WP bit even in supervisor
mode...Ok.
SLUB: Genslabs=11, HWalign=32, Order=0-1, MinObjects=4, CPUs=1, Nodes=1
Calibrating delay using timer specific routine.. 535.25 BogoMIPS
(lpj=1070500)
Mount-cache hash table entries: 512
CPU: L1 I cache: 16K, L1 D cache: 16K
Compat vDSO mapped to ffffe000.
CPU: Intel Celeron (Covington) stepping 00
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 0k freed
ACPI: Core revision 20070126
ACPI Exception (tbxface-0629): AE_NO_ACPI_TABLES, While loading
namespace from ACPI tables [20070126]
ACPI: Unable to load the System Description Tables
net_namespace: 448 bytes
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfda61, last bus=1
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
* Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
* this clock source is slow. Consider trying other clock sources
PCI quirk: region 6100-613f claimed by PIIX4 ACPI
PCI quirk: region 5f00-5f0f claimed by PIIX4 SMB
PCI: Using IRQ router PIIX/ICH [8086/7110] at 0000:00:07.0
Time: tsc clocksource has been installed.
PCI: Bridge: 0000:00:01.0
  IO window: b000-bfff
  MEM window: efe00000-efefffff
  PREFETCH window: e5c00000-e7cfffff
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
SGI XFS with no debug enabled
io scheduler noop registered (default)
Limiting direct PCI/PCI transfers.
Boot video device is 0000:01:00.0
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
PIIX4: IDE controller (0x8086:0x7111 rev 0x01) at  PCI slot 0000:00:07.1
PIIX4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio
Probing IDE interface ide0...
hda: QUANTUM FIREBALL EX3.2A, ATA DISK drive
hda: host max PIO4 wanted PIO255(auto-tune) selected PIO4
hda: UDMA/33 mode selected
Probing IDE interface ide1...
hdc: CRD-8160B, ATAPI CD/DVD-ROM drive
hdc: host max PIO4 wanted PIO255(auto-tune) selected PIO4
hdc: MWDMA2 mode selected
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 6306048 sectors (3228 MB) w/418KiB Cache, CHS=6256/16/63
hda: cache flushes not supported
 hda: hda1 hda2 < hda5 hda6 hda7 hda8 hda9 >
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
nf_conntrack version 0.5.0 (3072 buckets, 12288 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 15
Using IPI Shortcut mode
input: AT Raw Set 2 keyboard as
/devices/platform/i8042/serio0/input/input0
input: AT Raw Set 2 keyboard as
/devices/platform/i8042/serio0/input/input1
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda1
Starting XFS recovery on filesystem: hda1 (logdev: internal)
Ending XFS recovery on filesystem: hda1 (logdev: internal)
VFS: Mounted root (xfs filesystem) readonly.
Freeing unused kernel memory: 176k freed
Adding 330584k swap on /dev/hda9.  Priority:-1 extents:1 across:330584k
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
PCI: setting IRQ 10 as level-triggered
PCI: Found IRQ 10 for device 0000:00:09.0
3c59x: Donald Becker and others.
0000:00:09.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001dc00.
PCI: setting IRQ 11 as level-triggered
PCI: Found IRQ 11 for device 0000:00:0a.0
0000:00:0a.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001da00.
PCI: setting IRQ 9 as level-triggered
PCI: Found IRQ 9 for device 0000:00:0b.0
PCI: Sharing IRQ 9 with 0000:00:07.2
0000:00:0b.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001d800.
netconsole: local port 6665
netconsole: local IP 85.32.35.30
netconsole: interface eth0
netconsole: remote port 6666
netconsole: remote IP 80.204.235.230
netconsole: remote ethernet address 00:1b:d4:0a:43:d0
netconsole: device eth0 not up yet, forcing it
eth0:  setting full-duplex.
netconsole: carrier detect appears untrustworthy, waiting 4 seconds
console [netcon0] enabled
netconsole: network logging started
Filesystem "hda5": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda5
Ending clean XFS mount for filesystem: hda5
Filesystem "hda6": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda6
Starting XFS recovery on filesystem: hda6 (logdev: internal)
Ending XFS recovery on filesystem: hda6 (logdev: internal)
Filesystem "hda7": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda7
Ending clean XFS mount for filesystem: hda7
Filesystem "hda8": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda8
Ending clean XFS mount for filesystem: hda8
BUG: unable to handle kernel NULL pointer dereference at 00000224
IP: [<c02a07d3>] ip_route_input+0x33/0xe00
Oops: 0000 [#1]
Modules linked in: netconsole sch_sfq sch_htb cls_fw nf_nat_pptp
nf_nat_proto_gre nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_ftp
nf_conntrack_ftp 3c59x mii

Pid: 0, comm: swapper Not tainted (2.6.24 #1)
EIP: 0060:[<c02a07d3>] EFLAGS: 00010202 CPU: 0
EIP is at ip_route_input+0x33/0xe00
EAX: 00000000 EBX: 2e0112ac ECX: 00000000 EDX: 00000040
ESI: 00000002 EDI: c90bf000 EBP: 020217ac ESP: c038fc4c
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process swapper (pid: 0, ti=c038e000 task=c03672e0 task.ti=c038e000)
Stack: 00000000 c9e201e0 c9d14840 c038fd24 c02a010d 00000002 00000000
c038fc8c
       00000003 c038fd60 c03de660 c90bf120 c9f6b1c0 00f6bb60 00000001
00000001
       00000002 00000001 00000000 020217ac 1e232055 00000000 00000000
00000000
Call Trace:
 [<c02a010d>] __ip_route_output_key+0x4ad/0x840
 [<c02a04bc>] ip_route_output_flow+0x1c/0x200
 [<c02d38db>] ip_route_me_harder+0xab/0x290
 [<c02d87e2>] reject_tg+0x232/0x400
 [<c02d85b0>] reject_tg+0x0/0x400
 [<c02d793c>] ipt_do_table+0x2bc/0x4b0
 [<c02d7bc0>] ipt_hook+0x0/0x20
 [<c02968c7>] nf_iterate+0x57/0x80
 [<c02969c8>] nf_hook_slow+0x98/0xd0
 [<c02a4440>] ip_forward_finish+0x0/0x30
 [<c02a471a>] ip_forward+0x2aa/0x320
 [<c02a4440>] ip_forward_finish+0x0/0x30
 [<c02a307b>] ip_rcv_finish+0xeb/0x310
 [<c02a2f90>] ip_rcv_finish+0x0/0x310
 [<c02a3691>] ip_rcv+0x191/0x250
 [<c02a2f90>] ip_rcv_finish+0x0/0x310
 [<c02a3500>] ip_rcv+0x0/0x250
 [<c027dfb1>] netif_receive_skb+0x1f1/0x370
 [<c02806db>] process_backlog+0x5b/0xb0
 [<c0280224>] net_rx_action+0xb4/0x140
 [<c011a642>] __do_softirq+0x42/0x90
 [<c011a6b6>] do_softirq+0x26/0x30
 [<c010560a>] do_IRQ+0x4a/0x80
 [<c01039a7>] common_interrupt+0x23/0x28
 [<c012007b>] do_notify_parent+0x9b/0x170
 [<c0101c4d>] default_idle+0x4d/0x80
 [<c0101c00>] default_idle+0x0/0x80
 [<c0101a66>] cpu_idle+0x26/0x80
 =======================
Code: 81 ec b8 00 00 00 89 44 24 30 8b 84 24 d0 00 00 00 8b 54 24 30 0f
b6 8c 24 cc 00 00 00 8b 70 4c 8b 42 14 80 e1 1c 89 f2 c1 e2 05 <8b> b8
24 02 00 00 89 d8 31 ea 88 4c 24 53 e8 7a dd ff ff 8b 15
EIP: [<c02a07d3>] ip_route_input+0x33/0xe00 SS:ESP 0068:c038fc4c
Kernel panic - not syncing: Fatal exception in interrupt

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Herbert Xu]

On Mon, May 12, 2008 at 09:14:54AM +0200, Marco Berizzi wrote:
> 
> PS: linux version shown by dmesg is 2.6.24 but it
> is the results of 'git bisect good v2.6.24' and
> 'git bisect bad v2.6.25'

So what's the changeset hash for the source that produced this
kernel?

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Herbert Xu wrote:

> On Mon, May 12, 2008 at 09:14:54AM +0200, Marco Berizzi wrote:
> >
> > PS: linux version shown by dmesg is 2.6.24 but it
> > is the results of 'git bisect good v2.6.24' and
> > 'git bisect bad v2.6.25'
>
> So what's the changeset hash for the source that produced this
> kernel?

root@Calimero:/tmp/GIT/herbert# git bisect good v2.6.24
You need to start by "git bisect start"
Do you want me to do it for you [Y/n]? Y
root@Calimero:/tmp/GIT/herbert# git bisect bad v2.6.25
Bisecting: 6353 revisions left to test after this
[dd5f5fed6c9458a7aa81eeef3732cc3a9891cfdf] Merge branch 'audit.b46' of
git://git.kernel.org/pub/scm/linux/kernel/git/viro/audit-current

Then I have copied from 2.6.25 two files:

net/ipv4/xfrm4_tunnel.c and ipcomp.c because
of these bugs:

http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=commit;h=11b47c8828d4cd1df21636719603784ec5e26067
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=commit;h=363c11d7e1c2b2cc30e33416a518cea5ef9e0cc8
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=commit;h=cefe34bea77e194fd6b6a7a062e1620af2eef69f

and finally I have run 'make menuconfig' and
'make bzImage modules modules_install'

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Marco Berizzi wrote:

> Herbert Xu wrote:
>
> > On Mon, May 12, 2008 at 09:14:54AM +0200, Marco Berizzi wrote:
> > >
> > > PS: linux version shown by dmesg is 2.6.24 but it
> > > is the results of 'git bisect good v2.6.24' and
> > > 'git bisect bad v2.6.25'
> >
> > So what's the changeset hash for the source that produced this
> > kernel?
>
> root@Calimero:/tmp/GIT/herbert# git bisect good v2.6.24
> You need to start by "git bisect start"
> Do you want me to do it for you [Y/n]? Y
> root@Calimero:/tmp/GIT/herbert# git bisect bad v2.6.25
> Bisecting: 6353 revisions left to test after this
> [dd5f5fed6c9458a7aa81eeef3732cc3a9891cfdf] Merge branch 'audit.b46' of
> git://git.kernel.org/pub/scm/linux/kernel/git/viro/audit-current
>
> Then I have copied from 2.6.25 two files:
>
> net/ipv4/xfrm4_tunnel.c and ipcomp.c because
> of these bugs:
>
>
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=commit;h=11b47c8828d4cd1df21636719603784ec5e26067
>
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=commit;h=363c11d7e1c2b2cc30e33416a518cea5ef9e0cc8
>
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=commit;h=cefe34bea77e194fd6b6a7a062e1620af2eef69f
>
> and finally I have run 'make menuconfig' and
> 'make bzImage modules modules_install'

ok, I have git bisected again:

root@Calimero:/tmp/GIT/herbert# git bisect bad
Bisecting: 3176 revisions left to test after this
[fde3571fd8613483f1203d11394ae316c6b79a03] iwlwifi: avoid firmware
command sending if rfkill is enabled

and then copied always the two files from 2.6.25
xfrm4_tunnel.c and ipcomp.c

now this kernel is up for 4 hours (at least it
doesn't crash at startup)
I will keep you updated.
Thanks for the feedback Herbert.

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Marco Berizzi wrote:

> Marco Berizzi wrote:
>
> > Herbert Xu wrote:
> >
> > > On Mon, May 12, 2008 at 09:14:54AM +0200, Marco Berizzi wrote:
> > > >
> > > > PS: linux version shown by dmesg is 2.6.24 but it
> > > > is the results of 'git bisect good v2.6.24' and
> > > > 'git bisect bad v2.6.25'
> > >
> > > So what's the changeset hash for the source that produced this
> > > kernel?
> >
> > root@Calimero:/tmp/GIT/herbert# git bisect good v2.6.24
> > You need to start by "git bisect start"
> > Do you want me to do it for you [Y/n]? Y
> > root@Calimero:/tmp/GIT/herbert# git bisect bad v2.6.25
> > Bisecting: 6353 revisions left to test after this
> > [dd5f5fed6c9458a7aa81eeef3732cc3a9891cfdf] Merge branch 'audit.b46'
of
> > git://git.kernel.org/pub/scm/linux/kernel/git/viro/audit-current
> >
> > Then I have copied from 2.6.25 two files:
> >
> > net/ipv4/xfrm4_tunnel.c and ipcomp.c because
> > of these bugs:
> >
> >
>
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=commit;h=11b47c8828d4cd1df21636719603784ec5e26067
> >
>
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=commit;h=363c11d7e1c2b2cc30e33416a518cea5ef9e0cc8
> >
>
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=commit;h=cefe34bea77e194fd6b6a7a062e1620af2eef69f
> >
> > and finally I have run 'make menuconfig' and
> > 'make bzImage modules modules_install'
>
> ok, I have git bisected again:
>
> root@Calimero:/tmp/GIT/herbert# git bisect bad
> Bisecting: 3176 revisions left to test after this
> [fde3571fd8613483f1203d11394ae316c6b79a03] iwlwifi: avoid firmware
> command sending if rfkill is enabled
>
> and then copied always the two files from 2.6.25
> xfrm4_tunnel.c and ipcomp.c
>
> now this kernel is up for 4 hours (at least it
> doesn't crash at startup)
> I will keep you updated.
> Thanks for the feedback Herbert.

NAK: it crashed after 4 hours and few minutes with
this message:

BUG: unable to handle kernel paging request at virtual address 6b6b6b6f
printing eip: ca8931be *pde = 00000000
Oops: 0002 [#1]
Modules linked in: netconsole sch_sfq sch_htb cls_fw nf_nat_pptp
nf_nat_proto_gre nf_conntrack_pptp nf_conntrack_proto_gre nf_nat_ftp
nf_conntrack_ftp 3c59x mii
Pid: 881, comm: squid Not tainted (2.6.24 #1)
EIP: 0060:[<ca8931be>] EFLAGS: 00010203 CPU: 0
EIP is at sfq_dequeue+0x6e/0x1f0 [sch_sfq]
EAX: 6b6b6b6b EBX: c91d0d14 ECX: c91d0d08 EDX: 000005e8
ESI: c91d0080 EDI: c559c540 EBP: c91d0000 ESP: c9149e9c
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process squid (pid: 881, ti=c9148000 task=c9fbd020 task.ti=c9148000)
Stack: c559c540 7e0039c1 00000000 fffffffe c913ed20 00000000 ca89b146
c905b468
       c2468012 c9e3d840 00000282 c905b388 00000282 000005ea 00000000
c905b168
       c905b288 000006e0 c905b060 00000000 c905b0e0 00000007 5c9a608b
00000003
Call Trace:
 [<ca89b146>] htb_dequeue+0x226/0x860 [sch_htb]
 [<c028189b>] dev_hard_start_xmit+0x1cb/0x230
 [<c0290bf5>] __qdisc_run+0xd5/0x130
 [<c0283129>] net_tx_action+0x39/0xd0
 [<c0283129>] net_tx_action+0x39/0xd0
 [<c0283177>] net_tx_action+0x87/0xd0
 [<c0118ad2>] __do_softirq+0x42/0x90
 [<c0118b46>] do_softirq+0x26/0x30
 [<c0104a9a>] do_IRQ+0x4a/0x80
 [<c0154891>] sys_read+0x41/0x70
 [<c010303f>] common_interrupt+0x23/0x28
 =======================
Code: 8d 14 85 00 00 00 00 8d 8c 16 a0 06 00 00 8b 41 0c 8d 59 0c 39 d8
89 04 24 0f 84 72 01 00 00 8b 3c 24 8b 07 ff 8c 16 b4 06 00 00 <89> 58
04 89 41 0c c7 47 04 00 00 00 00 c7 07 00 00 00 00 0f b6
EIP: [<ca8931be>] sfq_dequeue+0x6e/0x1f0 [sch_sfq] SS:ESP 0068:c9149e9c
Kernel panic - not syncing: Fatal exception in interrupt
Rebooting in 5 seconds..Linux version 2.6.24 (root@Calimero) (gcc
version 4.2.3) #1 Mon May 12 11:12:56 CEST 2008
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
 BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
 BIOS-e820: 00000000000dc000 - 00000000000e0000 (reserved)
 BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
 BIOS-e820: 0000000000100000 - 000000000a000000 (usable)
 BIOS-e820: 00000000ffff0000 - 0000000100000000 (reserved)
160MB LOWMEM available.
Entering add_active_range(0, 0, 40960) 0 entries of 256 used
Zone PFN ranges:
  DMA             0 ->     4096
  Normal       4096 ->    40960
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0:        0 ->    40960
On node 0 totalpages: 40960
  DMA zone: 32 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 4064 pages, LIFO batch:0
  Normal zone: 288 pages used for memmap
  Normal zone: 36576 pages, LIFO batch:7
  Movable zone: 0 pages used for memmap
DMI 2.1 present.
ACPI: DMI detected: Hewlett-Packard
Allocating PCI resources starting at 10000000 (gap: 0a000000:f5ff0000)
Built 1 zonelists in Zone order, mobility grouping on.  Total pages:
40640
Kernel command line: auto BOOT_IMAGE=Linux ro root=301 slub_debug
Local APIC disabled by BIOS -- you can enable it with "lapic"
mapped APIC to ffffb000 (01141000)
Enabling fast FPU save and restore... done.
Initializing CPU#0
PID hash table entries: 1024 (order: 10, 4096 bytes)
Detected 267.281 MHz processor.
Console: colour VGA+ 80x25
console [tty0] enabled
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 158956k/163840k available (2010k kernel code, 4452k reserved,
606k data, 160k init, 0k highmem)
virtual kernel memory layout:
    fixmap  : 0xfffb5000 - 0xfffff000   ( 296 kB)
    vmalloc : 0xca800000 - 0xfffb3000   ( 855 MB)
    lowmem  : 0xc0000000 - 0xca000000   ( 160 MB)
      .init : 0xc0392000 - 0xc03ba000   ( 160 kB)
      .data : 0xc02f687e - 0xc038e280   ( 606 kB)
      .text : 0xc0100000 - 0xc02f687e   (2010 kB)
Checking if this processor honours the WP bit even in supervisor mode...
Ok.
SLUB: Genslabs=11, HWalign=32, Order=0-1, MinObjects=4, CPUs=1, Nodes=1
Calibrating delay using timer specific routine.. 535.28 BogoMIPS
(lpj=1070570)
Mount-cache hash table entries: 512
CPU: After generic identify, caps: 0183f9ff 00000000 00000000 00000000
00000000 00000000 00000000 00000000
CPU: L1 I cache: 16K, L1 D cache: 16K
CPU: After all inits, caps: 0183f9ff 00000000 00000000 00000040 00000000
00000000 00000000 00000000
Compat vDSO mapped to ffffe000.
CPU: Intel Celeron (Covington) stepping 00
Checking 'hlt' instruction... OK.
Freeing SMP alternatives: 0k freed
ACPI: Core revision 20070126
ACPI Exception (tbxface-0629): AE_NO_ACPI_TABLES, While loading
namespace from ACPI tables [20070126]
ACPI: Unable to load the System Description Tables
net_namespace: 128 bytes
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfda61, last bus=1
PCI: Using configuration type 1
Setting up standard PCI resources
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
* Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
* this clock source is slow. Consider trying other clock sources
PCI quirk: region 6100-613f claimed by PIIX4 ACPI
PCI quirk: region 5f00-5f0f claimed by PIIX4 SMB
PCI: Using IRQ router PIIX/ICH [8086/7110] at 0000:00:07.0
Time: tsc clocksource has been installed.
PCI: Bridge: 0000:00:01.0
  IO window: b000-bfff
  MEM window: efe00000-efefffff
  PREFETCH window: e5c00000-e7cfffff
NET: Registered protocol family 2
IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
TCP established hash table entries: 8192 (order: 4, 65536 bytes)
TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
TCP: Hash tables configured (established 8192 bind 8192)
TCP reno registered
SGI XFS with no debug enabled
io scheduler noop registered (default)
Limiting direct PCI/PCI transfers.
Boot video device is 0000:01:00.0
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with
idebus=xx
PIIX4: IDE controller (0x8086:0x7111 rev 0x01) at  PCI slot 0000:00:07.1
PIIX4: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:DMA, hdd:pio
Probing IDE interface ide0...
hda: QUANTUM FIREBALL EX3.2A, ATA DISK drive
hda: host max PIO4 wanted PIO255(auto-tune) selected PIO4
hda: UDMA/33 mode selected
Probing IDE interface ide1...
hdc: CRD-8160B, ATAPI CD/DVD-ROM drive
hdc: host max PIO4 wanted PIO255(auto-tune) selected PIO4
hdc: MWDMA2 mode selected
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: max request size: 128KiB
hda: 6306048 sectors (3228 MB) w/418KiB Cache, CHS=6256/16/63
hda: cache flushes not supported
 hda: hda1 hda2 < hda5 hda6 hda7 hda8 hda9 >
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
mice: PS/2 mouse device common for all mice
nf_conntrack version 0.5.0 (3072 buckets, 12288 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
TCP cubic registered
Initializing XFRM netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 15
Using IPI Shortcut mode
input: AT Translated Set 2 keyboard as
/devices/platform/i8042/serio0/input/input0
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda1
Starting XFS recovery on filesystem: hda1 (logdev: internal)
Ending XFS recovery on filesystem: hda1 (logdev: internal)
VFS: Mounted root (xfs filesystem) readonly.
Freeing unused kernel memory: 160k freed
Adding 330584k swap on /dev/hda9.  Priority:-1 extents:1 across:330584k
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
Filesystem "hda1": Disabling barriers, not supported by the underlying
device
PCI: setting IRQ 10 as level-triggered
PCI: Found IRQ 10 for device 0000:00:09.0
3c59x: Donald Becker and others.
0000:00:09.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001dc00.
PCI: setting IRQ 11 as level-triggered
PCI: Found IRQ 11 for device 0000:00:0a.0
0000:00:0a.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001da00.
PCI: setting IRQ 9 as level-triggered
PCI: Found IRQ 9 for device 0000:00:0b.0
PCI: Sharing IRQ 9 with 0000:00:07.2
0000:00:0b.0: 3Com PCI 3c905 Boomerang 100baseTx at 0001d800.
netconsole: local port 6665
netconsole: local IP 85.32.35.30
netconsole: interface eth0
netconsole: remote port 6666
netconsole: remote IP 80.204.235.230
netconsole: remote ethernet address 00:1b:d4:0a:43:d0
netconsole: device eth0 not up yet, forcing it
eth0:  setting full-duplex.
netconsole: carrier detect appears untrustworthy, waiting 4 seconds
console [netcon0] enabled
netconsole: network logging started
Filesystem "hda5": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda5
Starting XFS recovery on filesystem: hda5 (logdev: internal)
Ending XFS recovery on filesystem: hda5 (logdev: internal)
Filesystem "hda6": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda6
Starting XFS recovery on filesystem: hda6 (logdev: internal)
Ending XFS recovery on filesystem: hda6 (logdev: internal)
Filesystem "hda7": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda7
Ending clean XFS mount for filesystem: hda7
Filesystem "hda8": Disabling barriers, not supported by the underlying
device
XFS mounting filesystem hda8
Starting XFS recovery on filesystem: hda8 (logdev: internal)
Ending XFS recovery on filesystem: hda8 (logdev: internal)

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Marco Berizzi wrote:

> Marco Berizzi wrote:
>
> > Marco Berizzi wrote:
> >
> > > Herbert Xu wrote:
> > >
> > > > On Mon, May 12, 2008 at 09:14:54AM +0200, Marco Berizzi wrote:
> > > > >
> > > > > PS: linux version shown by dmesg is 2.6.24 but it
> > > > > is the results of 'git bisect good v2.6.24' and
> > > > > 'git bisect bad v2.6.25'
> > > >
> > > > So what's the changeset hash for the source that produced this
> > > > kernel?

This kernel (see below git bisect log) has not
been crashed after more than 17 hours. I want
to test it till this evening.
This is the git bisect log:

root@Calimero:/tmp/GIT/herbert# git bisect log
git-bisect start
# good: [49914084e797530d9baaf51df9eda77babc98fa8] Linux 2.6.24
git-bisect good 49914084e797530d9baaf51df9eda77babc98fa8
# bad: [4b119e21d0c66c22e8ca03df05d9de623d0eb50f] Linux 2.6.25
git-bisect bad 4b119e21d0c66c22e8ca03df05d9de623d0eb50f
# bad: [dd5f5fed6c9458a7aa81eeef3732cc3a9891cfdf] Merge branch
'audit.b46' of
git://git.kernel.org/pub/scm/linux/kernel/git/viro/audit-current
git-bisect bad dd5f5fed6c9458a7aa81eeef3732cc3a9891cfdf
# bad: [fde3571fd8613483f1203d11394ae316c6b79a03] iwlwifi: avoid
firmware command sending if rfkill is enabled
git-bisect bad fde3571fd8613483f1203d11394ae316c6b79a03
# good: [1c7c2cdec3a6b2873439096983794a550d7ff65b] Merge
git://git.kernel.org/pub/scm/linux/kernel/git/bart/ide-2.6
git-bisect good 1c7c2cdec3a6b2873439096983794a550d7ff65b
# bad: [4c37799ccf6c722e0dad6a0677af22d1c23fb897] [NETFILTER]: Use
lowercase names for matches in Kconfig
git-bisect bad 4c37799ccf6c722e0dad6a0677af22d1c23fb897

I hope this helps.

PS: I always replace from 2.6.25 ipcomp.c and
xfrm4_tunnel.c

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

game over :-((

I was going forward with git bisect but now kernel
doesn't compile anymore
  LD      drivers/misc/built-in.o
  CC      drivers/net/Space.o
  CC      drivers/net/loopback.o
In file included from include/net/sock.h:50,
                 from drivers/net/loopback.c:53:
include/linux/pcounter.h: In function 'pcounter_add':
include/linux/pcounter.h:87: error: 'struct pcounter' has no member
named 'value'
make[2]: *** [drivers/net/loopback.o] Error 1
make[1]: *** [drivers/net] Error 2

Here is my git bisect log output:

git-bisect start
# good: [49914084e797530d9baaf51df9eda77babc98fa8] Linux 2.6.24
git-bisect good 49914084e797530d9baaf51df9eda77babc98fa8
# bad: [4b119e21d0c66c22e8ca03df05d9de623d0eb50f] Linux 2.6.25
git-bisect bad 4b119e21d0c66c22e8ca03df05d9de623d0eb50f
# bad: [dd5f5fed6c9458a7aa81eeef3732cc3a9891cfdf] Merge branch
'audit.b46' of
git://git.kernel.org/pub/scm/linux/kernel/git/viro/audit-current
git-bisect bad dd5f5fed6c9458a7aa81eeef3732cc3a9891cfdf
# bad: [fde3571fd8613483f1203d11394ae316c6b79a03] iwlwifi: avoid
firmware command sending if rfkill is enabled
git-bisect bad fde3571fd8613483f1203d11394ae316c6b79a03
# good: [1c7c2cdec3a6b2873439096983794a550d7ff65b] Merge
git://git.kernel.org/pub/scm/linux/kernel/git/bart/ide-2.6
git-bisect good 1c7c2cdec3a6b2873439096983794a550d7ff65b
# bad: [4c37799ccf6c722e0dad6a0677af22d1c23fb897] [NETFILTER]: Use
lowercase names for matches in Kconfig
git-bisect bad 4c37799ccf6c722e0dad6a0677af22d1c23fb897
# good: [f4798748dee00c807a63f5518f08b3df161e0f6d] Merge branch
'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/hid
git-bisect good f4798748dee00c807a63f5518f08b3df161e0f6d
# good: [9c55e01c0cc835818475a6ce8c4d684df9949ac8] [TCP]: Splice receive
support.
git-bisect good 9c55e01c0cc835818475a6ce8c4d684df9949ac8

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Herbert Xu]

On Wed, May 14, 2008 at 02:03:05PM +0200, Marco Berizzi wrote:
> game over :-((
> 
> I was going forward with git bisect but now kernel
> doesn't compile anymore
>   LD      drivers/misc/built-in.o
>   CC      drivers/net/Space.o
>   CC      drivers/net/loopback.o
> In file included from include/net/sock.h:50,
>                  from drivers/net/loopback.c:53:
> include/linux/pcounter.h: In function 'pcounter_add':
> include/linux/pcounter.h:87: error: 'struct pcounter' has no member
> named 'value'
> make[2]: *** [drivers/net/loopback.o] Error 1
> make[1]: *** [drivers/net] Error 2

This patch should let you test that kernel:

commit e7d0362dd41e760f340c1b500646cc92522bd9d5
Author: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Date:   Mon Nov 26 23:34:54 2007 +0800

    [PCOUNTER] Fix build error without CONFIG_SMP
    
    I keep getting this build error and couldn't find anyone fixing
    it in archives. ...Maybe all net developers except me build
    just SMP kernels :-).
    
    In file included from include/net/sock.h:50,
                     from ipc/mqueue.c:35:
    include/linux/pcounter.h: In function 'pcounter_add':
    include/linux/pcounter.h:87: error: 'struct pcounter' has no
    member named 'value'
    make[1]: *** [ipc/mqueue.o] Error 1
    make: *** [ipc] Error 2
    
    Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
    Acked-by: Arnaldo Carvalho de Melo <acme@redhat.com>
    Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
    Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/include/linux/pcounter.h b/include/linux/pcounter.h
index 620aade..9c4760a 100644
--- a/include/linux/pcounter.h
+++ b/include/linux/pcounter.h
@@ -84,7 +84,7 @@ static inline int pcounter_getval(const struct pcounter *self)
 
 static inline void pcounter_add(struct pcounter *self, int inc)
 {
-	self->value += inc;
+	self->val += inc;
 }
 
 static inline int pcounter_getval(const struct pcounter *self)

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Herbert Xu wrote:

> This patch should let you test that kernel:

yes it works. Thanks.

[IPSEC]: Use the correct ip_local_out function

[Herbert Xu]

On Wed, May 14, 2008 at 10:19:57AM +0200, Marco Berizzi wrote:
> 
> I hope this helps.

OK found the problem, it was my fault after all :)

Dave, this patch needs to go into stable too.

[IPSEC]: Use the correct ip_local_out function

Because the IPsec output function xfrm_output_resume does its
own dst_output call it should always call __ip_local_output
instead of ip_local_output as the latter may invoke dst_output
directly.  Otherwise the return values from nf_hook and dst_output
may clash as they both use the value 1 but for different purposes.

When that clash occurs this can cause a packet to be used after
it has been freed which usually leads to a crash.  Because the
offending value is only returned from dst_output with qdiscs
such as HTB, this bug is normally not visible.

Thanks to Marco Berizzi for his perseverance in tracking this
down.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index 92f90ae..df41026 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -160,7 +160,7 @@ static struct dst_ops ipv4_dst_ops = {
 	.negative_advice =	ipv4_negative_advice,
 	.link_failure =		ipv4_link_failure,
 	.update_pmtu =		ip_rt_update_pmtu,
-	.local_out =		ip_local_out,
+	.local_out =		__ip_local_out,
 	.entry_size =		sizeof(struct rtable),
 	.entries =		ATOMIC_INIT(0),
 };
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 12bba08..849b78a 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -109,7 +109,7 @@ static struct dst_ops ip6_dst_ops_template = {
 	.negative_advice	=	ip6_negative_advice,
 	.link_failure		=	ip6_link_failure,
 	.update_pmtu		=	ip6_rt_update_pmtu,
-	.local_out		=	ip6_local_out,
+	.local_out		=	__ip6_local_out,
 	.entry_size		=	sizeof(struct rt6_info),
 	.entries		=	ATOMIC_INIT(0),
 };

Re: [IPSEC]: Use the correct ip_local_out function

[Marco Berizzi]

Herbert Xu wrote:

> On Wed, May 14, 2008 at 10:19:57AM +0200, Marco Berizzi wrote:
> >
> > I hope this helps.
>
> OK found the problem, it was my fault after all :)
>
> Dave, this patch needs to go into stable too.
>
> [IPSEC]: Use the correct ip_local_out function
>
> Because the IPsec output function xfrm_output_resume does its
> own dst_output call it should always call __ip_local_output
> instead of ip_local_output as the latter may invoke dst_output
> directly.  Otherwise the return values from nf_hook and dst_output
> may clash as they both use the value 1 but for different purposes.
>
> When that clash occurs this can cause a packet to be used after
> it has been freed which usually leads to a crash.  Because the
> offending value is only returned from dst_output with qdiscs
> such as HTB, this bug is normally not visible.
>
> Thanks to Marco Berizzi for his perseverance in tracking this
> down.

Herbert,
many many thanks to you for fixing this bug.
I was going on with git bisect, but I was not
able anymore to reproduce the bug after four
git bisect step.
I'm going to apply immediately your patch to
2.6.25.4
Just for record, this is my last git bisect log:
root@Venus:/tmp/GIT/my2.6.25.y# git bisect log
git-bisect start
# good: [49914084e797530d9baaf51df9eda77babc98fa8] Linux 2.6.24
git-bisect good 49914084e797530d9baaf51df9eda77babc98fa8
# bad: [4b119e21d0c66c22e8ca03df05d9de623d0eb50f] Linux 2.6.25
git-bisect bad 4b119e21d0c66c22e8ca03df05d9de623d0eb50f
# bad: [dd5f5fed6c9458a7aa81eeef3732cc3a9891cfdf] Merge branch
'audit.b46' of
git://git.kernel.org/pub/scm/linux/kernel/git/viro/audit-current
git-bisect bad dd5f5fed6c9458a7aa81eeef3732cc3a9891cfdf
# bad: [fde3571fd8613483f1203d11394ae316c6b79a03] iwlwifi: avoid
firmware command sending if rfkill is enabled
git-bisect bad fde3571fd8613483f1203d11394ae316c6b79a03
# good: [1c7c2cdec3a6b2873439096983794a550d7ff65b] Merge
git://git.kernel.org/pub/scm/linux/kernel/git/bart/ide-2.6
git-bisect good 1c7c2cdec3a6b2873439096983794a550d7ff65b
# bad: [4c37799ccf6c722e0dad6a0677af22d1c23fb897] [NETFILTER]: Use
lowercase names for matches in Kconfig
git-bisect bad 4c37799ccf6c722e0dad6a0677af22d1c23fb897
# good: [f4798748dee00c807a63f5518f08b3df161e0f6d] Merge branch
'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jikos/hid
git-bisect good f4798748dee00c807a63f5518f08b3df161e0f6d
# good: [9c55e01c0cc835818475a6ce8c4d684df9949ac8] [TCP]: Splice receive
support.
git-bisect good 9c55e01c0cc835818475a6ce8c4d684df9949ac8
# good: [8e8c71f1ab0ca1c4e74efad14533b991524dcb6c] [DCCP]: Honour and
make use of shutdown option set by user
git-bisect good 8e8c71f1ab0ca1c4e74efad14533b991524dcb6c
# good: [dd88590995de7c7ce108718a9ad52b3832e77814] [DECNET]: Remove
extra memset from dn_fib_check_nh
git-bisect good dd88590995de7c7ce108718a9ad52b3832e77814

Re: [IPSEC]: Use the correct ip_local_out function

[David Miller]

From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Tue, 20 May 2008 17:25:11 +0800

> On Wed, May 14, 2008 at 10:19:57AM +0200, Marco Berizzi wrote:
> > 
> > I hope this helps.
> 
> OK found the problem, it was my fault after all :)
> 
> Dave, this patch needs to go into stable too.
> 
> [IPSEC]: Use the correct ip_local_out function
> 
> Because the IPsec output function xfrm_output_resume does its
> own dst_output call it should always call __ip_local_output
> instead of ip_local_output as the latter may invoke dst_output
> directly.  Otherwise the return values from nf_hook and dst_output
> may clash as they both use the value 1 but for different purposes.
> 
> When that clash occurs this can cause a packet to be used after
> it has been freed which usually leads to a crash.  Because the
> offending value is only returned from dst_output with qdiscs
> such as HTB, this bug is normally not visible.
> 
> Thanks to Marco Berizzi for his perseverance in tracking this
> down.
> 
> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Applied and queued to -stable, thanks!

Re: [IPSEC]: Use the correct ip_local_out function

[Marco Berizzi]

David Miller wrote:

> From: Herbert Xu <herbert@gondor.apana.org.au>
> Date: Tue, 20 May 2008 17:25:11 +0800
>
> > On Wed, May 14, 2008 at 10:19:57AM +0200, Marco Berizzi wrote:
> > >
> > > I hope this helps.
> >
> > OK found the problem, it was my fault after all :)
> >
> > Dave, this patch needs to go into stable too.
> >
> > [IPSEC]: Use the correct ip_local_out function
> >
> > Because the IPsec output function xfrm_output_resume does its
> > own dst_output call it should always call __ip_local_output
> > instead of ip_local_output as the latter may invoke dst_output
> > directly.  Otherwise the return values from nf_hook and dst_output
> > may clash as they both use the value 1 but for different purposes.
> >
> > When that clash occurs this can cause a packet to be used after
> > it has been freed which usually leads to a crash.  Because the
> > offending value is only returned from dst_output with qdiscs
> > such as HTB, this bug is normally not visible.
> >
> > Thanks to Marco Berizzi for his perseverance in tracking this
> > down.
> >
> > Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
>
> Applied and queued to -stable, thanks!

Just a confirmation message that this bug has been fixed
(one week uptime).

Re: [patch 00/50] 2.6.25.6 -stable review

[Marco Berizzi]

David Miller wrote:

> From: Herbert Xu <herbert@gondor.apana.org.au>
> Date: Tue, 20 May 2008 17:25:11 +0800
> 
>> On Wed, May 14, 2008 at 10:19:57AM +0200, Marco Berizzi wrote:
>> > 
>> > I hope this helps.
>> 
>> OK found the problem, it was my fault after all :)
>> 
>> Dave, this patch needs to go into stable too.
>> 
>> [IPSEC]: Use the correct ip_local_out function
>> 
>> Because the IPsec output function xfrm_output_resume does its
>> own dst_output call it should always call __ip_local_output
>> instead of ip_local_output as the latter may invoke dst_output
>> directly.  Otherwise the return values from nf_hook and dst_output
>> may clash as they both use the value 1 but for different purposes.
>> 
>> When that clash occurs this can cause a packet to be used after
>> it has been freed which usually leads to a crash.  Because the
>> offending value is only returned from dst_output with qdiscs
>> such as HTB, this bug is normally not visible.
>> 
>> Thanks to Marco Berizzi for his perseverance in tracking this
>> down.
>> 
>> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
> 
> Applied and queued to -stable, thanks!

Hi David,

I don't see this patch in Chris 2.6.25.6 -stable review message.

Re: [patch 00/50] 2.6.25.6 -stable review

[Willy Tarreau]

On Sat, Jun 07, 2008 at 10:27:58PM +0200, Marco Berizzi wrote:
> David Miller wrote:
> 
> > From: Herbert Xu <herbert@gondor.apana.org.au>
> > Date: Tue, 20 May 2008 17:25:11 +0800
> > 
> >> On Wed, May 14, 2008 at 10:19:57AM +0200, Marco Berizzi wrote:
> >> > 
> >> > I hope this helps.
> >> 
> >> OK found the problem, it was my fault after all :)
> >> 
> >> Dave, this patch needs to go into stable too.
> >> 
> >> [IPSEC]: Use the correct ip_local_out function
> >> 
> >> Because the IPsec output function xfrm_output_resume does its
> >> own dst_output call it should always call __ip_local_output
> >> instead of ip_local_output as the latter may invoke dst_output
> >> directly.  Otherwise the return values from nf_hook and dst_output
> >> may clash as they both use the value 1 but for different purposes.
> >> 
> >> When that clash occurs this can cause a packet to be used after
> >> it has been freed which usually leads to a crash.  Because the
> >> offending value is only returned from dst_output with qdiscs
> >> such as HTB, this bug is normally not visible.
> >> 
> >> Thanks to Marco Berizzi for his perseverance in tracking this
> >> down.
> >> 
> >> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
> > 
> > Applied and queued to -stable, thanks!
> 
> Hi David,
> 
> I don't see this patch in Chris 2.6.25.6 -stable review message.

Is it already in mainline ? (stable should only contain already merged
patches). It generally helps the stable team if you indicate the commit
id, as it's easier to "git show" than to "git log|grep".

Regards,
Willy

Re: [patch 00/50] 2.6.25.6 -stable review

[Marco Berizzi]

Willy Tarreau wrote:

> On Sat, Jun 07, 2008 at 10:27:58PM +0200, Marco Berizzi wrote:
>> David Miller wrote:
>> 
>> > From: Herbert Xu <herbert@gondor.apana.org.au>
>> > Date: Tue, 20 May 2008 17:25:11 +0800
>> > 
>> >> On Wed, May 14, 2008 at 10:19:57AM +0200, Marco Berizzi wrote:
>> >> > 
>> >> > I hope this helps.
>> >> 
>> >> OK found the problem, it was my fault after all :)
>> >> 
>> >> Dave, this patch needs to go into stable too.
>> >> 
>> >> [IPSEC]: Use the correct ip_local_out function
>> >> 
>> >> Because the IPsec output function xfrm_output_resume does its
>> >> own dst_output call it should always call __ip_local_output
>> >> instead of ip_local_output as the latter may invoke dst_output
>> >> directly.  Otherwise the return values from nf_hook and dst_output
>> >> may clash as they both use the value 1 but for different purposes.
>> >> 
>> >> When that clash occurs this can cause a packet to be used after
>> >> it has been freed which usually leads to a crash.  Because the
>> >> offending value is only returned from dst_output with qdiscs
>> >> such as HTB, this bug is normally not visible.
>> >> 
>> >> Thanks to Marco Berizzi for his perseverance in tracking this
>> >> down.
>> >> 
>> >> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
>> > 
>> > Applied and queued to -stable, thanks!
>> 
>> Hi David,
>> 
>> I don't see this patch in Chris 2.6.25.6 -stable review message.
> 
> Is it already in mainline ?

yes, since 2008/05/20
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1ac06e0306d0192a7a4d9ea1c9e06d355ce7e7d3

Re: [patch 00/50] 2.6.25.6 -stable review

[Willy Tarreau]

On Sun, Jun 08, 2008 at 01:56:01PM +0200, Marco Berizzi wrote:
> Willy Tarreau wrote:
> 
> > On Sat, Jun 07, 2008 at 10:27:58PM +0200, Marco Berizzi wrote:
> >> David Miller wrote:
> >> 
> >> > From: Herbert Xu <herbert@gondor.apana.org.au>
> >> > Date: Tue, 20 May 2008 17:25:11 +0800
> >> > 
> >> >> On Wed, May 14, 2008 at 10:19:57AM +0200, Marco Berizzi wrote:
> >> >> > 
> >> >> > I hope this helps.
> >> >> 
> >> >> OK found the problem, it was my fault after all :)
> >> >> 
> >> >> Dave, this patch needs to go into stable too.
> >> >> 
> >> >> [IPSEC]: Use the correct ip_local_out function
> >> >> 
> >> >> Because the IPsec output function xfrm_output_resume does its
> >> >> own dst_output call it should always call __ip_local_output
> >> >> instead of ip_local_output as the latter may invoke dst_output
> >> >> directly.  Otherwise the return values from nf_hook and dst_output
> >> >> may clash as they both use the value 1 but for different purposes.
> >> >> 
> >> >> When that clash occurs this can cause a packet to be used after
> >> >> it has been freed which usually leads to a crash.  Because the
> >> >> offending value is only returned from dst_output with qdiscs
> >> >> such as HTB, this bug is normally not visible.
> >> >> 
> >> >> Thanks to Marco Berizzi for his perseverance in tracking this
> >> >> down.
> >> >> 
> >> >> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
> >> > 
> >> > Applied and queued to -stable, thanks!
> >> 
> >> Hi David,
> >> 
> >> I don't see this patch in Chris 2.6.25.6 -stable review message.
> > 
> > Is it already in mainline ?
> 
> yes, since 2008/05/20
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1ac06e0306d0192a7a4d9ea1c9e06d355ce7e7d3

Indeed. Most likely it was simply lost somewhere in the e-mail chain.
Then best thing to do is to retransmit it for next batch of patches.
Chris, here's the fix in question.

Thanks,
Willy
--

>From 1ac06e0306d0192a7a4d9ea1c9e06d355ce7e7d3 Mon Sep 17 00:00:00 2001
From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Tue, 20 May 2008 14:32:14 -0700
Subject: ipsec: Use the correct ip_local_out function

Because the IPsec output function xfrm_output_resume does its
own dst_output call it should always call __ip_local_output
instead of ip_local_output as the latter may invoke dst_output
directly.  Otherwise the return values from nf_hook and dst_output
may clash as they both use the value 1 but for different purposes.

When that clash occurs this can cause a packet to be used after
it has been freed which usually leads to a crash.  Because the
offending value is only returned from dst_output with qdiscs
such as HTB, this bug is normally not visible.

Thanks to Marco Berizzi for his perseverance in tracking this
down.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: David S. Miller <davem@davemloft.net>
---
 net/ipv4/route.c |    2 +-
 net/ipv6/route.c |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index 92f90ae..df41026 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -160,7 +160,7 @@ static struct dst_ops ipv4_dst_ops = {
 	.negative_advice =	ipv4_negative_advice,
 	.link_failure =		ipv4_link_failure,
 	.update_pmtu =		ip_rt_update_pmtu,
-	.local_out =		ip_local_out,
+	.local_out =		__ip_local_out,
 	.entry_size =		sizeof(struct rtable),
 	.entries =		ATOMIC_INIT(0),
 };
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index b7a4a87..48534c6 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -109,7 +109,7 @@ static struct dst_ops ip6_dst_ops_template = {
 	.negative_advice	=	ip6_negative_advice,
 	.link_failure		=	ip6_link_failure,
 	.update_pmtu		=	ip6_rt_update_pmtu,
-	.local_out		=	ip6_local_out,
+	.local_out		=	__ip6_local_out,
 	.entry_size		=	sizeof(struct rt6_info),
 	.entries		=	ATOMIC_INIT(0),
 };
-- 
1.5.3.8

Re: [patch 00/50] 2.6.25.6 -stable review

[David Miller]

From: Willy Tarreau <w@1wt.eu>
Date: Sun, 8 Jun 2008 14:36:01 +0200

> Indeed. Most likely it was simply lost somewhere in the e-mail chain.
> Then best thing to do is to retransmit it for next batch of patches.
> Chris, here's the fix in question.

It did not get lost at all, it's perfectly sitting in my
networking -stable queue waiting for me to have an
opportunity to submit it to the -stable folks after I do
some testing.

If you ask me in the future about the status of a -stable
patch from the networking, I'll let you know exactly what
is happening to that patch wrt. stable.  I rarely forget
to submit an appropriate patch, and when I do forget you
merely have to let me know (rather than submitting it
to -stable directly, please don't do that) so that I can
fit it in with what I plan to submit to -stable already.

Right now is an unusual situation where I am in a 3 week
period of near constant travel, otherwise all of this
stuff would have been submitted already.  As I stated in
another reply to you, that period is ending tomorrow so I
should be able to crank out all of these patches to -stable
some time this week.

Thanks.

Re: [patch 00/50] 2.6.25.6 -stable review

[Willy Tarreau]

On Sun, Jun 08, 2008 at 07:10:51AM -0700, David Miller wrote:
> From: Willy Tarreau <w@1wt.eu>
> Date: Sun, 8 Jun 2008 14:36:01 +0200
> 
> > Indeed. Most likely it was simply lost somewhere in the e-mail chain.
> > Then best thing to do is to retransmit it for next batch of patches.
> > Chris, here's the fix in question.
> 
> It did not get lost at all, it's perfectly sitting in my
> networking -stable queue waiting for me to have an
> opportunity to submit it to the -stable folks after I do
> some testing.
> 
> If you ask me in the future about the status of a -stable
> patch from the networking, I'll let you know exactly what
> is happening to that patch wrt. stable.  I rarely forget
> to submit an appropriate patch, and when I do forget you
> merely have to let me know (rather than submitting it
> to -stable directly, please don't do that) so that I can
> fit it in with what I plan to submit to -stable already.

OK

> Right now is an unusual situation where I am in a 3 week
> period of near constant travel, otherwise all of this
> stuff would have been submitted already.  As I stated in
> another reply to you, that period is ending tomorrow so I
> should be able to crank out all of these patches to -stable
> some time this week.

Perfect, thank you David.
Willy

Re: [patch 00/50] 2.6.25.6 -stable review

[Jay Cliburn]

On Sun, 08 Jun 2008 07:10:51 -0700 (PDT)
David Miller <davem@davemloft.net> wrote:


> If you ask me in the future about the status of a -stable
> patch from the networking, I'll let you know exactly what
> is happening to that patch wrt. stable.  I rarely forget
> to submit an appropriate patch, and when I do forget you
> merely have to let me know (rather than submitting it
> to -stable directly, please don't do that) so that I can
> fit it in with what I plan to submit to -stable already.


As a netdev driver maintainer, I've been following this workflow for
patches that need to go to -stable:

1.  I submit a mainline patch to Jeff Garzik.
2.  Jeff submits to David.
3.  David submits to Linus.
4.  Linus merges patch into mainline.
5.  I extract mainline commit ID.
6.  I apply and test patch against appropriate 2.6.x.y git tree.
7.  I submit patch directly to -stable.

David's admonition tells me I'm doing it wrong, and that I should
submit the stable patch to Jeff as well.  Am I right?

Jay

Re: [patch 00/50] 2.6.25.6 -stable review

[Willy Tarreau]

On Sun, Jun 08, 2008 at 10:38:35AM -0500, Jay Cliburn wrote:
> On Sun, 08 Jun 2008 07:10:51 -0700 (PDT)
> David Miller <davem@davemloft.net> wrote:
> 
> 
> > If you ask me in the future about the status of a -stable
> > patch from the networking, I'll let you know exactly what
> > is happening to that patch wrt. stable.  I rarely forget
> > to submit an appropriate patch, and when I do forget you
> > merely have to let me know (rather than submitting it
> > to -stable directly, please don't do that) so that I can
> > fit it in with what I plan to submit to -stable already.
> 
> 
> As a netdev driver maintainer, I've been following this workflow for
> patches that need to go to -stable:
> 
> 1.  I submit a mainline patch to Jeff Garzik.
> 2.  Jeff submits to David.
> 3.  David submits to Linus.
> 4.  Linus merges patch into mainline.
> 5.  I extract mainline commit ID.
> 6.  I apply and test patch against appropriate 2.6.x.y git tree.
> 7.  I submit patch directly to -stable.
> 
> David's admonition tells me I'm doing it wrong, and that I should
> submit the stable patch to Jeff as well.  Am I right?

The normal recommended method to get patches automatically sent to
stable is to add a "Cc: stable@kernel.org" line above your signed-off-by
line. That way it is automatically sent to stable when Linus merges it,
and David gets notified. This should *theorically* save him some of the
work consisting in checking his stable queue for unmerged patches, but
his workflow may be different. Also, this method is particularly suited
to ensure that patches don't get lost when the maintainer does not have
a specific stable queue, which is not the case here, according to David.

Regards,
Willy

Re: [patch 00/50] 2.6.25.6 -stable review

[Jeff Garzik]

Jay Cliburn wrote:
> On Sun, 08 Jun 2008 07:10:51 -0700 (PDT)
> David Miller <davem@davemloft.net> wrote:
> 
> 
>> If you ask me in the future about the status of a -stable
>> patch from the networking, I'll let you know exactly what
>> is happening to that patch wrt. stable.  I rarely forget
>> to submit an appropriate patch, and when I do forget you
>> merely have to let me know (rather than submitting it
>> to -stable directly, please don't do that) so that I can
>> fit it in with what I plan to submit to -stable already.
> 
> 
> As a netdev driver maintainer, I've been following this workflow for
> patches that need to go to -stable:
> 
> 1.  I submit a mainline patch to Jeff Garzik.
> 2.  Jeff submits to David.
> 3.  David submits to Linus.
> 4.  Linus merges patch into mainline.
> 5.  I extract mainline commit ID.
> 6.  I apply and test patch against appropriate 2.6.x.y git tree.
> 7.  I submit patch directly to -stable.
> 
> David's admonition tells me I'm doing it wrong, and that I should
> submit the stable patch to Jeff as well.  Am I right?

I usually encourage a more-parallel process where you simply email 
stable@kernel.org with the upstream commit id of the change(s) in question.

	Jeff

Re: [patch 00/50] 2.6.25.6 -stable review

[David Miller]

From: Jeff Garzik <jeff@garzik.org>
Date: Sun, 08 Jun 2008 16:07:47 -0400

> Jay Cliburn wrote:
> > David's admonition tells me I'm doing it wrong, and that I should
> > submit the stable patch to Jeff as well.  Am I right?
> 
> I usually encourage a more-parallel process where you simply email 
> stable@kernel.org with the upstream commit id of the change(s) in question.

Right, and if Jeff wants to work things that way for the
networking drivers that's fine.

Personally, I like to make sure some time passes between when
a fix goes into Linus's tree and when I push it into -stable
because time is often what shakes out the last remaining problems
introduced by some change no matter how seeming obvious the
patch is.

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Herbert Xu wrote:

> Perhaps you could enable SLAB debugging to see if we can get
> closer to the culprit? Alternatively you could try a git bisection.

I don't know if this is the same bug or something else.
This is a different box but same hw/.config

http://www.ussg.iu.edu/hypermail/linux/kernel/0805.0/1969.html

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Herbert Xu]

Marco Berizzi <pupilla@hotmail.com> wrote:
> I know, it is pretty useless this report,
> but I have nothing more than these bitmaps
> and the message in the subject :-(
> 
> http://80.204.235.230/1.jpg
> http://80.204.235.230/2.jpg
> http://80.204.235.230/2.jpg

Could you please take a better shot of this crash the next time
it happens? As it is the right-hand-side of the code section is
missing which makes it hard to determine where exactly it crashed.

Alternatively send me your xfrm_output.o file (or vmlinux).

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Herbert Xu wrote:

> Marco Berizzi <pupilla@hotmail.com> wrote:
> > I know, it is pretty useless this report,
> > but I have nothing more than these bitmaps
> > and the message in the subject :-(
> >
> > http://80.204.235.230/1.jpg
> > http://80.204.235.230/2.jpg
> > http://80.204.235.230/2.jpg
>
> Could you please take a better shot of this crash the next time
> it happens? As it is the right-hand-side of the code section is
> missing which makes it hard to determine where exactly it crashed.
>
> Alternatively send me your xfrm_output.o file (or vmlinux).

I have removed these modules from
the two boxes that are crashing:

sch_sfq
sch_htb
cls_fw

I will wait to see if they are crashing
again without these modules loaded.

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Herbert Xu]

Marco Berizzi <pupilla@hotmail.com> wrote:

> I have removed these modules from
> the two boxes that are crashing:

OK, I've matched up your xfrm_output.o with the crash and it
would appear that skb->dst is either NULL or crap after the
local_out call in xfrm_output_resume.  We know that it was valid
before the call because skb->dst is used to make that call.

So it would appear that some netfilter rule has freed skb->dst.
Do you have relevant netfilter rules, e.g., NAT entries or any
rules that might reroute the packet?

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Herbert Xu wrote:
 
> So it would appear that some netfilter rule has freed skb->dst.
> Do you have relevant netfilter rules, e.g., NAT entries

nat table is empty

> or any
> rules that might reroute the packet?

Yes indeed, this box is in the middle of a very poorly designed
network: it is the default gateway for lan clients, but it is
routing back all traffic to another lan box except for ipsec
traffic.

eth2 is internal lan and eth0 is the interface connected to
the isp router.

root@Halley:/# ip ru sh
0:      from all lookup local
601:    from 172.23.0.0/23 iif eth2 lookup isa
32766:  from all lookup main
32767:  from all lookup default

root@Halley:/# ip r s table isa
default via 172.23.1.254 dev eth2  metric 1

root@Halley:/# ip r s
88.51.228.225 dev eth0  scope link
88.51.228.224/28 dev eth1  proto kernel  scope link  src 88.51.228.238
172.22.1.0/24 via 88.51.228.225 dev eth0  src 172.23.2.254
172.18.1.0/24 via 88.51.228.225 dev eth0
172.25.5.0/24 via 88.51.228.225 dev eth0
172.25.1.0/24 via 88.51.228.225 dev eth0
172.21.1.0/24 via 88.51.228.225 dev eth0
172.17.1.0/24 via 88.51.228.225 dev eth0
172.23.2.0/23 via 88.51.228.225 dev eth0
172.23.0.0/23 dev eth2  proto kernel  scope link  src 172.23.1.8
172.16.0.0/23 via 88.51.228.225 dev eth0
127.0.0.0/8 dev lo  scope link
default via 88.51.228.225 dev eth0  metric 1

root@Halley:/# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:90:27:74:66:4c brd ff:ff:ff:ff:ff:ff
    inet 88.51.228.238/28 brd 88.51.228.239 scope global eth0
    inet 172.23.2.254/32 scope global eth0
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:d0:b7:18:14:bc brd ff:ff:ff:ff:ff:ff
    inet 88.51.228.238/28 brd 88.51.228.239 scope global eth1
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
    link/ether 00:60:08:6d:3a:84 brd ff:ff:ff:ff:ff:ff
    inet 172.23.1.8/23 brd 172.23.1.255 scope global eth2

Here is the info you asked:

root@Halley:~# iptables -nvxL -t nat
Chain PREROUTING (policy ACCEPT 72557 packets, 6148591 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 46575 packets, 3596771 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 191 packets, 26422 bytes)
    pkts      bytes target     prot opt in     out     source               destination

root@Halley:~# iptables -nvxL -t mangle
Chain PREROUTING (policy ACCEPT 13646213 packets, 12635102939 bytes)
    pkts      bytes target     prot opt in     out     source               destination
    2752  2713288 MARK       tcp  --  eth2   *       0.0.0.0/0            172.16.0.0/12       multiport dports 20,25 MARK set 0x6e
       0        0 MARK       all  --  *      *       0.0.0.0/0            172.16.0.0/12       helper match "ftp" MARK set 0x6e
  209912 10965262 MARK       tcp  --  eth2   *       0.0.0.0/0            172.16.0.0/12       multiport dports 1494,2598 MARK set 0x70

Chain INPUT (policy ACCEPT 7226005 packets, 8188427260 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 6413301 packets, 4441928803 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 2626545 packets, 513891469 bytes)
    pkts      bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 8985576 packets, 4948767535 bytes)
    pkts      bytes target     prot opt in     out     source               destination

root@Halley:~# iptables -nvxL -t filter
Chain INPUT (policy DROP 18528 packets, 1978955 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
    9788   662676 ACCEPT     all  --  *      *       172.16.1.247         0.0.0.0/0           
  107544 30848205 ACCEPT     all  --  *      *       80.204.235.254       0.0.0.0/0           
       0        0 ACCEPT     all  --  *      *       127.0.0.1            127.0.0.1           
 3721271 4246951537 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
   18232  1950857 green-me   all  --  eth2   *       172.23.0.0/23        0.0.0.0/0           
       0        0 dmz-me     all  --  eth1   *       88.51.228.224/28     0.0.0.0/0           
 3361526 3907062198 red-me     all  --  eth0   *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy DROP 3093 packets, 230687 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
 6211800 4421683285 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
  147228 15568498 ACCEPT     all  --  eth2   eth2    172.23.0.0/23        0.0.0.0/0           
    2366   220478 ACCEPT     all  --  *      *       172.23.0.0/23        172.16.0.0/23       
    5774   531670 ACCEPT     all  --  *      *       172.16.0.0/23        172.23.0.0/23       
    1201    90893 ACCEPT     all  --  *      *       172.23.0.0/23        172.23.2.0/23       
    3114   275588 ACCEPT     all  --  *      *       172.23.2.0/23        172.23.0.0/23       
    6141   505776 ACCEPT     all  --  *      *       172.23.0.0/23        172.18.1.0/24       
   10018   684510 ACCEPT     all  --  *      *       172.18.1.0/24        172.23.0.0/23       
    8620   755691 ACCEPT     all  --  *      *       172.23.0.0/23        172.25.1.0/24       
     572    86585 ACCEPT     all  --  *      *       172.25.1.0/24        172.23.0.0/23       
    2554   124898 ACCEPT     all  --  *      *       172.23.0.0/23        172.25.5.0/24       
    3713   286355 ACCEPT     all  --  *      *       172.25.5.0/24        172.23.0.0/23       
      66     3168 ACCEPT     all  --  *      *       172.23.0.0/23        172.17.1.0/24       
     298    30003 ACCEPT     all  --  *      *       172.17.1.0/24        172.23.0.0/23       
       0        0 ACCEPT     all  --  *      *       172.23.0.0/23        172.22.1.0/24       
      10     2674 ACCEPT     all  --  *      *       172.22.1.0/24        172.23.0.0/23       
       0        0 ACCEPT     all  --  *      *       172.23.0.0/23        172.21.1.0/24       
     108    26588 ACCEPT     all  --  *      *       172.21.1.0/24        172.23.0.0/23       
       0        0 ACCEPT     all  --  *      *       172.23.0.0/23        81.113.185.96/27    
       0        0 ACCEPT     all  --  *      *       81.113.185.96/27     172.23.0.0/23       
      30     1472 green-red  all  --  eth2   eth0    172.23.0.0/23        0.0.0.0/0           
       0        0 green-dmz  all  --  eth2   eth1    172.23.0.0/23        88.51.228.224/28    
       0        0 dmz-red    all  --  eth1   eth0    88.51.228.224/28     0.0.0.0/0           
       0        0 dmz-green  all  --  eth1   eth2    88.51.228.224/28     172.23.0.0/23       
    3063   229215 syn-flood-dmz  all  --  eth0   eth1    0.0.0.0/0            88.51.228.224/28    
       0        0 syn-flood-green  all  --  eth0   eth2    0.0.0.0/0            172.23.0.0/23       

Chain OUTPUT (policy DROP 4866 packets, 722169 bytes)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ACCEPT     all  --  *      *       127.0.0.1            127.0.0.1           
 2570465 506300871 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
   36483  5204284 me-green   all  --  *      eth2    0.0.0.0/0            172.23.0.0/23       
       0        0 me-dmz     all  --  *      eth1    0.0.0.0/0            88.51.228.224/28    
     191    26666 me-red     all  --  *      eth0    0.0.0.0/0            0.0.0.0/0           

Chain dmz-green (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 icmp-me    icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
       0        0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-unreachable 

Chain dmz-me (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain dmz-red (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 icmp-me    icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
       0        0 ACCEPT     all  --  *      *       88.51.228.224/28     85.32.35.0/27       
       0        0 ACCEPT     all  --  *      *       88.51.228.224/28     82.189.143.32/28    
       0        0 ACCEPT     all  --  *      *       88.51.228.224/28     88.45.249.192/27    
       0        0 ACCEPT     all  --  *      *       88.51.228.224/28     88.35.239.32/28     
       0        0 ACCEPT     all  --  *      *       88.51.228.224/28     81.113.185.96/27    
       0        0 ACCEPT     all  --  *      *       88.51.228.224/28     88.57.240.144/28    
       0        0 ACCEPT     all  --  *      *       88.51.228.224/28     88.40.176.96/28     
       0        0 ACCEPT     all  --  *      *       88.51.228.224/28     85.41.109.160/27    
       0        0 ACCEPT     all  --  *      *       88.51.228.224/28     80.204.235.224/27   
       0        0 ACCEPT     all  --  *      *       88.51.228.224/28     88.52.180.96/28     

Chain green-dmz (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain green-me (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ACCEPT     icmp --  *      *       172.23.1.4           0.0.0.0/0           icmp type 8 

Chain green-red (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain icmp-me (3 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 0 
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 8 
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 3 

Chain me-dmz (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain me-green (1 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain me-red (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 icmp-me    icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
       4      576 ACCEPT     esp  --  *      *       0.0.0.0/0            0.0.0.0/0           
       0        0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           multiport dports 53,123 
     183    25850 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           multiport dports 53,123,500,4500,6666 

Chain red-dmz (3 references)
    pkts      bytes target     prot opt in     out     source               destination         

Chain red-green (3 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ACCEPT     tcp  --  *      *       0.0.0.0/0            172.23.1.2          multiport dports 80 

Chain red-me (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           icmp type 3 
     179    36984 ACCEPT     esp  --  *      *       0.0.0.0/0            0.0.0.0/0           
 3360694 3906910252 ACCEPT     4    --  *      *       0.0.0.0/0            0.0.0.0/0           
     376    93408 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           multiport dports 500,4500 

Chain syn-flood-dmz (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
    1767   101324 red-dmz    tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x17/0x02 limit: avg 1/sec burst 5 
       7      280 red-dmz    tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x17/0x04 limit: avg 1/sec burst 5 
     214    66554 red-dmz    udp  --  *      *       0.0.0.0/0            0.0.0.0/0           
       0        0 ACCEPT     all  --  *      *       85.32.35.0/27        88.51.228.224/28    
       0        0 ACCEPT     all  --  *      *       82.189.143.32/28     88.51.228.224/28    
       0        0 ACCEPT     all  --  *      *       88.45.249.192/27     88.51.228.224/28    
       0        0 ACCEPT     all  --  *      *       88.35.239.32/28      88.51.228.224/28    
       0        0 ACCEPT     all  --  *      *       81.113.185.96/27     88.51.228.224/28    
       0        0 ACCEPT     all  --  *      *       88.57.240.144/28     88.51.228.224/28    
       0        0 ACCEPT     all  --  *      *       88.40.176.96/28      88.51.228.224/28    
       0        0 ACCEPT     all  --  *      *       85.41.109.160/27     88.51.228.224/28    
       0        0 ACCEPT     all  --  *      *       80.204.235.224/27    88.51.228.224/28    
       0        0 ACCEPT     all  --  *      *       88.52.180.96/28      88.51.228.224/28    

Chain syn-flood-green (1 references)
    pkts      bytes target     prot opt in     out     source               destination         
       0        0 red-green  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x17/0x02 limit: avg 20/min burst 5 
       0        0 red-green  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp flags:0x17/0x04 limit: avg 20/min burst 5 
       0        0 red-green  udp  --  *      *       0.0.0.0/0            0.0.0.0/0           

root@Halley:/# ip x p
src 172.25.1.0/24 dst 172.23.0.0/23
        dir in priority 2376
        tmpl src 85.41.109.190 dst 88.51.228.238
                proto comp reqid 16386 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16385 mode transport
src 172.18.1.0/24 dst 172.23.0.0/23
        dir in priority 2376
        tmpl src 85.32.35.30 dst 88.51.228.238
                proto comp reqid 16394 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16393 mode transport
src 172.21.1.0/24 dst 172.23.0.0/23
        dir in priority 2376
        tmpl src 88.35.239.46 dst 88.51.228.238
                proto comp reqid 16390 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16389 mode transport
src 172.22.1.0/24 dst 172.23.0.0/23
        dir in priority 2376
        tmpl src 88.45.249.222 dst 88.51.228.238
                proto comp reqid 16398 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16397 mode transport
src 172.25.5.0/24 dst 172.23.0.0/23
        dir in priority 2376
        tmpl src 88.57.240.158 dst 88.51.228.238
                proto comp reqid 16402 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16401 mode transport
src 172.17.1.0/24 dst 172.23.0.0/23
        dir in priority 2376
        tmpl src 82.189.143.46 dst 88.51.228.238
                proto comp reqid 16410 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16409 mode transport
src 172.16.0.0/23 dst 172.23.0.0/23
        dir in priority 2377
        tmpl src 80.204.235.254 dst 88.51.228.238
                proto comp reqid 16414 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16413 mode transport
src 172.23.2.0/23 dst 172.23.0.0/23
        dir in priority 2377
        tmpl src 88.52.180.110 dst 88.51.228.238
                proto comp reqid 16406 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16405 mode transport
src 172.23.0.0/23 dst 172.25.1.0/24
        dir out priority 2376
        tmpl src 88.51.228.238 dst 85.41.109.190
                proto comp reqid 16386 mode tunnel
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16385 mode transport
src 172.23.0.0/23 dst 172.21.1.0/24
        dir out priority 2376
        tmpl src 88.51.228.238 dst 88.35.239.46
                proto comp reqid 16390 mode tunnel
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16389 mode transport
src 172.23.0.0/23 dst 172.18.1.0/24
        dir out priority 2376
        tmpl src 88.51.228.238 dst 85.32.35.30
                proto comp reqid 16394 mode tunnel
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16393 mode transport
src 172.23.0.0/23 dst 172.22.1.0/24
        dir out priority 2376
        tmpl src 88.51.228.238 dst 88.45.249.222
                proto comp reqid 16398 mode tunnel
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16397 mode transport
src 172.23.0.0/23 dst 172.25.5.0/24
        dir out priority 2376
        tmpl src 88.51.228.238 dst 88.57.240.158
                proto comp reqid 16402 mode tunnel
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16401 mode transport
src 172.23.0.0/23 dst 172.17.1.0/24
        dir out priority 2376
        tmpl src 88.51.228.238 dst 82.189.143.46
                proto comp reqid 16410 mode tunnel
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16409 mode transport
src 172.23.0.0/23 dst 172.23.2.0/23
        dir out priority 2377
        tmpl src 88.51.228.238 dst 88.52.180.110
                proto comp reqid 16406 mode tunnel
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16405 mode transport
src 172.23.0.0/23 dst 172.16.0.0/23
        dir out priority 2377
        tmpl src 88.51.228.238 dst 80.204.235.254
                proto comp reqid 16414 mode tunnel
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16413 mode transport
src 172.25.1.0/24 dst 172.23.0.0/23
        dir fwd priority 2376
        tmpl src 85.41.109.190 dst 88.51.228.238
                proto comp reqid 16386 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16385 mode transport
src 172.18.1.0/24 dst 172.23.0.0/23
        dir fwd priority 2376
        tmpl src 85.32.35.30 dst 88.51.228.238
                proto comp reqid 16394 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16393 mode transport
src 172.21.1.0/24 dst 172.23.0.0/23
        dir fwd priority 2376
        tmpl src 88.35.239.46 dst 88.51.228.238
                proto comp reqid 16390 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16389 mode transport
src 172.22.1.0/24 dst 172.23.0.0/23
        dir fwd priority 2376
        tmpl src 88.45.249.222 dst 88.51.228.238
                proto comp reqid 16398 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16397 mode transport
src 172.25.5.0/24 dst 172.23.0.0/23
        dir fwd priority 2376
        tmpl src 88.57.240.158 dst 88.51.228.238
                proto comp reqid 16402 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16401 mode transport
src 172.17.1.0/24 dst 172.23.0.0/23
        dir fwd priority 2376
        tmpl src 82.189.143.46 dst 88.51.228.238
                proto comp reqid 16410 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16409 mode transport
src 172.16.0.0/23 dst 172.23.0.0/23
        dir fwd priority 2377
        tmpl src 80.204.235.254 dst 88.51.228.238
                proto comp reqid 16414 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16413 mode transport
src 172.23.2.0/23 dst 172.23.0.0/23
        dir fwd priority 2377
        tmpl src 88.52.180.110 dst 88.51.228.238
                proto comp reqid 16406 mode tunnel
                level use
        tmpl src 0.0.0.0 dst 0.0.0.0
                proto esp reqid 16405 mode transport

and this my rc.qos script for setting up qdisc:

root@Halley:/etc/rc.d# cat rc.qos
#!/bin/sh
#set -x

# cleaning environment
tc qdisc del dev eth0 root
# done

# 1600kbit -->> 200kbps
# 200kbit -->> 25kbps
# 400kbit -->> 50kbps
# 600kbit -->> 75kbps
# 1200kbit -->> 150kbps
# 800kbit -->> 100kbps

tc qdisc add dev eth0 root handle 1:0 htb default 11

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 1700kbit ceil 1700kbit burst 15k

tc class add dev eth0 parent 1:1 classid 1:10 htb rate 200kbit ceil 400kbit burst 5k
tc class add dev eth0 parent 1:1 classid 1:11 htb rate 600kbit ceil 1600kbit burst 5k
tc class add dev eth0 parent 1:1 classid 1:12 htb rate 800kbit ceil 1700kbit burst 15k

tc qdisc add dev eth0 parent 1:10 handle 10:0 sfq perturb 10
tc qdisc add dev eth0 parent 1:11 handle 20:0 sfq perturb 5
tc qdisc add dev eth0 parent 1:12 handle 30:0 sfq perturb 10

FW="tc filter add dev eth0 protocol ip parent 1:0 prio 1"
$FW handle 110 fw flowid 1:10
$FW handle 111 fw flowid 1:11
$FW handle 112 fw flowid 1:12

Re: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Herbert Xu]

On Thu, May 01, 2008 at 02:59:42PM +0200, Marco Berizzi wrote:
>
> Yes indeed, this box is in the middle of a very poorly designed
> network: it is the default gateway for lan clients, but it is
> routing back all traffic to another lan box except for ipsec
> traffic.

OK I don't see any rerouting actions in your rule dump.

Is your box SMP or have you enabled preemption? Just trying to
eliminate the case of another thread zapping skb->dst.

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

RE: 2.6.25 crash: EIP: [<c02e2f14>] xfrm_output_resume+0x64/0x100 ss:esp 0068:c03a1e5c

[Marco Berizzi]

Herbert Xu wrote:

> On Thu, May 01, 2008 at 02:59:42PM +0200, Marco Berizzi wrote:
>>
>> Yes indeed, this box is in the middle of a very poorly designed
>> network: it is the default gateway for lan clients, but it is
>> routing back all traffic to another lan box except for ipsec
>> traffic.
> 
> OK I don't see any rerouting actions in your rule dump.
> 
> Is your box SMP

nope, it is an old PII or PIII 

> or have you enabled preemption? 

.config is posted here:

http://www.ussg.iu.edu/hypermail/linux/kernel/0804.3/1763.html

CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
_________________________________________________________________
Discover the new Windows Vista
http://search.msn.com/results.aspx?q=windows+vista&mkt=en-US&form=QBRE