* Fw: 2.6.23.1-smp kernel panic (network-related)
@ 2007-11-07 17:51 Stephen Hemminger
2007-11-07 18:38 ` Marek Kierdelewicz
0 siblings, 1 reply; 4+ messages in thread
From: Stephen Hemminger @ 2007-11-07 17:51 UTC (permalink / raw)
To: Marek Kierdelewicz; +Cc: netdev
[-- Attachment #1: Type: text/plain, Size: 3597 bytes --]
What is the test input that causes the crash??
Forwarding message to proper group.
Begin forwarded message:
Date: Wed, 7 Nov 2007 17:52:11 +0100
From: Marek Kierdelewicz <marek@koba.pl>
To: linux-kernel@vger.kernel.org
Newsgroups: gmane.linux.kernel
Subject: 2.6.23.1-smp kernel panic (network-related)
Hi there,
My company's (ISP) bussines model requires dynamic resizing of the
client queues. It's achieved by regenerating shaping rules and loading
then using batch mode of a tc binary. On production systems it's done
once every 1 or 2 minutes. Unfortunately this causes smp kernels to
panic. Non-smp kernels don't have such problems. Bug is around a long
time. I first noticed it after migrating to shaping configs that use
IFB, it might have been 2.6.18 "era".
Test scenario:
I've put together a test machine with configuration copied from
production router. I'm feeding the machine with production traffic
by means of port mirroring. Test machine has the same config as
production one (including mac addresses), so it tries to route the
incoming traffic.
Tested kernels were 2.6.31.1 and 2.6.20.6 (config from 2.6.20.6 is in
attachment). Both panicked if compiled with SMP support and work stable
otherwise. Problem occurs only with cyclic "shaping restarts". For the
test, reload operation using "tc -b ..." was executed in an infinite
loop.
Box's CPU usage was approximately 15%. Panics occur with few hours -
one day intervals.
Below I attach the panic message captured via serial console:
----------------------------------------------------------------------
printk: 63 messages suppressed.
dst cache overflow
SMP
Modules linked in: ipt_LOG xt_hashlimit ipt_MASQUERADE ip_set_macipmap
ip_set_ipmap xt_state w83627hf hwmon_vid eeprom ifb ipt_SET ipt_set
ip_set ipip tunnel4 ip_gre e1000 i2c_i801 i2c_core CPU: 1 EIP:
0060:[<f255c08f>] Not tainted VLI EFLAGS: 00010202 (2.6.23.1-smp
#2) EIP is at 0xf255c08f
eax: c196a000 ebx: 00000100 ecx: ef2f408c edx: f3630000
esi: f0332029 edi: f255c08c ebp: 00000001 esp: f3631ea8
ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068
Process tc (pid: 27695, ti=f3630000 task=f26a9560 task.ti=f3630000)
Stack: c01280ad f26a9560 00000001 f3631eb4 c0106f57 ef2f408c f0e8c08c
00000031 c0495308 0000000a c0124eb6 00000046 00000000 f7657740 f3630000
c0124f4c c180f120 c0114e62 00000000 00000000 f7bfd224 f74ed95c c01047e0
f7bfd224 Call Trace:
[<c01280ad>] run_timer_softirq+0xf5/0x154
[<c0106f57>] profile_pc+0x21/0x4a
[<c0124eb6>] __do_softirq+0x5d/0xc1
[<c0124f4c>] do_softirq+0x32/0x36
[<c0114e62>] smp_apic_timer_interrupt+0x74/0x80
[<c01047e0>] apic_timer_interrupt+0x28/0x30
[<c014c82e>] remove_vma+0x1c/0x36
[<c014c912>] exit_mmap+0xca/0xe1
[<c011eedc>] mmput+0x1d/0x75
[<c0123688>] do_exit+0x1be/0x68a
[<c0123bc0>] sys_exit_group+0x0/0xd
[<c0103d12>] sysenter_past_esp+0x5f/0x85
=======================
Code: 00 52 41 f7 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 fa 00 00 00 ea 05 00 00 7f 00 00 00 34 a5 96 <c1> 8c
00 fd f3 a5 2b 09 00 d1 d8 32 c0 00 c0 55 f2 00 a0 96 c1 EIP:
[<f255c08f>] 0xf255c08f SS:ESP 0068:f3631ea8 Kernel panic - not
syncing: Fatal exception in interrupt
----------------------------------------------------------------------
--
Marek Kierdelewicz
Kierownik Działu Systemów Sieciowych, KoBa
Manager of Network Systems Department, KoBa
tel. (85) 7406466; fax. (85) 7406467
e-mail: admin@koba.pl
--
Stephen Hemminger <shemminger@linux-foundation.org>
[-- Attachment #2: kernel_config --]
[-- Type: application/octet-stream, Size: 14957 bytes --]
CONFIG_X86_32=y
CONFIG_GENERIC_TIME=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_SEMAPHORE_SLEEPERS=y
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_DMI=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION="-non-smp"
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_SYSFS_DEPRECATED=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_CC_OPTIMIZE_FOR_SIZE=y
CONFIG_SYSCTL=y
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_SLAB=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_CFQ=y
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_SMP=y
CONFIG_X86_PC=y
CONFIG_MPENTIUM4=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_NR_CPUS=8
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_NONE=y
CONFIG_PREEMPT_BKL=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_VM86=y
CONFIG_MICROCODE=m
CONFIG_MICROCODE_OLD_INTERFACE=y
CONFIG_X86_MSR=m
CONFIG_X86_CPUID=m
CONFIG_NOHIGHMEM=y
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_RESOURCES_64BIT=y
CONFIG_MTRR=y
CONFIG_IRQBALANCE=y
CONFIG_HZ_250=y
CONFIG_HZ=250
CONFIG_PHYSICAL_START=0x100000
CONFIG_PHYSICAL_ALIGN=0x100000
CONFIG_COMPAT_VDSO=y
CONFIG_PM=y
CONFIG_ACPI=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
CONFIG_APM=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_MSI=y
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
CONFIG_BINFMT_ELF=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=m
CONFIG_XFRM_SUB_POLICY=y
CONFIG_NET_KEY=m
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_MULTIPATH_CACHED=y
CONFIG_IP_ROUTE_MULTIPATH_RR=m
CONFIG_IP_ROUTE_MULTIPATH_RANDOM=m
CONFIG_IP_ROUTE_MULTIPATH_WRANDOM=m
CONFIG_IP_ROUTE_MULTIPATH_DRR=m
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_NET_IPIP=m
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=m
CONFIG_INET_ESP=m
CONFIG_INET_IPCOMP=m
CONFIG_INET_XFRM_TUNNEL=m
CONFIG_INET_TUNNEL=m
CONFIG_INET_XFRM_MODE_TRANSPORT=m
CONFIG_INET_XFRM_MODE_TUNNEL=m
CONFIG_INET_XFRM_MODE_BEET=m
CONFIG_INET_DIAG=y
CONFIG_INET_TCP_DIAG=y
CONFIG_TCP_CONG_ADVANCED=y
CONFIG_TCP_CONG_BIC=y
CONFIG_TCP_CONG_CUBIC=m
CONFIG_TCP_CONG_WESTWOOD=m
CONFIG_TCP_CONG_HTCP=m
CONFIG_DEFAULT_BIC=y
CONFIG_DEFAULT_TCP_CONG="bic"
CONFIG_NETFILTER=y
CONFIG_BRIDGE_NETFILTER=y
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_QUEUE=y
CONFIG_NETFILTER_NETLINK_LOG=y
CONFIG_NF_CONNTRACK_ENABLED=y
CONFIG_NF_CONNTRACK_SUPPORT=y
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CT_ACCT=y
CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_EVENTS=y
CONFIG_NF_CT_PROTO_GRE=m
CONFIG_NF_CT_PROTO_SCTP=m
CONFIG_NF_CONNTRACK_AMANDA=m
CONFIG_NF_CONNTRACK_FTP=m
CONFIG_NF_CONNTRACK_H323=m
CONFIG_NF_CONNTRACK_IRC=m
CONFIG_NF_CONNTRACK_NETBIOS_NS=m
CONFIG_NF_CONNTRACK_PPTP=m
CONFIG_NF_CONNTRACK_SIP=m
CONFIG_NF_CONNTRACK_TFTP=m
CONFIG_NF_CT_NETLINK=m
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
CONFIG_NETFILTER_XT_TARGET_CONNMARK=m
CONFIG_NETFILTER_XT_TARGET_DSCP=m
CONFIG_NETFILTER_XT_TARGET_MARK=y
CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
CONFIG_NETFILTER_XT_TARGET_NFLOG=m
CONFIG_NETFILTER_XT_TARGET_NOTRACK=m
CONFIG_NETFILTER_XT_MATCH_COMMENT=m
CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m
CONFIG_NETFILTER_XT_MATCH_CONNMARK=m
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m
CONFIG_NETFILTER_XT_MATCH_DCCP=m
CONFIG_NETFILTER_XT_MATCH_DSCP=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_NETFILTER_XT_MATCH_HELPER=m
CONFIG_NETFILTER_XT_MATCH_LENGTH=m
CONFIG_NETFILTER_XT_MATCH_LIMIT=y
CONFIG_NETFILTER_XT_MATCH_MAC=m
CONFIG_NETFILTER_XT_MATCH_MARK=y
CONFIG_NETFILTER_XT_MATCH_POLICY=m
CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m
CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m
CONFIG_NETFILTER_XT_MATCH_QUOTA=m
CONFIG_NETFILTER_XT_MATCH_REALM=m
CONFIG_NETFILTER_XT_MATCH_SCTP=m
CONFIG_NETFILTER_XT_MATCH_STATE=y
CONFIG_NETFILTER_XT_MATCH_STATISTIC=m
CONFIG_NETFILTER_XT_MATCH_STRING=m
CONFIG_NETFILTER_XT_MATCH_TCPMSS=m
CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=y
CONFIG_NF_CONNTRACK_IPV4=y
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_QUEUE=m
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_MATCH_IPRANGE=y
CONFIG_IP_NF_MATCH_TOS=y
CONFIG_IP_NF_MATCH_RECENT=y
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_AH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_MATCH_ADDRTYPE=m
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_NF_NAT=y
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_TARGET_NETMAP=m
CONFIG_IP_NF_TARGET_SAME=m
CONFIG_NF_NAT_SNMP_BASIC=m
CONFIG_NF_NAT_PROTO_GRE=m
CONFIG_NF_NAT_FTP=m
CONFIG_NF_NAT_IRC=m
CONFIG_NF_NAT_TFTP=m
CONFIG_NF_NAT_AMANDA=m
CONFIG_NF_NAT_PPTP=m
CONFIG_NF_NAT_H323=m
CONFIG_NF_NAT_SIP=m
CONFIG_IP_NF_MANGLE=y
CONFIG_IP_NF_TARGET_TOS=y
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_TTL=m
CONFIG_IP_NF_TARGET_CLUSTERIP=m
CONFIG_IP_NF_RAW=y
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
CONFIG_IP_NF_SET=y
CONFIG_IP_NF_SET_MAX=256
CONFIG_IP_NF_SET_HASHSIZE=1024
CONFIG_IP_NF_SET_IPMAP=y
CONFIG_IP_NF_SET_MACIPMAP=y
CONFIG_IP_NF_SET_PORTMAP=y
CONFIG_IP_NF_SET_IPHASH=y
CONFIG_IP_NF_SET_NETHASH=y
CONFIG_IP_NF_SET_IPPORTHASH=y
CONFIG_IP_NF_SET_IPTREE=y
CONFIG_IP_NF_MATCH_SET=y
CONFIG_IP_NF_TARGET_SET=y
CONFIG_IP_NF_MATCH_U32=y
CONFIG_BRIDGE_NF_EBTABLES=m
CONFIG_BRIDGE_EBT_BROUTE=m
CONFIG_BRIDGE_EBT_T_FILTER=m
CONFIG_BRIDGE_EBT_T_NAT=m
CONFIG_BRIDGE_EBT_802_3=m
CONFIG_BRIDGE_EBT_AMONG=m
CONFIG_BRIDGE_EBT_ARP=m
CONFIG_BRIDGE_EBT_IP=m
CONFIG_BRIDGE_EBT_LIMIT=m
CONFIG_BRIDGE_EBT_MARK=m
CONFIG_BRIDGE_EBT_PKTTYPE=m
CONFIG_BRIDGE_EBT_STP=m
CONFIG_BRIDGE_EBT_VLAN=m
CONFIG_BRIDGE_EBT_ARPREPLY=m
CONFIG_BRIDGE_EBT_DNAT=m
CONFIG_BRIDGE_EBT_MARK_T=m
CONFIG_BRIDGE_EBT_REDIRECT=m
CONFIG_BRIDGE_EBT_SNAT=m
CONFIG_BRIDGE_EBT_LOG=m
CONFIG_BRIDGE_EBT_ULOG=m
CONFIG_BRIDGE=y
CONFIG_VLAN_8021Q=y
CONFIG_LLC=y
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_FIFO=y
CONFIG_NET_SCH_CLK_GETTIMEOFDAY=y
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=y
CONFIG_NET_SCH_HFSC=m
CONFIG_NET_SCH_PRIO=m
CONFIG_NET_SCH_RED=m
CONFIG_NET_SCH_SFQ=m
CONFIG_NET_SCH_TEQL=m
CONFIG_NET_SCH_TBF=m
CONFIG_NET_SCH_GRED=m
CONFIG_NET_SCH_DSMARK=m
CONFIG_NET_SCH_NETEM=m
CONFIG_NET_SCH_INGRESS=m
CONFIG_NET_CLS=y
CONFIG_NET_CLS_BASIC=m
CONFIG_NET_CLS_TCINDEX=m
CONFIG_NET_CLS_ROUTE4=m
CONFIG_NET_CLS_ROUTE=y
CONFIG_NET_CLS_FW=m
CONFIG_NET_CLS_U32=m
CONFIG_CLS_U32_PERF=y
CONFIG_CLS_U32_MARK=y
CONFIG_NET_CLS_RSVP=m
CONFIG_NET_CLS_RSVP6=m
CONFIG_NET_EMATCH=y
CONFIG_NET_EMATCH_STACK=32
CONFIG_NET_EMATCH_CMP=m
CONFIG_NET_EMATCH_NBYTE=m
CONFIG_NET_EMATCH_U32=m
CONFIG_NET_EMATCH_META=m
CONFIG_NET_EMATCH_TEXT=m
CONFIG_NET_CLS_ACT=y
CONFIG_NET_ACT_POLICE=m
CONFIG_NET_ACT_GACT=m
CONFIG_GACT_PROB=y
CONFIG_NET_ACT_MIRRED=m
CONFIG_NET_ACT_IPT=m
CONFIG_NET_ACT_PEDIT=m
CONFIG_NET_ACT_SIMP=m
CONFIG_NET_CLS_IND=y
CONFIG_NET_ESTIMATOR=y
CONFIG_NET_PKTGEN=m
CONFIG_FIB_RULES=y
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=m
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
CONFIG_PARPORT_1284=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_CRYPTOLOOP=m
CONFIG_BLK_DEV_RAM=m
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
CONFIG_BLK_DEV_RAM_BLOCKSIZE=1024
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_BLK_DEV_IDECD=y
CONFIG_IDE_GENERIC=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_GENERIC=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_IDEDMA_PCI_AUTO=y
CONFIG_BLK_DEV_AMD74XX=y
CONFIG_BLK_DEV_PIIX=y
CONFIG_BLK_DEV_SIIMAGE=y
CONFIG_BLK_DEV_VIA82CXXX=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_AUTO=y
CONFIG_SCSI=y
CONFIG_BLK_DEV_SD=y
CONFIG_ATA=y
CONFIG_ATA_PIIX=y
CONFIG_SATA_NV=y
CONFIG_SATA_SIL=y
CONFIG_SATA_SIL24=y
CONFIG_SATA_INTEL_COMBINED=y
CONFIG_PATA_MPIIX=y
CONFIG_PATA_VIA=y
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_LINEAR=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
CONFIG_NETDEVICES=y
CONFIG_IFB=m
CONFIG_TUN=m
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_NET_PCI=y
CONFIG_E100=m
CONFIG_8139TOO=m
CONFIG_8139TOO_PIO=y
CONFIG_E1000=m
CONFIG_E1000_NAPI=y
CONFIG_TIGON3=m
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1280
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=1024
CONFIG_INPUT_EVDEV=y
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_LIBPS2=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_CORE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_PRINTER=y
CONFIG_NVRAM=y
CONFIG_RTC=y
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_ALGOBIT=y
CONFIG_I2C_ALGOPCF=m
CONFIG_I2C_ALGOPCA=m
CONFIG_I2C_ALI1535=m
CONFIG_I2C_ALI1563=m
CONFIG_I2C_ALI15X3=m
CONFIG_I2C_AMD756=m
CONFIG_I2C_AMD756_S4882=m
CONFIG_I2C_AMD8111=m
CONFIG_I2C_I801=m
CONFIG_I2C_I810=m
CONFIG_I2C_PIIX4=m
CONFIG_I2C_ISA=m
CONFIG_I2C_NFORCE2=m
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=m
CONFIG_I2C_PROSAVAGE=m
CONFIG_I2C_SAVAGE4=m
CONFIG_SCx200_ACB=m
CONFIG_I2C_SIS5595=m
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
CONFIG_I2C_STUB=m
CONFIG_I2C_VIA=m
CONFIG_I2C_VIAPRO=m
CONFIG_I2C_VOODOO3=m
CONFIG_I2C_PCA_ISA=m
CONFIG_SENSORS_DS1337=m
CONFIG_SENSORS_DS1374=m
CONFIG_SENSORS_EEPROM=m
CONFIG_SENSORS_PCF8574=m
CONFIG_SENSORS_PCA9539=m
CONFIG_SENSORS_PCF8591=m
CONFIG_SENSORS_MAX6875=m
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
CONFIG_SENSORS_ABITUGURU=m
CONFIG_SENSORS_ADM1021=m
CONFIG_SENSORS_ADM1025=m
CONFIG_SENSORS_ADM1026=m
CONFIG_SENSORS_ADM1031=m
CONFIG_SENSORS_ADM9240=m
CONFIG_SENSORS_ASB100=m
CONFIG_SENSORS_ATXP1=m
CONFIG_SENSORS_DS1621=m
CONFIG_SENSORS_F71805F=m
CONFIG_SENSORS_FSCHER=m
CONFIG_SENSORS_FSCPOS=m
CONFIG_SENSORS_GL518SM=m
CONFIG_SENSORS_GL520SM=m
CONFIG_SENSORS_IT87=m
CONFIG_SENSORS_LM63=m
CONFIG_SENSORS_LM75=m
CONFIG_SENSORS_LM77=m
CONFIG_SENSORS_LM78=m
CONFIG_SENSORS_LM80=m
CONFIG_SENSORS_LM83=m
CONFIG_SENSORS_LM85=m
CONFIG_SENSORS_LM87=m
CONFIG_SENSORS_LM90=m
CONFIG_SENSORS_LM92=m
CONFIG_SENSORS_MAX1619=m
CONFIG_SENSORS_PC87360=m
CONFIG_SENSORS_SIS5595=m
CONFIG_SENSORS_SMSC47M1=m
CONFIG_SENSORS_SMSC47M192=m
CONFIG_SENSORS_SMSC47B397=m
CONFIG_SENSORS_VIA686A=m
CONFIG_SENSORS_VT8231=m
CONFIG_SENSORS_W83781D=m
CONFIG_SENSORS_W83791D=m
CONFIG_SENSORS_W83792D=m
CONFIG_SENSORS_W83L785TS=m
CONFIG_SENSORS_W83627HF=m
CONFIG_SENSORS_W83627EHF=m
CONFIG_SENSORS_HDAPS=m
CONFIG_FIRMWARE_EDID=y
CONFIG_FB=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_VESA=y
CONFIG_VGA_CONSOLE=y
CONFIG_VIDEO_SELECT=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_HID=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
CONFIG_USB_DEVICEFS=y
CONFIG_USB_EHCI_HCD=y
CONFIG_USB_UHCI_HCD=y
CONFIG_USB_STORAGE=y
CONFIG_USB_STORAGE_DATAFAB=y
CONFIG_USB_STORAGE_FREECOM=y
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
CONFIG_USB_STORAGE_USBAT=y
CONFIG_USB_STORAGE_SDDR09=y
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y
CONFIG_USB_STORAGE_ALAUDA=y
CONFIG_USB_STORAGE_KARMA=y
CONFIG_USB_LIBUSUAL=y
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
CONFIG_RTC_HCTOSYS=y
CONFIG_RTC_HCTOSYS_DEVICE="rtc0"
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
CONFIG_RTC_DRV_X1205=m
CONFIG_RTC_DRV_DS1307=m
CONFIG_RTC_DRV_DS1553=m
CONFIG_RTC_DRV_ISL1208=m
CONFIG_RTC_DRV_DS1672=m
CONFIG_RTC_DRV_DS1742=m
CONFIG_RTC_DRV_PCF8563=m
CONFIG_RTC_DRV_PCF8583=m
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_M48T86=m
CONFIG_RTC_DRV_TEST=m
CONFIG_RTC_DRV_V3020=m
CONFIG_DMA_ENGINE=y
CONFIG_NET_DMA=y
CONFIG_INTEL_IOATDMA=m
CONFIG_EXT2_FS=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_JBD=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
CONFIG_DNOTIFY=y
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_FAT_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=850
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_NFS_FS=m
CONFIG_NFSD=m
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=m
CONFIG_EXPORTFS=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_CIFS=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-15"
CONFIG_NLS_CODEPAGE_850=y
CONFIG_NLS_CODEPAGE_852=m
CONFIG_NLS_CODEPAGE_1250=m
CONFIG_NLS_ISO8859_1=y
CONFIG_NLS_ISO8859_15=y
CONFIG_NLS_UTF8=y
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_LOG_BUF_SHIFT=15
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_EARLY_PRINTK=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_DOUBLEFAULT=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_BLKCIPHER=m
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_MD5=m
CONFIG_CRYPTO_SHA1=m
CONFIG_CRYPTO_ECB=m
CONFIG_CRYPTO_CBC=m
CONFIG_CRYPTO_DES=m
CONFIG_CRYPTO_AES=m
CONFIG_CRYPTO_AES_586=m
CONFIG_CRYPTO_DEFLATE=m
CONFIG_CRYPTO_CRC32C=m
CONFIG_CRYPTO_DEV_GEODE=m
CONFIG_BITREVERSE=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC32=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=m
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
CONFIG_TEXTSEARCH_FSM=m
CONFIG_PLIST=y
CONFIG_IOMAP_COPY=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_KTIME_SCALAR=y
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 2.6.23.1-smp kernel panic (network-related)
2007-11-07 17:51 Fw: 2.6.23.1-smp kernel panic (network-related) Stephen Hemminger
@ 2007-11-07 18:38 ` Marek Kierdelewicz
2007-11-07 18:47 ` Stephen Hemminger
0 siblings, 1 reply; 4+ messages in thread
From: Marek Kierdelewicz @ 2007-11-07 18:38 UTC (permalink / raw)
To: Stephen Hemminger; +Cc: netdev
>What is the test input that causes the crash??
Test box is treated with mirrored traffic that is routed by production
linux router (with non-smp kernel). It's usual traffic generated by
broadband clients. Some of the characteristics:
bandwidth used: ~40/40 Mbit (up/down)
pps: ~15k
number of clients: ~550
dump of packet sizes:
Packet size | Count
1 to 75: 186501
76 to 150: 14145
151 to 225: 3285
226 to 300: 2088
301 to 375: 3632
376 to 450: 2097
451 to 525: 1513
526 to 600: 3069
601 to 675: 20081
676 to 750: 1294
751 to 825: 1189
826 to 900: 885
901 to 975: 2207
976 to 1050: 1333
1051 to 1125: 1192
1201 to 1275: 3036
1276 to 1350: 3709
1351 to 1425: 3453
1426 to 1500+: 185318
protocol breakdown:
most of the traffic is IPv4, some UDP and a little bit of ICMP
Don't know if it's important, but box is connected to switch by a
802.1q trunk. Each of vlan interfaces has egress shaping on it +
dedicated ifb device attached to ingress qdisc and ingress shaping on
ifb device.
Is any additional information needed?
--
Marek Kierdelewicz
Kierownik Działu Systemów Sieciowych, KoBa
Manager of Network Systems Department, KoBa
tel. (85) 7406466; fax. (85) 7406467
e-mail: admin@koba.pl
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 2.6.23.1-smp kernel panic (network-related)
2007-11-07 18:38 ` Marek Kierdelewicz
@ 2007-11-07 18:47 ` Stephen Hemminger
0 siblings, 0 replies; 4+ messages in thread
From: Stephen Hemminger @ 2007-11-07 18:47 UTC (permalink / raw)
To: Marek Kierdelewicz; +Cc: netdev
On Wed, 7 Nov 2007 19:38:43 +0100
Marek Kierdelewicz <marek@koba.pl> wrote:
> >What is the test input that causes the crash??
>
> Test box is treated with mirrored traffic that is routed by production
> linux router (with non-smp kernel). It's usual traffic generated by
> broadband clients. Some of the characteristics:
>
> bandwidth used: ~40/40 Mbit (up/down)
> pps: ~15k
>
> number of clients: ~550
>
> dump of packet sizes:
> Packet size | Count
> 1 to 75: 186501
> 76 to 150: 14145
> 151 to 225: 3285
> 226 to 300: 2088
> 301 to 375: 3632
> 376 to 450: 2097
> 451 to 525: 1513
> 526 to 600: 3069
> 601 to 675: 20081
> 676 to 750: 1294
> 751 to 825: 1189
> 826 to 900: 885
> 901 to 975: 2207
> 976 to 1050: 1333
> 1051 to 1125: 1192
> 1201 to 1275: 3036
> 1276 to 1350: 3709
> 1351 to 1425: 3453
> 1426 to 1500+: 185318
>
> protocol breakdown:
> most of the traffic is IPv4, some UDP and a little bit of ICMP
>
> Don't know if it's important, but box is connected to switch by a
> 802.1q trunk. Each of vlan interfaces has egress shaping on it +
> dedicated ifb device attached to ingress qdisc and ingress shaping on
> ifb device.
>
> Is any additional information needed?
Could you get 'ethtool -S' output after it hangs.
I think there is a problem with fifo overflow. I can cause board
to latch if enough traffic arrives on both ports.
--
Stephen Hemminger <shemminger@linux-foundation.org>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 2.6.23.1-smp kernel panic (network-related)
[not found] <20071107175211.1a12c8c9@catlap>
@ 2007-11-07 20:23 ` Andrew Morton
0 siblings, 0 replies; 4+ messages in thread
From: Andrew Morton @ 2007-11-07 20:23 UTC (permalink / raw)
To: Marek Kierdelewicz; +Cc: linux-kernel, netdev
(cc netdev)
> On Wed, 7 Nov 2007 17:52:11 +0100 Marek Kierdelewicz <marek@koba.pl> wrote:
> Hi there,
>
> My company's (ISP) bussines model requires dynamic resizing of the
> client queues. It's achieved by regenerating shaping rules and loading
> then using batch mode of a tc binary. On production systems it's done
> once every 1 or 2 minutes. Unfortunately this causes smp kernels to
> panic. Non-smp kernels don't have such problems. Bug is around a long
> time. I first noticed it after migrating to shaping configs that use
> IFB, it might have been 2.6.18 "era".
>
> Test scenario:
>
> I've put together a test machine with configuration copied from
> production router. I'm feeding the machine with production traffic
> by means of port mirroring. Test machine has the same config as
> production one (including mac addresses), so it tries to route the
> incoming traffic.
>
> Tested kernels were 2.6.31.1 and 2.6.20.6 (config from 2.6.20.6 is in
> attachment). Both panicked if compiled with SMP support and work stable
> otherwise. Problem occurs only with cyclic "shaping restarts". For the
> test, reload operation using "tc -b ..." was executed in an infinite
> loop.
>
> Box's CPU usage was approximately 15%. Panics occur with few hours -
> one day intervals.
>
>
> Below I attach the panic message captured via serial console:
> ----------------------------------------------------------------------
> printk: 63 messages suppressed.
> dst cache overflow
> SMP
> Modules linked in: ipt_LOG xt_hashlimit ipt_MASQUERADE ip_set_macipmap
> ip_set_ipmap xt_state w83627hf hwmon_vid eeprom ifb ipt_SET ipt_set
> ip_set ipip tunnel4 ip_gre e1000 i2c_i801 i2c_core CPU: 1 EIP:
> 0060:[<f255c08f>] Not tainted VLI EFLAGS: 00010202 (2.6.23.1-smp
> #2) EIP is at 0xf255c08f
> eax: c196a000 ebx: 00000100 ecx: ef2f408c edx: f3630000
> esi: f0332029 edi: f255c08c ebp: 00000001 esp: f3631ea8
> ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0068
> Process tc (pid: 27695, ti=f3630000 task=f26a9560 task.ti=f3630000)
> Stack: c01280ad f26a9560 00000001 f3631eb4 c0106f57 ef2f408c f0e8c08c
> 00000031 c0495308 0000000a c0124eb6 00000046 00000000 f7657740 f3630000
> c0124f4c c180f120 c0114e62 00000000 00000000 f7bfd224 f74ed95c c01047e0
> f7bfd224 Call Trace:
> [<c01280ad>] run_timer_softirq+0xf5/0x154
> [<c0106f57>] profile_pc+0x21/0x4a
> [<c0124eb6>] __do_softirq+0x5d/0xc1
> [<c0124f4c>] do_softirq+0x32/0x36
> [<c0114e62>] smp_apic_timer_interrupt+0x74/0x80
> [<c01047e0>] apic_timer_interrupt+0x28/0x30
> [<c014c82e>] remove_vma+0x1c/0x36
> [<c014c912>] exit_mmap+0xca/0xe1
> [<c011eedc>] mmput+0x1d/0x75
> [<c0123688>] do_exit+0x1be/0x68a
> [<c0123bc0>] sys_exit_group+0x0/0xd
> [<c0103d12>] sysenter_past_esp+0x5f/0x85
> =======================
> Code: 00 52 41 f7 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 00 00 00 00 00 00 fa 00 00 00 ea 05 00 00 7f 00 00 00 34 a5 96 <c1> 8c
> 00 fd f3 a5 2b 09 00 d1 d8 32 c0 00 c0 55 f2 00 a0 96 c1 EIP:
> [<f255c08f>] 0xf255c08f SS:ESP 0068:f3631ea8 Kernel panic - not
> syncing: Fatal exception in interrupt
> ----------------------------------------------------------------------
>
>
> --
> Marek Kierdelewicz
> Kierownik Działu Systemów Sieciowych, KoBa
> Manager of Network Systems Department, KoBa
> tel. (85) 7406466; fax. (85) 7406467
> e-mail: admin@koba.pl
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2007-11-07 20:23 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-11-07 17:51 Fw: 2.6.23.1-smp kernel panic (network-related) Stephen Hemminger
2007-11-07 18:38 ` Marek Kierdelewicz
2007-11-07 18:47 ` Stephen Hemminger
[not found] <20071107175211.1a12c8c9@catlap>
2007-11-07 20:23 ` Andrew Morton
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).