From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nix Subject: 3.19+: (and quite probably earlier) VIA Rhine hanging under high network load, yet again: redux Date: Sat, 04 Apr 2015 19:03:57 +0100 Message-ID: <874mov7otu.fsf@spindle.srvr.nix> Mime-Version: 1.0 Content-Type: text/plain To: rl@hellgate.ch, Francois Romieu , Bjarke Istrup Pedersen , "David S. Miller" , Linux-Netdev Return-path: Received: from icebox.esperi.org.uk ([81.187.191.129]:47101 "EHLO mail.esperi.org.uk" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752322AbbDDSZF (ORCPT ); Sat, 4 Apr 2015 14:25:05 -0400 Sender: netdev-owner@vger.kernel.org List-ID: So I started to encounter hangs with high CPU load when trafficking on multiple NICs when I added a bunch of extra VIA Rhine NICs to my Soekris net5501 board via a lan1741 board. This was in 3.18, but of course the bug could well have predated that. The hangs appear to need simultaneous rx and tx load (as commonly seen in routing situations): a iperf -t 3600 -l 100 routed through the box is sufficient to cause a crash in half an hour or so. (UDP flows don't work, not even bidirectional ones, for reasons that remain unclear but are probably related to iperf's use of a ~five-times- higher packet flow rate when using TCP, or perhaps related to my iptables rules, though those are minimal for anything TCP or UDP on either of the interfaces I tested with. It is just barely possible that two flows are required for this, and three interfaces: this box is my firewall, and it seems statistically unlikely how often the crash happens just at the instant something is coming off the internet, in through an interface uninvolved in this testing and then out of one of the interfaces that is involved.) In 3.18, everything seemed to work for the built-in four NICs (and even worked for the non-built-in ones if you kept the traffic low enough, probably because no interrupts came in while it was handling an existing packet), but in 3.19, even the built-in NICs are affected. A really crude bisection suggests that things went wrong around 87545899b, but there's no way I trust that that commit was actually at fault. (I have the ocmplete bisection log if anyone trusts it at all). The absence of crashes associated with the built-in NICs in 3.18 and below is more likely to be due to differences in the timing of IRQs coming across the single-slot PCI expansion bus than anything else, I suspect. I ran across which culminated in , moving all the rx work into the tx handler, which seemed to culminate in 7ab87ff4c. Unfortunately, this appears not to fix things, or this fix appears to have regressed -- but it's hard to say, since the only symptom I see is a hang so hard that nothing can get written out anywhere at all (if the serial console is in the middle of emitting something, it halts mid-message). The only thing that can get any response out of the machine after a hang is a powerdown or the ever-handy hardware watchdog. We sometimes see extremely high CPU loads in bursts before the point of the hang, but this appears to be unrelated, since extremely high CPU load is also observed doing perfectly hang-free bidirectional iperf flows over a single NIC (up to 75% of time consumed in the interrupt handler in both cases). I suspect that everything goes fine until an interrupt comes in at the wrong moment, and then everything hangs, so instantaneous CPU load from a machine that hasn't hung yet won't tell us anything. This is, like all Soekris net5501s, an AMD Geode LX, a UP x86-32 system, so a hang or loop in an interrupt handler means you are as dead as a very dead thing with no chance of recovery. .config follows: CONFIG_X86_32=y CONFIG_X86=y CONFIG_INSTRUCTION_DECODER=y CONFIG_PERF_EVENTS_INTEL_UNCORE=y CONFIG_OUTPUT_FORMAT="elf32-i386" CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig" CONFIG_LOCKDEP_SUPPORT=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_HAVE_LATENCYTOP_SUPPORT=y CONFIG_MMU=y CONFIG_NEED_SG_DMA_LENGTH=y CONFIG_GENERIC_ISA_DMA=y CONFIG_GENERIC_BUG=y CONFIG_GENERIC_HWEIGHT=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_RWSEM_XCHGADD_ALGORITHM=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_ARCH_HAS_CPU_RELAX=y CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y CONFIG_HAVE_SETUP_PER_CPU_AREA=y CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y CONFIG_ARCH_HIBERNATION_POSSIBLE=y CONFIG_ARCH_SUSPEND_POSSIBLE=y CONFIG_ARCH_WANT_HUGE_PMD_SHARE=y CONFIG_ARCH_WANT_GENERAL_HUGETLB=y CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y CONFIG_X86_32_LAZY_GS=y CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-ecx -fcall-saved-edx" CONFIG_ARCH_SUPPORTS_UPROBES=y CONFIG_FIX_EARLYCON_MEM=y CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config" CONFIG_IRQ_WORK=y CONFIG_BUILDTIME_EXTABLE_SORT=y CONFIG_BROKEN_ON_SMP=y CONFIG_INIT_ENV_ARG_LIMIT=32 CONFIG_CROSS_COMPILE="" CONFIG_LOCALVERSION="" CONFIG_HAVE_KERNEL_GZIP=y CONFIG_HAVE_KERNEL_BZIP2=y CONFIG_HAVE_KERNEL_LZMA=y CONFIG_HAVE_KERNEL_XZ=y CONFIG_HAVE_KERNEL_LZO=y CONFIG_HAVE_KERNEL_LZ4=y CONFIG_KERNEL_GZIP=y CONFIG_DEFAULT_HOSTNAME="fold" CONFIG_SWAP=y CONFIG_SYSVIPC=y CONFIG_SYSVIPC_SYSCTL=y CONFIG_POSIX_MQUEUE=y CONFIG_POSIX_MQUEUE_SYSCTL=y CONFIG_FHANDLE=y CONFIG_HAVE_ARCH_AUDITSYSCALL=y CONFIG_GENERIC_IRQ_PROBE=y CONFIG_GENERIC_IRQ_SHOW=y CONFIG_GENERIC_IRQ_LEGACY_ALLOC_HWIRQ=y CONFIG_IRQ_DOMAIN=y CONFIG_GENERIC_MSI_IRQ=y CONFIG_IRQ_FORCED_THREADING=y CONFIG_SPARSE_IRQ=y CONFIG_CLOCKSOURCE_WATCHDOG=y CONFIG_ARCH_CLOCKSOURCE_DATA=y CONFIG_CLOCKSOURCE_VALIDATE_LAST_CYCLE=y CONFIG_GENERIC_TIME_VSYSCALL=y CONFIG_GENERIC_CLOCKEVENTS=y CONFIG_GENERIC_CLOCKEVENTS_BUILD=y CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y CONFIG_GENERIC_CMOS_UPDATE=y CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ_COMMON=y CONFIG_NO_HZ_IDLE=y CONFIG_HIGH_RES_TIMERS=y CONFIG_TICK_CPU_ACCOUNTING=y CONFIG_BSD_PROCESS_ACCT=y CONFIG_TINY_RCU=y CONFIG_LOG_BUF_SHIFT=14 CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y CONFIG_CGROUPS=y CONFIG_CGROUP_SCHED=y CONFIG_FAIR_GROUP_SCHED=y CONFIG_SCHED_AUTOGROUP=y CONFIG_RELAY=y CONFIG_INIT_FALLBACK=y CONFIG_CC_OPTIMIZE_FOR_SIZE=y CONFIG_SYSCTL=y CONFIG_ANON_INODES=y CONFIG_HAVE_UID16=y CONFIG_SYSCTL_EXCEPTION_TRACE=y CONFIG_HAVE_PCSPKR_PLATFORM=y CONFIG_BPF=y CONFIG_EXPERT=y CONFIG_UID16=y CONFIG_SYSCTL_SYSCALL=y CONFIG_KALLSYMS=y CONFIG_PRINTK=y CONFIG_BUG=y CONFIG_ELF_CORE=y CONFIG_BASE_FULL=y CONFIG_FUTEX=y CONFIG_EPOLL=y CONFIG_SIGNALFD=y CONFIG_TIMERFD=y CONFIG_EVENTFD=y CONFIG_SHMEM=y CONFIG_AIO=y CONFIG_ADVISE_SYSCALLS=y CONFIG_PCI_QUIRKS=y CONFIG_EMBEDDED=y CONFIG_HAVE_PERF_EVENTS=y CONFIG_PERF_EVENTS=y CONFIG_VM_EVENT_COUNTERS=y CONFIG_SLUB_DEBUG=y CONFIG_SLUB=y CONFIG_HAVE_OPROFILE=y CONFIG_OPROFILE_NMI_TIMER=y CONFIG_JUMP_LABEL=y CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y CONFIG_ARCH_USE_BUILTIN_BSWAP=y CONFIG_HAVE_IOREMAP_PROT=y CONFIG_HAVE_KPROBES=y CONFIG_HAVE_KRETPROBES=y CONFIG_HAVE_OPTPROBES=y CONFIG_HAVE_KPROBES_ON_FTRACE=y CONFIG_HAVE_ARCH_TRACEHOOK=y CONFIG_HAVE_DMA_ATTRS=y CONFIG_HAVE_DMA_CONTIGUOUS=y CONFIG_GENERIC_SMP_IDLE_THREAD=y CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y CONFIG_HAVE_DMA_API_DEBUG=y CONFIG_HAVE_HW_BREAKPOINT=y CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y CONFIG_HAVE_USER_RETURN_NOTIFIER=y CONFIG_HAVE_PERF_EVENTS_NMI=y CONFIG_HAVE_PERF_REGS=y CONFIG_HAVE_PERF_USER_STACK_DUMP=y CONFIG_HAVE_ARCH_JUMP_LABEL=y CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y CONFIG_HAVE_CMPXCHG_LOCAL=y CONFIG_HAVE_CMPXCHG_DOUBLE=y CONFIG_ARCH_WANT_IPC_PARSE_VERSION=y CONFIG_HAVE_ARCH_SECCOMP_FILTER=y CONFIG_SECCOMP_FILTER=y CONFIG_HAVE_CC_STACKPROTECTOR=y CONFIG_CC_STACKPROTECTOR_NONE=y CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y CONFIG_MODULES_USE_ELF_REL=y CONFIG_CLONE_BACKWARDS=y CONFIG_OLD_SIGSUSPEND3=y CONFIG_OLD_SIGACTION=y CONFIG_ARCH_HAS_GCOV_PROFILE_ALL=y CONFIG_HAVE_GENERIC_DMA_COHERENT=y CONFIG_SLABINFO=y CONFIG_RT_MUTEXES=y CONFIG_BASE_SMALL=0 CONFIG_BLOCK=y CONFIG_BLK_DEV_BSG=y CONFIG_PARTITION_ADVANCED=y CONFIG_MSDOS_PARTITION=y CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_DEADLINE=y CONFIG_DEFAULT_DEADLINE=y CONFIG_DEFAULT_IOSCHED="deadline" CONFIG_INLINE_SPIN_UNLOCK_IRQ=y CONFIG_INLINE_READ_UNLOCK=y CONFIG_INLINE_READ_UNLOCK_IRQ=y CONFIG_INLINE_WRITE_UNLOCK=y CONFIG_INLINE_WRITE_UNLOCK_IRQ=y CONFIG_ARCH_SUPPORTS_ATOMIC_RMW=y CONFIG_ARCH_USE_QUEUE_RWLOCK=y CONFIG_ZONE_DMA=y CONFIG_X86_FEATURE_NAMES=y CONFIG_X86_MPPARSE=y CONFIG_IOSF_MBI=y CONFIG_SCHED_OMIT_FRAME_POINTER=y CONFIG_NO_BOOTMEM=y CONFIG_MGEODE_LX=y CONFIG_X86_INTERNODE_CACHE_SHIFT=5 CONFIG_X86_L1_CACHE_SHIFT=5 CONFIG_X86_USE_PPRO_CHECKSUM=y CONFIG_X86_USE_3DNOW=y CONFIG_X86_TSC=y CONFIG_X86_CMOV=y CONFIG_X86_MINIMUM_CPU_FAMILY=4 CONFIG_X86_DEBUGCTLMSR=y CONFIG_CPU_SUP_INTEL=y CONFIG_CPU_SUP_CYRIX_32=y CONFIG_CPU_SUP_AMD=y CONFIG_CPU_SUP_CENTAUR=y CONFIG_CPU_SUP_TRANSMETA_32=y CONFIG_CPU_SUP_UMC_32=y CONFIG_NR_CPUS=1 CONFIG_PREEMPT_NONE=y CONFIG_X86_UP_APIC=y CONFIG_X86_UP_APIC_MSI=y CONFIG_X86_LOCAL_APIC=y CONFIG_VM86=y CONFIG_X86_MSR=y CONFIG_X86_CPUID=y CONFIG_NOHIGHMEM=y CONFIG_VMSPLIT_3G=y CONFIG_PAGE_OFFSET=0xC0000000 CONFIG_ARCH_FLATMEM_ENABLE=y CONFIG_ARCH_SPARSEMEM_ENABLE=y CONFIG_ARCH_SELECT_MEMORY_MODEL=y CONFIG_ILLEGAL_POINTER_VALUE=0 CONFIG_SELECT_MEMORY_MODEL=y CONFIG_FLATMEM_MANUAL=y CONFIG_FLATMEM=y CONFIG_FLAT_NODE_MEM_MAP=y CONFIG_SPARSEMEM_STATIC=y CONFIG_HAVE_MEMBLOCK=y CONFIG_HAVE_MEMBLOCK_NODE_MAP=y CONFIG_ARCH_DISCARD_MEMBLOCK=y CONFIG_PAGEFLAGS_EXTENDED=y CONFIG_SPLIT_PTLOCK_CPUS=4 CONFIG_ZONE_DMA_FLAG=1 CONFIG_BOUNCE=y CONFIG_VIRT_TO_BUS=y CONFIG_DEFAULT_MMAP_MIN_ADDR=4096 CONFIG_NEED_PER_CPU_KM=y CONFIG_ZSMALLOC=y CONFIG_GENERIC_EARLY_IOREMAP=y CONFIG_X86_RESERVE_LOW=4 CONFIG_SECCOMP=y CONFIG_HZ_100=y CONFIG_HZ=100 CONFIG_SCHED_HRTICK=y CONFIG_PHYSICAL_START=0x100000 CONFIG_PHYSICAL_ALIGN=0x100000 CONFIG_PCI=y CONFIG_PCI_GOANY=y CONFIG_PCI_BIOS=y CONFIG_PCI_DIRECT=y CONFIG_PCI_DOMAINS=y CONFIG_PCI_MSI=y CONFIG_ISA_DMA_API=y CONFIG_SCx200=y CONFIG_SCx200HR_TIMER=y CONFIG_NET5501=y CONFIG_AMD_NB=y CONFIG_BINFMT_ELF=y CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y CONFIG_BINFMT_SCRIPT=y CONFIG_HAVE_AOUT=y CONFIG_BINFMT_MISC=y CONFIG_COREDUMP=y CONFIG_HAVE_ATOMIC_IOMAP=y CONFIG_PMC_ATOM=y CONFIG_NET=y CONFIG_PACKET=y CONFIG_PACKET_DIAG=y CONFIG_UNIX=y CONFIG_UNIX_DIAG=y CONFIG_INET=y CONFIG_IP_MULTICAST=y CONFIG_IP_ADVANCED_ROUTER=y CONFIG_IP_MULTIPLE_TABLES=y CONFIG_IP_ROUTE_MULTIPATH=y CONFIG_IP_ROUTE_CLASSID=y CONFIG_NET_IP_TUNNEL=y CONFIG_SYN_COOKIES=y CONFIG_INET_TUNNEL=y CONFIG_INET_DIAG=y CONFIG_INET_TCP_DIAG=y CONFIG_INET_UDP_DIAG=y CONFIG_TCP_CONG_ADVANCED=y CONFIG_TCP_CONG_CUBIC=y CONFIG_DEFAULT_CUBIC=y CONFIG_DEFAULT_TCP_CONG="cubic" CONFIG_IPV6=y CONFIG_IPV6_SIT=y CONFIG_IPV6_NDISC_NODETYPE=y CONFIG_NETFILTER=y CONFIG_NETFILTER_ADVANCED=y CONFIG_NETFILTER_NETLINK=y CONFIG_NETFILTER_NETLINK_ACCT=y CONFIG_NF_CONNTRACK=y CONFIG_NF_LOG_COMMON=y CONFIG_NF_CONNTRACK_MARK=y CONFIG_NF_CONNTRACK_TIMEOUT=y CONFIG_NF_CT_PROTO_DCCP=y CONFIG_NF_CT_PROTO_UDPLITE=y CONFIG_NF_CONNTRACK_FTP=y CONFIG_NF_CONNTRACK_IRC=y CONFIG_NF_CONNTRACK_BROADCAST=y CONFIG_NF_CONNTRACK_SNMP=y CONFIG_NF_CONNTRACK_SIP=y CONFIG_NF_CT_NETLINK=y CONFIG_NF_CT_NETLINK_TIMEOUT=y CONFIG_NF_NAT=y CONFIG_NF_NAT_NEEDED=y CONFIG_NF_NAT_PROTO_DCCP=y CONFIG_NF_NAT_PROTO_UDPLITE=y CONFIG_NF_NAT_FTP=y CONFIG_NF_NAT_IRC=y CONFIG_NF_NAT_SIP=y CONFIG_NF_NAT_REDIRECT=y CONFIG_NETFILTER_XTABLES=y CONFIG_NETFILTER_XT_TARGET_CLASSIFY=y CONFIG_NETFILTER_XT_TARGET_DSCP=y CONFIG_NETFILTER_XT_TARGET_LOG=y CONFIG_NETFILTER_XT_NAT=y CONFIG_NETFILTER_XT_TARGET_REDIRECT=y CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=y CONFIG_NETFILTER_XT_MATCH_DSCP=y CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=y CONFIG_NETFILTER_XT_MATCH_HELPER=y CONFIG_NETFILTER_XT_MATCH_IPRANGE=y CONFIG_NETFILTER_XT_MATCH_LIMIT=y CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y CONFIG_NETFILTER_XT_MATCH_NFACCT=y CONFIG_NETFILTER_XT_MATCH_OWNER=y CONFIG_NETFILTER_XT_MATCH_PKTTYPE=y CONFIG_NETFILTER_XT_MATCH_QUOTA=y CONFIG_NETFILTER_XT_MATCH_STATE=y CONFIG_NF_DEFRAG_IPV4=y CONFIG_NF_CONNTRACK_IPV4=y CONFIG_NF_LOG_IPV4=y CONFIG_NF_REJECT_IPV4=y CONFIG_NF_NAT_IPV4=y CONFIG_IP_NF_IPTABLES=y CONFIG_IP_NF_FILTER=y CONFIG_IP_NF_TARGET_REJECT=y CONFIG_IP_NF_NAT=y CONFIG_IP_NF_TARGET_REDIRECT=y CONFIG_IP_NF_MANGLE=y CONFIG_NF_DEFRAG_IPV6=y CONFIG_NF_CONNTRACK_IPV6=y CONFIG_NF_REJECT_IPV6=y CONFIG_NF_LOG_IPV6=y CONFIG_IP6_NF_IPTABLES=y CONFIG_IP6_NF_FILTER=y CONFIG_IP6_NF_TARGET_REJECT=y CONFIG_IP6_NF_MANGLE=y CONFIG_HAVE_NET_DSA=y CONFIG_NET_SCHED=y CONFIG_NET_SCH_CBQ=y CONFIG_NET_SCH_HTB=y CONFIG_NET_SCH_HFSC=y CONFIG_NET_SCH_PRIO=y CONFIG_NET_SCH_RED=y CONFIG_NET_SCH_SFQ=y CONFIG_NET_SCH_TEQL=y CONFIG_NET_SCH_TBF=y CONFIG_NET_SCH_GRED=y CONFIG_NET_SCH_DSMARK=y CONFIG_NET_SCH_INGRESS=y CONFIG_NET_CLS=y CONFIG_NET_CLS_BASIC=y CONFIG_NET_CLS_TCINDEX=y CONFIG_NET_CLS_ROUTE4=y CONFIG_NET_CLS_FW=y CONFIG_NET_CLS_U32=y CONFIG_CLS_U32_PERF=y CONFIG_CLS_U32_MARK=y CONFIG_NET_CLS_RSVP=y CONFIG_NET_EMATCH=y CONFIG_NET_EMATCH_STACK=32 CONFIG_NET_EMATCH_CMP=y CONFIG_NET_EMATCH_NBYTE=y CONFIG_NET_EMATCH_U32=y CONFIG_NET_EMATCH_META=y CONFIG_NET_CLS_ACT=y CONFIG_NET_ACT_POLICE=y CONFIG_NET_ACT_GACT=y CONFIG_NET_ACT_PEDIT=y CONFIG_NET_SCH_FIFO=y CONFIG_NETLINK_DIAG=y CONFIG_NET_RX_BUSY_POLL=y CONFIG_BQL=y CONFIG_FIB_RULES=y CONFIG_PREVENT_FIRMWARE_BUILD=y CONFIG_FW_LOADER=y CONFIG_FIRMWARE_IN_KERNEL=y CONFIG_EXTRA_FIRMWARE="" CONFIG_GENERIC_CPU_AUTOPROBE=y CONFIG_ARCH_MIGHT_HAVE_PC_PARPORT=y CONFIG_BLK_DEV=y CONFIG_ZRAM=y CONFIG_BLK_DEV_LOOP=y CONFIG_BLK_DEV_LOOP_MIN_COUNT=8 CONFIG_BLK_DEV_CRYPTOLOOP=y CONFIG_CS5535_MFGPT=y CONFIG_CS5535_MFGPT_DEFAULT_IRQ=7 CONFIG_CS5535_CLOCK_EVENT_SRC=y CONFIG_HAVE_IDE=y CONFIG_SCSI_MOD=y CONFIG_SCSI=y CONFIG_SCSI_DMA=y CONFIG_BLK_DEV_SD=y CONFIG_CHR_DEV_SG=y CONFIG_SCSI_CONSTANTS=y CONFIG_SCSI_SCAN_ASYNC=y CONFIG_SCSI_SPI_ATTRS=y CONFIG_ATA=y CONFIG_ATA_VERBOSE_ERROR=y CONFIG_ATA_SFF=y CONFIG_ATA_BMDMA=y CONFIG_PATA_CS5536=y CONFIG_NETDEVICES=y CONFIG_MII=y CONFIG_NET_CORE=y CONFIG_DUMMY=y CONFIG_NETCONSOLE=y CONFIG_NETPOLL=y CONFIG_NET_POLL_CONTROLLER=y CONFIG_ETHERNET=y CONFIG_NET_VENDOR_VIA=y CONFIG_VIA_RHINE=y CONFIG_VIA_RHINE_MMIO=y CONFIG_ARCH_MIGHT_HAVE_PC_SERIO=y CONFIG_TTY=y CONFIG_UNIX98_PTYS=y CONFIG_SERIAL_EARLYCON=y CONFIG_SERIAL_8250=y CONFIG_SERIAL_8250_CONSOLE=y CONFIG_SERIAL_8250_PCI=y CONFIG_SERIAL_8250_NR_UARTS=1 CONFIG_SERIAL_8250_RUNTIME_UARTS=1 CONFIG_SERIAL_CORE=y CONFIG_SERIAL_CORE_CONSOLE=y CONFIG_HW_RANDOM=y CONFIG_HW_RANDOM_GEODE=y CONFIG_HANGCHECK_TIMER=y CONFIG_DEVPORT=y CONFIG_I2C=y CONFIG_I2C_BOARDINFO=y CONFIG_I2C_CHARDEV=y CONFIG_I2C_HELPER_AUTO=y CONFIG_SCx200_ACB=y CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y CONFIG_GPIOLIB=y CONFIG_GPIO_DEVRES=y CONFIG_GPIO_SYSFS=y CONFIG_GPIO_CS5535=y CONFIG_HWMON=y CONFIG_HWMON_VID=y CONFIG_SENSORS_PC87360=y CONFIG_WATCHDOG=y CONFIG_WATCHDOG_CORE=y CONFIG_GEODE_WDT=y CONFIG_SSB_POSSIBLE=y CONFIG_BCMA_POSSIBLE=y CONFIG_MFD_CORE=y CONFIG_MFD_CS5535=y CONFIG_USB_OHCI_LITTLE_ENDIAN=y CONFIG_USB_SUPPORT=y CONFIG_USB_COMMON=y CONFIG_USB_ARCH_HAS_HCD=y CONFIG_USB=y CONFIG_USB_DEFAULT_PERSIST=y CONFIG_USB_DYNAMIC_MINORS=y CONFIG_USB_EHCI_HCD=y CONFIG_USB_EHCI_PCI=y CONFIG_USB_OHCI_HCD=y CONFIG_USB_OHCI_HCD_PCI=y CONFIG_USB_ACM=y CONFIG_USB_STORAGE=y CONFIG_RTC_LIB=y CONFIG_STAGING=y CONFIG_CLKSRC_I8253=y CONFIG_CLKEVT_I8253=y CONFIG_CLKBLD_I8253=y CONFIG_DCACHE_WORD_ACCESS=y CONFIG_EXT2_FS=y CONFIG_EXT2_FS_XATTR=y CONFIG_EXT2_FS_POSIX_ACL=y CONFIG_EXT2_FS_SECURITY=y CONFIG_FS_MBCACHE=y CONFIG_FS_POSIX_ACL=y CONFIG_EXPORTFS=y CONFIG_FILE_LOCKING=y CONFIG_FSNOTIFY=y CONFIG_DNOTIFY=y CONFIG_INOTIFY_USER=y CONFIG_QUOTA=y CONFIG_PRINT_QUOTA_WARNING=y CONFIG_QUOTA_TREE=y CONFIG_QFMT_V2=y CONFIG_QUOTACTL=y CONFIG_PROC_FS=y CONFIG_PROC_SYSCTL=y CONFIG_PROC_PAGE_MONITOR=y CONFIG_KERNFS=y CONFIG_SYSFS=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y CONFIG_TMPFS_XATTR=y CONFIG_NETWORK_FILESYSTEMS=y CONFIG_NFS_FS=y CONFIG_NFS_V3=y CONFIG_NFS_V3_ACL=y CONFIG_NFS_SWAP=y CONFIG_GRACE_PERIOD=y CONFIG_LOCKD=y CONFIG_LOCKD_V4=y CONFIG_NFS_ACL_SUPPORT=y CONFIG_NFS_COMMON=y CONFIG_SUNRPC=y CONFIG_SUNRPC_SWAP=y CONFIG_NLS=y CONFIG_NLS_DEFAULT="utf8" CONFIG_NLS_CODEPAGE_437=y CONFIG_NLS_CODEPAGE_850=y CONFIG_NLS_CODEPAGE_852=y CONFIG_NLS_ASCII=y CONFIG_NLS_ISO8859_1=y CONFIG_NLS_ISO8859_2=y CONFIG_NLS_ISO8859_15=y CONFIG_NLS_UTF8=y CONFIG_TRACE_IRQFLAGS_SUPPORT=y CONFIG_PRINTK_TIME=y CONFIG_MESSAGE_LOGLEVEL_DEFAULT=4 CONFIG_ENABLE_WARN_DEPRECATED=y CONFIG_ENABLE_MUST_CHECK=y CONFIG_FRAME_WARN=1024 CONFIG_STRIP_ASM_SYMS=y CONFIG_ARCH_WANT_FRAME_POINTERS=y CONFIG_DEBUG_KERNEL=y CONFIG_HAVE_DEBUG_KMEMLEAK=y CONFIG_HAVE_DEBUG_STACKOVERFLOW=y CONFIG_HAVE_ARCH_KMEMCHECK=y CONFIG_DETECT_HUNG_TASK=y CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120 CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0 CONFIG_PANIC_ON_OOPS=y CONFIG_PANIC_ON_OOPS_VALUE=1 CONFIG_PANIC_TIMEOUT=5 CONFIG_SCHED_STACK_END_CHECK=y CONFIG_DEBUG_BUGVERBOSE=y CONFIG_ARCH_HAS_DEBUG_STRICT_USER_COPY_CHECKS=y CONFIG_USER_STACKTRACE_SUPPORT=y CONFIG_HAVE_FUNCTION_TRACER=y CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y CONFIG_HAVE_DYNAMIC_FTRACE=y CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y CONFIG_HAVE_SYSCALL_TRACEPOINTS=y CONFIG_HAVE_C_RECORDMCOUNT=y CONFIG_TRACING_SUPPORT=y CONFIG_HAVE_ARCH_KGDB=y CONFIG_STRICT_DEVMEM=y CONFIG_DEBUG_RODATA=y CONFIG_DOUBLEFAULT=y CONFIG_HAVE_MMIOTRACE_SUPPORT=y CONFIG_IO_DELAY_TYPE_0X80=0 CONFIG_IO_DELAY_TYPE_0XED=1 CONFIG_IO_DELAY_TYPE_UDELAY=2 CONFIG_IO_DELAY_TYPE_NONE=3 CONFIG_IO_DELAY_0X80=y CONFIG_DEFAULT_IO_DELAY_TYPE=0 CONFIG_SECURITY_DMESG_RESTRICT=y CONFIG_SECURITY=y CONFIG_SECURITYFS=y CONFIG_SECURITY_PATH=y CONFIG_SECURITY_YAMA=y CONFIG_SECURITY_YAMA_STACKED=y CONFIG_DEFAULT_SECURITY_YAMA=y CONFIG_DEFAULT_SECURITY="yama" CONFIG_CRYPTO=y CONFIG_CRYPTO_ALGAPI=y CONFIG_CRYPTO_ALGAPI2=y CONFIG_CRYPTO_AEAD2=y CONFIG_CRYPTO_BLKCIPHER=y CONFIG_CRYPTO_BLKCIPHER2=y CONFIG_CRYPTO_HASH2=y CONFIG_CRYPTO_RNG2=y CONFIG_CRYPTO_PCOMP2=y CONFIG_CRYPTO_MANAGER=y CONFIG_CRYPTO_MANAGER2=y CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y CONFIG_CRYPTO_WORKQUEUE=y CONFIG_CRYPTO_CBC=y CONFIG_CRYPTO_AES=y CONFIG_CRYPTO_HW=y CONFIG_CRYPTO_DEV_GEODE=y CONFIG_HAVE_KVM=y CONFIG_BITREVERSE=y CONFIG_GENERIC_STRNCPY_FROM_USER=y CONFIG_GENERIC_STRNLEN_USER=y CONFIG_GENERIC_NET_UTILS=y CONFIG_GENERIC_FIND_FIRST_BIT=y CONFIG_GENERIC_PCI_IOMAP=y CONFIG_GENERIC_IOMAP=y CONFIG_GENERIC_IO=y CONFIG_ARCH_HAS_FAST_MULTIPLIER=y CONFIG_CRC32=y CONFIG_CRC32_SLICEBY8=y CONFIG_LZO_COMPRESS=y CONFIG_LZO_DECOMPRESS=y CONFIG_HAS_IOMEM=y CONFIG_HAS_IOPORT_MAP=y CONFIG_HAS_DMA=y CONFIG_DQL=y CONFIG_GLOB=y CONFIG_NLATTR=y CONFIG_ARCH_HAS_ATOMIC64_DEC_IF_POSITIVE=y CONFIG_ARCH_HAS_SG_CHAIN=y