x86/mce merge, integration hickup + crash, design thoughts

x86/mce merge, integration hickup + crash, design thoughts

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 5954 bytes --]


hi,

today i (belatedly ...) started looking into the status of the tip/x86/mce 
branch, and merged it into tip/master as a first step.

firstly there's a small complication, it triggers this crash with the 
attached config:

  Enabling fast FPU save and restore... done.
  Enabling unmasked SIMD FPU exception support... done.
  Initializing CPU#0
  RCU-based detection of stalled CPUs is enabled.
  ------------[ cut here ]------------
  Kernel BUG at c0b2a7e8 [verbose debug info unavailable]
  invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
  last sysfs file: 
  Modules linked in:
  
  Pid: 0, comm: swapper Not tainted (2.6.28-tip #12389) 
  EIP: 0060:[<c0b2a7e8>] EFLAGS: 00010002 CPU: 0
  EIP is at native_init_IRQ+0x3a8/0x3d0
  EAX: c0108e00 EBX: c0b1ffb8 ECX: c0ba8500 EDX: 00603b1c
  ESI: c0b1ffb4 EDI: c0b14000 EBP: c0b1ffc4 ESP: c0b1ff64
   DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
  Process swapper (pid: 0, ti=c0b1e000 task=c0a703c0 task.ti=c0b1e000)
  Stack:
   c0a74d20 c0b1ff70 c015547b c0b1ff94 00603b1c c0108e00 00603b50 c0108e00
   00603ae8 c0108e00 00603a84 c0108e00 00603a50 c0108e00 00603a1c c0108e00
   006039e8 c0108e00 006039b4 c0108e00 00603978 c0108e00 00000001 c0b673a0
  Call Trace:
   [<c015547b>] ? trace_hardirqs_off+0xb/0x10
   [<c0108e00>] ? mwait_idle_with_hints+0x30/0x60
   [<c0108e00>] ? mwait_idle_with_hints+0x30/0x60
   [<c0108e00>] ? mwait_idle_with_hints+0x30/0x60
   [<c0108e00>] ? mwait_idle_with_hints+0x30/0x60
   [<c0108e00>] ? mwait_idle_with_hints+0x30/0x60
   [<c0108e00>] ? mwait_idle_with_hints+0x30/0x60
   [<c0108e00>] ? mwait_idle_with_hints+0x30/0x60
   [<c0108e00>] ? mwait_idle_with_hints+0x30/0x60
   [<c0108e00>] ? mwait_idle_with_hints+0x30/0x60
   [<c0b266ef>] ? start_kernel+0x1af/0x300
   [<c0b26220>] ? unknown_bootoption+0x0/0x220
   [<c0b2606b>] ? i386_start_kernel+0x6b/0x80
  Code: 18 f6 05 98 85 b1 c0 01 75 0f ba 20 23 a7 c0 b8 0d 00 00 00 e8 fa 4d 64 ff 83 c4 58 5b 5e 5d c3 8d 76 00 0f 0b eb fe 0f 0b eb fe <0f> 0b eb fe 0f 0b 89 f6 eb fc 0f 0b eb fe 0f 0b eb fe 0f 0b 8d 
  EIP: [<c0b2a7e8>] native_init_IRQ+0x3a8/0x3d0 SS:ESP 0068:c0b1ff64
  ---[ end trace 4eaa2a86a8e2da22 ]---
  Kernel panic - not syncing: Attempted to kill the idle task!
  ------------[ cut here ]------------

i've pushed out the broken branch here:

   git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git tmp.x86.mce.broken

and have attached the bad config. I think this must be some integration 
artifact - recent changes in the x86 tree interacting with the x86/mce 
changes. Havent had time to track it down to a specific commit.

If this is fixed i'll merge it into tip/master and will push it out to 
linux-next as well as the code looks good otherwise (with a few 
reservations, see below).

I also looked at the code itself, and generally it's pretty nice. I have a 
few general observations though that need to be addressed - i think these 
can all be solved in this (elongated) merge window so that we can merge it 
all into v2.6.29:

We really need to get rid of /dev/mcelog. It's a quirky binary logging 
facility not available on 32-bit on current kernels and it has a couple of 
limitations:

  - it squeezes all MCE errors from the whole system into a small,
    32-entry ringbuffer. 

  - it puts all the MCE logging info into an intermediary binary log 
    record format: 'struct mce' - just for userspace to in essence
    printf out those entries with minimal post-processing. The fact that 
    we squeeze all information into a fixed-size binary record makes it 
    hard to extend and complicates the code needlessly.

  - these design aspects are also quite harmful to usability: by
    default all MCEs are fatal currently (pre-Nehalem anyway), so 
    /dev/mcelog will only be used if a user goes out on a limb to 
    configure it and sets the tolerant flag.

A far more useful design for handling MCE events would be to feed them 
into printk logging. So instead of printing such rather cryptic error 
messages:

   MCE 0
   HARDWARE ERROR. This is *NOT* a software problem!
   Please contact your hardware vendor
   CPU 0 BANK 6 MISC 202d ADDR ffeef740
   This is not a software problem!
   Run through mcelog --ascii to decode and contact your hardware vendor

and expecting people to run mcelog, we should print plain-text something 
like:

   MCE 0
   HARDWARE ERROR. This is *NOT* a software problem!
   Please contact your hardware vendor
   CPU 1 4 northbridge TSC 89a560bb249
   ADDR 1dfa49690
     Northbridge Chipkill ECC error
     Chipkill ECC syndrome = 2021
          bit46 = corrected ecc error
     bus error 'local node response, request didn't time out
         generic read mem transaction
         memory access, level generic'
   STATUS 9410c00020080a13 MCGSTATUS 0

straight from the kernel. This means that the MCEs will make a lot more 
sense at a glance - and the user can figure out the suspected trouble 
area, without having to find some other box to run mcelog on, etc. We can 
eliminate the user-space mcelog utility/daemon component altogether - it 
buys us little but needless complexity and inflexibility.

If we want to enable userspace to capture MCE events, then it must be done 
in a way that benefits the whole kernel, not just x86: a structured 
logging facility that is in essence a printk variant and is ASCII driven. 

Such event sources should be discoverable, and only 'aware' printouts 
should go into this new facility (not all printks). Demultiplexing should 
be easy and well-defined.

I.e. we could use this opportunity of the MCE code unification to bring 
the code to the next level - and not prolongue to broken concepts of the 
past.

I'd be glad to help out with any portion of this, it should be easy to 
solve and it will clearly improve the code. For .29 we could just do a raw 
printk based approach with no decoding just yet, and layer smart decoding 
and structured logging for .30.

Hm?

	Ingo

[-- Attachment #2: config-Sat_Dec_27_12_47_38_CET_2008.bad --]
[-- Type: text/plain, Size: 60002 bytes --]

# e0243ce4
#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.28
# Sat Dec 27 12:47:38 2008
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
# CONFIG_X86_64 is not set
CONFIG_X86=y
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/i386_defconfig"
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
# CONFIG_RWSEM_GENERIC_SPINLOCK is not set
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
# CONFIG_GENERIC_TIME_VSYSCALL is not set
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
# CONFIG_HAVE_CPUMASK_OF_CPU_MAP is not set
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
# CONFIG_ZONE_DMA32 is not set
CONFIG_ARCH_POPULATES_NODE_MAP=y
# CONFIG_AUDIT_ARCH is not set
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_X86_SMP=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_X86_32_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_KTIME_SCALAR=y
CONFIG_BOOTPARAM_SUPPORT_NOT_WANTED=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_BROKEN_BOOT_ALLOWED4=y
CONFIG_BROKEN_BOOT_ALLOWED3=y
CONFIG_BROKEN_BOOT_ALLOWED2=y
# CONFIG_BROKEN_BOOT_ALLOWED is not set
CONFIG_BROKEN_BOOT_EUROPE=y
CONFIG_BROKEN_BOOT_TITAN=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
# CONFIG_LOCALVERSION_AUTO is not set
# CONFIG_SWAP is not set
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
# CONFIG_BSD_PROCESS_ACCT is not set
CONFIG_TASKSTATS=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_XACCT=y
# CONFIG_TASK_IO_ACCOUNTING is not set
CONFIG_AUDIT=y
CONFIG_AUDITSYSCALL=y
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=21
# CONFIG_CGROUPS is not set
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_EMBEDDED=y
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_COMPAT_BRK=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_ANON_INODES=y
# CONFIG_EPOLL is not set
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
# CONFIG_SHMEM is not set
# CONFIG_AIO is not set
CONFIG_HAVE_PERF_COUNTERS=y

#
# Performance Counters
#
CONFIG_PERF_COUNTERS=y
# CONFIG_VM_EVENT_COUNTERS is not set
# CONFIG_PCI_QUIRKS is not set
CONFIG_SLUB_DEBUG=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
CONFIG_TRACEPOINTS=y
CONFIG_MARKERS=y
CONFIG_HAVE_OPROFILE=y
CONFIG_KPROBES=y
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_KRETPROBES=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_GENERIC_DMA_COHERENT=y
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_TINY_SHMEM=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
CONFIG_MODULE_SRCVERSION_ALL=y
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_LBD=y
CONFIG_BLK_DEV_IO_TRACE=y
# CONFIG_LSF is not set
CONFIG_BLK_DEV_BSG=y
CONFIG_BLK_DEV_INTEGRITY=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
# CONFIG_IOSCHED_CFQ is not set
CONFIG_DEFAULT_AS=y
# CONFIG_DEFAULT_DEADLINE is not set
# CONFIG_DEFAULT_CFQ is not set
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="anticipatory"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_CLASSIC_RCU=y
# CONFIG_TREE_RCU is not set
# CONFIG_PREEMPT_RCU is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_PREEMPT_RCU_TRACE is not set
# CONFIG_FREEZER is not set

#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
# CONFIG_SMP_SUPPORT is not set
CONFIG_SPARSE_IRQ=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
# CONFIG_UP_WANTED_1 is not set
CONFIG_SMP=y
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_VSMP is not set
CONFIG_X86_RDC321X=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
# CONFIG_PARAVIRT_GUEST is not set
CONFIG_MEMTEST=y
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
CONFIG_M686=y
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_GENERIC=y
CONFIG_X86_CPU=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_X86_XADD=y
CONFIG_X86_PPRO_FENCE=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=4
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_PROCESSOR_SELECT=y
# CONFIG_CPU_SUP_INTEL is not set
# CONFIG_CPU_SUP_CYRIX_32 is not set
# CONFIG_CPU_SUP_AMD is not set
CONFIG_CPU_SUP_CENTAUR_32=y
CONFIG_CPU_SUP_TRANSMETA_32=y
CONFIG_CPU_SUP_UMC_32=y
CONFIG_X86_DS=y
CONFIG_X86_PTRACE_BTS=y
# CONFIG_HPET_TIMER is not set
# CONFIG_DMI is not set
# CONFIG_IOMMU_HELPER is not set
CONFIG_NR_CPUS=8
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_X86_MCE_P4THERMAL=y
CONFIG_VM86=y
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
CONFIG_X86_REBOOTFIXUPS=y
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_VMSPLIT_3G=y
# CONFIG_VMSPLIT_3G_OPT is not set
# CONFIG_VMSPLIT_2G is not set
# CONFIG_VMSPLIT_2G_OPT is not set
# CONFIG_VMSPLIT_1G is not set
CONFIG_PAGE_OFFSET=0xC0000000
CONFIG_HIGHMEM=y
# CONFIG_ARCH_PHYS_ADDR_T_64BIT is not set
CONFIG_ARCH_FLATMEM_ENABLE=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ILLEGAL_POINTER_VALUE=0
CONFIG_SELECT_MEMORY_MODEL=y
CONFIG_FLATMEM_MANUAL=y
# CONFIG_DISCONTIGMEM_MANUAL is not set
# CONFIG_SPARSEMEM_MANUAL is not set
CONFIG_FLATMEM=y
CONFIG_FLAT_NODE_MEM_MAP=y
CONFIG_SPARSEMEM_STATIC=y
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_RESOURCES_64BIT=y
# CONFIG_PHYS_ADDR_T_64BIT is not set
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_UNEVICTABLE_LRU=y
CONFIG_MMU_NOTIFIER=y
CONFIG_HIGHPTE=y
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
# CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK is not set
CONFIG_X86_RESERVE_LOW_64K=y
CONFIG_MATH_EMULATION=y
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=0
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
# CONFIG_X86_PAT is not set
# CONFIG_SECCOMP is not set
# CONFIG_HZ_100 is not set
CONFIG_HZ_250=y
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=250
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PHYSICAL_START=0x100000
CONFIG_RELOCATABLE=y
CONFIG_PHYSICAL_ALIGN=0x100000
CONFIG_HOTPLUG_CPU=y
# CONFIG_COMPAT_VDSO is not set
CONFIG_CMDLINE_BOOL=y
CONFIG_CMDLINE=""
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

#
# Power management and ACPI options
#
# CONFIG_PM is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
# CONFIG_CPU_FREQ_DEBUG is not set
CONFIG_CPU_FREQ_STAT=y
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_GOV_ONDEMAND=m
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=m

#
# CPUFreq processor drivers
#
CONFIG_X86_POWERNOW_K6=y
CONFIG_X86_POWERNOW_K7=m
CONFIG_X86_POWERNOW_K8=m
CONFIG_X86_GX_SUSPMOD=m
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
CONFIG_X86_SPEEDSTEP_ICH=m
CONFIG_X86_SPEEDSTEP_SMI=y
CONFIG_X86_P4_CLOCKMOD=y
CONFIG_X86_CPUFREQ_NFORCE2=m
CONFIG_X86_LONGRUN=m
# CONFIG_X86_E_POWERSAVER is not set

#
# shared options
#
CONFIG_X86_SPEEDSTEP_LIB=y
CONFIG_X86_SPEEDSTEP_RELAXED_CAP_CHECK=y
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
# CONFIG_PCI_GOOLPC is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_OLPC=y
CONFIG_PCI_DOMAINS=y
# CONFIG_PCIEPORTBUS is not set
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
# CONFIG_PCI_LEGACY is not set
# CONFIG_PCI_DEBUG is not set
CONFIG_HT_IRQ=y
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
CONFIG_EISA=y
# CONFIG_EISA_VLB_PRIMING is not set
CONFIG_EISA_PCI_EISA=y
CONFIG_EISA_VIRTUAL_ROOT=y
CONFIG_EISA_NAMES=y
CONFIG_MCA=y
CONFIG_MCA_LEGACY=y
# CONFIG_MCA_PROC_FS is not set
CONFIG_SCx200=y
# CONFIG_SCx200HR_TIMER is not set
CONFIG_OLPC=y
CONFIG_K8_NB=y
CONFIG_PCCARD=y
CONFIG_PCMCIA_DEBUG=y
CONFIG_PCMCIA=m
CONFIG_PCMCIA_LOAD_CIS=y
# CONFIG_PCMCIA_IOCTL is not set
# CONFIG_CARDBUS is not set

#
# PC-card bridges
#
# CONFIG_YENTA is not set
CONFIG_PD6729=m
CONFIG_I82092=m
CONFIG_I82365=m
CONFIG_TCIC=m
CONFIG_PCMCIA_PROBE=y
CONFIG_PCCARD_NONSTATIC=m
CONFIG_HOTPLUG_PCI=m
CONFIG_HOTPLUG_PCI_FAKE=m
CONFIG_HOTPLUG_PCI_IBM=m
CONFIG_HOTPLUG_PCI_CPCI=y
# CONFIG_HOTPLUG_PCI_CPCI_ZT5550 is not set
# CONFIG_HOTPLUG_PCI_CPCI_GENERIC is not set
# CONFIG_HOTPLUG_PCI_SHPC is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
CONFIG_HAVE_AOUT=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=y
CONFIG_HAVE_ATOMIC_IOMAP=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
CONFIG_XFRM_USER=y
# CONFIG_XFRM_SUB_POLICY is not set
CONFIG_XFRM_MIGRATE=y
# CONFIG_XFRM_STATISTICS is not set
CONFIG_NET_KEY=y
# CONFIG_NET_KEY_MIGRATE is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_VERBOSE=y
CONFIG_IP_PNP=y
# CONFIG_IP_PNP_DHCP is not set
CONFIG_IP_PNP_BOOTP=y
CONFIG_IP_PNP_RARP=y
CONFIG_NET_IPIP=y
CONFIG_NET_IPGRE=m
CONFIG_NET_IPGRE_BROADCAST=y
CONFIG_IP_MROUTE=y
CONFIG_IP_PIMSM_V1=y
CONFIG_IP_PIMSM_V2=y
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
CONFIG_INET_ESP=y
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
CONFIG_INET_TUNNEL=y
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
# CONFIG_IPV6 is not set
CONFIG_NETLABEL=y
CONFIG_NETWORK_SECMARK=y
# CONFIG_NETFILTER is not set
# CONFIG_IP_DCCP is not set
CONFIG_IP_SCTP=m
CONFIG_SCTP_DBG_MSG=y
CONFIG_SCTP_DBG_OBJCNT=y
# CONFIG_SCTP_HMAC_NONE is not set
# CONFIG_SCTP_HMAC_SHA1 is not set
CONFIG_SCTP_HMAC_MD5=y
# CONFIG_TIPC is not set
CONFIG_ATM=m
# CONFIG_ATM_CLIP is not set
CONFIG_ATM_LANE=m
CONFIG_ATM_MPOA=m
CONFIG_ATM_BR2684=m
CONFIG_ATM_BR2684_IPFILTER=y
# CONFIG_BRIDGE is not set
# CONFIG_NET_DSA is not set
CONFIG_VLAN_8021Q=y
# CONFIG_VLAN_8021Q_GVRP is not set
CONFIG_DECNET=m
CONFIG_DECNET_ROUTER=y
CONFIG_LLC=y
CONFIG_LLC2=y
CONFIG_IPX=y
CONFIG_IPX_INTERN=y
CONFIG_ATALK=m
# CONFIG_DEV_APPLETALK is not set
CONFIG_X25=y
# CONFIG_LAPB is not set
CONFIG_ECONET=y
CONFIG_ECONET_AUNUDP=y
CONFIG_ECONET_NATIVE=y
CONFIG_WAN_ROUTER=y
CONFIG_NET_SCHED=y

#
# Queueing/Scheduling
#
CONFIG_NET_SCH_CBQ=m
CONFIG_NET_SCH_HTB=m
CONFIG_NET_SCH_HFSC=y
CONFIG_NET_SCH_ATM=m
CONFIG_NET_SCH_PRIO=y
CONFIG_NET_SCH_MULTIQ=y
# CONFIG_NET_SCH_RED is not set
CONFIG_NET_SCH_SFQ=y
CONFIG_NET_SCH_TEQL=y
# CONFIG_NET_SCH_TBF is not set
CONFIG_NET_SCH_GRED=y
CONFIG_NET_SCH_DSMARK=m
# CONFIG_NET_SCH_NETEM is not set
# CONFIG_NET_SCH_INGRESS is not set

#
# Classification
#
CONFIG_NET_CLS=y
# CONFIG_NET_CLS_BASIC is not set
# CONFIG_NET_CLS_TCINDEX is not set
# CONFIG_NET_CLS_ROUTE4 is not set
CONFIG_NET_CLS_FW=m
# CONFIG_NET_CLS_U32 is not set
CONFIG_NET_CLS_RSVP=y
# CONFIG_NET_CLS_RSVP6 is not set
# CONFIG_NET_CLS_FLOW is not set
# CONFIG_NET_EMATCH is not set
CONFIG_NET_CLS_ACT=y
# CONFIG_NET_ACT_POLICE is not set
# CONFIG_NET_ACT_GACT is not set
CONFIG_NET_ACT_MIRRED=m
# CONFIG_NET_ACT_NAT is not set
CONFIG_NET_ACT_PEDIT=y
# CONFIG_NET_ACT_SIMP is not set
# CONFIG_NET_ACT_SKBEDIT is not set
CONFIG_NET_CLS_IND=y
CONFIG_NET_SCH_FIFO=y

#
# Network testing
#
CONFIG_NET_PKTGEN=y
CONFIG_NET_TCPPROBE=y
# CONFIG_HAMRADIO is not set
CONFIG_CAN=m
# CONFIG_CAN_RAW is not set
# CONFIG_CAN_BCM is not set

#
# CAN Device Drivers
#
CONFIG_CAN_VCAN=m
CONFIG_CAN_DEBUG_DEVICES=y
CONFIG_IRDA=y

#
# IrDA protocols
#
# CONFIG_IRLAN is not set
# CONFIG_IRNET is not set
CONFIG_IRCOMM=m
# CONFIG_IRDA_ULTRA is not set

#
# IrDA options
#
CONFIG_IRDA_CACHE_LAST_LSAP=y
# CONFIG_IRDA_FAST_RR is not set
CONFIG_IRDA_DEBUG=y

#
# Infrared-port device drivers
#

#
# SIR device drivers
#
CONFIG_IRTTY_SIR=m

#
# Dongle support
#
CONFIG_DONGLE=y
CONFIG_ESI_DONGLE=m
# CONFIG_ACTISYS_DONGLE is not set
CONFIG_TEKRAM_DONGLE=m
CONFIG_TOIM3232_DONGLE=m
CONFIG_LITELINK_DONGLE=m
CONFIG_MA600_DONGLE=m
CONFIG_GIRBIL_DONGLE=m
CONFIG_MCP2120_DONGLE=m
CONFIG_OLD_BELKIN_DONGLE=m
CONFIG_ACT200L_DONGLE=m
CONFIG_KINGSUN_DONGLE=m
# CONFIG_KSDAZZLE_DONGLE is not set
CONFIG_KS959_DONGLE=y

#
# FIR device drivers
#
CONFIG_USB_IRDA=m
# CONFIG_SIGMATEL_FIR is not set
CONFIG_NSC_FIR=y
CONFIG_WINBOND_FIR=y
CONFIG_TOSHIBA_FIR=m
CONFIG_SMC_IRCC_FIR=y
CONFIG_ALI_FIR=y
CONFIG_VLSI_FIR=m
CONFIG_VIA_FIR=y
# CONFIG_MCS_FIR is not set
# CONFIG_BT is not set
CONFIG_AF_RXRPC=y
# CONFIG_AF_RXRPC_DEBUG is not set
CONFIG_RXKAD=m
# CONFIG_PHONET is not set
CONFIG_FIB_RULES=y
CONFIG_WIRELESS=y
CONFIG_CFG80211=y
CONFIG_NL80211=y
CONFIG_WIRELESS_OLD_REGULATORY=y
CONFIG_WIRELESS_EXT=y
CONFIG_WIRELESS_EXT_SYSFS=y
# CONFIG_MAC80211 is not set
CONFIG_IEEE80211=y
CONFIG_IEEE80211_DEBUG=y
# CONFIG_IEEE80211_CRYPT_WEP is not set
CONFIG_IEEE80211_CRYPT_CCMP=m
CONFIG_IEEE80211_CRYPT_TKIP=y
CONFIG_RFKILL=y
# CONFIG_RFKILL_INPUT is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
CONFIG_DEBUG_DRIVER=y
CONFIG_DEBUG_DEVRES=y
# CONFIG_SYS_HYPERVISOR is not set
# CONFIG_CONNECTOR is not set
CONFIG_PARPORT=y
CONFIG_PARPORT_PC=y
CONFIG_PARPORT_SERIAL=m
CONFIG_PARPORT_PC_FIFO=y
# CONFIG_PARPORT_PC_SUPERIO is not set
CONFIG_PARPORT_PC_PCMCIA=m
# CONFIG_PARPORT_GSC is not set
CONFIG_PARPORT_AX88796=m
# CONFIG_PARPORT_1284 is not set
CONFIG_PARPORT_NOT_PC=y
CONFIG_PNP=y
# CONFIG_PNP_DEBUG_MESSAGES is not set

#
# Protocols
#
CONFIG_ISAPNP=y
CONFIG_PNPBIOS=y
# CONFIG_PNPBIOS_PROC_FS is not set
# CONFIG_PNPACPI is not set
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
CONFIG_BLK_DEV_XD=y
CONFIG_BLK_CPQ_DA=y
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_CRYPTOLOOP=m
# CONFIG_BLK_DEV_NBD is not set
CONFIG_BLK_DEV_SX8=m
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=y
CONFIG_CDROM_PKTCDVD_BUFFERS=8
CONFIG_CDROM_PKTCDVD_WCACHE=y
CONFIG_ATA_OVER_ETH=m
# CONFIG_BLK_DEV_HD is not set
# CONFIG_MISC_DEVICES is not set
CONFIG_TIFM_CORE=y
CONFIG_HAVE_IDE=y

#
# SCSI device support
#
CONFIG_RAID_ATTRS=y
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=y
# CONFIG_SCSI_NETLINK is not set
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=y
CONFIG_CHR_DEV_OSST=m
# CONFIG_BLK_DEV_SR is not set
# CONFIG_CHR_DEV_SG is not set
CONFIG_CHR_DEV_SCH=y

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
# CONFIG_SCSI_FC_ATTRS is not set
CONFIG_SCSI_ISCSI_ATTRS=y
CONFIG_SCSI_SAS_ATTRS=m
# CONFIG_SCSI_SRP_ATTRS is not set
# CONFIG_SCSI_LOWLEVEL is not set
CONFIG_SCSI_AIC7XXX=y
# CONFIG_SCSI_LOWLEVEL_PCMCIA is not set
# CONFIG_SCSI_DH is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_SATA_PMP=y
CONFIG_SATA_AHCI=y
# CONFIG_SATA_SIL24 is not set
CONFIG_ATA_SFF=y
CONFIG_SATA_SVW=m
CONFIG_ATA_PIIX=y
# CONFIG_SATA_MV is not set
CONFIG_SATA_NV=y
CONFIG_PDC_ADMA=y
CONFIG_SATA_QSTOR=m
CONFIG_SATA_PROMISE=y
CONFIG_SATA_SX4=y
CONFIG_SATA_SIL=y
CONFIG_SATA_SIS=y
# CONFIG_SATA_ULI is not set
CONFIG_SATA_VIA=m
CONFIG_SATA_VITESSE=y
CONFIG_SATA_INIC162X=y
CONFIG_PATA_ALI=y
CONFIG_PATA_AMD=y
CONFIG_PATA_ARTOP=m
CONFIG_PATA_ATIIXP=m
# CONFIG_PATA_CMD640_PCI is not set
CONFIG_PATA_CMD64X=y
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
CONFIG_PATA_CS5535=m
CONFIG_PATA_CS5536=y
CONFIG_PATA_CYPRESS=m
CONFIG_PATA_EFAR=y
CONFIG_ATA_GENERIC=y
# CONFIG_PATA_HPT366 is not set
CONFIG_PATA_HPT37X=y
CONFIG_PATA_HPT3X2N=m
CONFIG_PATA_HPT3X3=y
CONFIG_PATA_HPT3X3_DMA=y
# CONFIG_PATA_ISAPNP is not set
# CONFIG_PATA_IT821X is not set
CONFIG_PATA_IT8213=m
CONFIG_PATA_JMICRON=m
# CONFIG_PATA_LEGACY is not set
CONFIG_PATA_TRIFLEX=m
CONFIG_PATA_MARVELL=m
# CONFIG_PATA_MPIIX is not set
CONFIG_PATA_OLDPIIX=y
CONFIG_PATA_NETCELL=y
# CONFIG_PATA_NINJA32 is not set
CONFIG_PATA_NS87410=y
CONFIG_PATA_NS87415=m
# CONFIG_PATA_OPTI is not set
CONFIG_PATA_OPTIDMA=y
CONFIG_PATA_PCMCIA=m
CONFIG_PATA_PDC_OLD=y
CONFIG_PATA_QDI=y
# CONFIG_PATA_RADISYS is not set
CONFIG_PATA_RZ1000=m
CONFIG_PATA_SC1200=m
CONFIG_PATA_SERVERWORKS=y
CONFIG_PATA_PDC2027X=m
# CONFIG_PATA_SIL680 is not set
CONFIG_PATA_SIS=y
CONFIG_PATA_VIA=m
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_WINBOND_VLB is not set
CONFIG_PATA_PLATFORM=m
# CONFIG_PATA_SCH is not set
# CONFIG_MD is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#

#
# Enable only one of the two stacks, unless you know what you are doing
#
# CONFIG_FIREWIRE is not set
CONFIG_IEEE1394=y
# CONFIG_IEEE1394_OHCI1394 is not set
CONFIG_IEEE1394_PCILYNX=m
# CONFIG_IEEE1394_SBP2 is not set
CONFIG_IEEE1394_ETH1394_ROM_ENTRY=y
CONFIG_IEEE1394_ETH1394=y
CONFIG_IEEE1394_RAWIO=m
CONFIG_IEEE1394_VERBOSEDEBUG=y
CONFIG_I2O=m
CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y
# CONFIG_I2O_EXT_ADAPTEC is not set
CONFIG_I2O_CONFIG=m
# CONFIG_I2O_CONFIG_OLD_IOCTL is not set
CONFIG_I2O_BUS=m
CONFIG_I2O_BLOCK=m
CONFIG_I2O_SCSI=m
CONFIG_I2O_PROC=m
CONFIG_MACINTOSH_DRIVERS=y
CONFIG_MAC_EMUMOUSEBTN=y
CONFIG_NETDEVICES=y
CONFIG_IFB=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
CONFIG_MACVLAN=y
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_VETH is not set
CONFIG_NET_SB1000=y
CONFIG_ARCNET=y
CONFIG_ARCNET_1201=y
CONFIG_ARCNET_1051=y
CONFIG_ARCNET_RAW=m
CONFIG_ARCNET_CAP=m
CONFIG_ARCNET_COM90xx=m
CONFIG_ARCNET_COM90xxIO=y
# CONFIG_ARCNET_RIM_I is not set
CONFIG_ARCNET_COM20020=y
CONFIG_ARCNET_COM20020_ISA=m
CONFIG_ARCNET_COM20020_PCI=y
CONFIG_PHYLIB=y

#
# MII PHY device drivers
#
CONFIG_MARVELL_PHY=y
CONFIG_DAVICOM_PHY=y
CONFIG_QSEMI_PHY=y
# CONFIG_LXT_PHY is not set
CONFIG_CICADA_PHY=y
CONFIG_VITESSE_PHY=m
CONFIG_SMSC_PHY=y
# CONFIG_BROADCOM_PHY is not set
CONFIG_ICPLUS_PHY=m
CONFIG_REALTEK_PHY=m
# CONFIG_FIXED_PHY is not set
CONFIG_MDIO_BITBANG=m
CONFIG_NET_ETHERNET=y
CONFIG_MII=y
CONFIG_HAPPYMEAL=y
CONFIG_SUNGEM=y
CONFIG_CASSINI=y
# CONFIG_NET_VENDOR_3COM is not set
CONFIG_VORTEX=y
CONFIG_LANCE=y
CONFIG_NET_VENDOR_SMC=y
CONFIG_ULTRAMCA=y
CONFIG_ULTRA=y
CONFIG_ULTRA32=m
CONFIG_SMC9194=y
# CONFIG_ENC28J60 is not set
# CONFIG_NET_VENDOR_RACAL is not set
# CONFIG_NET_TULIP is not set
CONFIG_AT1700=m
# CONFIG_DEPCA is not set
# CONFIG_HP100 is not set
CONFIG_NET_ISA=y
CONFIG_E2100=y
CONFIG_EWRK3=m
# CONFIG_EEXPRESS is not set
# CONFIG_EEXPRESS_PRO is not set
# CONFIG_HPLAN is not set
# CONFIG_LP486E is not set
# CONFIG_ETH16I is not set
CONFIG_NE2000=m
CONFIG_ZNET=m
CONFIG_SEEQ8005=m
CONFIG_NE2_MCA=m
CONFIG_IBMLANA=y
# CONFIG_IBM_NEW_EMAC_ZMII is not set
# CONFIG_IBM_NEW_EMAC_RGMII is not set
# CONFIG_IBM_NEW_EMAC_TAH is not set
# CONFIG_IBM_NEW_EMAC_EMAC4 is not set
# CONFIG_IBM_NEW_EMAC_NO_FLOW_CTRL is not set
# CONFIG_IBM_NEW_EMAC_MAL_CLR_ICINTSTAT is not set
# CONFIG_IBM_NEW_EMAC_MAL_COMMON_ERR is not set
CONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
CONFIG_AMD8111_ETH=y
CONFIG_ADAPTEC_STARFIRE=m
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_B44 is not set
CONFIG_FORCEDETH=y
CONFIG_FORCEDETH_NAPI=y
CONFIG_CS89x0=y
CONFIG_EEPRO100=m
CONFIG_E100=y
CONFIG_LNE390=y
CONFIG_FEALNX=y
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
CONFIG_NE3210=m
# CONFIG_ES3210 is not set
CONFIG_8139CP=y
CONFIG_8139TOO=y
CONFIG_8139TOO_PIO=y
CONFIG_8139TOO_TUNE_TWISTER=y
CONFIG_8139TOO_8129=y
CONFIG_8139_OLD_RX_RESET=y
CONFIG_R6040=y
CONFIG_SIS900=m
CONFIG_EPIC100=y
CONFIG_SUNDANCE=y
# CONFIG_SUNDANCE_MMIO is not set
# CONFIG_TLAN is not set
CONFIG_VIA_RHINE=y
CONFIG_VIA_RHINE_MMIO=y
CONFIG_SC92031=m
CONFIG_NET_POCKET=y
CONFIG_ATP=y
# CONFIG_DE600 is not set
CONFIG_DE620=y
# CONFIG_ATL2 is not set
CONFIG_NETDEV_1000=y
CONFIG_ACENIC=y
CONFIG_ACENIC_OMIT_TIGON_I=y
CONFIG_DL2K=m
CONFIG_E1000=y
CONFIG_E1000E=y
CONFIG_IP1000=y
CONFIG_IGB=m
CONFIG_IGB_LRO=y
# CONFIG_IGB_DCA is not set
CONFIG_NS83820=y
CONFIG_HAMACHI=y
CONFIG_YELLOWFIN=y
CONFIG_R8169=y
CONFIG_R8169_VLAN=y
# CONFIG_SIS190 is not set
CONFIG_SKGE=m
# CONFIG_SKGE_DEBUG is not set
CONFIG_SKY2=m
CONFIG_SKY2_DEBUG=y
CONFIG_VIA_VELOCITY=m
CONFIG_TIGON3=y
CONFIG_BNX2=y
CONFIG_QLA3XXX=m
CONFIG_ATL1=y
CONFIG_ATL1E=m
CONFIG_JME=y
CONFIG_NETDEV_10000=y
CONFIG_CHELSIO_T1=m
CONFIG_CHELSIO_T1_1G=y
CONFIG_CHELSIO_T3=y
# CONFIG_ENIC is not set
# CONFIG_IXGBE is not set
CONFIG_IXGB=y
CONFIG_S2IO=m
CONFIG_MYRI10GE=m
CONFIG_MYRI10GE_DCA=y
# CONFIG_NETXEN_NIC is not set
CONFIG_NIU=y
# CONFIG_MLX4_EN is not set
# CONFIG_MLX4_CORE is not set
# CONFIG_TEHUTI is not set
CONFIG_BNX2X=y
CONFIG_QLGE=m
CONFIG_SFC=y
CONFIG_TR=y
# CONFIG_IBMTR is not set
CONFIG_IBMOL=m
CONFIG_IBMLS=y
# CONFIG_3C359 is not set
CONFIG_TMS380TR=m
# CONFIG_TMSPCI is not set
CONFIG_SKISA=m
CONFIG_PROTEON=m
# CONFIG_ABYSS is not set
CONFIG_MADGEMC=m
# CONFIG_SMCTR is not set

#
# Wireless LAN
#
CONFIG_WLAN_PRE80211=y
CONFIG_STRIP=m
CONFIG_ARLAN=y
CONFIG_WAVELAN=y
CONFIG_PCMCIA_WAVELAN=m
# CONFIG_PCMCIA_NETWAVE is not set
CONFIG_WLAN_80211=y
# CONFIG_PCMCIA_RAYCS is not set
CONFIG_IPW2100=y
CONFIG_IPW2100_MONITOR=y
CONFIG_IPW2100_DEBUG=y
CONFIG_IPW2200=m
CONFIG_IPW2200_MONITOR=y
CONFIG_IPW2200_RADIOTAP=y
CONFIG_IPW2200_PROMISCUOUS=y
CONFIG_IPW2200_QOS=y
CONFIG_IPW2200_DEBUG=y
# CONFIG_LIBERTAS is not set
CONFIG_AIRO=y
# CONFIG_HERMES is not set
CONFIG_ATMEL=m
CONFIG_PCI_ATMEL=m
# CONFIG_PCMCIA_ATMEL is not set
CONFIG_AIRO_CS=m
# CONFIG_PCMCIA_WL3501 is not set
# CONFIG_PRISM54 is not set
CONFIG_USB_ZD1201=y
CONFIG_USB_NET_RNDIS_WLAN=y
# CONFIG_IWLWIFI_LEDS is not set
# CONFIG_HOSTAP is not set

#
# USB Network Adapters
#
CONFIG_USB_CATC=m
CONFIG_USB_KAWETH=y
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
CONFIG_USB_USBNET=y
CONFIG_USB_NET_AX8817X=m
CONFIG_USB_NET_CDCETHER=y
CONFIG_USB_NET_DM9601=m
CONFIG_USB_NET_SMSC95XX=m
CONFIG_USB_NET_GL620A=y
# CONFIG_USB_NET_NET1080 is not set
CONFIG_USB_NET_PLUSB=y
CONFIG_USB_NET_MCS7830=y
CONFIG_USB_NET_RNDIS_HOST=y
CONFIG_USB_NET_CDC_SUBSET=y
# CONFIG_USB_ALI_M5632 is not set
CONFIG_USB_AN2720=y
CONFIG_USB_BELKIN=y
# CONFIG_USB_ARMLINUX is not set
# CONFIG_USB_EPSON2888 is not set
CONFIG_USB_KC2190=y
CONFIG_USB_NET_ZAURUS=m
CONFIG_USB_HSO=y
CONFIG_NET_PCMCIA=y
CONFIG_PCMCIA_3C589=m
CONFIG_PCMCIA_3C574=m
# CONFIG_PCMCIA_FMVJ18X is not set
CONFIG_PCMCIA_PCNET=m
CONFIG_PCMCIA_NMCLAN=m
# CONFIG_PCMCIA_SMC91C92 is not set
CONFIG_PCMCIA_XIRC2PS=m
# CONFIG_PCMCIA_AXNET is not set
CONFIG_ARCNET_COM20020_CS=m
# CONFIG_PCMCIA_IBMTR is not set
CONFIG_WAN=y
# CONFIG_HOSTESS_SV11 is not set
# CONFIG_COSA is not set
# CONFIG_LANMEDIA is not set
CONFIG_SEALEVEL_4021=m
CONFIG_HDLC=m
CONFIG_HDLC_RAW=m
CONFIG_HDLC_RAW_ETH=m
CONFIG_HDLC_CISCO=m
# CONFIG_HDLC_FR is not set
# CONFIG_HDLC_PPP is not set

#
# X.25/LAPB support is disabled
#
CONFIG_PCI200SYN=m
CONFIG_WANXL=m
CONFIG_PC300TOO=m
# CONFIG_N2 is not set
# CONFIG_C101 is not set
# CONFIG_FARSYNC is not set
# CONFIG_DSCC4 is not set
CONFIG_DLCI=m
CONFIG_DLCI_MAX=8
CONFIG_SDLA=m
# CONFIG_WAN_ROUTER_DRIVERS is not set
CONFIG_SBNI=y
CONFIG_SBNI_MULTILINE=y
CONFIG_ATM_DRIVERS=y
CONFIG_ATM_DUMMY=m
CONFIG_ATM_TCP=m
CONFIG_ATM_LANAI=m
CONFIG_ATM_ENI=m
CONFIG_ATM_ENI_DEBUG=y
CONFIG_ATM_ENI_TUNE_BURST=y
# CONFIG_ATM_ENI_BURST_TX_16W is not set
CONFIG_ATM_ENI_BURST_TX_8W=y
CONFIG_ATM_ENI_BURST_TX_4W=y
CONFIG_ATM_ENI_BURST_TX_2W=y
CONFIG_ATM_ENI_BURST_RX_16W=y
CONFIG_ATM_ENI_BURST_RX_8W=y
CONFIG_ATM_ENI_BURST_RX_4W=y
# CONFIG_ATM_ENI_BURST_RX_2W is not set
CONFIG_ATM_FIRESTREAM=m
# CONFIG_ATM_ZATM is not set
# CONFIG_ATM_NICSTAR is not set
CONFIG_ATM_IDT77252=m
# CONFIG_ATM_IDT77252_DEBUG is not set
# CONFIG_ATM_IDT77252_RCV_ALL is not set
CONFIG_ATM_IDT77252_USE_SUNI=y
# CONFIG_ATM_AMBASSADOR is not set
CONFIG_ATM_HORIZON=m
CONFIG_ATM_HORIZON_DEBUG=y
CONFIG_ATM_IA=m
CONFIG_ATM_IA_DEBUG=y
# CONFIG_ATM_FORE200E is not set
# CONFIG_ATM_HE is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PLIP=y
CONFIG_PPP=m
# CONFIG_PPP_MULTILINK is not set
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=m
# CONFIG_PPP_SYNC_TTY is not set
CONFIG_PPP_DEFLATE=m
CONFIG_PPP_BSDCOMP=m
# CONFIG_PPP_MPPE is not set
CONFIG_PPPOE=m
CONFIG_PPPOATM=m
CONFIG_PPPOL2TP=m
# CONFIG_SLIP is not set
CONFIG_SLHC=m
# CONFIG_NET_FC is not set
CONFIG_NETCONSOLE=y
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
CONFIG_ISDN=y
# CONFIG_ISDN_I4L is not set
CONFIG_ISDN_CAPI=m
CONFIG_ISDN_DRV_AVMB1_VERBOSE_REASON=y
CONFIG_CAPI_TRACE=y
CONFIG_ISDN_CAPI_MIDDLEWARE=y
# CONFIG_ISDN_CAPI_CAPI20 is not set

#
# CAPI hardware drivers
#
# CONFIG_CAPI_AVM is not set
CONFIG_CAPI_EICON=y
CONFIG_ISDN_DIVAS=m
CONFIG_ISDN_DIVAS_BRIPCI=y
CONFIG_ISDN_DIVAS_PRIPCI=y
# CONFIG_ISDN_DIVAS_DIVACAPI is not set
CONFIG_ISDN_DIVAS_USERIDI=m
# CONFIG_ISDN_DIVAS_MAINT is not set
CONFIG_PHONE=y

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=y
CONFIG_INPUT_EVDEV=m
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_KEYBOARD_SUNKBD=y
CONFIG_KEYBOARD_LKKBD=m
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_KEYBOARD_NEWTON=y
# CONFIG_KEYBOARD_STOWAWAY is not set
CONFIG_INPUT_MOUSE=y
# CONFIG_MOUSE_PS2 is not set
# CONFIG_MOUSE_SERIAL is not set
CONFIG_MOUSE_APPLETOUCH=m
CONFIG_MOUSE_BCM5974=m
# CONFIG_MOUSE_INPORT is not set
CONFIG_MOUSE_LOGIBM=m
CONFIG_MOUSE_PC110PAD=y
CONFIG_MOUSE_VSXXXAA=y
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
# CONFIG_JOYSTICK_A3D is not set
CONFIG_JOYSTICK_ADI=y
CONFIG_JOYSTICK_COBRA=y
CONFIG_JOYSTICK_GF2K=y
CONFIG_JOYSTICK_GRIP=m
CONFIG_JOYSTICK_GRIP_MP=m
CONFIG_JOYSTICK_GUILLEMOT=y
# CONFIG_JOYSTICK_INTERACT is not set
CONFIG_JOYSTICK_SIDEWINDER=m
CONFIG_JOYSTICK_TMDC=m
CONFIG_JOYSTICK_IFORCE=m
CONFIG_JOYSTICK_IFORCE_USB=y
# CONFIG_JOYSTICK_IFORCE_232 is not set
CONFIG_JOYSTICK_WARRIOR=m
CONFIG_JOYSTICK_MAGELLAN=m
CONFIG_JOYSTICK_SPACEORB=m
CONFIG_JOYSTICK_SPACEBALL=m
CONFIG_JOYSTICK_STINGER=m
CONFIG_JOYSTICK_TWIDJOY=y
# CONFIG_JOYSTICK_ZHENHUA is not set
CONFIG_JOYSTICK_DB9=m
CONFIG_JOYSTICK_GAMECON=m
CONFIG_JOYSTICK_TURBOGRAFX=m
CONFIG_JOYSTICK_JOYDUMP=m
CONFIG_JOYSTICK_XPAD=y
CONFIG_JOYSTICK_XPAD_FF=y
CONFIG_INPUT_TABLET=y
CONFIG_TABLET_USB_ACECAD=y
CONFIG_TABLET_USB_AIPTEK=m
CONFIG_TABLET_USB_GTCO=y
# CONFIG_TABLET_USB_KBTAB is not set
# CONFIG_TABLET_USB_WACOM is not set
CONFIG_INPUT_TOUCHSCREEN=y
CONFIG_TOUCHSCREEN_ADS7846=m
CONFIG_TOUCHSCREEN_FUJITSU=y
CONFIG_TOUCHSCREEN_GUNZE=y
# CONFIG_TOUCHSCREEN_ELO is not set
# CONFIG_TOUCHSCREEN_MTOUCH is not set
# CONFIG_TOUCHSCREEN_INEXIO is not set
CONFIG_TOUCHSCREEN_MK712=y
CONFIG_TOUCHSCREEN_HTCPEN=m
CONFIG_TOUCHSCREEN_PENMOUNT=y
# CONFIG_TOUCHSCREEN_TOUCHRIGHT is not set
CONFIG_TOUCHSCREEN_TOUCHWIN=y
# CONFIG_TOUCHSCREEN_USB_COMPOSITE is not set
# CONFIG_TOUCHSCREEN_TOUCHIT213 is not set
# CONFIG_INPUT_MISC is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
# CONFIG_SERIO_CT82C710 is not set
CONFIG_SERIO_PARKBD=y
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
CONFIG_GAMEPORT=y
CONFIG_GAMEPORT_NS558=m
# CONFIG_GAMEPORT_L4 is not set
CONFIG_GAMEPORT_EMU10K1=y
CONFIG_GAMEPORT_FM801=m

#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_VT_HW_CONSOLE_BINDING is not set
CONFIG_DEVKMEM=y
# CONFIG_SERIAL_NONSTANDARD is not set
CONFIG_NOZOMI=y

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=m
# CONFIG_SERIAL_8250_CS is not set
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
CONFIG_SERIAL_8250_MANY_PORTS=y
CONFIG_SERIAL_8250_FOURPORT=y
# CONFIG_SERIAL_8250_ACCENT is not set
CONFIG_SERIAL_8250_BOCA=m
CONFIG_SERIAL_8250_EXAR_ST16C554=y
CONFIG_SERIAL_8250_HUB6=y
# CONFIG_SERIAL_8250_SHARE_IRQ is not set
# CONFIG_SERIAL_8250_DETECT_IRQ is not set
CONFIG_SERIAL_8250_RSA=y
# CONFIG_SERIAL_8250_MCA is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set
CONFIG_PRINTER=y
# CONFIG_LP_CONSOLE is not set
CONFIG_PPDEV=y
CONFIG_HVC_DRIVER=y
# CONFIG_IPMI_HANDLER is not set
CONFIG_HW_RANDOM=y
# CONFIG_HW_RANDOM_INTEL is not set
CONFIG_HW_RANDOM_AMD=y
CONFIG_HW_RANDOM_GEODE=y
CONFIG_HW_RANDOM_VIA=y
# CONFIG_NVRAM is not set
CONFIG_DTLK=m
CONFIG_R3964=y
# CONFIG_APPLICOM is not set
CONFIG_SONYPI=m

#
# PCMCIA character devices
#
CONFIG_SYNCLINK_CS=m
CONFIG_CARDMAN_4000=m
# CONFIG_CARDMAN_4040 is not set
CONFIG_IPWIRELESS=m
CONFIG_MWAVE=m
CONFIG_SCx200_GPIO=m
CONFIG_PC8736x_GPIO=y
CONFIG_NSC_GPIO=y
CONFIG_CS5535_GPIO=y
CONFIG_RAW_DRIVER=m
CONFIG_MAX_RAW_DEVS=256
CONFIG_HANGCHECK_TIMER=m
# CONFIG_TCG_TPM is not set
CONFIG_TELCLOCK=y
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
# CONFIG_I2C_CHARDEV is not set
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=y

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
CONFIG_I2C_ALI1535=y
CONFIG_I2C_ALI1563=y
CONFIG_I2C_ALI15X3=m
# CONFIG_I2C_AMD756 is not set
CONFIG_I2C_AMD8111=y
# CONFIG_I2C_I801 is not set
CONFIG_I2C_ISCH=m
# CONFIG_I2C_PIIX4 is not set
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_SIS5595 is not set
CONFIG_I2C_SIS630=m
CONFIG_I2C_SIS96X=m
# CONFIG_I2C_VIA is not set
CONFIG_I2C_VIAPRO=m

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
CONFIG_I2C_OCORES=m
# CONFIG_I2C_SIMTEC is not set

#
# External I2C/SMBus adapter drivers
#
CONFIG_I2C_PARPORT=m
CONFIG_I2C_PARPORT_LIGHT=y
CONFIG_I2C_TAOS_EVM=m
CONFIG_I2C_TINY_USB=y

#
# Graphics adapter I2C/DDC channel drivers
#
CONFIG_I2C_VOODOO3=m

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_PCA_PLATFORM is not set
CONFIG_I2C_STUB=m
# CONFIG_SCx200_I2C is not set
# CONFIG_SCx200_ACB is not set

#
# Miscellaneous I2C Chip support
#
# CONFIG_DS1682 is not set
CONFIG_AT24=m
CONFIG_SENSORS_EEPROM=y
CONFIG_SENSORS_PCF8574=m
# CONFIG_PCF8575 is not set
CONFIG_SENSORS_PCA9539=m
CONFIG_SENSORS_PCF8591=y
# CONFIG_SENSORS_MAX6875 is not set
CONFIG_SENSORS_TSL2550=m
CONFIG_I2C_DEBUG_CORE=y
CONFIG_I2C_DEBUG_ALGO=y
# CONFIG_I2C_DEBUG_BUS is not set
CONFIG_I2C_DEBUG_CHIP=y
CONFIG_SPI=y
# CONFIG_SPI_DEBUG is not set
CONFIG_SPI_MASTER=y

#
# SPI Master Controller Drivers
#
CONFIG_SPI_BITBANG=y
# CONFIG_SPI_BUTTERFLY is not set
# CONFIG_SPI_LM70_LLP is not set

#
# SPI Protocol Masters
#
# CONFIG_SPI_AT25 is not set
# CONFIG_SPI_SPIDEV is not set
CONFIG_SPI_TLE62X0=m
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
CONFIG_W1=y

#
# 1-wire Bus Masters
#
CONFIG_W1_MASTER_MATROX=y
# CONFIG_W1_MASTER_DS2490 is not set
# CONFIG_W1_MASTER_DS2482 is not set

#
# 1-wire Slaves
#
CONFIG_W1_SLAVE_THERM=m
CONFIG_W1_SLAVE_SMEM=y
CONFIG_W1_SLAVE_DS2433=y
# CONFIG_W1_SLAVE_DS2433_CRC is not set
CONFIG_W1_SLAVE_DS2760=m
# CONFIG_W1_SLAVE_BQ27000 is not set
CONFIG_POWER_SUPPLY=m
# CONFIG_POWER_SUPPLY_DEBUG is not set
CONFIG_PDA_POWER=m
CONFIG_BATTERY_DS2760=m
CONFIG_BATTERY_OLPC=m
# CONFIG_BATTERY_BQ27x00 is not set
# CONFIG_HWMON is not set
# CONFIG_THERMAL is not set
CONFIG_THERMAL_HWMON=y
CONFIG_WATCHDOG=y
CONFIG_WATCHDOG_NOWAYOUT=y

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
CONFIG_ACQUIRE_WDT=y
CONFIG_ADVANTECH_WDT=m
CONFIG_ALIM1535_WDT=m
CONFIG_ALIM7101_WDT=y
# CONFIG_SC520_WDT is not set
# CONFIG_IB700_WDT is not set
CONFIG_IBMASR=m
CONFIG_WAFER_WDT=y
CONFIG_I6300ESB_WDT=y
# CONFIG_ITCO_WDT is not set
CONFIG_IT8712F_WDT=m
CONFIG_IT87_WDT=m
CONFIG_HP_WATCHDOG=m
CONFIG_SC1200_WDT=y
CONFIG_SCx200_WDT=y
CONFIG_PC87413_WDT=y
CONFIG_RDC321X_WDT=y
CONFIG_60XX_WDT=y
CONFIG_SBC8360_WDT=y
# CONFIG_SBC7240_WDT is not set
# CONFIG_CPU5_WDT is not set
CONFIG_SMSC37B787_WDT=y
CONFIG_W83627HF_WDT=y
CONFIG_W83697HF_WDT=y
# CONFIG_W83697UG_WDT is not set
# CONFIG_W83877F_WDT is not set
# CONFIG_W83977F_WDT is not set
# CONFIG_MACHZ_WDT is not set
CONFIG_SBC_EPX_C3_WATCHDOG=m

#
# ISA-based Watchdog Cards
#
CONFIG_PCWATCHDOG=m
CONFIG_MIXCOMWD=y

#
# PCI-based Watchdog Cards
#
# CONFIG_PCIPCWATCHDOG is not set
# CONFIG_WDTPCI is not set

#
# USB-based Watchdog Cards
#
CONFIG_USBPCWATCHDOG=y
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
CONFIG_SSB=y
CONFIG_SSB_SPROM=y
CONFIG_SSB_PCIHOST_POSSIBLE=y
CONFIG_SSB_PCIHOST=y
# CONFIG_SSB_B43_PCI_BRIDGE is not set
CONFIG_SSB_SILENT=y
CONFIG_SSB_DRIVER_PCICORE_POSSIBLE=y
CONFIG_SSB_DRIVER_PCICORE=y

#
# Multifunction device drivers
#
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_MFD_TMIO is not set
CONFIG_PMIC_DA903X=y
# CONFIG_MFD_WM8400 is not set
CONFIG_REGULATOR=y
CONFIG_REGULATOR_DEBUG=y
# CONFIG_REGULATOR_FIXED_VOLTAGE is not set
CONFIG_REGULATOR_VIRTUAL_CONSUMER=m
# CONFIG_REGULATOR_BQ24022 is not set
CONFIG_REGULATOR_DA903X=m

#
# Multimedia devices
#

#
# Multimedia core support
#
CONFIG_VIDEO_DEV=y
CONFIG_VIDEO_V4L2_COMMON=y
# CONFIG_VIDEO_ALLOW_V4L1 is not set
# CONFIG_VIDEO_V4L1_COMPAT is not set
# CONFIG_DVB_CORE is not set
CONFIG_VIDEO_MEDIA=y

#
# Multimedia drivers
#
CONFIG_VIDEO_SAA7146=m
CONFIG_VIDEO_SAA7146_VV=m
CONFIG_MEDIA_ATTACH=y
CONFIG_MEDIA_TUNER=y
CONFIG_MEDIA_TUNER_CUSTOMIZE=y
CONFIG_MEDIA_TUNER_SIMPLE=m
CONFIG_MEDIA_TUNER_TDA8290=m
CONFIG_MEDIA_TUNER_TDA827X=m
CONFIG_MEDIA_TUNER_TDA18271=y
CONFIG_MEDIA_TUNER_TDA9887=m
CONFIG_MEDIA_TUNER_TEA5761=m
CONFIG_MEDIA_TUNER_TEA5767=m
# CONFIG_MEDIA_TUNER_MT20XX is not set
# CONFIG_MEDIA_TUNER_MT2060 is not set
CONFIG_MEDIA_TUNER_MT2266=y
CONFIG_MEDIA_TUNER_MT2131=y
CONFIG_MEDIA_TUNER_QT1010=y
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=y
CONFIG_MEDIA_TUNER_MXL5005S=m
CONFIG_MEDIA_TUNER_MXL5007T=m
CONFIG_VIDEO_V4L2=y
CONFIG_VIDEOBUF_GEN=y
CONFIG_VIDEOBUF_DMA_SG=y
CONFIG_VIDEOBUF_VMALLOC=y
CONFIG_VIDEO_BTCX=y
CONFIG_VIDEO_IR=y
CONFIG_VIDEO_TVEEPROM=y
CONFIG_VIDEO_TUNER=y
CONFIG_VIDEO_CAPTURE_DRIVERS=y
CONFIG_VIDEO_ADV_DEBUG=y
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
# CONFIG_VIDEO_HELPER_CHIPS_AUTO is not set
# CONFIG_VIDEO_IR_I2C is not set

#
# Encoders/decoders and other helper chips
#

#
# Audio decoders
#
# CONFIG_VIDEO_TVAUDIO is not set
# CONFIG_VIDEO_TDA7432 is not set
CONFIG_VIDEO_TDA9840=y
CONFIG_VIDEO_TDA9875=m
CONFIG_VIDEO_TEA6415C=y
CONFIG_VIDEO_TEA6420=m
CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_CS5345=m
CONFIG_VIDEO_CS53L32A=m
CONFIG_VIDEO_M52790=m
# CONFIG_VIDEO_TLV320AIC23B is not set
CONFIG_VIDEO_WM8775=y
CONFIG_VIDEO_WM8739=m
CONFIG_VIDEO_VP27SMPX=y

#
# Video decoders
#
CONFIG_VIDEO_OV7670=y
CONFIG_VIDEO_TCM825X=m
CONFIG_VIDEO_SAA711X=y
CONFIG_VIDEO_SAA717X=m
CONFIG_VIDEO_TVP5150=y

#
# Video and audio decoders
#
CONFIG_VIDEO_CX25840=y

#
# MPEG video encoders
#
CONFIG_VIDEO_CX2341X=y

#
# Video encoders
#
CONFIG_VIDEO_SAA7127=m

#
# Video improvement chips
#
CONFIG_VIDEO_UPD64031A=y
CONFIG_VIDEO_UPD64083=m
CONFIG_VIDEO_VIVI=y
CONFIG_VIDEO_BT848=y
# CONFIG_VIDEO_SAA6588 is not set
CONFIG_VIDEO_SAA5246A=m
CONFIG_VIDEO_SAA5249=y
CONFIG_VIDEO_SAA7134=m
CONFIG_VIDEO_HEXIUM_ORION=m
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
CONFIG_VIDEO_IVTV=m
CONFIG_VIDEO_CAFE_CCIC=y
CONFIG_SOC_CAMERA=m
CONFIG_SOC_CAMERA_MT9M001=m
# CONFIG_SOC_CAMERA_MT9M111 is not set
# CONFIG_SOC_CAMERA_MT9V022 is not set
# CONFIG_SOC_CAMERA_PLATFORM is not set
# CONFIG_VIDEO_SH_MOBILE_CEU is not set
# CONFIG_V4L_USB_DRIVERS is not set
CONFIG_RADIO_ADAPTERS=y
# CONFIG_RADIO_CADET is not set
CONFIG_RADIO_RTRACK=y
CONFIG_RADIO_RTRACK_PORT=20f
CONFIG_RADIO_RTRACK2=y
CONFIG_RADIO_RTRACK2_PORT=30c
CONFIG_RADIO_AZTECH=y
CONFIG_RADIO_AZTECH_PORT=350
CONFIG_RADIO_GEMTEK=y
CONFIG_RADIO_GEMTEK_PORT=34c
CONFIG_RADIO_GEMTEK_PROBE=y
# CONFIG_RADIO_GEMTEK_PCI is not set
# CONFIG_RADIO_MAXIRADIO is not set
CONFIG_RADIO_MAESTRO=m
CONFIG_RADIO_SF16FMI=y
CONFIG_RADIO_SF16FMR2=y
# CONFIG_RADIO_TERRATEC is not set
CONFIG_RADIO_TRUST=m
# CONFIG_RADIO_TYPHOON is not set
# CONFIG_RADIO_ZOLTRIX is not set
CONFIG_USB_DSBR=y
CONFIG_USB_SI470X=y
CONFIG_USB_MR800=y
CONFIG_DAB=y
CONFIG_USB_DABUSB=m

#
# Graphics support
#
CONFIG_AGP=m
CONFIG_AGP_ALI=m
CONFIG_AGP_ATI=m
CONFIG_AGP_AMD=m
CONFIG_AGP_AMD64=m
CONFIG_AGP_INTEL=m
# CONFIG_AGP_NVIDIA is not set
CONFIG_AGP_SIS=m
# CONFIG_AGP_SWORKS is not set
CONFIG_AGP_VIA=m
CONFIG_AGP_EFFICEON=m
CONFIG_DRM=m
CONFIG_DRM_TDFX=m
# CONFIG_DRM_R128 is not set
# CONFIG_DRM_RADEON is not set
CONFIG_DRM_I810=m
# CONFIG_DRM_I830 is not set
# CONFIG_DRM_I915 is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
CONFIG_DRM_VIA=m
CONFIG_DRM_SAVAGE=m
# CONFIG_VGASTATE is not set
CONFIG_VIDEO_OUTPUT_CONTROL=m
# CONFIG_FB is not set
CONFIG_BACKLIGHT_LCD_SUPPORT=y
CONFIG_LCD_CLASS_DEVICE=m
CONFIG_LCD_LTV350QV=m
CONFIG_LCD_ILI9320=m
CONFIG_LCD_TDO24M=m
CONFIG_LCD_VGG2432A4=m
# CONFIG_LCD_PLATFORM is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=m
# CONFIG_BACKLIGHT_CORGI is not set
CONFIG_BACKLIGHT_PROGEAR=m
CONFIG_BACKLIGHT_DA903X=m
CONFIG_BACKLIGHT_MBP_NVIDIA=m
CONFIG_BACKLIGHT_SAHARA=m

#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64
CONFIG_DUMMY_CONSOLE=y
# CONFIG_SOUND is not set
CONFIG_HID_SUPPORT=y
CONFIG_HID=m
CONFIG_HID_DEBUG=y
CONFIG_HIDRAW=y

#
# USB Input Devices
#
CONFIG_USB_HID=m
CONFIG_HID_PID=y
# CONFIG_USB_HIDDEV is not set

#
# USB HID Boot Protocol drivers
#
CONFIG_USB_KBD=y
CONFIG_USB_MOUSE=y

#
# Special HID drivers
#
# CONFIG_HID_COMPAT is not set
# CONFIG_HID_A4TECH is not set
CONFIG_HID_APPLE=m
CONFIG_HID_BELKIN=m
# CONFIG_HID_BRIGHT is not set
CONFIG_HID_CHERRY=m
CONFIG_HID_CHICONY=m
# CONFIG_HID_CYPRESS is not set
CONFIG_HID_DELL=m
CONFIG_HID_EZKEY=m
CONFIG_HID_GYRATION=m
# CONFIG_HID_LOGITECH is not set
CONFIG_HID_MICROSOFT=m
# CONFIG_HID_MONTEREY is not set
CONFIG_HID_PANTHERLORD=m
CONFIG_PANTHERLORD_FF=y
CONFIG_HID_PETALYNX=m
CONFIG_HID_SAMSUNG=m
CONFIG_HID_SONY=m
CONFIG_HID_SUNPLUS=m
# CONFIG_THRUSTMASTER_FF is not set
CONFIG_ZEROPLUS_FF=m
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
CONFIG_USB_DEBUG=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

#
# Miscellaneous USB options
#
# CONFIG_USB_DEVICEFS is not set
CONFIG_USB_DEVICE_CLASS=y
CONFIG_USB_DYNAMIC_MINORS=y
# CONFIG_USB_OTG is not set
CONFIG_USB_OTG_WHITELIST=y
# CONFIG_USB_OTG_BLACKLIST_HUB is not set
CONFIG_USB_MON=y
CONFIG_USB_WUSB=y
CONFIG_USB_WUSB_CBAF=m
CONFIG_USB_WUSB_CBAF_DEBUG=y

#
# USB Host Controller Drivers
#
CONFIG_USB_C67X00_HCD=y
CONFIG_USB_EHCI_HCD=y
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_USB_ISP116X_HCD=m
# CONFIG_USB_ISP1760_HCD is not set
CONFIG_USB_OHCI_HCD=y
# CONFIG_USB_OHCI_HCD_SSB is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=y
# CONFIG_USB_U132_HCD is not set
# CONFIG_USB_SL811_HCD is not set
CONFIG_USB_R8A66597_HCD=m
CONFIG_USB_HWA_HCD=y

#
# USB Device Class drivers
#
CONFIG_USB_ACM=y
CONFIG_USB_PRINTER=y
CONFIG_USB_WDM=y
CONFIG_USB_TMC=y

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may also be needed;
#

#
# see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_DATAFAB=y
# CONFIG_USB_STORAGE_FREECOM is not set
CONFIG_USB_STORAGE_ISD200=y
CONFIG_USB_STORAGE_DPCM=y
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
CONFIG_USB_STORAGE_SDDR55=y
CONFIG_USB_STORAGE_JUMPSHOT=y
CONFIG_USB_STORAGE_ALAUDA=y
# CONFIG_USB_STORAGE_ONETOUCH is not set
CONFIG_USB_STORAGE_KARMA=y
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
CONFIG_USB_LIBUSUAL=y

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
CONFIG_USB_MICROTEK=m

#
# USB port drivers
#
CONFIG_USB_USS720=y
CONFIG_USB_SERIAL=m
CONFIG_USB_EZUSB=y
CONFIG_USB_SERIAL_GENERIC=y
CONFIG_USB_SERIAL_AIRCABLE=m
# CONFIG_USB_SERIAL_ARK3116 is not set
CONFIG_USB_SERIAL_BELKIN=m
# CONFIG_USB_SERIAL_CH341 is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m
CONFIG_USB_SERIAL_CP2101=m
CONFIG_USB_SERIAL_CYPRESS_M8=m
CONFIG_USB_SERIAL_EMPEG=m
CONFIG_USB_SERIAL_FTDI_SIO=m
CONFIG_USB_SERIAL_FUNSOFT=m
CONFIG_USB_SERIAL_VISOR=m
CONFIG_USB_SERIAL_IPAQ=m
# CONFIG_USB_SERIAL_IR is not set
CONFIG_USB_SERIAL_EDGEPORT=m
CONFIG_USB_SERIAL_EDGEPORT_TI=m
# CONFIG_USB_SERIAL_GARMIN is not set
CONFIG_USB_SERIAL_IPW=m
# CONFIG_USB_SERIAL_IUU is not set
CONFIG_USB_SERIAL_KEYSPAN_PDA=m
CONFIG_USB_SERIAL_KEYSPAN=m
CONFIG_USB_SERIAL_KEYSPAN_MPR=y
CONFIG_USB_SERIAL_KEYSPAN_USA28=y
CONFIG_USB_SERIAL_KEYSPAN_USA28X=y
CONFIG_USB_SERIAL_KEYSPAN_USA28XA=y
# CONFIG_USB_SERIAL_KEYSPAN_USA28XB is not set
CONFIG_USB_SERIAL_KEYSPAN_USA19=y
# CONFIG_USB_SERIAL_KEYSPAN_USA18X is not set
CONFIG_USB_SERIAL_KEYSPAN_USA19W=y
CONFIG_USB_SERIAL_KEYSPAN_USA19QW=y
# CONFIG_USB_SERIAL_KEYSPAN_USA19QI is not set
CONFIG_USB_SERIAL_KEYSPAN_USA49W=y
CONFIG_USB_SERIAL_KEYSPAN_USA49WLC=y
CONFIG_USB_SERIAL_KLSI=m
CONFIG_USB_SERIAL_KOBIL_SCT=m
CONFIG_USB_SERIAL_MCT_U232=m
# CONFIG_USB_SERIAL_MOS7720 is not set
CONFIG_USB_SERIAL_MOS7840=m
CONFIG_USB_SERIAL_MOTOROLA=m
CONFIG_USB_SERIAL_NAVMAN=m
# CONFIG_USB_SERIAL_PL2303 is not set
CONFIG_USB_SERIAL_OTI6858=m
CONFIG_USB_SERIAL_SPCP8X5=m
CONFIG_USB_SERIAL_HP4X=m
# CONFIG_USB_SERIAL_SAFE is not set
CONFIG_USB_SERIAL_SIERRAWIRELESS=m
CONFIG_USB_SERIAL_TI=m
# CONFIG_USB_SERIAL_CYBERJACK is not set
CONFIG_USB_SERIAL_XIRCOM=m
# CONFIG_USB_SERIAL_OPTION is not set
# CONFIG_USB_SERIAL_OMNINET is not set
CONFIG_USB_SERIAL_DEBUG=m

#
# USB Miscellaneous drivers
#
CONFIG_USB_EMI62=y
# CONFIG_USB_EMI26 is not set
CONFIG_USB_ADUTUX=y
CONFIG_USB_SEVSEG=y
CONFIG_USB_RIO500=m
CONFIG_USB_LEGOTOWER=m
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
CONFIG_USB_LED=m
CONFIG_USB_CYPRESS_CY7C63=y
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGET is not set
CONFIG_USB_IDMOUSE=y
CONFIG_USB_FTDI_ELAN=m
# CONFIG_USB_APPLEDISPLAY is not set
CONFIG_USB_SISUSBVGA=m
# CONFIG_USB_SISUSBVGA_CON is not set
CONFIG_USB_LD=y
CONFIG_USB_TRANCEVIBRATOR=y
CONFIG_USB_IOWARRIOR=m
CONFIG_USB_ISIGHTFW=y
CONFIG_USB_VST=m
CONFIG_USB_ATM=m
CONFIG_USB_SPEEDTOUCH=m
CONFIG_USB_CXACRU=m
CONFIG_USB_UEAGLEATM=m
CONFIG_USB_XUSBATM=m
CONFIG_UWB=y
CONFIG_UWB_HWA=y
CONFIG_UWB_WHCI=m
# CONFIG_UWB_WLP is not set
# CONFIG_UWB_I1480U is not set
CONFIG_MMC=y
# CONFIG_MMC_DEBUG is not set
# CONFIG_MMC_UNSAFE_RESUME is not set

#
# MMC/SD/SDIO Card Drivers
#
CONFIG_MMC_BLOCK=y
CONFIG_MMC_BLOCK_BOUNCE=y
# CONFIG_SDIO_UART is not set
CONFIG_MMC_TEST=m

#
# MMC/SD/SDIO Host Controller Drivers
#
CONFIG_MMC_SDHCI=y
CONFIG_MMC_SDHCI_PCI=y
CONFIG_MMC_RICOH_MMC=m
CONFIG_MMC_WBSD=y
CONFIG_MMC_TIFM_SD=y
CONFIG_MMC_SDRICOH_CS=m
CONFIG_MEMSTICK=m
CONFIG_MEMSTICK_DEBUG=y

#
# MemoryStick drivers
#
CONFIG_MEMSTICK_UNSAFE_RESUME=y
# CONFIG_MSPRO_BLOCK is not set

#
# MemoryStick Host Controller Drivers
#
CONFIG_MEMSTICK_TIFM_MS=m
# CONFIG_MEMSTICK_JMICRON_38X is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
CONFIG_EDAC=y

#
# Reporting subsystems
#
CONFIG_EDAC_DEBUG=y
CONFIG_EDAC_MM_EDAC=m
CONFIG_EDAC_AMD76X=m
CONFIG_EDAC_E7XXX=m
CONFIG_EDAC_E752X=m
CONFIG_EDAC_I82875P=m
CONFIG_EDAC_I82975X=m
CONFIG_EDAC_I3000=m
CONFIG_EDAC_X38=m
CONFIG_EDAC_I82860=m
CONFIG_EDAC_R82600=m
CONFIG_EDAC_I5000=m
CONFIG_EDAC_I5100=m
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
# CONFIG_RTC_HCTOSYS is not set
# CONFIG_RTC_DEBUG is not set

#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
# CONFIG_RTC_INTF_DEV is not set
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#
CONFIG_RTC_DRV_DS1307=m
# CONFIG_RTC_DRV_DS1374 is not set
CONFIG_RTC_DRV_DS1672=m
CONFIG_RTC_DRV_MAX6900=y
CONFIG_RTC_DRV_RS5C372=m
CONFIG_RTC_DRV_ISL1208=y
# CONFIG_RTC_DRV_X1205 is not set
CONFIG_RTC_DRV_PCF8563=y
CONFIG_RTC_DRV_PCF8583=y
CONFIG_RTC_DRV_M41T80=y
CONFIG_RTC_DRV_M41T80_WDT=y
# CONFIG_RTC_DRV_S35390A is not set
CONFIG_RTC_DRV_FM3130=m
# CONFIG_RTC_DRV_RX8581 is not set

#
# SPI RTC drivers
#
CONFIG_RTC_DRV_M41T94=m
CONFIG_RTC_DRV_DS1305=y
# CONFIG_RTC_DRV_DS1390 is not set
# CONFIG_RTC_DRV_MAX6902 is not set
CONFIG_RTC_DRV_R9701=m
# CONFIG_RTC_DRV_RS5C348 is not set
# CONFIG_RTC_DRV_DS3234 is not set

#
# Platform RTC drivers
#
# CONFIG_RTC_DRV_CMOS is not set
CONFIG_RTC_DRV_DS1286=m
# CONFIG_RTC_DRV_DS1511 is not set
CONFIG_RTC_DRV_DS1553=y
# CONFIG_RTC_DRV_DS1742 is not set
CONFIG_RTC_DRV_STK17TA8=m
CONFIG_RTC_DRV_M48T86=y
CONFIG_RTC_DRV_M48T35=m
CONFIG_RTC_DRV_M48T59=m
# CONFIG_RTC_DRV_BQ4802 is not set
# CONFIG_RTC_DRV_V3020 is not set

#
# on-CPU RTC drivers
#
CONFIG_DMADEVICES=y

#
# DMA Devices
#
CONFIG_INTEL_IOATDMA=m
CONFIG_DMA_ENGINE=y

#
# DMA Clients
#
# CONFIG_NET_DMA is not set
CONFIG_DMATEST=m
CONFIG_DCA=m
# CONFIG_AUXDISPLAY is not set
CONFIG_UIO=y
CONFIG_UIO_CIF=m
CONFIG_UIO_PDRV=m
CONFIG_UIO_PDRV_GENIRQ=y
CONFIG_UIO_SMX=y
CONFIG_UIO_SERCOS3=y

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
# CONFIG_DELL_RBU is not set
CONFIG_DCDBAS=m
# CONFIG_ISCSI_IBFT_FIND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT2_FS_XIP=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
# CONFIG_EXT4_FS is not set
CONFIG_FS_XIP=y
CONFIG_JBD=y
CONFIG_JBD_DEBUG=y
CONFIG_JBD2=m
CONFIG_JBD2_DEBUG=y
CONFIG_FS_MBCACHE=y
CONFIG_REISERFS_FS=y
CONFIG_REISERFS_CHECK=y
CONFIG_REISERFS_PROC_INFO=y
CONFIG_REISERFS_FS_XATTR=y
# CONFIG_REISERFS_FS_POSIX_ACL is not set
CONFIG_REISERFS_FS_SECURITY=y
CONFIG_JFS_FS=m
CONFIG_JFS_POSIX_ACL=y
CONFIG_JFS_SECURITY=y
CONFIG_JFS_DEBUG=y
CONFIG_JFS_STATISTICS=y
CONFIG_FS_POSIX_ACL=y
CONFIG_FILE_LOCKING=y
CONFIG_XFS_FS=y
CONFIG_XFS_QUOTA=y
CONFIG_XFS_POSIX_ACL=y
# CONFIG_XFS_RT is not set
CONFIG_XFS_DEBUG=y
CONFIG_OCFS2_FS=m
# CONFIG_OCFS2_FS_O2CB is not set
# CONFIG_OCFS2_FS_STATS is not set
CONFIG_OCFS2_DEBUG_MASKLOG=y
CONFIG_OCFS2_DEBUG_FS=y
# CONFIG_OCFS2_COMPAT_JBD is not set
# CONFIG_DNOTIFY is not set
# CONFIG_INOTIFY is not set
CONFIG_QUOTA=y
CONFIG_QUOTA_NETLINK_INTERFACE=y
# CONFIG_PRINT_QUOTA_WARNING is not set
CONFIG_QFMT_V1=y
CONFIG_QFMT_V2=y
CONFIG_QUOTACTL=y
CONFIG_AUTOFS_FS=y
CONFIG_AUTOFS4_FS=m
CONFIG_FUSE_FS=y

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
# CONFIG_ZISOFS is not set
# CONFIG_UDF_FS is not set

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
# CONFIG_PROC_SYSCTL is not set
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
CONFIG_AFFS_FS=m
CONFIG_HFS_FS=y
CONFIG_HFSPLUS_FS=y
CONFIG_BEFS_FS=m
CONFIG_BEFS_DEBUG=y
CONFIG_BFS_FS=m
CONFIG_EFS_FS=y
# CONFIG_CRAMFS is not set
CONFIG_VXFS_FS=y
CONFIG_MINIX_FS=y
CONFIG_OMFS_FS=m
CONFIG_HPFS_FS=y
CONFIG_QNX4FS_FS=y
CONFIG_ROMFS_FS=m
CONFIG_SYSV_FS=m
CONFIG_UFS_FS=y
CONFIG_UFS_FS_WRITE=y
CONFIG_UFS_DEBUG=y
CONFIG_NETWORK_FILESYSTEMS=y
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
CONFIG_NFS_V3_ACL=y
CONFIG_NFS_V4=y
CONFIG_NFSD=m
CONFIG_NFSD_V2_ACL=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V3_ACL=y
CONFIG_NFSD_V4=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=m
CONFIG_NFS_ACL_SUPPORT=m
CONFIG_NFS_COMMON=y
CONFIG_SUNRPC=m
CONFIG_SUNRPC_GSS=m
CONFIG_SUNRPC_REGISTER_V4=y
CONFIG_RPCSEC_GSS_KRB5=m
CONFIG_RPCSEC_GSS_SPKM3=m
CONFIG_SMB_FS=y
CONFIG_SMB_NLS_DEFAULT=y
CONFIG_SMB_NLS_REMOTE="cp437"
CONFIG_CIFS=m
CONFIG_CIFS_STATS=y
CONFIG_CIFS_STATS2=y
CONFIG_CIFS_WEAK_PW_HASH=y
CONFIG_CIFS_UPCALL=y
CONFIG_CIFS_XATTR=y
# CONFIG_CIFS_POSIX is not set
CONFIG_CIFS_DEBUG2=y
CONFIG_CIFS_EXPERIMENTAL=y
# CONFIG_CIFS_DFS_UPCALL is not set
# CONFIG_NCP_FS is not set
CONFIG_CODA_FS=y
CONFIG_AFS_FS=y
CONFIG_AFS_DEBUG=y

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
CONFIG_ACORN_PARTITION=y
CONFIG_ACORN_PARTITION_CUMANA=y
CONFIG_ACORN_PARTITION_EESOX=y
# CONFIG_ACORN_PARTITION_ICS is not set
# CONFIG_ACORN_PARTITION_ADFS is not set
CONFIG_ACORN_PARTITION_POWERTEC=y
# CONFIG_ACORN_PARTITION_RISCIX is not set
CONFIG_OSF_PARTITION=y
CONFIG_AMIGA_PARTITION=y
CONFIG_ATARI_PARTITION=y
CONFIG_MAC_PARTITION=y
CONFIG_MSDOS_PARTITION=y
CONFIG_BSD_DISKLABEL=y
# CONFIG_MINIX_SUBPARTITION is not set
CONFIG_SOLARIS_X86_PARTITION=y
CONFIG_UNIXWARE_DISKLABEL=y
# CONFIG_LDM_PARTITION is not set
CONFIG_SGI_PARTITION=y
CONFIG_ULTRIX_PARTITION=y
CONFIG_SUN_PARTITION=y
# CONFIG_KARMA_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
CONFIG_SYSV68_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_CODEPAGE_737=y
CONFIG_NLS_CODEPAGE_775=m
CONFIG_NLS_CODEPAGE_850=y
# CONFIG_NLS_CODEPAGE_852 is not set
CONFIG_NLS_CODEPAGE_855=m
# CONFIG_NLS_CODEPAGE_857 is not set
CONFIG_NLS_CODEPAGE_860=y
CONFIG_NLS_CODEPAGE_861=y
# CONFIG_NLS_CODEPAGE_862 is not set
CONFIG_NLS_CODEPAGE_863=m
CONFIG_NLS_CODEPAGE_864=y
CONFIG_NLS_CODEPAGE_865=y
# CONFIG_NLS_CODEPAGE_866 is not set
CONFIG_NLS_CODEPAGE_869=m
CONFIG_NLS_CODEPAGE_936=m
CONFIG_NLS_CODEPAGE_950=m
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
CONFIG_NLS_CODEPAGE_874=m
CONFIG_NLS_ISO8859_8=m
# CONFIG_NLS_CODEPAGE_1250 is not set
CONFIG_NLS_CODEPAGE_1251=m
CONFIG_NLS_ASCII=y
CONFIG_NLS_ISO8859_1=m
CONFIG_NLS_ISO8859_2=y
CONFIG_NLS_ISO8859_3=y
CONFIG_NLS_ISO8859_4=y
# CONFIG_NLS_ISO8859_5 is not set
CONFIG_NLS_ISO8859_6=m
# CONFIG_NLS_ISO8859_7 is not set
CONFIG_NLS_ISO8859_9=m
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
CONFIG_NLS_KOI8_R=y
CONFIG_NLS_KOI8_U=m
CONFIG_NLS_UTF8=y
# CONFIG_DLM is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
# CONFIG_PRINTK_TIME is not set
# CONFIG_ALLOW_WARNINGS is not set
CONFIG_FRAME_WARN=1024
CONFIG_MAGIC_SYSRQ=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
CONFIG_HEADERS_CHECK=y
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
# CONFIG_DETECT_SOFTLOCKUP is not set
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_TIMER_STATS=y
CONFIG_DEBUG_OBJECTS=y
# CONFIG_DEBUG_OBJECTS_SELFTEST is not set
# CONFIG_DEBUG_OBJECTS_FREE is not set
# CONFIG_DEBUG_OBJECTS_TIMERS is not set
CONFIG_DEBUG_OBJECTS_ENABLE_DEFAULT=1
CONFIG_SLUB_DEBUG_ON=y
# CONFIG_SLUB_STATS is not set
CONFIG_DEBUG_RT_MUTEXES=y
CONFIG_DEBUG_PI_LIST=y
CONFIG_RT_MUTEX_TESTER=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y
CONFIG_LOCKDEP=y
# CONFIG_LOCK_STAT is not set
CONFIG_DEBUG_LOCKDEP=y
CONFIG_TRACE_IRQFLAGS=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_HIGHMEM is not set
# CONFIG_DEBUG_BUGVERBOSE is not set
# CONFIG_DEBUG_VM is not set
CONFIG_DEBUG_VIRTUAL=y
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_SG=y
CONFIG_DEBUG_NOTIFIERS=y
CONFIG_FRAME_POINTER=y
CONFIG_BOOT_PRINTK_DELAY=y
# CONFIG_RCU_TORTURE_TEST is not set
CONFIG_RCU_CPU_STALL_DETECTOR=y
CONFIG_KPROBES_SANITY_TEST=y
CONFIG_BACKTRACE_SELF_TEST=m
# CONFIG_LKDTM is not set
# CONFIG_FAULT_INJECTION is not set
CONFIG_LATENCYTOP=y
CONFIG_SYSCTL_SYSCALL_CHECK=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_HW_BRANCH_TRACER=y
CONFIG_TRACER_MAX_TRACE=y
CONFIG_RING_BUFFER=y
CONFIG_TRACING=y

#
# Tracers
#
CONFIG_FUNCTION_TRACER=y
# CONFIG_FUNCTION_GRAPH_TRACER is not set
CONFIG_IRQSOFF_TRACER=y
CONFIG_SYSPROF_TRACER=y
# CONFIG_SCHED_TRACER is not set
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_BOOT_TRACER=y
# CONFIG_POWER_TRACER is not set
CONFIG_STACK_TRACER=y
# CONFIG_HW_BRANCH_TRACER is not set
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FTRACE_MCOUNT_RECORD=y
CONFIG_PROVIDE_OHCI1394_DMA_INIT=y
# CONFIG_BUILD_DOCSRC is not set
CONFIG_DYNAMIC_PRINTK_DEBUG=y
CONFIG_SAMPLES=y
# CONFIG_SAMPLE_MARKERS is not set
CONFIG_SAMPLE_TRACEPOINTS=m
CONFIG_SAMPLE_KOBJECT=y
CONFIG_SAMPLE_KPROBES=m
CONFIG_SAMPLE_KRETPROBES=m
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
CONFIG_EARLY_PRINTK_DBGP=y
CONFIG_DEBUG_STACKOVERFLOW=y
CONFIG_DEBUG_STACK_USAGE=y
CONFIG_DEBUG_PAGEALLOC=y
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
# CONFIG_DEBUG_RODATA is not set
CONFIG_DEBUG_NX_TEST=m
# CONFIG_4KSTACKS is not set
CONFIG_DOUBLEFAULT=y
CONFIG_MMIOTRACE=y
CONFIG_MMIOTRACE_TEST=m
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
CONFIG_DEBUG_BOOT_PARAMS=y
CONFIG_CPA_DEBUG=y
CONFIG_OPTIMIZE_INLINING=y

#
# Security options
#
CONFIG_KEYS=y
CONFIG_KEYS_DEBUG_PROC_KEYS=y
CONFIG_SECURITY=y
# CONFIG_SECURITYFS is not set
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY_NETWORK_XFRM=y
CONFIG_SECURITY_FILE_CAPABILITIES=y
CONFIG_SECURITY_DEFAULT_MMAP_MIN_ADDR=0
CONFIG_SECURITY_SELINUX=y
CONFIG_SECURITY_SELINUX_BOOTPARAM=y
CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1
CONFIG_SECURITY_SELINUX_DISABLE=y
# CONFIG_SECURITY_SELINUX_DEVELOP is not set
CONFIG_SECURITY_SELINUX_AVC_STATS=y
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX=y
CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX_VALUE=19
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
CONFIG_CRYPTO_FIPS=y
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_GF128MUL=y
CONFIG_CRYPTO_NULL=m
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=y
CONFIG_CRYPTO_TEST=m

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
CONFIG_CRYPTO_GCM=y
CONFIG_CRYPTO_SEQIV=y

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=y
CONFIG_CRYPTO_CTS=y
CONFIG_CRYPTO_ECB=y
# CONFIG_CRYPTO_LRW is not set
CONFIG_CRYPTO_PCBC=m
CONFIG_CRYPTO_XTS=m

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set

#
# Digest
#
# CONFIG_CRYPTO_CRC32C is not set
CONFIG_CRYPTO_CRC32C_INTEL=m
# CONFIG_CRYPTO_MD4 is not set
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=y
CONFIG_CRYPTO_RMD128=y
CONFIG_CRYPTO_RMD160=y
CONFIG_CRYPTO_RMD256=m
CONFIG_CRYPTO_RMD320=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
CONFIG_CRYPTO_TGR192=m
CONFIG_CRYPTO_WP512=y

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_586=y
CONFIG_CRYPTO_ANUBIS=y
# CONFIG_CRYPTO_ARC4 is not set
CONFIG_CRYPTO_BLOWFISH=y
CONFIG_CRYPTO_CAMELLIA=y
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=y
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_FCRYPT=y
CONFIG_CRYPTO_KHAZAD=y
# CONFIG_CRYPTO_SALSA20 is not set
CONFIG_CRYPTO_SALSA20_586=m
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
CONFIG_CRYPTO_TEA=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_TWOFISH_COMMON=m
CONFIG_CRYPTO_TWOFISH_586=m

#
# Compression
#
CONFIG_CRYPTO_DEFLATE=y
# CONFIG_CRYPTO_LZO is not set

#
# Random Number Generation
#
CONFIG_CRYPTO_ANSI_CPRNG=m
CONFIG_CRYPTO_HW=y
CONFIG_CRYPTO_DEV_PADLOCK=m
CONFIG_CRYPTO_DEV_PADLOCK_AES=m
CONFIG_CRYPTO_DEV_PADLOCK_SHA=m
CONFIG_CRYPTO_DEV_GEODE=y
# CONFIG_CRYPTO_DEV_HIFN_795X is not set
CONFIG_HAVE_KVM=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
CONFIG_KVM_INTEL=m
# CONFIG_KVM_AMD is not set
# CONFIG_KVM_TRACE is not set
CONFIG_LGUEST=y
# CONFIG_VIRTIO_PCI is not set
# CONFIG_VIRTIO_BALLOON is not set

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
CONFIG_CRC7=m
CONFIG_LIBCRC32C=y
CONFIG_AUDIT_GENERIC=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_CHECK_SIGNATURE=y
CONFIG_FORCE_SUCCESSFUL_BUILD=y
CONFIG_FORCE_MINIMAL_CONFIG=y
CONFIG_FORCE_MINIMAL_CONFIG_PHYS=y
CONFIG_X86_32_ALWAYS_ON=y

Re: x86/mce merge, integration hickup + crash, design thoughts

[Ingo Molnar]

[ resend - i have restored the Cc: list ]

* Andi Kleen <ak@linux.intel.com> wrote:

>> We really need to get rid of /dev/mcelog. It's a quirky binary logging 
>> facility not available on 32-bit on current kernels and it has a couple 
>> of limitations:
>
> It's a bit more complicated than it looks first. I'm not in principle 
> opposed to an ASCII interface, but there are some complications in 
> practice. I'll write up all the details tomorrow too
>
> (just some quick comments below, but there's more to it)
>
>> A far more useful design for handling MCE events would be to feed them 
>> into printk logging.
>
> If there's ASCII logging it should be separate from normal printk.

Well, why? Sure, MCE exceptions themselves cannot generally printk 
[because they are in essence non-maskable contexts] unless they get a 
fatal MCE [in which case we have no other choice but to try to printk and 
hope for the best]. But they can sprintf into a buffer which then gets 
printk-d (or passed to whatever ASCII based facility).

'struct mce' is pointless complexity and a pointless restriction - and so 
is /dev/mcelog.

> I started that originally because I was sick of machine checks getting 
> reported as kernel bugs, and I got a lot of feedback over the years that 
> people like that.  Later on it turned out there are more good reasons to 
> separate logging.

Hm, such as? Right now i see mcelog as a facility that gets used only in 
the rarest of circumstances. 99% of the time mcelog is just used in mcelog 
--ascii mode to decode something quirky that the kernel could have (and 
should have) printed out in a much more human-accessible format.

>> So instead of printing such rather cryptic error messages:
>>
>>    MCE 0
>>    HARDWARE ERROR. This is *NOT* a software problem!
>>    Please contact your hardware vendor
>>    CPU 0 BANK 6 MISC 202d ADDR ffeef740
>>    This is not a software problem!
>>    Run through mcelog --ascii to decode and contact your hardware vendor
>>
>> and expecting people to run mcelog, we should print plain-text 
>> something like:
>>
>>    MCE 0
>>    HARDWARE ERROR. This is *NOT* a software problem!
>>    Please contact your hardware vendor
>>    CPU 1 4 northbridge TSC 89a560bb249
>>    ADDR 1dfa49690
>>      Northbridge Chipkill ECC error
>
> It turns out that users don't really find this more enlightening (most 
> users have no clue what a Northbridge is).  They think it's some kind of 
> kernel bug even with the HARDWARE ERROR header.

You should not assume that administrators/users reading kernel crash 
messages are dumb. (an ordinary user wont see it most of the time anyway) 

The usage patterns i see is that admins who get an MCE crash often fail to 
write down the whole MCE message (not realizing that it is important) and 
have to go back and reproduce the MCE crash once again before they can get 
any meaningful information.

Printing out cryptic hexadecimal error codes, requiring people to write 
them down and decode them in user-space is the technology of the 80s - i 
didnt think i'd have to argue too much about this ;-)

Reducing the amount of information presented to the user in such a crash 
situation is a dumb idea. (especially here where the MCE information is 
rather dense and single-screen anyway - so there's no screen real estate 
considerations.)

> The only people who really care about the micro architectural details in 
> full are chip developers, and those typically decode using other methods 
> anyways.

People do care about getting meaningful crash information from the kernel. 
That's why we by default print out something like:

Call Trace:
 [<c013463d>] warn_slowpath+0x6d/0x90
 [<c07a03fb>] ? _spin_lock_irqsave+0x1b/0x60
 [<c07a07fc>] ? _spin_unlock_irqrestore+0x3c/0x60
 [<c07a07fc>] ? _spin_unlock_irqrestore+0x3c/0x60
 [<c015547b>] ? trace_hardirqs_off+0xb/0x10
 [<c079e77c>] ? __mutex_unlock_slowpath+0x9c/0x170
 [<c015fc1c>] smp_call_function_mask+0x1ac/0x1c0

and not:

Call Trace:
 [<c013463d>]
 [<c07a03fb>]
 [<c07a07fc>]
 [<c07a07fc>]
 [<c015547b>]
 [<c079e77c>]
 [<c015fc1c>]

This is a basic principle in the Linux kernel. We try to print out as 
useful information as possible - and only cut down on it if the 
information physically does not fit on the screen. (which is not a problem 
here)

	Ingo

Re: x86/mce merge, integration hickup + crash, design thoughts

[Andi Kleen]

Ingo Molnar wrote:

>>> A far more useful design for handling MCE events would be to feed them 
>>> into printk logging.
>> If there's ASCII logging it should be separate from normal printk.
> 
> Well, why? 

Mostly because the problem is not a kernel issue. Especially large systems
with a lot of memory can generate a lot of corrected events (one bit
flips in DIMMs are not that uncommon) and it's not
good to mix that all up into other kernel messages. It also makes it more
clear that it's not a kernel problem, but a hardware problem. I've
got feedback over the years that confirm this sight.

> Hm, such as? Right now i see mcelog as a facility that gets used only in 
> the rarest of circumstances. 99% of the time mcelog is just used in mcelog 
> --ascii mode to decode something quirky that the kernel could have (and 
> should have) printed out in a much more human-accessible format.

Nope, it's used all the time without --ascii to report corrected by hardware
errors, like for example single bit errors in memory or cache.
With the current trend in hardware (more and more memory, more and
more transistors) corrected errors are getting more common all the time.

Another case is handling uncorrected exceptions with tolerant == 3: while that
sounds obscure it's used in practice by a large user base.

> You should not assume that administrators/users reading kernel crash 
> messages are dumb. (an ordinary user wont see it most of the time anyway) 

It has nothing to do with dumb, it's not a useful level of detail
and doesn't provide the information needed to fix the problem.
In particular it says nothing about which hardware component failed,
which is the main important information. mcelog goes to some effort
to provide more information, like for example trying to figure
out which DIMM actually failed [unfortunately there are a couple
of problems with this too, so its WIP, but it's improving]

Then you're right the panics are often hidden by X currently, but there are
ways to address this. The most simple one I'm consider is to make a panic
timeout default for machine checks. Then with a warm reboot the mces
can be logged after reboot because they stay around in the CPU register.

The other is with kernel mode setting it's actually possible to report
a MCE (or a oops) on top of the X server [my old PPC ibook did this
 >6 years ago, but finally x86 has caught up]. However the soft reboot
method above is better in my opinion because it doesn't require
anyone to scribble anything down.

There are a couple of other approaches to solve this that are also
getting explored to keep the information over reboot.


> The usage patterns i see is that admins who get an MCE crash often fail to 
> write down the whole MCE message (not realizing that it is important) and 
> have to go back and reproduce the MCE crash once again before they can get 
> any meaningful information.

Most MCEs are not panics but corrected (by hardware) errors, currently
/dev/mcelog + mcelog is primarily for this case (but also for the "log event after warm
reboot" case and of course the tolerant==3 case)

Now the reasons why ASCII MCE logging is problematic:

- First when I started the 64bit mce.c some years ago I started with NMI
safe ASCII logging. But I found it personally quite difficult to write
a well performing correct lockless variable record length buffer that
handles all the corner cases well.

Using fixed size records simplified the problem considerably [although the
current mce_log() implementation still has issues, but Ying has a rewrite in
the pipeline)
This is probably solvable with some effort, but it's not easy, especially
when you want to make sure the result is fully correct.

- Then mce/nmi logging has different requirements from a standard printk.
In particular it requires tail dropping instead of the usual head dropping
[the first events are more important than the later ones]. While
this could be probably done in a generic implementation, it's an awkward
special case there.

- Then to do a generic lockless buffer you need a suitable atomic primitive.
Currently that's CMPXCHG, but that's not available on all architectures,
so you can't easily make it a generic facility. You would likely need
different implementations for different architectures.

- Even if you only consider x86 there is only
CMPXCHG4/8 generally available on x86, severly limiting the maximum buffer
size in the most obvious variable record sized buffer. It might be possible
to write a buffer with something else (like XADD), but it would be hard
(and definitely unportable)

Ying's new mcelog implementation uses a per CPU buffer. That avoids
some problems with the old implementation, in particular scalability
with a lot of CPUs.

- But there is some concern about the size of the per CPU buffer. It requires
a minimum margin per CPU (16+ events/CPU) ASCII will be a lot more bloated so it would
require more memory per CPU.
That's not a serious concern, but I don't like it particularly.

- Then lockless buffering has a couple of drawbacks, in particular it's slower
in the worst case [at least my implementation was] and they are more prone
to livelock and harder to verify. And ultimatively there are very few
loggers who really need full NMI semantics, because the dominating
majority code runs fine with standard interrupt locking rules. And I suspect
that code majority will be more happy with a better performing locked
buffer facility.

- Then we have a machine check injector upcoming for inclusion which turned out to
be essential for debugging and improving the mce code. The injector
right now is just a write() on mcelog with an internal struct mce storage
that is then faked on MSR accesses.  That's nice and simple and small
enough that it doesn't need much CONFIGery.
If the interface was switched to ASCII we would first need a ASCII
parser and then some internal storage that would be equivalent to
struct mce. So you would still need struct mce anyways.
And while it's possible to write a parser of course it would
be large (and likely somewhat ugly) code and make the injector code a lot more
complicated.

So at least for the injector I think binary /dev/mcelog should be kept

- Full decoding is a lot of code.

First modern mcelog does a lot more than decoding bits. It has a database
to keep track of errors per component, can decode addresses to DIMMs and some
other things. Some of this (like the database) clearly doesn't belong
into kernel space, and even the other bits are quite a lot of code combined.

Even basic decoding can be quite complicated (not that this is not even
complete, e.g. Atom support is missing currently):

  text    data     bss     dec     hex filename
    1267    1424       0    2691     a83 core2.o
     791       0       0     791     317 bitfield.o
    1290    1008       0    2298     8fa dunnington.o
    3999     544      64    4607    11ff p4.o
    2169    1856       0    4025     fb9 nehalem.o
     239       0       0     239      ef intel.o
    3783    1264      64    5111    13f7 k8.o

About 20k code alone. If you add all the other related
decoding code you're going towards 40k. I expect it also
to become more complicated in the future.

That would be quite a lot of kernel bloat.

- One of the upcoming changes for the main MC exception handler
is Monarch support, aka synchronizing machine checks
over all CPUs. This is required to correctly handle the error containment
model in the hardware and also allows avoiding a couple of other issues.
The Monarch code requires different CPUs to communicate their
machine check information to others. While they don't need all of struct mce, they
need several fields of it. At least a subset of it would be still
needed. It's also convenient in this code to just copy the single
data structure around.

- The requirements for printing a panic is different from talking to mcelog
(or some other user backend). For a panic you should do at least some
decoding and add some information (like the HARDWARE ERROR header),
but when talking to another program that's in the way.

- Then machine check events are relatively complicated as far as kernel
events go. They are structured records with multiple fields with
different semantics. They also require extensibility (the current
/dev/mcelog interface is fully extensible with full backwards
compatibility in both directions). So it's not just writing a single
string out, you would need a record with start/end markers and a
special version which his different from a "human cooked" version.
I'm not suggesting it needs XML @), but at least some elements
of it (e.g. some nested format) would be probably needed.
I don't think it would be very nice.

- Then mcelog in some cases poll()s for events. This is useful
in the tolerant==3 case for example when it wants to run as quickly
as possible.  But the variable length records above do not really
fit well into a poll() interface (or into the sysfs frame work
in general). You would probably need at least a nasty state
machine in the kernel poll handler and some buffer preallocation.
It's doable of course, but the code would be ugly I think

- Finally the currently only known user of /dev/mcelog (or whatever
interface is there to communicate MC events) uses struct mce internally.
So if this was changed to an ASCII interface it would just parse
it and then put it back into the same structure. The only
known user wouldn't benefit from this change. Would seem a little
bit like a waste.

It's also not really an interface that is expected to be used
by a lot of different programs.

- Then a lot of people run old mcelog and I would be opposed to
breaking compatibility, so the old interface would need to be kept
for some time. That's especially important because mcelog breaks
not very obviously, and silent breakages are always bad.

None of the points above are real show stoppers for an ASCII interface,
but I think with all of this above together considered it's not really an
attractive change.

I think what could be done is:

- Investigate how to make the panic message more information without adding full
decoders.
- Implement the default panic timeout method described above to get automatic
on disk logging in common cases.

Would that address your concerns?

-Andi

Re: x86/mce merge, integration hickup + crash, design thoughts

[Andi Kleen]

[-- Attachment #1: Type: text/plain, Size: 723 bytes --]

Ingo Molnar wrote:

> today i (belatedly ...) started looking into the status of the tip/x86/mce 
> branch, and merged it into tip/master as a first step.
> 
> firstly there's a small complication, it triggers this crash with the 
> attached config:
> 

I tested the config on a couple of different systems. Unfortunately I
wasn't able to reproduce the problem. It booted always fine even on multiple
tries.

I had to adapt the configuration slightly to boot in my setup.

The configuration also was not complete, i had
to press return a few times (are you sure you sent me the correct file?)

One merging issue I noted is that you put the perfctrs and mce self
on the same vector. The attached patch fixes that.

-Andi




[-- Attachment #2: mce-merge --]
[-- Type: text/plain, Size: 649 bytes --]

commit c736d118dcad93929dfc21cb72f33bc94c1b314a
Author: Andi Kleen <ak@linux.intel.com>
Date:   Sun Dec 28 14:16:43 2008 +0100

    Avoid conflict of apic error and local perfmon vector
    
    Signed-off-by: Andi Kleen <ak@linux.intel.com>

diff --git a/arch/x86/include/asm/irq_vectors.h b/arch/x86/include/asm/irq_vectors.h
index 73957a2..0fcb682 100644
--- a/arch/x86/include/asm/irq_vectors.h
+++ b/arch/x86/include/asm/irq_vectors.h
@@ -88,7 +88,7 @@
 /*
  * Performance monitoring interrupt vector:
  */
-#define LOCAL_PERF_VECTOR	0xee
+#define LOCAL_PERF_VECTOR	0xed
 
 /*
  * First APIC vector available to drivers: (vectors 0x30-0xee) we

Re: x86/mce merge, integration hickup + crash, design thoughts

[Ingo Molnar]

* Andi Kleen <ak@linux.intel.com> wrote:

> Ingo Molnar wrote:
>
>> today i (belatedly ...) started looking into the status of the 
>> tip/x86/mce branch, and merged it into tip/master as a first step.
>>
>> firstly there's a small complication, it triggers this crash with the  
>> attached config:
>>
>
> I tested the config on a couple of different systems. Unfortunately I 
> wasn't able to reproduce the problem. It booted always fine even on 
> multiple tries.
>
> I had to adapt the configuration slightly to boot in my setup.
>
> The configuration also was not complete, i had to press return a few 
> times (are you sure you sent me the correct file?)
>
> One merging issue I noted is that you put the perfctrs and mce self on 
> the same vector. The attached patch fixes that.

hm, will re-test and see whether i can get more debug info out of the 
system.

> commit c736d118dcad93929dfc21cb72f33bc94c1b314a
> Author: Andi Kleen <ak@linux.intel.com>
> Date:   Sun Dec 28 14:16:43 2008 +0100
> 
>     Avoid conflict of apic error and local perfmon vector
>     
>     Signed-off-by: Andi Kleen <ak@linux.intel.com>
> 
> diff --git a/arch/x86/include/asm/irq_vectors.h b/arch/x86/include/asm/irq_vectors.h
> index 73957a2..0fcb682 100644
> --- a/arch/x86/include/asm/irq_vectors.h
> +++ b/arch/x86/include/asm/irq_vectors.h
> @@ -88,7 +88,7 @@
>  /*
>   * Performance monitoring interrupt vector:
>   */
> -#define LOCAL_PERF_VECTOR	0xee
> +#define LOCAL_PERF_VECTOR	0xed

this reminds me, i dont particularly like the way you added a 
self-interrupt:

+       __send_IPI_shortcut(APIC_DEST_SELF, MCE_SELF_VECTOR, 0);

IPI self-sends are generally more fragile than task-flag based methods, 
and they also waste IRQ vector real-estate.

	Ingo

Re: x86/mce merge, integration hickup + crash, design thoughts

[Andi Kleen]

> this reminds me, i dont particularly like the way you added a 
> self-interrupt:
> 
> +       __send_IPI_shortcut(APIC_DEST_SELF, MCE_SELF_VECTOR, 0);
> 
> IPI self-sends are generally more fragile than task-flag based methods, 
> and they also waste IRQ vector real-estate.

I believe the self IPI method is superior than going through the idle
notifiers using TIF flags as it was done before.

There's no guarantee idle (or other tif checkers like syscall return)
runs anytime soon after the event. They do usually, but it's not guaranteed.
And at least in the tolerant==3 case it's pretty important
to run this code ASAP.

It also is slightly cleaner code IMHO and better expresses the semantics
(execute as soon as the interrupts are on again) the code wants.
Admittedly temporarily it merging slightly harder, but that should
be only a temporary issue.

The original reason I did it this way  was because 32bit didn't have
them (but I think that's obsolete now, 32bit needs them now too
for the idle power management code)

The other rationale was also that with everything moving to per CPU
interrupt spaces vectors are not as precious as they used to be, so
it's ok to use more.

And I'm not aware of a fundamental unreliability of self IPI.
After all the IRQ migration code already relies on it.

-Andi

Re: x86/mce merge, integration hickup + crash, design thoughts

[Russ Anderson]

On Sat, Dec 27, 2008 at 11:51:02PM +0100, Ingo Molnar wrote:
> * Andi Kleen <ak@linux.intel.com> wrote:
> 
> >> A far more useful design for handling MCE events would be to feed them 
> >> into printk logging.
> >
> > If there's ASCII logging it should be separate from normal printk.
> 
> Well, why? Sure, MCE exceptions themselves cannot generally printk 
> [because they are in essence non-maskable contexts] unless they get a 
> fatal MCE [in which case we have no other choice but to try to printk and 
> hope for the best]. But they can sprintf into a buffer which then gets 
> printk-d (or passed to whatever ASCII based facility).

FWIW, that is how ia64 MCA messages are handled.  The messages are 
written to a pre-allocated buffer.  If the mca is recovered, the
buffer gets printked from a safe context.  If the mca is fatal,
the buffer is printked immediately because the system is going down.
The message is short summary information (that ends up in
/var/log/messages).

See mprintk() in arch/ia64/kernel/mca.c .

Formatting of ia64 MCA/CMCI/CPE/INIT records is done by the salinfo
daemon process.  The records are written to NVRAM, so if the
system crashes after reboot salinfo reads the records and writes
them to /var/log/salinfo/decoded.  This is the full ASCII error
record.

> 'struct mce' is pointless complexity and a pointless restriction - and so 
> is /dev/mcelog.

I agree with Andi.  In general that type interface is needed, though
some of the specifics could change.  (ie Ying's new mcelog
implementation uses a per CPU buffer.)

> >> So instead of printing such rather cryptic error messages:
> >>
> >>    MCE 0
> >>    HARDWARE ERROR. This is *NOT* a software problem!
> >>    Please contact your hardware vendor
> >>    CPU 0 BANK 6 MISC 202d ADDR ffeef740
> >>    This is not a software problem!
> >>    Run through mcelog --ascii to decode and contact your hardware vendor
> >>
> >> and expecting people to run mcelog, we should print plain-text 
> >> something like:
> >>
> >>    MCE 0
> >>    HARDWARE ERROR. This is *NOT* a software problem!
> >>    Please contact your hardware vendor
> >>    CPU 1 4 northbridge TSC 89a560bb249
> >>    ADDR 1dfa49690
> >>      Northbridge Chipkill ECC error

Summary ASCII information is useful, especially if the error
is clearly a hardware error.  Andi is right that decoding the
information to print the specific failing hardware (ie which 
DIMM) may be too dificult to decode on the way down.  It would
be great to identify the failing hardware component on the
way down, when possible.

> > It turns out that users don't really find this more enlightening (most 
> > users have no clue what a Northbridge is).  They think it's some kind of 
> > kernel bug even with the HARDWARE ERROR header.
> 
> You should not assume that administrators/users reading kernel crash 
> messages are dumb. (an ordinary user wont see it most of the time anyway) 
> 
> The usage patterns i see is that admins who get an MCE crash often fail to 
> write down the whole MCE message (not realizing that it is important) and 
> have to go back and reproduce the MCE crash once again before they can get 
> any meaningful information.

This is why saving the error records to MVRAM is so useful.
After reboot the records can be read, formatted, and logged.

> Printing out cryptic hexadecimal error codes, requiring people to write 
> them down and decode them in user-space is the technology of the 80s - i 
> didnt think i'd have to argue too much about this ;-)

I agree.  Printing out the failing hardware component in
human readable form, when possible, is the best thing.

> Reducing the amount of information presented to the user in such a crash 
> situation is a dumb idea. (especially here where the MCE information is 
> rather dense and single-screen anyway - so there's no screen real estate 
> considerations.)

On ia64 a summary message indicating a fatal MCA is printed.
The full useful information is written to NVRAM.

> > The only people who really care about the micro architectural details in 
> > full are chip developers, and those typically decode using other methods 
> > anyways.
> 
> People do care about getting meaningful crash information from the kernel. 

I agree with Andi.  Full error info needs to be captured at the
time of the crash.  On ia64, MCA records from crashes have been used
by chip developers to identify problems.

> This is a basic principle in the Linux kernel. We try to print out as 
> useful information as possible - and only cut down on it if the 
> information physically does not fit on the screen. (which is not a problem 
> here)

As long as the info is catpured and available on reboot, I
do not think the full information needs to be printed
on the way down.

-- 
Russ Anderson, OS RAS/Partitioning Project Lead  
SGI - Silicon Graphics Inc          rja@sgi.com

Re: x86/mce merge, integration hickup + crash, design thoughts

[Russ Anderson]

On Sat, Dec 27, 2008 at 04:50:19PM +0100, Ingo Molnar wrote:
> 
> If we want to enable userspace to capture MCE events, then it must be done 
> in a way that benefits the whole kernel, not just x86: a structured 
> logging facility that is in essence a printk variant and is ASCII driven. 

It would be very useful to implement error handling in a way
that has an arch independed framework.  ia64 has already
implemented much of this functionality.

> Such event sources should be discoverable, and only 'aware' printouts 
> should go into this new facility (not all printks). Demultiplexing should 
> be easy and well-defined.
> 
> I.e. we could use this opportunity of the MCE code unification to bring 
> the code to the next level - and not prolongue to broken concepts of the 
> past.
> 
> I'd be glad to help out with any portion of this, it should be easy to 
> solve and it will clearly improve the code. For .29 we could just do a raw 
> printk based approach with no decoding just yet, and layer smart decoding 
> and structured logging for .30.
> 
> Hm?

Something along those lines, though I would support more of
Andi's code getting in for .29 (even if it means putting code
into .29 and then changing it for .30).  It will take time to
come up with a cross arch framework, longer than the .29
merge window.

Thanks,
-- 
Russ Anderson, OS RAS/Partitioning Project Lead  
SGI - Silicon Graphics Inc          rja@sgi.com

Re: x86/mce merge, integration hickup + crash, design thoughts

[Andi Kleen]

Russ Anderson wrote:

> Summary ASCII information is useful, especially if the error
> is clearly a hardware error.  Andi is right that decoding the
> information to print the specific failing hardware (ie which 
> DIMM) may be too dificult to decode on the way down.  It would
> be great to identify the failing hardware component on the
> way down, when possible.

This will hopefully happen in the future. In fact mcelog has support
to decode using DMI tables, but it turns out this doesn't work
very well in practice (both because of BIOS problems and because
the DMI standard was not really designed for this). That is why
I wouldn't advocate right now to move this code from mcelog
into the kernel. This might change later.


>>> It turns out that users don't really find this more enlightening (most 
>>> users have no clue what a Northbridge is).  They think it's some kind of 
>>> kernel bug even with the HARDWARE ERROR header.
>> You should not assume that administrators/users reading kernel crash 
>> messages are dumb. (an ordinary user wont see it most of the time anyway) 
>>
>> The usage patterns i see is that admins who get an MCE crash often fail to 
>> write down the whole MCE message (not realizing that it is important) and 
>> have to go back and reproduce the MCE crash once again before they can get 
>> any meaningful information.
> 
> This is why saving the error records to MVRAM is so useful.
> After reboot the records can be read, formatted, and logged.

I've been looking at using EFI runtime services for this, but it's
also somewhat problematic (e.g. getting the messages out again in
a useful way without risking duplicating events)

There are also a few other candidates like the Management Engine
interface on many Intel platforms, but it's also not available everywhere.

Short term just changing the MCE panic to timeout by default
is the best option. I'll probably submit that for .30

-Andi

Re: x86/mce merge, integration hickup + crash, design thoughts

[Russ Anderson]

On Wed, Dec 31, 2008 at 02:32:07PM +0100, Andi Kleen wrote:
> Russ Anderson wrote:
> 
> >Summary ASCII information is useful, especially if the error
> >is clearly a hardware error.  Andi is right that decoding the
> >information to print the specific failing hardware (ie which 
> >DIMM) may be too dificult to decode on the way down.  It would
> >be great to identify the failing hardware component on the
> >way down, when possible.
> 
> This will hopefully happen in the future. In fact mcelog has support
> to decode using DMI tables, but it turns out this doesn't work
> very well in practice (both because of BIOS problems and because
> the DMI standard was not really designed for this). That is why
> I wouldn't advocate right now to move this code from mcelog
> into the kernel. This might change later.

Yes.  The way ia64 deals with this is the SAL (bios) does much of the
platform specific interpretation to identify the failing hardware
and passes that information up to linux.  That allows the kernel
to be more platform independent while the SAL (bios) deals with the
platform specifics.  

The problem of not having a spec, such as the ia64 SAL spec, is
that other interfaces end up getting overloaded, such as 
the DMI tables.  Should this be part of the EFI standard?
It really does need to be part of a standard specification.

> >>>It turns out that users don't really find this more enlightening (most 
> >>>users have no clue what a Northbridge is).  They think it's some kind of 
> >>>kernel bug even with the HARDWARE ERROR header.
> >>You should not assume that administrators/users reading kernel crash 
> >>messages are dumb. (an ordinary user wont see it most of the time anyway) 
> >>
> >>The usage patterns i see is that admins who get an MCE crash often fail 
> >>to write down the whole MCE message (not realizing that it is important) 
> >>and have to go back and reproduce the MCE crash once again before they 
> >>can get any meaningful information.
> >
> >This is why saving the error records to MVRAM is so useful.
> >After reboot the records can be read, formatted, and logged.
> 
> I've been looking at using EFI runtime services for this, but it's

I believe that is the right approach.  Error handling was one
of my primary motivations for getting in EFI runtime services
for UV.  Having a more standard interface, rather than platform
specific, would be good. 

> also somewhat problematic (e.g. getting the messages out again in
> a useful way without risking duplicating events)

In ia64, salinfo_decode looks for duplicate records when reading
from SAL (bios).  SGI SAL uses hardware timestamps when building 
error records to avoid duplicates.  There are still occasional
duplicate records in a few cases, but rare enough to not be a
problem.  

One case that always generates duplicate records is a recovered
MCA.  The MCA record is logged to NVRAM before linux tries to 
recover (in case the system crashes).  After recovery, the MCA
record is logged with a recovered status, making the record look
different than the record logged to NVRAM (to salinfo_decoded).
salinfo_decoded logs both the original MCA record and the MCA
record marked recovered (writing them to /var/log/salinfo/decoded).
It turned out to be useful to have both records when explaining
to customers how error recovery worked (in the case of an
application hitting a memory uncorrectable error).  When they
realize that each duplicate MCA record represents a time that the
system did not crash, they suddenly like those duplicate records. :-)

The bigger concern is not losing records (especially with a burst
of errors).  The issue is keeping the older (more significant)
error records and dropping the newer (less significant), not
to be confused with thresholding correctable errors...

-- 
Russ Anderson, OS RAS/Partitioning Project Lead  
SGI - Silicon Graphics Inc          rja@sgi.com

Re: x86/mce merge, integration hickup + crash, design thoughts

[Tim Hockin]

On Sat, Dec 27, 2008 at 7:50 AM, Ingo Molnar <mingo@elte.hu> wrote:
>
> We really need to get rid of /dev/mcelog. It's a quirky binary logging
> facility not available on 32-bit on current kernels and it has a couple of
> limitations:

It's not quirky, it's structured.  Until and unless there is a better,
structured interface, this needs to stay.

>  - it squeezes all MCE errors from the whole system into a small,
>    32-entry ringbuffer.

Yes 32 is small, but not really a big problem in practice.  The MCE daemon
(http://mcedaemon.googlecode.com) goes a long way towards making this
a non-issue.  Every distro should ship mced.

>  - it puts all the MCE logging info into an intermediary binary log
>    record format: 'struct mce' - just for userspace to in essence
>    printf out those entries with minimal post-processing. The fact that
>    we squeeze all information into a fixed-size binary record makes it
>    hard to extend and complicates the code needlessly.

This is not true, we do EXTENSIVE post-processing of MCEs.  Yes it is hard
to extend, but that's not the same as saying it is useless.

>  - these design aspects are also quite harmful to usability: by
>    default all MCEs are fatal currently (pre-Nehalem anyway), so
>    /dev/mcelog will only be used if a user goes out on a limb to
>    configure it and sets the tolerant flag.

Not true.  The vast vast vast majority of MCEs are corrected, as per our
experience, and I suspect we've got more experience with MCEs than just
about any other consumer.

> A far more useful design for handling MCE events would be to feed them
> into printk logging. So instead of printing such rather cryptic error
> messages:
>
>   MCE 0
>   HARDWARE ERROR. This is *NOT* a software problem!
>   Please contact your hardware vendor
>   CPU 0 BANK 6 MISC 202d ADDR ffeef740
>   This is not a software problem!
>   Run through mcelog --ascii to decode and contact your hardware vendor
>
> and expecting people to run mcelog, we should print plain-text something
> like:
>
>   MCE 0
>   HARDWARE ERROR. This is *NOT* a software problem!
>   Please contact your hardware vendor
>   CPU 1 4 northbridge TSC 89a560bb249
>   ADDR 1dfa49690
>     Northbridge Chipkill ECC error
>     Chipkill ECC syndrome = 2021
>          bit46 = corrected ecc error
>     bus error 'local node response, request didn't time out
>         generic read mem transaction
>         memory access, level generic'
>   STATUS 9410c00020080a13 MCGSTATUS 0

NO!  This moves in the completely wrong direction.  This output is just
about impossible to parse sanely.  And don't get me started about how much
of a pain it is to encode all of the vendor-centric and per-platform
specifics so that it decodes all the useful bits.

I promise you that there are things that are useful to us that the kernel
will not parse and print.

> straight from the kernel. This means that the MCEs will make a lot more
> sense at a glance - and the user can figure out the suspected trouble
> area, without having to find some other box to run mcelog on, etc. We can
> eliminate the user-space mcelog utility/daemon component altogether - it
> buys us little but needless complexity and inflexibility.

I disagree totally.  Putting the logic for decoding MCEs into the kernel
adds a large amount of complex, hard to maintain code precisely where it
is most destructive.  Any sort of decoding of this data in-kernel is
probably a mistake (yes, I am looking at you EDAC).

It would be much cleaner to instead make the kernel really good at
publishing errors and let userspace decode them.  A naive end user won't
know the difference - they will see a message on the syslog or the
console.  Power users will be able to do better things.

> If we want to enable userspace to capture MCE events, then it must be done
> in a way that benefits the whole kernel, not just x86: a structured
> logging facility that is in essence a printk variant and is ASCII driven.

I would be fine with this (though I disagree that it *must* be ASCII).  I
don't care how the raw data comes out of the kernel, just that it is the
raw data, not some half-assedly parsed data.

> Such event sources should be discoverable, and only 'aware' printouts
> should go into this new facility (not all printks). Demultiplexing should
> be easy and well-defined.

I agree 1000% with this - it is something I have wanted for a long time,
and that some of my team are starting to look at again.

> I.e. we could use this opportunity of the MCE code unification to bring
> the code to the next level - and not prolongue to broken concepts of the
> past.

Don't dicard the past until the present can stand on it's own.  Please.

> I'd be glad to help out with any portion of this, it should be easy to
> solve and it will clearly improve the code. For .29 we could just do a raw
> printk based approach with no decoding just yet, and layer smart decoding
> and structured logging for .30.

Please leave /dev/mcelog as a CONFIG option until such time as an
appropriate replacement is *done*.  We rely on it.

Tim

Re: x86/mce merge, integration hickup + crash, design thoughts

[Andi Kleen]

Tim Hockin wrote:

> 
>>  - it squeezes all MCE errors from the whole system into a small,
>>    32-entry ringbuffer.
> 
> Yes 32 is small, but not really a big problem in practice.  The MCE daemon
> (http://mcedaemon.googlecode.com)

Interesting.

> goes a long way towards making this
> a non-issue.  Every distro should ship mced.

The latest mcelog version (not yet released) also supports a daemon
mode btw. But the limited buffer has been also fixed already
here and replaced with more flexible per CPU buffers.

>>  - it puts all the MCE logging info into an intermediary binary log
>>    record format: 'struct mce' - just for userspace to in essence
>>    printf out those entries with minimal post-processing. The fact that
>>    we squeeze all information into a fixed-size binary record makes it
>>    hard to extend and complicates the code needlessly.
> 
> This is not true, we do EXTENSIVE post-processing of MCEs.  Yes it is hard
> to extend, but that's not the same as saying it is useless.

It's not hard to extend at all (I did it several times)
Adding fields is extremly simple and fully forwards and
backwards compatible. That works because mcelog asks
the kernel about the record size and just ignores any
excess data.

The only thing that cannot be done is removing old fields, but
that is difficult with ASCII parsers too (text parsers tend
to bail out when they can't find some field they expect)

They can be obsoleted fine however (happened with cpu -> extcpu)

> 
>>  - these design aspects are also quite harmful to usability: by
>>    default all MCEs are fatal currently (pre-Nehalem anyway), so
>>    /dev/mcelog will only be used if a user goes out on a limb to
>>    configure it and sets the tolerant flag.
> 
> Not true.  The vast vast vast majority of MCEs are corrected, as per our
> experience, and I suspect we've got more experience with MCEs than just
> about any other consumer.

My employer has a lot of experience with MCEs too...  And it matches your
experience.

-Andi

Re: x86/mce merge, integration hickup + crash, design thoughts

[Ingo Molnar]

* Andi Kleen <ak@linux.intel.com> wrote:

> Ingo Molnar wrote:
>
>>>> A far more useful design for handling MCE events would be to feed 
>>>> them into printk logging.
>>> If there's ASCII logging it should be separate from normal printk.
>>
>> Well, why? 
>
> Mostly because the problem is not a kernel issue. Especially large 
> systems with a lot of memory can generate a lot of corrected events (one 
> bit flips in DIMMs are not that uncommon) and it's not good to mix that 
> all up into other kernel messages. It also makes it more clear that it's 
> not a kernel problem, but a hardware problem. I've got feedback over the 
> years that confirm this sight.

Is your argument that syslog is not suitable for the logging of hw events?

If that is your argument then the answer is to extend syslog with those 
aspects, instead of widening the quirky /dev based mce ABIs to achieve 
something similar.

If you think that it's suitable then that contradicts your point above.

> [...]
>
> None of the points above are real show stoppers for an ASCII interface, 
> but I think with all of this above together considered it's not really 
> an attractive change.
>
> I think what could be done is:
>
> - Investigate how to make the panic message more information without 
>   adding full decoders.
>
> - Implement the default panic timeout method 
>   described above to get automatic on disk logging in common cases.
>
> Would that address your concerns?

For me the main blocking point is that mcelog uses a quirky, binary 
side-channel instead of using our main ASCII based logging abstraction 
that we have in Linux: printk + syslog.

That is a high level argument, while most of your arguments are low level. 
I dont think you can understand my argument if you concentrate on the low 
level only.

We need to resolve this instead of expanding the broken /dev/mcelog 
interface. If we expand the broken interface first then that removes all 
the incentives to enhance the primary logging facility of Linux in this 
area.

Anyway, this merge window has been very crowded in the x86 space already, 
and the MCE topic is not particularly super-important to have right now 
either, so lets skip it for this cycle so that we have more time to 
cleanly work out these details.

Let me know if there are must-have fixes in it and we can cherry-pick it 
over into x86/urgent.

Thanks,

	Ingo

Re: x86/mce merge, integration hickup + crash, design thoughts

[Tim Hockin]

On Tue, Jan 13, 2009 at 9:45 AM, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Andi Kleen <ak@linux.intel.com> wrote:
>
>> Ingo Molnar wrote:
>>
>>>>> A far more useful design for handling MCE events would be to feed
>>>>> them into printk logging.
>>>> If there's ASCII logging it should be separate from normal printk.
>>>
>>> Well, why?
>>
>> Mostly because the problem is not a kernel issue. Especially large
>> systems with a lot of memory can generate a lot of corrected events (one
>> bit flips in DIMMs are not that uncommon) and it's not good to mix that
>> all up into other kernel messages. It also makes it more clear that it's
>> not a kernel problem, but a hardware problem. I've got feedback over the
>> years that confirm this sight.
>
> Is your argument that syslog is not suitable for the logging of hw events?

I will argue that, yes.

> If that is your argument then the answer is to extend syslog with those
> aspects, instead of widening the quirky /dev based mce ABIs to achieve
> something similar.

I don't like "extend" in this context.  I'd prefer to think of it as a
side-band solution that we need.  And yes, such a solution COULD
obsolete mcelog.  Do you have such a solution done?  Specced?

> If you think that it's suitable then that contradicts your point above.
>
>> [...]
>>
>> None of the points above are real show stoppers for an ASCII interface,
>> but I think with all of this above together considered it's not really
>> an attractive change.
>>
>> I think what could be done is:
>>
>> - Investigate how to make the panic message more information without
>>   adding full decoders.
>>
>> - Implement the default panic timeout method
>>   described above to get automatic on disk logging in common cases.
>>
>> Would that address your concerns?
>
> For me the main blocking point is that mcelog uses a quirky, binary
> side-channel instead of using our main ASCII based logging abstraction
> that we have in Linux: printk + syslog.

It's not particularly quirky - it's a fixed-size ringbuffer.  You're
trying to spin it as some eccentric interface designed by tripped-out
hippies, but it's not really.  It's just a simple piece of plumbing
for a very specific purpose, which exists because there was no better
answer at the time (is there one now?)

> That is a high level argument, while most of your arguments are low level.
> I dont think you can understand my argument if you concentrate on the low
> level only.
>
> We need to resolve this instead of expanding the broken /dev/mcelog
> interface. If we expand the broken interface first then that removes all
> the incentives to enhance the primary logging facility of Linux in this
> area.

I'm 100% on board with that and will even help staff the effort.  This
is something that is VERY HIGHLY desired here.  I already have a
couple peopel looking at this and other HW-error reporting issues.

> Anyway, this merge window has been very crowded in the x86 space already,
> and the MCE topic is not particularly super-important to have right now
> either, so lets skip it for this cycle so that we have more time to
> cleanly work out these details.

Thanks!

> Let me know if there are must-have fixes in it and we can cherry-pick it
> over into x86/urgent.

Tim

Re: x86/mce merge, integration hickup + crash, design thoughts

[Huang Ying]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

On Wed, 2009-01-14 at 01:45 +0800, Ingo Molnar wrote:
> * Andi Kleen <ak@linux.intel.com> wrote:
> 
> > Ingo Molnar wrote:
> >
> >>>> A far more useful design for handling MCE events would be to feed 
> >>>> them into printk logging.
> >>> If there's ASCII logging it should be separate from normal printk.
> >>
> >> Well, why? 
> >
> > Mostly because the problem is not a kernel issue. Especially large 
> > systems with a lot of memory can generate a lot of corrected events (one 
> > bit flips in DIMMs are not that uncommon) and it's not good to mix that 
> > all up into other kernel messages. It also makes it more clear that it's 
> > not a kernel problem, but a hardware problem. I've got feedback over the 
> > years that confirm this sight.
> 
> Is your argument that syslog is not suitable for the logging of hw events?
> 
> If that is your argument then the answer is to extend syslog with those 
> aspects, instead of widening the quirky /dev based mce ABIs to achieve 
> something similar.

In current /dev based mce ABI implementation, syslog is used for logging
hw events, not though printk, but through /dev/mcelog and /sbin/mcelog.

For uncorrected MCE, they should be logged via printk. But for corrected
MCE, there could be thousands/millions ones (imagining you have a DIMM
with one data pin corrupted). I don't think it's a good idea to blend
these hardware events with other kernel software events in printk.

Best Regards,
Huang Ying


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: x86/mce merge, integration hickup + crash, design thoughts

[Andi Kleen]

> 
> I'm 100% on board with that and will even help staff the effort. 

Well if you want to change anything the code would be a good idea first to
establish clearly what is actually broken. I know various areas that need
improvement (and I have patches fixes most of them), but to my knowledge
none of them would be fixed by ASCII logging.

Perhaps a good start would be if Ingo could expand what exactly
he believes is broken currently. At least his earlier "high level" argument
seems to be large  based on clear misunderstandings of what kind
of MCE events are common and what not. I don't really blame
him for that since MCEs are obscure and difficult and badly
documented (I had a hard time getting up to speed on them myself
and it took me quite some time). But I hope he doesn't
dismiss the advice from people who have more experience with
them than him though.

I wrote a long email earlier in the thread with all the reasons why
ASCII logging is difficult (like the various atomicity issues and also others)
I haven't heard anyone refuting any of the arguments in there, so I assume they
are agreed one by everyone.

I would appreciate if the people who continue to propose ASCII
logging would explain how they plan to solve these problems.

> This
> is something that is VERY HIGHLY desired here.  

What is exactly desired?

> I already have a
> couple peopel looking at this and other HW-error reporting issues.

I have lots of patches pending for over half a year (including
tons of bug fixes) and they get all delayed again and again with
very little justification why. So before writing any new code
it would be good to just get the already pending improvements in.

-Andi

Re: x86/mce merge, integration hickup + crash, design thoughts

[Tim Hockin]

On Wed, Jan 14, 2009 at 1:29 AM, Andi Kleen <ak@linux.intel.com> wrote:
>
>>
>> I'm 100% on board with that and will even help staff the effort.
>
> Well if you want to change anything the code would be a good idea first to
> establish clearly what is actually broken. I know various areas that need
> improvement (and I have patches fixes most of them), but to my knowledge
> none of them would be fixed by ASCII logging.
>
> Perhaps a good start would be if Ingo could expand what exactly
> he believes is broken currently. At least his earlier "high level" argument
> seems to be large  based on clear misunderstandings of what kind
> of MCE events are common and what not. I don't really blame
> him for that since MCEs are obscure and difficult and badly
> documented (I had a hard time getting up to speed on them myself
> and it took me quite some time). But I hope he doesn't
> dismiss the advice from people who have more experience with
> them than him though.
>
> I wrote a long email earlier in the thread with all the reasons why
> ASCII logging is difficult (like the various atomicity issues and also
> others)
> I haven't heard anyone refuting any of the arguments in there, so I assume
> they
> are agreed one by everyone.
>
> I would appreciate if the people who continue to propose ASCII
> logging would explain how they plan to solve these problems.
>
>> This
>> is something that is VERY HIGHLY desired here.
>
> What is exactly desired?

>From my point of view: a single, consistent, easy logging interface
for the kernel to send *structured data* about hardware/system events
and errors up to userspace.

I don't care if it is ASCII, but it probably can be done in ASCII.
That's the cart before the horse, IMHO.  I just want something more
structured and better suited than printk().

>> I already have a
>> couple peopel looking at this and other HW-error reporting issues.
>
> I have lots of patches pending for over half a year (including
> tons of bug fixes) and they get all delayed again and again with
> very little justification why. So before writing any new code
> it would be good to just get the already pending improvements in.

We'd LOVE your improvements, if they work.  MCE is always a sore point
for us, in that we take too many of them.  Anything to reduce their
impact is a win.

Tim

Re: x86/mce merge, integration hickup + crash, design thoughts

[Andi Kleen]

> 
> From my point of view: a single, consistent, easy logging interface
> for the kernel to send *structured data* about hardware/system events
> and errors up to userspace.

Which kinds of events were you thinking of?

So far we managed by cramming some other CPU events like thermal
trip into "pseudo banks" in struct mce. Admittedly it's not the
most pretty solution in the world, but it worked.

-Andi

Re: x86/mce merge, integration hickup + crash, design thoughts

[Tim Hockin]

On Wed, Jan 14, 2009 at 10:05 AM, Andi Kleen <ak@linux.intel.com> wrote:
>
>>
>> From my point of view: a single, consistent, easy logging interface
>> for the kernel to send *structured data* about hardware/system events
>> and errors up to userspace.
>
> Which kinds of events were you thinking of?
>
> So far we managed by cramming some other CPU events like thermal
> trip into "pseudo banks" in struct mce. Admittedly it's not the
> most pretty solution in the world, but it worked.

Yeah, no offense, but that's horrible :)

Ideally, I'd rather see a more generic conduit for all sorts of
events.  Polled and exception MCEs.  Thermal interrupts.  MCE
threshold interrupts.  EDAC polled errors.  PCI-express errors.  SATA
disk timeouts.

Now I know there are different conduits for some events - netlink
tells me about netif link up/down events I think.  I would settle for
a small number of interfaces.  What I don't want is what we have today
- EVERYTHING has a different interface.  Some are poll()-able.  Some
have to be actively polled.  Some have to have a daemon listening or
else messages are dropped.  Some have to parse logs.  Puke.

Put it this way:  Given a thousand machines, I want to gather,
collate, and correlate all these events.  I want to be able to produce
a "life story" of sorts for a machine and for a data center.  Once I
can do that, I can start to make predictive diagnoses more accurately,
and I can know how much these things actually COST us.

Tim

Re: x86/mce merge, integration hickup + crash, design thoughts

[Andi Kleen]

Tim Hockin wrote:
> Yeah, no offense, but that's horrible :)

I'm not sure it's worse than the XML like format proposals that seem to
get thrown around. That is I am the only one who mentioned the
X word yet, but the structured ASCII records that have been hinted
at would be exactly like that.

> 
> Ideally, I'd rather see a more generic conduit for all sorts of
> events.  Polled and exception MCEs.  Thermal interrupts.  MCE
> threshold interrupts. 

Actually I think now MCE threshold interrupts should have never been
separate events. That was a design mistake in the AMD implementation
(together with all the sysfs complications)

An MCE threshold interrupt is just a slightly different internal
notification mechanism and it should only trigger the events it reads
from the MCE banks. Nothing more.
My upcoming CMCI code works exactly this way.

> PCI-express errors. 

Yes we need some mechanism for those. Fortunately that's easier
because it doesn't need to handle NMIs.

> SATA
> disk timeouts.

Now that's a different issue. Generalized driver error reporting for everyone.

There was a lot of discussion some years ago from a IBM proposal to do
in general structured error reporting. But that was quite unpopular
and no-one really liked it.

What came out of it was the dev_printk() stuff that allows
to match error messages to devices. So you already have some
baby steps in this direction.

I suspect doing this fully generalized would be quite difficult
because there would be so many people you have to convince.



> Now I know there are different conduits for some events - netlink
> tells me about netif link up/down events I think.  I would settle for
> a small number of interfaces.  What I don't want is what we have today
> - EVERYTHING has a different interface.  Some are poll()-able.  Some
> have to be actively polled.  Some have to have a daemon listening or
> else messages are dropped.  

Well the kernel will always have limited buffers, so the someone
needs to listen problem will be always there.

There are not __that__ many I think.

Also whatever code handles this has to have special code for
all of these anyways, so having a variety of interfaces for them
doesn't seem like the end of the world to me.

> 
> Put it this way:  Given a thousand machines, I want to gather,
> collate, and correlate all these events.  I want to be able to produce
> a "life story" of sorts for a machine and for a data center.  Once I
> can do that, I can start to make predictive diagnoses more accurately,
> and I can know how much these things actually COST us.

Sure sounds nice. But frankly I don't see it happening. It would
be just too radical a change of too much code.

-Andi

Re: x86/mce merge, integration hickup + crash, design thoughts

[Tim Hockin]

On Thu, Jan 15, 2009 at 2:56 PM, Andi Kleen <ak@linux.intel.com> wrote:
> Tim Hockin wrote:
>>
>> Yeah, no offense, but that's horrible :)
>
> I'm not sure it's worse than the XML like format proposals that seem to
> get thrown around. That is I am the only one who mentioned the
> X word yet, but the structured ASCII records that have been hinted
> at would be exactly like that.

Yeah, if anyone proposes XML, I'll punch them :)  I'm not that firm on
it being ASCII myself, but it would be OK.  Google's protocol buffers
are a good example of a very efficient encoding of flexible data.
This is not an unsolvable problem.

>> SATA
>> disk timeouts.
>
> Now that's a different issue. Generalized driver error reporting for
> everyone.
>
> There was a lot of discussion some years ago from a IBM proposal to do
> in general structured error reporting. But that was quite unpopular
> and no-one really liked it.
>
> What came out of it was the dev_printk() stuff that allows
> to match error messages to devices. So you already have some
> baby steps in this direction.

I remember - I was at Sun at the time.  We wanted the same thing.

> I suspect doing this fully generalized would be quite difficult
> because there would be so many people you have to convince.

Yes, it will be hard.  But I think it will not be impossible.

>> Now I know there are different conduits for some events - netlink
>> tells me about netif link up/down events I think.  I would settle for
>> a small number of interfaces.  What I don't want is what we have today
>> - EVERYTHING has a different interface.  Some are poll()-able.  Some
>> have to be actively polled.  Some have to have a daemon listening or
>> else messages are dropped.
>
> Well the kernel will always have limited buffers, so the someone
> needs to listen problem will be always there.

Sure, but just having a small buffer, like mcelog, and being
poll()able goes a log way.

>> Put it this way:  Given a thousand machines, I want to gather,
>> collate, and correlate all these events.  I want to be able to produce
>> a "life story" of sorts for a machine and for a data center.  Once I
>> can do that, I can start to make predictive diagnoses more accurately,
>> and I can know how much these things actually COST us.
>
> Sure sounds nice. But frankly I don't see it happening. It would
> be just too radical a change of too much code.

We do a lot of this now, but it's far less accurate than it could be,
because we drop so many events.  And programming to all the different
interfaces is difficult and error-prone.

I'm hoping we can find a solution that makes everyone only a little bit unhappy.

Tim