Race condition in module load causing undefined symbols

Race condition in module load causing undefined symbols

[Stephen Lord]

[-- Attachment #1: Type: text/plain, Size: 2442 bytes --]


Hi,

I am having troubles getting any recent kernel to boot successfully
on one of my machines, a generic 2.6GHz P4 box with HT enabled
running an updated Fedora Core 3 distro. This is present in
2.6.12-rc6. It does not manifest itself with the Fedora Core
kernels which have identical initrd contents as far as the
init script and the set of modules included goes.

The problem manifests itself as various undefined symbols from
module loads.  Here is the relevant section from the init script:

echo Starting udev
/sbin/udevstart
echo -n "/sbin/hotplug" > /proc/sys/kernel/hotplug
echo "Loading scsi_mod.ko module"
insmod /lib/scsi_mod.ko
echo "Loading sd_mod.ko module"
insmod /lib/sd_mod.ko
echo "Loading libata.ko module"
insmod /lib/libata.ko
echo "Loading ata_piix.ko module"
insmod /lib/ata_piix.ko
echo "Loading ieee1394.ko module"
insmod /lib/ieee1394.ko
echo "Loading ohci1394.ko module"
insmod /lib/ohci1394.ko
echo "Loading sbp2.ko module"
insmod /lib/sbp2.ko
echo "Loading dm-mod.ko module"
insmod /lib/dm-mod.ko
echo "Loading jbd.ko module"
insmod /lib/jbd.ko
echo "Loading ext3.ko module"
insmod /lib/ext3.ko
echo "Loading dm-mirror.ko module"
insmod /lib/dm-mirror.ko
echo "Loading dm-zero.ko module"
insmod /lib/dm-zero.ko
echo "Loading dm-snapshot.ko module"
insmod /lib/dm-snapshot.ko
/sbin/udevstart

The failures are different on different boots, sometimes the ata_piix
module cannot find symbols from libata, sometimes ext3 cannot find jbd
symbols, sometimes dm modules cannot find things from dm-mod, usually
it is a combination of these. End result is a panic when it cannot
find the root device.

 From the behavior, it appears that a module load is returning
control to user space before the previous one has got its symbols
loaded.

The loadable module section of my config file looks like this:

CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y

Since the fedora core kernels work consistently, this is either
something about my config triggering this, or something
redhat has in their kernels - I suspect the former.

My module tools are "module-init-tools version 3.1-pre5"
which seems to be as far as Redhat has updated their rpms
to. I will try a later version of these to see if it makes
any difference.

Complete .config attached.

Steve


[-- Attachment #2: config --]
[-- Type: text/plain, Size: 31367 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.12-rc6
# Fri Jun 10 08:00:11 2005
#
CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y

#
# Code maturity level options
#
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32

#
# General setup
#
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_BSD_PROCESS_ACCT=y
# CONFIG_BSD_PROCESS_ACCT_V3 is not set
CONFIG_SYSCTL=y
# CONFIG_AUDIT is not set
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
# CONFIG_CPUSETS is not set
# CONFIG_EMBEDDED is not set
CONFIG_KALLSYMS=y
# CONFIG_KALLSYMS_ALL is not set
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
# CONFIG_TINY_SHMEM is not set
CONFIG_BASE_SMALL=0

#
# Loadable module support
#
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
CONFIG_OBSOLETE_MODPARM=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_KMOD=y
CONFIG_STOP_MACHINE=y

#
# Processor type and features
#
CONFIG_X86_PC=y
# CONFIG_X86_ELAN is not set
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_NUMAQ is not set
# CONFIG_X86_SUMMIT is not set
# CONFIG_X86_BIGSMP is not set
# CONFIG_X86_VISWS is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_ES7000 is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
CONFIG_MPENTIUM4=y
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_X86_GENERIC is not set
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=7
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_SMP=y
CONFIG_NR_CPUS=4
CONFIG_SCHED_SMT=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
# CONFIG_X86_MCE_P4THERMAL is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_X86_REBOOTFIXUPS is not set
CONFIG_MICROCODE=y
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y

#
# Firmware Drivers
#
# CONFIG_EDD is not set
# CONFIG_NOHIGHMEM is not set
CONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not set
CONFIG_HIGHMEM=y
# CONFIG_HIGHPTE is not set
# CONFIG_MATH_EMULATION is not set
CONFIG_MTRR=y
# CONFIG_EFI is not set
CONFIG_IRQBALANCE=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_REGPARM=y
CONFIG_SECCOMP=y

#
# Power management options (ACPI, APM)
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SOFTWARE_SUSPEND is not set

#
# ACPI (Advanced Configuration and Power Interface) Support
#
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_SLEEP_PROC_FS=y
CONFIG_ACPI_AC=m
# CONFIG_ACPI_BATTERY is not set
CONFIG_ACPI_BUTTON=m
# CONFIG_ACPI_VIDEO is not set
CONFIG_ACPI_FAN=m
CONFIG_ACPI_PROCESSOR=m
CONFIG_ACPI_THERMAL=m
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_IBM is not set
# CONFIG_ACPI_TOSHIBA is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y
CONFIG_X86_PM_TIMER=y
# CONFIG_ACPI_CONTAINER is not set

#
# APM (Advanced Power Management) BIOS Support
#
# CONFIG_APM is not set

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set

#
# Bus options (PCI, PCMCIA, EISA, MCA, ISA)
#
CONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GOMMCONFIG is not set
# CONFIG_PCI_GODIRECT is not set
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
# CONFIG_PCIEPORTBUS is not set
# CONFIG_PCI_MSI is not set
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
# CONFIG_PCI_DEBUG is not set
CONFIG_ISA_DMA_API=y
# CONFIG_ISA is not set
# CONFIG_MCA is not set
# CONFIG_SCx200 is not set

#
# PCCARD (PCMCIA/CardBus) support
#
# CONFIG_PCCARD is not set

#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats
#
CONFIG_BINFMT_ELF=y
# CONFIG_BINFMT_AOUT is not set
# CONFIG_BINFMT_MISC is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
# CONFIG_DEBUG_DRIVER is not set

#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set

#
# Parallel port support
#
# CONFIG_PARPORT is not set

#
# Plug and Play support
#
# CONFIG_PNP is not set

#
# Block devices
#
CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=16384
CONFIG_BLK_DEV_INITRD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_LBD=y
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=8
# CONFIG_CDROM_PKTCDVD_WCACHE is not set

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_ATA_OVER_ETH is not set

#
# ATA/ATAPI/MFM/RLL support
#
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y

#
# Please see Documentation/ide.txt for help/info on IDE drives
#
# CONFIG_BLK_DEV_IDE_SATA is not set
# CONFIG_BLK_DEV_HD_IDE is not set
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
CONFIG_BLK_DEV_IDEFLOPPY=y
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set

#
# IDE chipset support/bugfixes
#
CONFIG_IDE_GENERIC=y
# CONFIG_BLK_DEV_CMD640 is not set
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
# CONFIG_BLK_DEV_OFFBOARD is not set
CONFIG_BLK_DEV_GENERIC=y
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_RZ1000 is not set
CONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_IDEDMA_FORCED is not set
CONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_BLK_DEV_ATIIXP is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5520 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_BLK_DEV_HPT366 is not set
# CONFIG_BLK_DEV_SC1200 is not set
# CONFIG_BLK_DEV_PIIX is not set
# CONFIG_BLK_DEV_NS87415 is not set
CONFIG_BLK_DEV_PDC202XX_OLD=y
CONFIG_PDC202XX_BURST=y
CONFIG_BLK_DEV_PDC202XX_NEW=y
CONFIG_PDC202XX_FORCE=y
# CONFIG_BLK_DEV_SVWKS is not set
# CONFIG_BLK_DEV_SIIMAGE is not set
CONFIG_BLK_DEV_SIS5513=y
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_ARM is not set
CONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_IVB is not set
CONFIG_IDEDMA_AUTO=y
# CONFIG_BLK_DEV_HD is not set

#
# SCSI device support
#
CONFIG_SCSI=m
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=m
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=m
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=m

#
# Some SCSI devices (e.g. CD jukebox) support multiple LUNs
#
# CONFIG_SCSI_MULTI_LUN is not set
CONFIG_SCSI_CONSTANTS=y
# CONFIG_SCSI_LOGGING is not set

#
# SCSI Transport Attributes
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set

#
# SCSI low-level drivers
#
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
# CONFIG_SCSI_AIC7XXX is not set
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
CONFIG_SCSI_SATA=y
# CONFIG_SCSI_SATA_AHCI is not set
# CONFIG_SCSI_SATA_SVW is not set
CONFIG_SCSI_ATA_PIIX=m
# CONFIG_SCSI_SATA_NV is not set
# CONFIG_SCSI_SATA_PROMISE is not set
# CONFIG_SCSI_SATA_QSTOR is not set
# CONFIG_SCSI_SATA_SX4 is not set
# CONFIG_SCSI_SATA_SIL is not set
# CONFIG_SCSI_SATA_SIS is not set
# CONFIG_SCSI_SATA_ULI is not set
# CONFIG_SCSI_SATA_VIA is not set
# CONFIG_SCSI_SATA_VITESSE is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
CONFIG_SCSI_QLA2XXX=m
# CONFIG_SCSI_QLA21XX is not set
# CONFIG_SCSI_QLA22XX is not set
# CONFIG_SCSI_QLA2300 is not set
# CONFIG_SCSI_QLA2322 is not set
# CONFIG_SCSI_QLA6312 is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set

#
# Multi-device support (RAID and LVM)
#
CONFIG_MD=y
CONFIG_BLK_DEV_MD=m
CONFIG_MD_LINEAR=m
# CONFIG_MD_RAID0 is not set
# CONFIG_MD_RAID1 is not set
# CONFIG_MD_RAID10 is not set
# CONFIG_MD_RAID5 is not set
# CONFIG_MD_RAID6 is not set
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
CONFIG_BLK_DEV_DM=m
# CONFIG_DM_CRYPT is not set
CONFIG_DM_SNAPSHOT=m
CONFIG_DM_MIRROR=m
CONFIG_DM_ZERO=m
# CONFIG_DM_MULTIPATH is not set

#
# Fusion MPT device support
#
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#
CONFIG_IEEE1394=m

#
# Subsystem Options
#
# CONFIG_IEEE1394_VERBOSEDEBUG is not set
CONFIG_IEEE1394_OUI_DB=y
CONFIG_IEEE1394_EXTRA_CONFIG_ROMS=y
# CONFIG_IEEE1394_CONFIG_ROM_IP1394 is not set

#
# Device Drivers
#
# CONFIG_IEEE1394_PCILYNX is not set
CONFIG_IEEE1394_OHCI1394=m

#
# Protocol Drivers
#
CONFIG_IEEE1394_VIDEO1394=m
CONFIG_IEEE1394_SBP2=m
CONFIG_IEEE1394_SBP2_PHYS_DMA=y
# CONFIG_IEEE1394_ETH1394 is not set
CONFIG_IEEE1394_DV1394=m
CONFIG_IEEE1394_RAWIO=m
# CONFIG_IEEE1394_CMP is not set

#
# I2O device support
#
# CONFIG_I2O is not set

#
# Networking support
#
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
# CONFIG_IP_MULTICAST is not set
# CONFIG_IP_ADVANCED_ROUTER is not set
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
CONFIG_INET_AH=y
CONFIG_INET_ESP=y
CONFIG_INET_IPCOMP=y
CONFIG_INET_TUNNEL=y
CONFIG_IP_TCPDIAG=y
# CONFIG_IP_TCPDIAG_IPV6 is not set
# CONFIG_IPV6 is not set
# CONFIG_NETFILTER is not set
CONFIG_XFRM=y
# CONFIG_XFRM_USER is not set

#
# SCTP Configuration (EXPERIMENTAL)
#
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set

#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set
# CONFIG_NET_CLS_ROUTE is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_HAMRADIO is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
CONFIG_TUN=m

#
# ARCnet devices
#
# CONFIG_ARCNET is not set

#
# Ethernet (10 or 100Mbit)
#
# CONFIG_NET_ETHERNET is not set

#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
CONFIG_E1000=m
CONFIG_E1000_NAPI=y
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SK98LIN is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set

#
# Ethernet (10000 Mbit)
#
# CONFIG_IXGB is not set
# CONFIG_S2IO is not set

#
# Token Ring devices
#
# CONFIG_TR is not set

#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set

#
# Wan interfaces
#
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_SHAPER is not set
# CONFIG_NETCONSOLE is not set

#
# ISDN subsystem
#
# CONFIG_ISDN is not set

#
# Telephony Support
#
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_TSDEV is not set
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
# CONFIG_INPUT_MISC is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=m
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
# CONFIG_SERIAL_NONSTANDARD is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
# CONFIG_SERIAL_8250_ACPI is not set
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_EXTENDED=y
# CONFIG_SERIAL_8250_MANY_PORTS is not set
CONFIG_SERIAL_8250_SHARE_IRQ=y
CONFIG_SERIAL_8250_DETECT_IRQ=y
CONFIG_SERIAL_8250_MULTIPORT=y
CONFIG_SERIAL_8250_RSA=y

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_LEGACY_PTYS is not set

#
# IPMI
#
CONFIG_IPMI_HANDLER=m
# CONFIG_IPMI_PANIC_EVENT is not set
CONFIG_IPMI_DEVICE_INTERFACE=m
CONFIG_IPMI_SI=m
CONFIG_IPMI_WATCHDOG=m
# CONFIG_IPMI_POWEROFF is not set

#
# Watchdog Cards
#
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
CONFIG_SOFT_WATCHDOG=m
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
# CONFIG_ALIM1535_WDT is not set
# CONFIG_ALIM7101_WDT is not set
# CONFIG_SC520_WDT is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
# CONFIG_WAFER_WDT is not set
CONFIG_I8XX_TCO=m
# CONFIG_SC1200_WDT is not set
# CONFIG_60XX_WDT is not set
# CONFIG_CPU5_WDT is not set
# CONFIG_W83627HF_WDT is not set
# CONFIG_W83877F_WDT is not set
# CONFIG_MACHZ_WDT is not set

#
# PCI-based Watchdog Cards
#
CONFIG_PCIPCWATCHDOG=m
# CONFIG_WDTPCI is not set

#
# USB-based Watchdog Cards
#
# CONFIG_USBPCWATCHDOG is not set
CONFIG_HW_RANDOM=m
CONFIG_NVRAM=m
CONFIG_RTC=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set

#
# Ftape, the floppy tape device driver
#
CONFIG_AGP=y
# CONFIG_AGP_ALI is not set
# CONFIG_AGP_ATI is not set
# CONFIG_AGP_AMD is not set
# CONFIG_AGP_AMD64 is not set
CONFIG_AGP_INTEL=y
# CONFIG_AGP_NVIDIA is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_SWORKS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_AGP_EFFICEON is not set
CONFIG_DRM=y
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
CONFIG_DRM_RADEON=m
# CONFIG_DRM_I810 is not set
# CONFIG_DRM_I830 is not set
# CONFIG_DRM_I915 is not set
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_MWAVE is not set
# CONFIG_RAW_DRIVER is not set
# CONFIG_HPET is not set
CONFIG_HANGCHECK_TIMER=m

#
# TPM devices
#
# CONFIG_TCG_TPM is not set

#
# I2C support
#
CONFIG_I2C=y
# CONFIG_I2C_CHARDEV is not set

#
# I2C Algorithms
#
CONFIG_I2C_ALGOBIT=m
# CONFIG_I2C_ALGOPCF is not set
# CONFIG_I2C_ALGOPCA is not set

#
# I2C Hardware Bus support
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
CONFIG_I2C_I801=m
# CONFIG_I2C_I810 is not set
CONFIG_I2C_PIIX4=m
# CONFIG_I2C_ISA is not set
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_PROSAVAGE is not set
# CONFIG_I2C_SAVAGE4 is not set
# CONFIG_SCx200_ACB is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_STUB is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set
# CONFIG_I2C_VOODOO3 is not set
# CONFIG_I2C_PCA_ISA is not set

#
# Hardware Sensors Chip support
#
# CONFIG_I2C_SENSOR is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_FSCPOS is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83627HF is not set

#
# Other I2C Chip support
#
# CONFIG_SENSORS_DS1337 is not set
# CONFIG_SENSORS_EEPROM is not set
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_RTC8564 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set

#
# Dallas's 1-wire bus
#
# CONFIG_W1 is not set

#
# Misc devices
#
# CONFIG_IBM_ASM is not set

#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set

#
# Digital Video Broadcasting Devices
#
# CONFIG_DVB is not set

#
# Graphics support
#
CONFIG_FB=y
CONFIG_FB_CFB_FILLRECT=y
CONFIG_FB_CFB_COPYAREA=y
CONFIG_FB_CFB_IMAGEBLIT=y
CONFIG_FB_SOFT_CURSOR=y
# CONFIG_FB_MACMODES is not set
CONFIG_FB_MODE_HELPERS=y
# CONFIG_FB_TILEBLITTING is not set
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ASILIANT is not set
# CONFIG_FB_IMSTT is not set
# CONFIG_FB_VGA16 is not set
CONFIG_FB_VESA=y
CONFIG_VIDEO_SELECT=y
# CONFIG_FB_HGA is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_I810 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON_OLD is not set
CONFIG_FB_RADEON=m
CONFIG_FB_RADEON_I2C=y
# CONFIG_FB_RADEON_DEBUG is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_VIRTUAL is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_FRAMEBUFFER_CONSOLE=y
CONFIG_FONTS=y
# CONFIG_FONT_8x8 is not set
CONFIG_FONT_8x16=y
# CONFIG_FONT_6x11 is not set
# CONFIG_FONT_PEARL_8x8 is not set
# CONFIG_FONT_ACORN_8x8 is not set
# CONFIG_FONT_MINI_4x6 is not set
# CONFIG_FONT_SUN8x16 is not set
# CONFIG_FONT_SUN12x22 is not set

#
# Logo configuration
#
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set

#
# Sound
#
CONFIG_SOUND=m

#
# Advanced Linux Sound Architecture
#
CONFIG_SND=m
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_RTCTIMER=m
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set

#
# Generic devices
#
CONFIG_SND_MPU401_UART=m
CONFIG_SND_DUMMY=m
CONFIG_SND_VIRMIDI=m
CONFIG_SND_MTPAV=m
# CONFIG_SND_SERIAL_U16550 is not set
CONFIG_SND_MPU401=m

#
# PCI devices
#
CONFIG_SND_AC97_CODEC=m
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_YMFPCI is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_FM801 is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
CONFIG_SND_INTEL8X0=m
CONFIG_SND_INTEL8X0M=m
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_HDA_INTEL is not set

#
# USB devices
#
# CONFIG_SND_USB_AUDIO is not set
# CONFIG_SND_USB_USX2Y is not set

#
# Open Sound System
#
# CONFIG_SOUND_PRIME is not set

#
# USB support
#
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set

#
# USB Host Controller Drivers
#
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_SPLIT_ISO=y
CONFIG_USB_EHCI_ROOT_HUB_TT=y
# CONFIG_USB_OHCI_HCD is not set
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set

#
# USB Device Class drivers
#
# CONFIG_USB_AUDIO is not set
# CONFIG_USB_BLUETOOTH_TTY is not set
# CONFIG_USB_MIDI is not set
# CONFIG_USB_ACM is not set
CONFIG_USB_PRINTER=m

#
# NOTE: USB_STORAGE enables SCSI, and 'SCSI disk support' may also be needed; see USB_STORAGE Help for more information
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set

#
# USB Input Devices
#
CONFIG_USB_HID=y
CONFIG_USB_HIDINPUT=y
# CONFIG_HID_FF is not set
# CONFIG_USB_HIDDEV is not set
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_MTOUCH is not set
# CONFIG_USB_EGALAX is not set
# CONFIG_USB_XPAD is not set
# CONFIG_USB_ATI_REMOTE is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB Multimedia devices
#
# CONFIG_USB_DABUSB is not set

#
# Video4Linux support is needed for USB Multimedia device support
#

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
CONFIG_USB_MON=y

#
# USB port drivers
#

#
# USB Serial Converter support
#
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_PHIDGETKIT is not set
# CONFIG_USB_PHIDGETSERVO is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_TEST is not set

#
# USB ATM/DSL drivers
#

#
# USB Gadget Support
#
# CONFIG_USB_GADGET is not set

#
# MMC/SD Card support
#
# CONFIG_MMC is not set

#
# InfiniBand support
#
# CONFIG_INFINIBAND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
CONFIG_EXT3_FS=m
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_JBD=m
# CONFIG_JBD_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y

#
# XFS support
#
CONFIG_XFS_FS=y
CONFIG_XFS_EXPORT=y
# CONFIG_XFS_RT is not set
# CONFIG_XFS_QUOTA is not set
# CONFIG_XFS_SECURITY is not set
# CONFIG_XFS_POSIX_ACL is not set
# CONFIG_MINIX_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_QUOTA is not set
CONFIG_DNOTIFY=y
# CONFIG_AUTOFS_FS is not set
CONFIG_AUTOFS4_FS=m

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_ZISOFS_FS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
# CONFIG_DEVFS_FS is not set
CONFIG_DEVPTS_FS_XATTR=y
CONFIG_DEVPTS_FS_SECURITY=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_XATTR is not set
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_RAMFS=y

#
# Miscellaneous filesystems
#
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set

#
# Network File Systems
#
CONFIG_NFS_FS=m
CONFIG_NFS_V3=y
# CONFIG_NFS_V4 is not set
# CONFIG_NFS_DIRECTIO is not set
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V4 is not set
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=m
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=y
CONFIG_SUNRPC=m
# CONFIG_RPCSEC_GSS_KRB5 is not set
# CONFIG_RPCSEC_GSS_SPKM3 is not set
# CONFIG_SMB_FS is not set
# CONFIG_CIFS is not set
# CONFIG_NCP_FS is not set
# CONFIG_CODA_FS is not set
# CONFIG_AFS_FS is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_EFI_PARTITION is not set

#
# Native Language Support
#
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=m
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ASCII is not set
# CONFIG_NLS_ISO8859_1 is not set
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=m

#
# Profiling support
#
CONFIG_PROFILING=y
CONFIG_OPROFILE=m

#
# Kernel hacking
#
CONFIG_PRINTK_TIME=y
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_LOG_BUF_SHIFT=17
# CONFIG_SCHEDSTATS is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_SPINLOCK is not set
CONFIG_DEBUG_SPINLOCK_SLEEP=y
# CONFIG_DEBUG_KOBJECT is not set
# CONFIG_DEBUG_HIGHMEM is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_FS is not set
# CONFIG_FRAME_POINTER is not set
CONFIG_EARLY_PRINTK=y
CONFIG_DEBUG_STACKOVERFLOW=y
# CONFIG_KPROBES is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PAGEALLOC is not set
# CONFIG_4KSTACKS is not set
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set

#
# Cryptographic options
#
CONFIG_CRYPTO=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_NULL=m
CONFIG_CRYPTO_MD4=m
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_SHA1=y
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_SHA512=m
# CONFIG_CRYPTO_WP512 is not set
# CONFIG_CRYPTO_TGR192 is not set
CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_BLOWFISH=m
CONFIG_CRYPTO_TWOFISH=m
CONFIG_CRYPTO_SERPENT=m
# CONFIG_CRYPTO_AES_586 is not set
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
# CONFIG_CRYPTO_TEA is not set
CONFIG_CRYPTO_ARC4=m
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_ANUBIS is not set
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_CRC32C=m
# CONFIG_CRYPTO_TEST is not set

#
# Hardware crypto devices
#
# CONFIG_CRYPTO_DEV_PADLOCK is not set

#
# Library routines
#
# CONFIG_CRC_CCITT is not set
CONFIG_CRC32=y
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_X86_TRAMPOLINE=y
CONFIG_PC=y

Re: Race condition in module load causing undefined symbols

[Andrew Morton]

Stephen Lord <lord@xfs.org> wrote:
>
> I am having troubles getting any recent kernel to boot successfully
>  on one of my machines, a generic 2.6GHz P4 box with HT enabled
>  running an updated Fedora Core 3 distro. This is present in
>  2.6.12-rc6. It does not manifest itself with the Fedora Core
>  kernels which have identical initrd contents as far as the
>  init script and the set of modules included goes.
> 
>  The problem manifests itself as various undefined symbols from
>  module loads.

Peculiar.  Module loading is all synchronous, isn't it?

> ...
>  The failures are different on different boots, sometimes the ata_piix
>  module cannot find symbols from libata, sometimes ext3 cannot find jbd
>  symbols, sometimes dm modules cannot find things from dm-mod, usually
>  it is a combination of these. End result is a panic when it cannot
>  find the root device.
> 
>   From the behavior, it appears that a module load is returning
>  control to user space before the previous one has got its symbols
>  loaded.

I wonder if rather than the intermittency being time-based, it is
load-address-based?  For example, suppose there's a bug in the symbol
lookup code?

Have you tried using a different gcc version?

Re: Race condition in module load causing undefined symbols

[Steve Lord]

Andrew Morton wrote:
> Stephen Lord <lord@xfs.org> wrote:
> 
>>I am having troubles getting any recent kernel to boot successfully
>> on one of my machines, a generic 2.6GHz P4 box with HT enabled
>> running an updated Fedora Core 3 distro. This is present in
>> 2.6.12-rc6. It does not manifest itself with the Fedora Core
>> kernels which have identical initrd contents as far as the
>> init script and the set of modules included goes.
>>
>> The problem manifests itself as various undefined symbols from
>> module loads.
> 
> 
> Peculiar.  Module loading is all synchronous, isn't it?


Hmm, now that I found the code, yes it is. insmod itself appears
to do no fancy foot work either.

> 
> 
>>...
>> The failures are different on different boots, sometimes the ata_piix
>> module cannot find symbols from libata, sometimes ext3 cannot find jbd
>> symbols, sometimes dm modules cannot find things from dm-mod, usually
>> it is a combination of these. End result is a panic when it cannot
>> find the root device.
>>
>>  From the behavior, it appears that a module load is returning
>> control to user space before the previous one has got its symbols
>> loaded.
> 
> 
> I wonder if rather than the intermittency being time-based, it is
> load-address-based?  For example, suppose there's a bug in the symbol
> lookup code?
> 
> Have you tried using a different gcc version?
> 

Don't have one handy at the moment, I am away from the machine right
now as well. I have been updating the machine using redhat's update
tools, so the compiler should be the same one I have here:

gcc (GCC) 3.4.3 20050227 (Red Hat 3.4.3-22.fc3)

That should also be a fairly common compiler variant.

I presume this is what redhat does their kernel builds with, so that
should be the same too. Shouldn't the memory map be pretty much
identical on each boot? Things are pretty deterministic at this
stage in the process, and the symbol match failures are not always
the same.

If this was a memory problem it seems like I would see more random
oopses than this. I added more memory to the machine a month or so
back, and had to detune the bios settings a little to make it stable.
It would be odd that a 2.6.11 kernel was rock solid and a 2.6.12-rc6
falls over so quickly if that was the case.

I can play with the init script some and maybe dump out the symbol
table after an insmod.

Steve

Re: Race condition in module load causing undefined symbols

[Stephen Lord]

Andrew Morton wrote:
> Stephen Lord <lord@xfs.org> wrote:
> 
>>I am having troubles getting any recent kernel to boot successfully
>> on one of my machines, a generic 2.6GHz P4 box with HT enabled
>> running an updated Fedora Core 3 distro. This is present in
>> 2.6.12-rc6. It does not manifest itself with the Fedora Core
>> kernels which have identical initrd contents as far as the
>> init script and the set of modules included goes.
>>
>> The problem manifests itself as various undefined symbols from
>> module loads.
> 
> 
> Peculiar.  Module loading is all synchronous, isn't it?
> 

Well, things are getting more bizarre, adding sleeps between
module loads cures the problem with missing symbols. I then
run into a problem with device mapper/lvm which seems to be
having problems setting up devices. In this section of
the init script:

umount /sys
echo Mounting root filesystem
mount -o defaults --ro -t ext3 /dev/root /sysroot
mount -t tmpfs --bind /dev /sysroot/dev
echo Switching to new root
switchroot /sysroot
umount /initrd/dev

The correct number of volumes are found, but adding a showlabels
command to the init script fails to display them, it spits out
errors about readdir failures in /dev/Volume00

The umount of /sys fails, the root mount fails and obviously, the
switchroot then fails.

I tried using the same config options as the redhat supplied
kernel without any success, this still has module symbol
problems.

I am baffled, but it looks like it is not a symbol table problem.

Steve

Re: Race condition in module load causing undefined symbols

[Pozsár Balázs]

On Fri, Jun 10, 2005 at 11:25:15AM -0700, Andrew Morton wrote:
> I wonder if rather than the intermittency being time-based, it is
> load-address-based?  For example, suppose there's a bug in the symbol
> lookup code?

Just a data point: I met the same problem with 2.6.12-rc5, using
gcc 3.3.4.
I think it's time-based issue, because I was playing around with the 
initscripts, and the bug shows up when there are lots of modprobes in a 
short time.



-- 
pozsy

Re: Race condition in module load causing undefined symbols

[Steve Lord]

Pozsár Balázs wrote:
> On Fri, Jun 10, 2005 at 11:25:15AM -0700, Andrew Morton wrote:
> 
>>I wonder if rather than the intermittency being time-based, it is
>>load-address-based?  For example, suppose there's a bug in the symbol
>>lookup code?
> 
> 
> Just a data point: I met the same problem with 2.6.12-rc5, using
> gcc 3.3.4.
> I think it's time-based issue, because I was playing around with the 
> initscripts, and the bug shows up when there are lots of modprobes in a 
> short time.
> 
>

I think this is not actually module loading itself, but a problem
between the fork/exec/wait code in nash and the kernel. The
commands which have problems are the ones which are not built
into nash. So this looks more like a problem with wait. This
would explain sleep fixing it and the fact that I have device
issues after module load.

Steve

Re: Race condition in module load causing undefined symbols

[Pozsár Balázs]

On Sat, Jun 11, 2005 at 08:23:20AM -0500, Steve Lord wrote:
> I think this is not actually module loading itself, but a problem
> between the fork/exec/wait code in nash and the kernel.

I do not use nash, only bash, so this is not a nash-specific issue.


-- 
pozsy

Re: Race condition in module load causing undefined symbols

[Stephen Lord]

Pozsár Balázs wrote:
> On Sat, Jun 11, 2005 at 08:23:20AM -0500, Steve Lord wrote:
> 
>>I think this is not actually module loading itself, but a problem
>>between the fork/exec/wait code in nash and the kernel.
> 
> 
> I do not use nash, only bash, so this is not a nash-specific issue.
> 
> 

I disabled hyperthreading and things started working, so are there any
HT related scheduling bugs right now?

Steve

Here is my cpu information:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Intel(R) Pentium(R) 4 CPU 2.60GHz
stepping        : 9
cpu MHz         : 2606.469
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge 
mca cmov pat pse36 clflush dts acpi mmx f

And here is my system successfully coming up with 2.6.12-rc6 with HT
disabled in the bios:

Linux version 2.6.12-rc6 (slord@fubar.frontiernet.net) (gcc version 
3.4.3 200502 27 (Red Hat 3.4.3-22.fc3)) #3 SMP Fri Jun 10 22:14:34 CDT 2005
BIOS-provided physical RAM map:
  BIOS-e820: 0000000000000000 - 000000000009f800 (usable)
  BIOS-e820: 000000000009f800 - 00000000000a0000 (reserved)
  BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
  BIOS-e820: 0000000000100000 - 000000007fff0000 (usable)
  BIOS-e820: 000000007fff0000 - 000000007fff3000 (ACPI NVS)
  BIOS-e820: 000000007fff3000 - 0000000080000000 (ACPI data)
  BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
1151MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000f5d80
On node 0 totalpages: 524272
   DMA zone: 4096 pages, LIFO batch:1
   Normal zone: 225280 pages, LIFO batch:31
   HighMem zone: 294896 pages, LIFO batch:31
DMI 2.2 present.
Using APIC driver default
ACPI: RSDP (v000 IntelR                                ) @ 0x000f7810
ACPI: RSDT (v001 IntelR AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x7fff3000
ACPI: FADT (v001 IntelR AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x7fff3040
ACPI: MADT (v001 IntelR AWRDACPI 0x42302e31 AWRD 0x00000000) @ 0x7fff7a40
ACPI: DSDT (v001 INTELR AWRDACPI 0x00001000 MSFT 0x0100000e) @ 0x00000000
ACPI: PM-Timer IO Port: 0x408
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 15:2 APIC version 20
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] disabled)
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
ACPI: IRQ0 used by override.
ACPI: IRQ2 used by override.
ACPI: IRQ9 used by override.
Enabling APIC mode:  Flat.  Using 1 I/O APICs
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 80000000 (gap: 80000000:7ec00000)
Built 1 zonelists
Kernel command line: ro root=/dev/Volume00/Vol_Root vga=9
mapped APIC to ffffd000 (fee00000)
mapped IOAPIC to ffffc000 (fec00000)
Initializing CPU#0
CPU 0 irqstacks, hard=c0410000 soft=c03f0000
PID hash table entries: 4096 (order: 12, 65536 bytes)
Detected 2606.469 MHz processor.
Using pmtmr for high-res timesource
Console: colour VGA+ 132x44
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 2072744k/2097088k available (1985k kernel code, 23152k reserved, 
769k da ta, 228k init, 1179584k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay loop... 5160.96 BogoMIPS (lpj=2580480)
Security Framework v1.0.0 initialized
SELinux:  Initializing.
SELinux:  Starting in permissive mode
selinux_register_security:  Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 512
CPU: After generic identify, caps: bfebfbff 00000000 00000000 00000000 
00004400 00000000 00000000
CPU: After vendor identify, caps: bfebfbff 00000000 00000000 00000000 
00004400 0 0000000 00000000
CPU: Trace cache: 12K uops, L1 D cache: 8K
CPU: L2 cache: 512K
CPU: Hyper-Threading is disabled
CPU: After all inits, caps: bfebfbff 00000000 00000000 00000080 00004400 
0000000 0 00000000
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU0: Intel P4/Xeon Extended MCE MSRs (12) available
CPU0: Thermal monitoring enabled
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
CPU0: Intel(R) Pentium(R) 4 CPU 2.60GHz stepping 09
Total of 1 processors activated (5160.96 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 pin1=2 pin2=-1
Brought up 1 CPUs
CPU0 attaching sched-domain:
  domain 0: span 00000001
   groups: 00000001
   domain 1: span 00000001
    groups: 00000001
checking if image is initramfs... it is
Freeing initrd memory: 1214k freed
NET: Registered protocol family 16
PCI: PCI BIOS revision 2.10 entry at 0xfba20, last bus=3
PCI: Using configuration type 1
mtrr: v2.0 (20020519)
ACPI: Subsystem revision 20050309
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI: Probing PCI hardware (bus 00)
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.2
Boot video device is 0000:01:00.0
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.HUB0._PRT]
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 7 9 *10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 7 9 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 7 9 10 *11 12 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 7 *9 10 11 12 14 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 7 9 10 11 12 14 15) *0, 
disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 7 9 10 11 12 14 15) *0, 
disabled.
ACPI: PCI Interrupt Link [LNK0] (IRQs 3 4 5 7 9 10 11 12 14 15) *0, 
disabled.
ACPI: PCI Interrupt Link [LNK1] (IRQs *3 4 5 7 9 10 11 12 14 15)
ACPI: Power Resource [PFAN] (off)
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI init
pnp: PnP ACPI: found 13 devices
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a 
report
pnp: 00:0a: ioport range 0x400-0x4bf could not be reserved
apm: BIOS not found.
highmem bounce pool size: 64 pages
Total HugeTLB memory allocated, 0
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
SELinux:  Registering netfilter hooks
Initializing Cryptographic API
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI: Fan [FAN] (off)
ACPI: Thermal Zone [THRM] (45 C)
isapnp: Scanning for PnP cards...
isapnp: No Plug & Play device found
Real Time Clock Driver v1.12
Linux agpgart interface v0.101 (c) Dave Jones
agpgart: Detected an Intel i875 Chipset.
agpgart: AGP aperture is 128M @ 0xf0000000
PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 76 ports, IRQ sharing enabled
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Probing IDE interface ide0...
hda: PLEXTOR DVDR PX-716A, ATAPI CD/DVD-ROM drive
ide1: I/O resource 0x170-0x177 not free.
ide1: ports already in use, skipping probe
Probing IDE interface ide2...
Probing IDE interface ide3...
Probing IDE interface ide4...
Probing IDE interface ide5...
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: ATAPI 40X DVD-ROM DVD-R CD-R/RW drive, 8192kB Cache
Uniform CD-ROM driver Revision: 3.20
ide-floppy driver 0.99.newide
usbmon: debugs is not available
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.01:USB HID core driver
mice: PS/2 mouse device common for all mice
md: md driver 0.90.1 MAX_MD_DEVS=256, MD_SB_DISKS=27
NET: Registered protocol family 2
IP: routing cache hash table of 8192 buckets, 128Kbytes
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 7, 786432 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
ACPI wakeup devices:
PCI0 CSAD HUB0 UAR1 USB0 USB1 USB2 USB3 USBE MODM
ACPI: (supports S0 S1 S4 S5)
Freeing unused kernel memory: 228k freed
input: AT Translated Set 2 keyboard on isa0060/serio0
SCSI subsystem initialized
libata version 1.11 loaded.
ata_piix version 1.03
ata_piix: combined mode detected
ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 18 (level, low) -> IRQ 169
ata: 0x1f0 IDE port busy
PCI: Setting latency timer of device 0000:00:1f.2 to 64
ata1: SATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0xF008 irq 15
ata1: dev 0 cfg 49:2f00 82:74eb 83:7fea 84:4023 85:74e9 86:3c02 87:4023 
88:203f
ata1: dev 0 ATA, max UDMA/100, 241254720 sectors: lba48
ata1: dev 1 cfg 49:2f00 82:74eb 83:7fea 84:4023 85:74e9 86:3c02 87:4023 
88:203f
ata1: dev 1 ATA, max UDMA/100, 160836480 sectors: lba48
ata1: dev 0 configured for UDMA/100
ata1: dev 1 configured for UDMA/100
scsi0 : ata_piix
   Vendor: ATA       Model: HDS722512VLSA80   Rev: V33O
   Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 241254720 512-byte hdwr sectors (123522 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 241254720 512-byte hdwr sectors (123522 MB)
SCSI device sda: drive cache: write back
  sda: sda1 sda2 sda3 sda4 < sda5 >
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
   Vendor: ATA       Model: HDS722580VLSA80   Rev: V32O
   Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sdb: 160836480 512-byte hdwr sectors (82348 MB)
SCSI device sdb: drive cache: write back
SCSI device sdb: 160836480 512-byte hdwr sectors (82348 MB)
SCSI device sdb: drive cache: write back
  sdb: sdb1
Attached scsi disk sdb at scsi0, channel 0, id 1, lun 0
ieee1394: Initialized config rom entry `ip1394'
ohci1394: $Rev: 1250 $ Ben Collins <bcollins@debian.org>
ACPI: PCI Interrupt 0000:03:02.0[A] -> GSI 18 (level, low) -> IRQ 169
ohci1394: fw-host0: OHCI-1394 1.1 (PCI): IRQ=[169] 
MMIO=[fa104000-fa1047ff]  Ma x Packet=[2048]
sbp2: $Rev: 1219 $ Ben Collins <bcollins@debian.org>
device-mapper: 4.4.0-ioctl (2005-01-12) initialised: dm-devel@redhat.com
input: ImExPS/2 Generic Explorer Mouse on isa0060/serio1
cdrom: open failed.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
SELinux:  Disabled at runtime.
SELinux:  Unregistering netfilter hooks
ieee1394: Node added: ID:BUS[0-00:1023]  GUID[0010b92000afe64b]
ieee1394: Host added: ID:BUS[0-01:1023]  GUID[00508d0000f4320b]
scsi1 : SCSI emulation for IEEE-1394 SBP-2 Devices
ieee1394: sbp2: Logged into SBP-2 device
ieee1394: Node 0-00:1023: Max speed [S400] - Max payload [2048]
   Vendor: Maxtor    Model: OneTouch          Rev: 0200
   Type:   Direct-Access                      ANSI SCSI revision: 06
SCSI device sdc: 320171008 512-byte hdwr sectors (163928 MB)
sdc: cache data unavailable
sdc: assuming drive cache: write through
SCSI device sdc: 320171008 512-byte hdwr sectors (163928 MB)
sdc: cache data unavailable
sdc: assuming drive cache: write through
  sdc: sdc1
Attached scsi disk sdc at scsi1, channel 0, id 1, lun 0
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
Intel(R) PRO/1000 Network Driver - version 6.0.54-k2-NAPI
Copyright (c) 1999-2004 Intel Corporation.
ACPI: PCI Interrupt 0000:02:01.0[A] -> GSI 18 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:02:01.0 to 64
e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
ACPI: PCI Interrupt 0000:00:1f.5[B] -> GSI 17 (level, low) -> IRQ 177
PCI: Setting latency timer of device 0000:00:1f.5 to 64
intel8x0_measure_ac97_clock: measured 49800 usecs
intel8x0: clocking to 48000
ACPI: PCI Interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 185
PCI: Setting latency timer of device 0000:00:1d.7 to 64
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 1
ehci_hcd 0000:00:1d.7: irq 185, io mem 0xfa200000
PCI: cache line size of 128 is not supported by device 0000:00:1d.7
ehci_hcd 0000:00:1d.7: USB 2.0 initialized, EHCI 1.00, driver 10 Dec 2004
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 8 ports detected
USB Universal Host Controller Interface driver v2.2
ACPI: PCI Interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 193
PCI: Setting latency timer of device 0000:00:1d.0 to 64
uhci_hcd 0000:00:1d.0: UHCI Host Controller
usb 1-2: new high speed USB device using ehci_hcd and address 2
hub 1-2:1.0: USB hub found
hub 1-2:1.0: 4 ports detected
uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 2
uhci_hcd 0000:00:1d.0: irq 193, io base 0x0000b000
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 201
PCI: Setting latency timer of device 0000:00:1d.1 to 64
uhci_hcd 0000:00:1d.1: UHCI Host Controller
usb 1-2.4: new high speed USB device using ehci_hcd and address 3
uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 3
uhci_hcd 0000:00:1d.1: irq 201, io base 0x0000b400
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 169
PCI: Setting latency timer of device 0000:00:1d.2 to 64
uhci_hcd 0000:00:1d.2: UHCI Host Controller
uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.2: irq 169, io base 0x0000b800
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
ACPI: PCI Interrupt 0000:00:1d.3[A] -> GSI 16 (level, low) -> IRQ 193
PCI: Setting latency timer of device 0000:00:1d.3 to 64
uhci_hcd 0000:00:1d.3: UHCI Host Controller
uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 5
uhci_hcd 0000:00:1d.3: irq 193, io base 0x0000bc00
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 2 ports detected
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
Initializing USB Mass Storage driver...
scsi2 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 3
usb-storage: waiting for device to settle before scanning
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
ACPI: Power Button (FF) [PWRF]
ibm_acpi: ec object not found
EXT3 FS on dm-0, internal journal
cdrom: open failed.
kjournald starting.  Commit interval 5 seconds
EXT3 FS on sda2, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
kjournald starting.  Commit interval 5 seconds

Re: Race condition in module load causing undefined symbols

[Andrew Morton]

Stephen Lord <lord@xfs.org> wrote:
>
> Pozsár Balázs wrote:
>  > On Sat, Jun 11, 2005 at 08:23:20AM -0500, Steve Lord wrote:
>  > 
>  >>I think this is not actually module loading itself, but a problem
>  >>between the fork/exec/wait code in nash and the kernel.
>  > 
>  > 
>  > I do not use nash, only bash, so this is not a nash-specific issue.
>  > 
>  > 
> 
>  I disabled hyperthreading and things started working, so are there any
>  HT related scheduling bugs right now?

There haven't been any scheduler changes for some time.  There have been a
few low-level SMT changes I think.

Are you able to identify which kernel version broke it?

Re: Race condition in module load causing undefined symbols

[Pozsár Balázs]

On Sat, Jun 11, 2005 at 12:00:40PM -0700, Andrew Morton wrote:
> >  I disabled hyperthreading and things started working, so are there any
> >  HT related scheduling bugs right now?
> 
> There haven't been any scheduler changes for some time.  There have been a
> few low-level SMT changes I think.
> 
> Are you able to identify which kernel version broke it?

I do not have HT or SMP, though the kernel is smp. 2.6.9 works for me.
That's all I can tell now.


-- 
pozsy

Re: Race condition in module load causing undefined symbols

[Steve Lord]

Well, the bizarre part is that I think this has been around for a while, but 
it
does not exhibit itself in redhat kernels which postdate the earliest
recolloction of me seeing it. 2.6.11-rc1 is the earliest I remember,
but I am not religious about updating the kernel on this box so
my samples are spotty.

The difference between the two may be that I recompile for a P4
while redhat uses a lowest common denominator cpu type.

If I get a chance this weekend I will try some other kernels and
report back. Maybe just start out by dumbing down my cpu
type.

Steve

> Stephen Lord <lord@xfs.org> wrote:
>>
>>  I disabled hyperthreading and things started working, so are there any
>>  HT related scheduling bugs right now?
>
> There haven't been any scheduler changes for some time.  There have been a
> few low-level SMT changes I think.
>
> Are you able to identify which kernel version broke it?
>
> 

Re: Race condition in module load causing undefined symbols

[Pozsár Balázs]

On Sat, Jun 11, 2005 at 03:09:08PM -0500, Steve Lord wrote:
> Well, the bizarre part is that I think this has been around for a while, 
> but it
> does not exhibit itself in redhat kernels which postdate the earliest
> recolloction of me seeing it. 2.6.11-rc1 is the earliest I remember,
> but I am not religious about updating the kernel on this box so
> my samples are spotty.
> 
> The difference between the two may be that I recompile for a P4
> while redhat uses a lowest common denominator cpu type.
> 
> If I get a chance this weekend I will try some other kernels and
> report back. Maybe just start out by dumbing down my cpu
> type.

I always used CONFIG_M586=y.

Did you turn preempt on? I did with the 2.6.12-rc, but not with the 
(working) 2.6.9, so this migth be the difference.
Sorry, I do not have currently for testing.


-- 
pozsy

Re: Race condition in module load causing undefined symbols

[Rusty Russell]

On Fri, 2005-06-10 at 09:03 -0500, Stephen Lord wrote:
> Hi,
> 
> I am having troubles getting any recent kernel to boot successfully
> on one of my machines, a generic 2.6GHz P4 box with HT enabled
> running an updated Fedora Core 3 distro. This is present in
> 2.6.12-rc6. It does not manifest itself with the Fedora Core
> kernels which have identical initrd contents as far as the
> init script and the set of modules included goes.
> 
> The problem manifests itself as various undefined symbols from
> module loads.  Here is the relevant section from the init script

Module loading is synchronous.  All I can think of is that a module is
pulling in another module which requires it asynchronously (you need to
do this because your own module symbols are not available until *after*
init succeeds), or a hotplug interaction (hotplug is async, too).

Rusty.
-- 
A bad analogy is like a leaky screwdriver -- Richard Braakman

Re: Race condition in module load causing undefined symbols

[Steve Lord]

Andrew Morton wrote:
> Stephen Lord <lord@xfs.org> wrote:
> 
>>Pozsár Balázs wrote:
>> > On Sat, Jun 11, 2005 at 08:23:20AM -0500, Steve Lord wrote:
>> > 
>> >>I think this is not actually module loading itself, but a problem
>> >>between the fork/exec/wait code in nash and the kernel.
>> > 
>> > 
>> > I do not use nash, only bash, so this is not a nash-specific issue.
>> > 
>> > 
>>
>> I disabled hyperthreading and things started working, so are there any
>> HT related scheduling bugs right now?
> 
> 
> There haven't been any scheduler changes for some time.  There have been a
> few low-level SMT changes I think.
> 
> Are you able to identify which kernel version broke it?
> 

Still have not narrowed this down too far, disabling SMT made no
difference, disabling SMP did, which I was expecting.

Steve

Re: Race condition in module load causing undefined symbols

[K.R. Foley]

Steve Lord wrote:
> Andrew Morton wrote:
> 
>> Stephen Lord <lord@xfs.org> wrote:
>>
>>> Pozsár Balázs wrote:
>>> > On Sat, Jun 11, 2005 at 08:23:20AM -0500, Steve Lord wrote:
>>> > >>I think this is not actually module loading itself, but a problem
>>> >>between the fork/exec/wait code in nash and the kernel.
>>> > > > I do not use nash, only bash, so this is not a nash-specific 
>>> issue.
>>> > >
>>> I disabled hyperthreading and things started working, so are there any
>>> HT related scheduling bugs right now?
>>
>>
>>
>> There haven't been any scheduler changes for some time.  There have 
>> been a
>> few low-level SMT changes I think.
>>
>> Are you able to identify which kernel version broke it?
>>
> 
> Still have not narrowed this down too far, disabling SMT made no
> difference, disabling SMP did, which I was expecting.
> 
> Steve
> 

I initially saw this with 2.6.12-rc1 and every version up through rc3. I
haven't tried with later versions. :-/ I initially reported here:
http://marc.theaimsgroup.com/?l=linux-kernel&m=111235814529008&w=2

The way that I got around it was to compile in my aic7xxx driver instead
of making it a module. I have also recently received an email from
someone saying that disabling module unloading would also solve it. That
very well may be true since I did run into another booting problem
(2.6.12-rc5) that disabling module unloading fixed :-/ I haven't had a
chance to go back and check this out though.

So to summarize: I have a dual 933 with aic7xxx compiled in to get
passed the problem described above. I have a dual 2.6 w/HT that I have
disabled module unloading to get passed another boot condition.


-- 
    kr

Re: Race condition in module load causing undefined symbols

[K.R. Foley]

Steve Lord wrote:
> Andrew Morton wrote:
> 
>> Stephen Lord <lord@xfs.org> wrote:
>>
>>> Pozsár Balázs wrote:
>>> > On Sat, Jun 11, 2005 at 08:23:20AM -0500, Steve Lord wrote:
>>> > >>I think this is not actually module loading itself, but a problem
>>> >>between the fork/exec/wait code in nash and the kernel.
>>> > > > I do not use nash, only bash, so this is not a nash-specific 
>>> issue.
>>> > >
>>> I disabled hyperthreading and things started working, so are there any
>>> HT related scheduling bugs right now?
>>
>>
>>
>> There haven't been any scheduler changes for some time.  There have 
>> been a
>> few low-level SMT changes I think.
>>
>> Are you able to identify which kernel version broke it?
>>
> 
> Still have not narrowed this down too far, disabling SMT made no
> difference, disabling SMP did, which I was expecting.
> 
> Steve
> 

I initially saw this with 2.6.12-rc1 and every version up through rc3. I
haven't tried with later versions. :-/ I initially reported here:
http://marc.theaimsgroup.com/?l=linux-kernel&m=111235814529008&w=2

The way that I got around it was to compile in my aic7xxx driver instead
of making it a module. I have also recently received an email from
someone saying that disabling module unloading would also solve it. That
very well may be true since I did run into another booting problem
(2.6.12-rc5) that disabling module unloading fixed :-/ I haven't had a
chance to go back and check this out though.

So to summarize: I have a dual 933 with aic7xxx compiled in to get
passed the problem described above. I have a dual 2.6 w/HT that I have
disabled module unloading to get passed another boot condition.


-- 
    kr

Re: Race condition in module load causing undefined symbols

[Steve Lord]

K.R. Foley wrote:
> Steve Lord wrote:
> 
>> Andrew Morton wrote:
>>
>>> Stephen Lord <lord@xfs.org> wrote:
>>>
>>>> Pozsár Balázs wrote:
>>>> > On Sat, Jun 11, 2005 at 08:23:20AM -0500, Steve Lord wrote:
>>>> > >>I think this is not actually module loading itself, but a problem
>>>> >>between the fork/exec/wait code in nash and the kernel.
>>>> > > > I do not use nash, only bash, so this is not a nash-specific 
>>>> issue.
>>>> > >
>>>> I disabled hyperthreading and things started working, so are there any
>>>> HT related scheduling bugs right now?
>>>
>>>
>>>
>>>
>>> There haven't been any scheduler changes for some time.  There have 
>>> been a
>>> few low-level SMT changes I think.
>>>
>>> Are you able to identify which kernel version broke it?
>>>
>>
>> Still have not narrowed this down too far, disabling SMT made no
>> difference, disabling SMP did, which I was expecting.
>>
>> Steve
>>
> 
> I initially saw this with 2.6.12-rc1 and every version up through rc3. I
> haven't tried with later versions. :-/ I initially reported here:
> http://marc.theaimsgroup.com/?l=linux-kernel&m=111235814529008&w=2
> 
> The way that I got around it was to compile in my aic7xxx driver instead
> of making it a module. I have also recently received an email from
> someone saying that disabling module unloading would also solve it. That
> very well may be true since I did run into another booting problem
> (2.6.12-rc5) that disabling module unloading fixed :-/ I haven't had a
> chance to go back and check this out though.
> 
> So to summarize: I have a dual 933 with aic7xxx compiled in to get
> passed the problem described above. I have a dual 2.6 w/HT that I have
> disabled module unloading to get passed another boot condition.
> 
> 

I found another system which exhibits the problem, a dual Xeon
with HT support.

Here is one of the cpus from /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 1
model name      : Intel(R) Xeon(TM) CPU 1.40GHz
stepping        : 1
cpu MHz         : 1393.851
cache size      : 256 KB
physical id     : 0
siblings        : 2
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips        : 2752.51

I discovered that if I disable P4 support on this host and run with
P3 Xeon support instead, things start working. The host type in the
boot up is identified as a P4/Xeon:

Jun 14 11:25:19 k4 kernel: Booting processor 2/2 eip 3000
Jun 14 11:25:19 k4 kernel: CPU 2 irqstacks, hard=c03e7000 soft=c03df000
Jun 14 11:25:19 k4 kernel: Initializing CPU#2
Jun 14 11:25:19 k4 kernel: CPU: Trace cache: 12K uops, L1 D cache: 8K
Jun 14 11:25:19 k4 kernel: CPU: L2 cache: 256K
Jun 14 11:25:19 k4 kernel: CPU: L3 cache: 512K
Jun 14 11:25:19 k4 kernel: CPU: Physical Processor ID: 1
Jun 14 11:25:19 k4 kernel: Intel machine check architecture supported.
Jun 14 11:25:19 k4 kernel: Intel machine check reporting enabled on CPU#2.
Jun 14 11:25:19 k4 kernel: CPU2: Intel P4/Xeon Extended MCE MSRs (12) available
Jun 14 11:25:19 k4 kernel: CPU2: Intel(R) Xeon(TM) CPU 1.40GHz stepping 01

So is this some P4 specific optimization which is not working as
intended?

Steve

Re: Race condition in module load causing undefined symbols

[Andi Kleen]

Steve Lord <lord@xfs.org> writes:
>
> So is this some P4 specific optimization which is not working as
> intended?

The only pentium specific optimizations that are enabled by MPENTIUM4
is to tell the compiler to compile for pentium4 and a few settings
in arch/i386/Kconfig.

You could enable/disable these individually and see if you can track
it down with a binary search.

Most of this stuff should be fairly harmless though and be only
microoptimizations; I cannot see how they should cause user visible
races.

-Andi

Re: Race condition in module load causing undefined symbols

[K.R. Foley]

Steve Lord wrote:
<snip>
> So is this some P4 specific optimization which is not working as
> intended?
> 
> Steve
> 
> 

I'd say not since the first system I saw this on was a dual PIII Xeon. 
While I am not 100% sure that the problems are related, the problem that 
I saw on my 2.6 system also went away when I disabled hyper-threading in 
the bios. It really just seems to me like it is some hard-to-trigger race.

-- 
    kr

Re: Race condition in module load causing undefined symbols

[Steve Lord]

Andi Kleen wrote:
> Steve Lord <lord@xfs.org> writes:
> 
>>So is this some P4 specific optimization which is not working as
>>intended?
> 
> 
> The only pentium specific optimizations that are enabled by MPENTIUM4
> is to tell the compiler to compile for pentium4 and a few settings
> in arch/i386/Kconfig.
> 
> You could enable/disable these individually and see if you can track
> it down with a binary search.
> 
> Most of this stuff should be fairly harmless though and be only
> microoptimizations; I cannot see how they should cause user visible
> races.
> 
> -Andi
> 

I see what you mean about there not being a lot of difference
between the two, however, the other difference between the builds
would be -mtune=pentium3 and -mtune=pentium4.

Steve

Re: Race condition in module load causing undefined symbols

[Steve Lord]

K.R. Foley wrote:
> Steve Lord wrote:
> <snip>
> 
>> So is this some P4 specific optimization which is not working as
>> intended?
>>
>> Steve
>>
>>
> 
> I'd say not since the first system I saw this on was a dual PIII Xeon. 
> While I am not 100% sure that the problems are related, the problem that 
> I saw on my 2.6 system also went away when I disabled hyper-threading in 
> the bios. It really just seems to me like it is some hard-to-trigger race.
> 

Not too hard for me :-(

Definitely a race, and it appears to be somewhere in the fork/exec/wait
complex at the very least. insmod is not built into nash, so is getting
run as a seperate process. Since module loading itself is synchronous,
the error would seem most likely to be happening in sys_wait4.

It could be the compiler doing a bad optimization, it could be
some other optimization code triggered by the Pentium 4 config
option, or, as you say, it could be a race which is being
opened up by the changed build flags.

Steve

Re: Race condition in module load causing undefined symbols

[Prarit Bhargava]

Colleagues,

(Copied and edited from a post I made on linux-hotplug-devel last month.)

I've privately emailed Steve with a quick-and-dirty solution for the problems he 
was experiencing with the system boot.  I wasn't sure if he was having the same 
problems I've had with 2.6.12 and old packages but it looks like he was.

I'm surprised we haven't had more people on this list wondering about the 
strange behaviour of their initrd/initramfs :) .

When I looked at the original output Steve had posted I noticed that it looked 
like drivers were attempting to load at the same time and because of this he 
eventually hit an oops.  I (and an engineer from another company working on 
another arch) have hit the same problem due to the requirements of our current work.

(Unfortunately, I'm more familiar with RedHat/Fedora than I am with other 
distro's -- please bear with me.)

The issue is that David Howells posted a patch that changed the behaviour of 
kallsyms/insmod/rmmod sometime ago.  The patch *is correct* in what it does, 
however, the patch requires that /sbin/sh must be aware of pid returns by wait().

     http://lkml.org/lkml/2005/1/17/132

There are two fixes that I'm aware of, and depending on what you're doing they 
are both "correct" (although in the case of developing in 2.6.12, IMO, you
_must_ do the latter).

The first fix is for the situation where you're developing for a specific 
distribution.  If this is the case, then you should back out the patch above and 
continue moving forward.

The second fix, and again you must do this if you're developing 2.6.12, is to 
*update the mkinitrd package* which has a new version of /bin/sh.

P.

Re: Race condition in module load causing undefined symbols

[Steve Lord]

Thanks Prarit,

I updated mkinitrd from 4.1.18 to 4.2.15 and udev from 039 to 058.
This appears to have cured it on my work machine, I will try the
other box later.

Looking at Documentation/Changes, which appears to still be the
official repository for required tool versions, it seems somewhat
dated, and makes no mention of mkinitrd version requirements.

Steve

Prarit Bhargava wrote:
> Colleagues,
> 
> (Copied and edited from a post I made on linux-hotplug-devel last month.)
> 
> I've privately emailed Steve with a quick-and-dirty solution for the 
> problems he was experiencing with the system boot.  I wasn't sure if he 
> was having the same problems I've had with 2.6.12 and old packages but 
> it looks like he was.
> 
> I'm surprised we haven't had more people on this list wondering about 
> the strange behaviour of their initrd/initramfs :) .
> 
> When I looked at the original output Steve had posted I noticed that it 
> looked like drivers were attempting to load at the same time and because 
> of this he eventually hit an oops.  I (and an engineer from another 
> company working on another arch) have hit the same problem due to the 
> requirements of our current work.
> 
> (Unfortunately, I'm more familiar with RedHat/Fedora than I am with 
> other distro's -- please bear with me.)
> 
> The issue is that David Howells posted a patch that changed the 
> behaviour of kallsyms/insmod/rmmod sometime ago.  The patch *is correct* 
> in what it does, however, the patch requires that /sbin/sh must be aware 
> of pid returns by wait().
> 
>     http://lkml.org/lkml/2005/1/17/132
> 
> There are two fixes that I'm aware of, and depending on what you're 
> doing they are both "correct" (although in the case of developing in 
> 2.6.12, IMO, you
> _must_ do the latter).
> 
> The first fix is for the situation where you're developing for a 
> specific distribution.  If this is the case, then you should back out 
> the patch above and continue moving forward.
> 
> The second fix, and again you must do this if you're developing 2.6.12, 
> is to *update the mkinitrd package* which has a new version of /bin/sh.
> 
> P.
> 

Re: Race condition in module load causing undefined symbols

[Christoph Hellwig]

On Tue, Jun 14, 2005 at 02:27:29PM -0500, Steve Lord wrote:
> Thanks Prarit,
> 
> I updated mkinitrd from 4.1.18 to 4.2.15 and udev from 039 to 058.
> This appears to have cured it on my work machine, I will try the
> other box later.
> 
> Looking at Documentation/Changes, which appears to still be the
> official repository for required tool versions, it seems somewhat
> dated, and makes no mention of mkinitrd version requirements.

One of the reasons for that is that there is not generic mkinitrd.
Every distribution has it's own variant.

Re: Race condition in module load causing undefined symbols

[Pozsár Balázs]

On Tue, Jun 14, 2005 at 06:56:22PM +0200, Andi Kleen wrote:
> Steve Lord <lord@xfs.org> writes:
> >
> > So is this some P4 specific optimization which is not working as
> > intended?
> 
> The only pentium specific optimizations that are enabled by MPENTIUM4
> is to tell the compiler to compile for pentium4 and a few settings
> in arch/i386/Kconfig.
> 
> You could enable/disable these individually and see if you can track
> it down with a binary search.
> 
> Most of this stuff should be fairly harmless though and be only
> microoptimizations; I cannot see how they should cause user visible
> races.

I am 100% sure this is not a P4 optimization problem since I compiled my 
kernel for i586 and saw the same problem.


-- 
pozsy

Re: Race condition in module load causing undefined symbols

[Pozsár Balázs]

On Tue, Jun 14, 2005 at 02:23:04PM -0400, Prarit Bhargava wrote:
> The second fix, and again you must do this if you're developing 2.6.12, is 
> to *update the mkinitrd package* which has a new version of /bin/sh.

This sounds insane to me. I am using bash in my initrd, does this mean 
that every shell and whatever has to be updated? Exactly what 
modifications has to be made?


-- 
pozsy

Re: Race condition in module load causing undefined symbols

[Prarit Bhargava]

Pozsár Balázs wrote:
> On Tue, Jun 14, 2005 at 02:23:04PM -0400, Prarit Bhargava wrote:
> 
>>The second fix, and again you must do this if you're developing 2.6.12, is 
>>to *update the mkinitrd package* which has a new version of /bin/sh.
> 
> 
> This sounds insane to me. I am using bash in my initrd, does this mean 
> that every shell and whatever has to be updated? Exactly what 
> modifications has to be made?
> 
> 

If you're using bash, I would suggest starting with an update of the bash package.

It's interesting to note that Steve also needed to update his udev package. 
Steve, IIRC you were using Fedora 3/4?

P.

Re: Race condition in module load causing undefined symbols

[Pozsár Balázs]

On Wed, Jun 15, 2005 at 07:28:00AM -0400, Prarit Bhargava wrote:
> Pozsár Balázs wrote:
> >On Tue, Jun 14, 2005 at 02:23:04PM -0400, Prarit Bhargava wrote:
> >
> >>The second fix, and again you must do this if you're developing 2.6.12, 
> >>is to *update the mkinitrd package* which has a new version of /bin/sh.
> >
> >
> >This sounds insane to me. I am using bash in my initrd, does this mean 
> >that every shell and whatever has to be updated? Exactly what 
> >modifications has to be made?
> >
> >
> 
> If you're using bash, I would suggest starting with an update of the bash 
> package.

Well, I'm using 3.0 and afaik there's no newer version, but I don't 
think this is the problem either.

Exactlywhat modifications have to be made and to what to work around 
this kernel regression?


-- 
pozsy

Re: Race condition in module load causing undefined symbols

[Prarit Bhargava]

Pozsár Balázs wrote:

>>If you're using bash, I would suggest starting with an update of the bash 
>>package.
> 
> 
> Well, I'm using 3.0 and afaik there's no newer version, but I don't 
> think this is the problem either.
> 
> Exactlywhat modifications have to be made and to what to work around 
> this kernel regression?
> 

Just to be clear, this isn't a kernel regression -- it's a problem with packages ;).

Try backing out the patch I pointed to earlier in the thread.  Let's see if that 
works...

P.

Re: Race condition in module load causing undefined symbols

[Pozsár Balázs]

On Wed, Jun 15, 2005 at 07:35:22AM -0400, Prarit Bhargava wrote:
> Pozsár Balázs wrote:
> 
> >>If you're using bash, I would suggest starting with an update of the bash 
> >>package.
> >
> >
> >Well, I'm using 3.0 and afaik there's no newer version, but I don't 
> >think this is the problem either.
> >
> >Exactlywhat modifications have to be made and to what to work around 
> >this kernel regression?
> >
> 
> Just to be clear, this isn't a kernel regression -- it's a problem with 
> packages ;).

Let me this very clear: I've got a few initscripts written in bash, 
which load some modules.
These are very basic and trivial straightforward, basically some
"modprobe whatever".


All works fine with 2.6.9.
It does not work properly with 2.6.12-rc5.
This is a regression.



-- 
pozsy

Re: Race condition in module load causing undefined symbols

[Stephen Lord]

Prarit Bhargava wrote:
> 
> If you're using bash, I would suggest starting with an update of the 
> bash package.
> 
> It's interesting to note that Steve also needed to update his udev 
> package. Steve, IIRC you were using Fedora 3/4?
> 
> P.
> 

I am running an updated Fedora Core 3 - on both machines which have the
problem. I used rpms from Fedora Core 4 to update the system, this led
to a chain of updates, mkinitrd, udev, initscripts and a few others
with cross dependencies.

Still puzzled about what could have been fixed in user space since this
appears to affect more than one shell. Module loading appears to be
very synchronous, so unless the shell was not waiting for exit status
on children correctly, it seems hard to explain in user space.

Steve

Re: Race condition in module load causing undefined symbols

[David Howells]

Hi Steve,

Someone's finally waved this discussion in my direction.

> Still puzzled about what could have been fixed in user space since this
> appears to affect more than one shell. Module loading appears to be
> very synchronous, so unless the shell was not waiting for exit status
> on children correctly, it seems hard to explain in user space.

The problem with nash is very simple, and may be duplicated in other shells:

 (1) The patch to the module wangling patch that I made makes use of
     stop_machine_run() to insert a module into the module list or remove it
     from the module list.

     This is done because certain things that look at the list have to run
     without locks, so the only way to be certain they aren't going to run is
     to ensure that _nothing_ else is going to run.

 (2) stop_machine_run() creates a bunch of kernel threads to hog the other
     CPUs with interrupts disabled whilst one CPU does the actual work.

 (3) These kernel threads are reparented to the init process (PID 1).

 (4) When "parentless" threads exit, whatever process is running as PID 1 gets
     to deal with the zombies and will get a wait() event for each.

 (5) nash runs as PID 1 during boot.

 (6) nash was NOT checking the pid returned by its calls to wait(); in
     especial, when it forked off an insmod process, it would then simply wait
     for the first wait event to happen and continue on, without checking that
     the process it was waiting for had actually finished.

That is the basic problem being seen. It's just that it rarely happens without
this patch, but the problem is still there in nash, and could be triggered due
to other parentless processes exiting or dying.

I got nash patched, but it seems to be taking awhile to percolate.

David