* PROBLEM: memory corrupting bug, bisected to 6dda9d55 @ 2010-10-09 9:57 pacman 2010-10-11 12:52 ` Christoph Lameter 2010-10-11 14:30 ` Mel Gorman 0 siblings, 2 replies; 24+ messages in thread From: pacman @ 2010-10-09 9:57 UTC (permalink / raw) To: linux-mm Cc: Andrew Morton, Mel Gorman, Christoph Lameter, KOSAKI Motohiro, Yinghai Lu, linux-kernel (What a big Cc: list... scripts/get_maintainer.pl made me do it.) This will be a long story with a weak conclusion, sorry about that, but it's been a long bug-hunt. With recent kernels I've seen a bug that appears to corrupt random 4-byte chunks of memory. It's not easy to reproduce. It seems to happen only once per boot, pretty quickly after userspace has gotten started, and sometimes it doesn't happen at all. Symptoms that I have seen multiple times include: #1. Oops during modprobe usbcore (in apply_relocate_add) #2. (more frequent than #1) e2fsck dies of SIGSEGV or SIGILL I gdb'ed one of the e2fsck crashes and found that the SIGILL was indeed an illegal instruction. A single instruction had been replaced by 4 seemingly random bytes which did not form a valid instruction. So I began doing an md5 check of e2fsck and its dependent libs on every boot. This made detection easier, as I found that about 50% of the time, booting a bad kernel would cause an md5 mismatch in /lib/libe2p.so.2.3. None of this corruption was actually present on disk. I was always able to boot my old known-good kernel and md5 all the suspect files, and they were always fine. Using that test procedure, all the bad kernels showed the symptom on the second boot, and all the good kernels had 6 consecutive boots without any trouble. The git bisect ended here: commit 6dda9d55bf545013597724bf0cd79d01bd2bd944 Author: Corrado Zoccolo <czoccolo@gmail.com> page allocator: reduce fragmentation in buddy allocator by adding buddies that are merging to the tail of the free lists mm/page_alloc.c | 30 +++++++++++++++++++++++++----- 1 files changed, 25 insertions(+), 5 deletions(-) which is way back before 2.6.35-rc1. Since this is code that has obviously been tested by a lot of people and hasn't hurt most of them, I figure it must be very sensitive to hardware and/or kernel config options. I also considered the possibility of a compiler bug. Most of my testing was done with gcc 4.3.2, but I also tried 4.4.2 and that didn't make a difference. This is all happening on Pegasos2 (32-bit PPC). The latest kernel I've confirmed the bug on was 2.6.35.7. The bad commit reverts cleanly on top of 2.6.35.7, and that results in a good kernel as expected. (I can't test the latest Linus git tree until I solve the unrelated bug that has apparently killed the keyboard driver.) Can someone familiar with the code take a fresh look at 6dda9d55 and spot a bug? If not, what should I try next? Here's .config :r!grep '^[^\#]' .config CONFIG_PPC_BOOK3S_32=y CONFIG_PPC_BOOK3S=y CONFIG_6xx=y CONFIG_PPC_FPU=y CONFIG_PPC_STD_MMU=y CONFIG_PPC_STD_MMU_32=y CONFIG_PPC_HAVE_PMU_SUPPORT=y CONFIG_PPC_PERF_CTRS=y CONFIG_PPC32=y CONFIG_WORD_SIZE=32 CONFIG_MMU=y CONFIG_GENERIC_CMOS_UPDATE=y CONFIG_GENERIC_TIME=y CONFIG_GENERIC_TIME_VSYSCALL=y CONFIG_GENERIC_CLOCKEVENTS=y CONFIG_GENERIC_HARDIRQS=y CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y CONFIG_IRQ_PER_CPU=y CONFIG_NR_IRQS=512 CONFIG_STACKTRACE_SUPPORT=y CONFIG_HAVE_LATENCYTOP_SUPPORT=y CONFIG_TRACE_IRQFLAGS_SUPPORT=y CONFIG_LOCKDEP_SUPPORT=y CONFIG_RWSEM_XCHGADD_ALGORITHM=y CONFIG_ARCH_HAS_ILOG2_U32=y CONFIG_GENERIC_HWEIGHT=y CONFIG_GENERIC_FIND_NEXT_BIT=y CONFIG_PPC=y CONFIG_EARLY_PRINTK=y CONFIG_GENERIC_NVRAM=y CONFIG_SCHED_OMIT_FRAME_POINTER=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_PPC_OF=y CONFIG_OF=y CONFIG_PPC_UDBG_16550=y CONFIG_AUDIT_ARCH=y CONFIG_GENERIC_BUG=y CONFIG_DTC=y CONFIG_ARCH_HIBERNATION_POSSIBLE=y CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config" CONFIG_CONSTRUCTORS=y CONFIG_EXPERIMENTAL=y CONFIG_BROKEN_ON_SMP=y CONFIG_INIT_ENV_ARG_LIMIT=32 CONFIG_CROSS_COMPILE="" CONFIG_LOCALVERSION="" CONFIG_SWAP=y CONFIG_SYSVIPC=y CONFIG_SYSVIPC_SYSCTL=y CONFIG_POSIX_MQUEUE=y CONFIG_POSIX_MQUEUE_SYSCTL=y CONFIG_BSD_PROCESS_ACCT=y CONFIG_TREE_RCU=y CONFIG_RCU_FANOUT=32 CONFIG_IKCONFIG=y CONFIG_IKCONFIG_PROC=y CONFIG_LOG_BUF_SHIFT=14 CONFIG_CC_OPTIMIZE_FOR_SIZE=y CONFIG_SYSCTL=y CONFIG_ANON_INODES=y CONFIG_EMBEDDED=y CONFIG_SYSCTL_SYSCALL=y CONFIG_KALLSYMS=y CONFIG_HOTPLUG=y CONFIG_PRINTK=y CONFIG_BUG=y CONFIG_ELF_CORE=y CONFIG_PCSPKR_PLATFORM=y CONFIG_BASE_FULL=y CONFIG_FUTEX=y CONFIG_EPOLL=y CONFIG_SIGNALFD=y CONFIG_TIMERFD=y CONFIG_EVENTFD=y CONFIG_SHMEM=y CONFIG_AIO=y CONFIG_HAVE_PERF_EVENTS=y CONFIG_PERF_EVENTS=y CONFIG_PERF_COUNTERS=y CONFIG_VM_EVENT_COUNTERS=y CONFIG_PCI_QUIRKS=y CONFIG_COMPAT_BRK=y CONFIG_SLAB=y CONFIG_HAVE_OPROFILE=y CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y CONFIG_HAVE_IOREMAP_PROT=y CONFIG_HAVE_KPROBES=y CONFIG_HAVE_KRETPROBES=y CONFIG_HAVE_ARCH_TRACEHOOK=y CONFIG_HAVE_DMA_ATTRS=y CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y CONFIG_HAVE_DMA_API_DEBUG=y CONFIG_SLOW_WORK=y CONFIG_SLABINFO=y CONFIG_RT_MUTEXES=y CONFIG_BASE_SMALL=0 CONFIG_MODULES=y CONFIG_MODULE_UNLOAD=y CONFIG_MODULE_FORCE_UNLOAD=y CONFIG_MODVERSIONS=y CONFIG_BLOCK=y CONFIG_LBDAF=y CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_CFQ=y CONFIG_DEFAULT_CFQ=y CONFIG_DEFAULT_IOSCHED="cfq" CONFIG_INLINE_SPIN_UNLOCK=y CONFIG_INLINE_SPIN_UNLOCK_IRQ=y CONFIG_INLINE_READ_UNLOCK=y CONFIG_INLINE_READ_UNLOCK_IRQ=y CONFIG_INLINE_WRITE_UNLOCK=y CONFIG_INLINE_WRITE_UNLOCK_IRQ=y CONFIG_PPC_CHRP=y CONFIG_PPC_NATIVE=y CONFIG_PPC_OF_BOOT_TRAMPOLINE=y CONFIG_MPIC=y CONFIG_PPC_I8259=y CONFIG_PPC_RTAS=y CONFIG_RTAS_ERROR_LOGGING=y CONFIG_PPC_RTAS_DAEMON=y CONFIG_RTAS_PROC=y CONFIG_PPC_MPC106=y CONFIG_HIGHMEM=y CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ=y CONFIG_HIGH_RES_TIMERS=y CONFIG_GENERIC_CLOCKEVENTS_BUILD=y CONFIG_HZ_250=y CONFIG_HZ=250 CONFIG_SCHED_HRTICK=y CONFIG_PREEMPT_NONE=y CONFIG_BINFMT_ELF=y CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y CONFIG_ARCH_HAS_WALK_MEMORY=y CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y CONFIG_SPARSE_IRQ=y CONFIG_MAX_ACTIVE_REGIONS=32 CONFIG_ARCH_FLATMEM_ENABLE=y CONFIG_ARCH_POPULATES_NODE_MAP=y CONFIG_SELECT_MEMORY_MODEL=y CONFIG_FLATMEM_MANUAL=y CONFIG_FLATMEM=y CONFIG_FLAT_NODE_MEM_MAP=y CONFIG_HAVE_MEMBLOCK=y CONFIG_PAGEFLAGS_EXTENDED=y CONFIG_SPLIT_PTLOCK_CPUS=4 CONFIG_ZONE_DMA_FLAG=1 CONFIG_BOUNCE=y CONFIG_VIRT_TO_BUS=y CONFIG_DEFAULT_MMAP_MIN_ADDR=4096 CONFIG_PPC_4K_PAGES=y CONFIG_FORCE_MAX_ZONEORDER=11 CONFIG_PROC_DEVICETREE=y CONFIG_EXTRA_TARGETS="" CONFIG_SECCOMP=y CONFIG_ISA_DMA_API=y CONFIG_ZONE_DMA=y CONFIG_NEED_SG_DMA_LENGTH=y CONFIG_GENERIC_ISA_DMA=y CONFIG_PPC_INDIRECT_PCI=y CONFIG_PCI=y CONFIG_PCI_DOMAINS=y CONFIG_PCI_SYSCALL=y CONFIG_ARCH_SUPPORTS_MSI=y CONFIG_LOWMEM_SIZE=0x30000000 CONFIG_PAGE_OFFSET=0xc0000000 CONFIG_KERNEL_START=0xc0000000 CONFIG_PHYSICAL_START=0x00000000 CONFIG_TASK_SIZE=0xc0000000 CONFIG_NET=y CONFIG_PACKET=m CONFIG_UNIX=m CONFIG_XFRM=y CONFIG_XFRM_USER=m CONFIG_XFRM_IPCOMP=m CONFIG_NET_KEY=m CONFIG_INET=y CONFIG_IP_MULTICAST=y CONFIG_IP_FIB_HASH=y CONFIG_NET_IPIP=m CONFIG_NET_IPGRE=m CONFIG_NET_IPGRE_BROADCAST=y CONFIG_IP_MROUTE=y CONFIG_IP_PIMSM_V1=y CONFIG_IP_PIMSM_V2=y CONFIG_SYN_COOKIES=y CONFIG_INET_AH=m CONFIG_INET_ESP=m CONFIG_INET_IPCOMP=m CONFIG_INET_XFRM_TUNNEL=m CONFIG_INET_TUNNEL=m CONFIG_INET_XFRM_MODE_TRANSPORT=y CONFIG_INET_XFRM_MODE_TUNNEL=y CONFIG_INET_XFRM_MODE_BEET=y CONFIG_INET_LRO=y CONFIG_INET_DIAG=y CONFIG_INET_TCP_DIAG=y CONFIG_TCP_CONG_CUBIC=y CONFIG_DEFAULT_TCP_CONG="cubic" CONFIG_NETFILTER=y CONFIG_NETFILTER_ADVANCED=y CONFIG_BRIDGE_NETFILTER=y CONFIG_NETFILTER_NETLINK=m CONFIG_NETFILTER_NETLINK_QUEUE=m CONFIG_NETFILTER_NETLINK_LOG=m CONFIG_NF_CONNTRACK=m CONFIG_NF_CT_ACCT=y CONFIG_NF_CONNTRACK_MARK=y CONFIG_NF_CT_PROTO_GRE=m CONFIG_NF_CONNTRACK_AMANDA=m CONFIG_NF_CONNTRACK_FTP=m CONFIG_NF_CONNTRACK_H323=m CONFIG_NF_CONNTRACK_IRC=m CONFIG_NF_CONNTRACK_PPTP=m CONFIG_NF_CONNTRACK_SANE=m CONFIG_NF_CONNTRACK_SIP=m CONFIG_NF_CONNTRACK_TFTP=m CONFIG_NETFILTER_XTABLES=m CONFIG_NETFILTER_XT_MARK=m CONFIG_NETFILTER_XT_CONNMARK=m CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m CONFIG_NETFILTER_XT_TARGET_CONNMARK=m CONFIG_NETFILTER_XT_TARGET_DSCP=m CONFIG_NETFILTER_XT_TARGET_HL=m CONFIG_NETFILTER_XT_TARGET_MARK=m CONFIG_NETFILTER_XT_TARGET_NFLOG=m CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m CONFIG_NETFILTER_XT_TARGET_NOTRACK=m CONFIG_NETFILTER_XT_TARGET_RATEEST=m CONFIG_NETFILTER_XT_TARGET_TEE=m CONFIG_NETFILTER_XT_TARGET_TRACE=m CONFIG_NETFILTER_XT_TARGET_TCPMSS=m CONFIG_NETFILTER_XT_TARGET_TCPOPTSTRIP=m CONFIG_NETFILTER_XT_MATCH_COMMENT=m CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m CONFIG_NETFILTER_XT_MATCH_CONNMARK=m CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m CONFIG_NETFILTER_XT_MATCH_DSCP=m CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=m CONFIG_NETFILTER_XT_MATCH_HELPER=m CONFIG_NETFILTER_XT_MATCH_HL=m CONFIG_NETFILTER_XT_MATCH_LENGTH=m CONFIG_NETFILTER_XT_MATCH_LIMIT=m CONFIG_NETFILTER_XT_MATCH_MAC=m CONFIG_NETFILTER_XT_MATCH_MARK=m CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m CONFIG_NETFILTER_XT_MATCH_OSF=m CONFIG_NETFILTER_XT_MATCH_OWNER=m CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m CONFIG_NETFILTER_XT_MATCH_QUOTA=m CONFIG_NETFILTER_XT_MATCH_RATEEST=m CONFIG_NETFILTER_XT_MATCH_STATE=m CONFIG_NETFILTER_XT_MATCH_STATISTIC=m CONFIG_NETFILTER_XT_MATCH_STRING=m CONFIG_NETFILTER_XT_MATCH_TCPMSS=m CONFIG_NETFILTER_XT_MATCH_TIME=m CONFIG_NETFILTER_XT_MATCH_U32=m CONFIG_NF_DEFRAG_IPV4=m CONFIG_NF_CONNTRACK_IPV4=m CONFIG_NF_CONNTRACK_PROC_COMPAT=y CONFIG_IP_NF_IPTABLES=m CONFIG_IP_NF_MATCH_ADDRTYPE=m CONFIG_IP_NF_MATCH_ECN=m CONFIG_IP_NF_MATCH_TTL=m CONFIG_IP_NF_FILTER=m CONFIG_IP_NF_TARGET_REJECT=m CONFIG_IP_NF_TARGET_LOG=m CONFIG_NF_NAT=m CONFIG_NF_NAT_NEEDED=y CONFIG_IP_NF_TARGET_MASQUERADE=m CONFIG_IP_NF_TARGET_NETMAP=m CONFIG_IP_NF_TARGET_REDIRECT=m CONFIG_NF_NAT_SNMP_BASIC=m CONFIG_NF_NAT_PROTO_GRE=m CONFIG_NF_NAT_FTP=m CONFIG_NF_NAT_IRC=m CONFIG_NF_NAT_TFTP=m CONFIG_NF_NAT_AMANDA=m CONFIG_NF_NAT_PPTP=m CONFIG_NF_NAT_H323=m CONFIG_NF_NAT_SIP=m CONFIG_IP_NF_MANGLE=m CONFIG_IP_NF_TARGET_ECN=m CONFIG_IP_NF_TARGET_TTL=m CONFIG_IP_NF_RAW=m CONFIG_IP_SCTP=m CONFIG_SCTP_HMAC_MD5=y CONFIG_STP=m CONFIG_GARP=m CONFIG_BRIDGE=m CONFIG_BRIDGE_IGMP_SNOOPING=y CONFIG_VLAN_8021Q=m CONFIG_VLAN_8021Q_GVRP=y CONFIG_LLC=m CONFIG_LLC2=m CONFIG_NET_SCHED=y CONFIG_NET_SCH_CBQ=m CONFIG_NET_SCH_HTB=m CONFIG_NET_SCH_HFSC=m CONFIG_NET_SCH_PRIO=m CONFIG_NET_SCH_RED=m CONFIG_NET_SCH_SFQ=m CONFIG_NET_SCH_TEQL=m CONFIG_NET_SCH_TBF=m CONFIG_NET_SCH_GRED=m CONFIG_NET_SCH_DSMARK=m CONFIG_NET_SCH_NETEM=m CONFIG_NET_SCH_DRR=m CONFIG_NET_CLS=y CONFIG_NET_CLS_BASIC=m CONFIG_NET_CLS_TCINDEX=m CONFIG_NET_CLS_ROUTE4=m CONFIG_NET_CLS_ROUTE=y CONFIG_NET_CLS_FW=m CONFIG_NET_CLS_U32=m CONFIG_CLS_U32_MARK=y CONFIG_NET_CLS_RSVP=m CONFIG_NET_CLS_RSVP6=m CONFIG_NET_SCH_FIFO=y CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug" CONFIG_STANDALONE=y CONFIG_PREVENT_FIRMWARE_BUILD=y CONFIG_FW_LOADER=m CONFIG_EXTRA_FIRMWARE="" CONFIG_OF_FLATTREE=y CONFIG_OF_DYNAMIC=y CONFIG_OF_DEVICE=y CONFIG_OF_I2C=y CONFIG_OF_MDIO=m CONFIG_PARPORT=m CONFIG_PARPORT_PC=m CONFIG_PARPORT_PC_SUPERIO=y CONFIG_PARPORT_1284=y CONFIG_BLK_DEV=y CONFIG_BLK_DEV_FD=m CONFIG_BLK_DEV_LOOP=m CONFIG_BLK_DEV_CRYPTOLOOP=m CONFIG_BLK_DEV_NBD=m CONFIG_BLK_DEV_RAM=m CONFIG_BLK_DEV_RAM_COUNT=16 CONFIG_BLK_DEV_RAM_SIZE=8192 CONFIG_CDROM_PKTCDVD=m CONFIG_CDROM_PKTCDVD_BUFFERS=8 CONFIG_MISC_DEVICES=y CONFIG_HAVE_IDE=y CONFIG_IDE=y CONFIG_IDE_XFER_MODE=y CONFIG_IDE_TIMINGS=y CONFIG_IDE_ATAPI=y CONFIG_IDE_GD=y CONFIG_IDE_GD_ATA=y CONFIG_BLK_DEV_IDECD=m CONFIG_BLK_DEV_IDECD_VERBOSE_ERRORS=y CONFIG_IDE_PROC_FS=y CONFIG_BLK_DEV_IDEDMA_SFF=y CONFIG_BLK_DEV_IDEPCI=y CONFIG_IDEPCI_PCIBUS_ORDER=y CONFIG_BLK_DEV_GENERIC=y CONFIG_BLK_DEV_IDEDMA_PCI=y CONFIG_BLK_DEV_VIA82CXXX=y CONFIG_BLK_DEV_IDEDMA=y CONFIG_SCSI_MOD=m CONFIG_RAID_ATTRS=m CONFIG_SCSI=m CONFIG_SCSI_DMA=y CONFIG_SCSI_NETLINK=y CONFIG_SCSI_PROC_FS=y CONFIG_BLK_DEV_SD=m CONFIG_CHR_DEV_SG=m CONFIG_SCSI_MULTI_LUN=y CONFIG_SCSI_LOGGING=y CONFIG_SCSI_WAIT_SCAN=m CONFIG_SCSI_SPI_ATTRS=m CONFIG_SCSI_FC_ATTRS=m CONFIG_SCSI_ISCSI_ATTRS=m CONFIG_SCSI_LOWLEVEL=y CONFIG_MD=y CONFIG_BLK_DEV_MD=m CONFIG_MD_LINEAR=m CONFIG_MD_RAID0=m CONFIG_MD_RAID1=m CONFIG_MD_RAID10=m CONFIG_MD_RAID456=m CONFIG_MD_RAID6_PQ=m CONFIG_MD_MULTIPATH=m CONFIG_MD_FAULTY=m CONFIG_BLK_DEV_DM=m CONFIG_DM_CRYPT=m CONFIG_DM_SNAPSHOT=m CONFIG_DM_MIRROR=m CONFIG_DM_ZERO=m CONFIG_DM_MULTIPATH=m CONFIG_IEEE1394=m CONFIG_IEEE1394_OHCI1394=m CONFIG_IEEE1394_PCILYNX=m CONFIG_IEEE1394_SBP2=m CONFIG_IEEE1394_ETH1394_ROM_ENTRY=y CONFIG_IEEE1394_ETH1394=m CONFIG_IEEE1394_RAWIO=m CONFIG_IEEE1394_VIDEO1394=m CONFIG_IEEE1394_DV1394=m CONFIG_I2O=m CONFIG_I2O_LCT_NOTIFY_ON_CHANGES=y CONFIG_I2O_EXT_ADAPTEC=y CONFIG_I2O_CONFIG=m CONFIG_I2O_BLOCK=m CONFIG_I2O_SCSI=m CONFIG_I2O_PROC=m CONFIG_MACINTOSH_DRIVERS=y CONFIG_NETDEVICES=y CONFIG_DUMMY=m CONFIG_BONDING=m CONFIG_TUN=m CONFIG_VETH=m CONFIG_PHYLIB=m CONFIG_MARVELL_PHY=m CONFIG_DAVICOM_PHY=m CONFIG_QSEMI_PHY=m CONFIG_LXT_PHY=m CONFIG_CICADA_PHY=m CONFIG_NET_ETHERNET=y CONFIG_MII=y CONFIG_NET_PCI=y CONFIG_VIA_RHINE=m CONFIG_VIA_RHINE_MMIO=y CONFIG_NETDEV_1000=y CONFIG_VIA_VELOCITY=m CONFIG_MV643XX_ETH=m CONFIG_USB_CATC=m CONFIG_USB_PEGASUS=m CONFIG_USB_RTL8150=m CONFIG_USB_USBNET=m CONFIG_USB_NET_AX8817X=m CONFIG_USB_NET_CDCETHER=m CONFIG_USB_NET_DM9601=m CONFIG_USB_NET_ZAURUS=m CONFIG_PPP=m CONFIG_PPP_MULTILINK=y CONFIG_PPP_FILTER=y CONFIG_PPP_ASYNC=m CONFIG_PPP_DEFLATE=m CONFIG_PPP_BSDCOMP=m CONFIG_PPPOE=m CONFIG_SLHC=m CONFIG_NET_FC=y CONFIG_INPUT=y CONFIG_INPUT_MOUSEDEV=y CONFIG_INPUT_MOUSEDEV_PSAUX=y CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024 CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768 CONFIG_INPUT_EVDEV=m CONFIG_INPUT_EVBUG=m CONFIG_INPUT_KEYBOARD=y CONFIG_KEYBOARD_ATKBD=y CONFIG_INPUT_MOUSE=y CONFIG_MOUSE_PS2=m CONFIG_MOUSE_PS2_ALPS=y CONFIG_MOUSE_PS2_LOGIPS2PP=y CONFIG_MOUSE_PS2_SYNAPTICS=y CONFIG_MOUSE_PS2_TRACKPOINT=y CONFIG_INPUT_MISC=y CONFIG_INPUT_PCSPKR=m CONFIG_INPUT_UINPUT=m CONFIG_SERIO=y CONFIG_SERIO_I8042=y CONFIG_SERIO_SERPORT=m CONFIG_SERIO_LIBPS2=y CONFIG_SERIO_RAW=m CONFIG_GAMEPORT=m CONFIG_GAMEPORT_NS558=m CONFIG_GAMEPORT_EMU10K1=m CONFIG_GAMEPORT_FM801=m CONFIG_VT=y CONFIG_CONSOLE_TRANSLATIONS=y CONFIG_VT_CONSOLE=y CONFIG_HW_CONSOLE=y CONFIG_VT_HW_CONSOLE_BINDING=y CONFIG_DEVKMEM=y CONFIG_SERIAL_8250=m CONFIG_SERIAL_8250_PCI=m CONFIG_SERIAL_8250_NR_UARTS=4 CONFIG_SERIAL_8250_RUNTIME_UARTS=4 CONFIG_SERIAL_CORE=m CONFIG_SERIAL_OF_PLATFORM=m CONFIG_UNIX98_PTYS=y CONFIG_PRINTER=m CONFIG_IPMI_HANDLER=m CONFIG_IPMI_DEVICE_INTERFACE=m CONFIG_IPMI_SI=m CONFIG_IPMI_WATCHDOG=m CONFIG_IPMI_POWEROFF=m CONFIG_HW_RANDOM=y CONFIG_NVRAM=y CONFIG_GEN_RTC=y CONFIG_GEN_RTC_X=y CONFIG_DEVPORT=y CONFIG_RAMOOPS=m CONFIG_I2C=y CONFIG_I2C_BOARDINFO=y CONFIG_I2C_CHARDEV=m CONFIG_I2C_ALGOBIT=y CONFIG_I2C_VIAPRO=m CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y CONFIG_HWMON=m CONFIG_HWMON_VID=m CONFIG_SENSORS_VT8231=m CONFIG_WATCHDOG=y CONFIG_SOFT_WATCHDOG=m CONFIG_PCIPCWATCHDOG=m CONFIG_WDTPCI=m CONFIG_USBPCWATCHDOG=m CONFIG_SSB_POSSIBLE=y CONFIG_MEDIA_SUPPORT=y CONFIG_VIDEO_DEV=m CONFIG_VIDEO_V4L2_COMMON=m CONFIG_VIDEO_V4L1_COMPAT=y CONFIG_VIDEO_MEDIA=m CONFIG_IR_CORE=y CONFIG_VIDEO_IR=y CONFIG_MEDIA_ATTACH=y CONFIG_MEDIA_TUNER=m CONFIG_MEDIA_TUNER_SIMPLE=m CONFIG_MEDIA_TUNER_TDA8290=m CONFIG_MEDIA_TUNER_TDA9887=m CONFIG_MEDIA_TUNER_TEA5761=m CONFIG_MEDIA_TUNER_TEA5767=m CONFIG_MEDIA_TUNER_MT20XX=m CONFIG_MEDIA_TUNER_XC2028=m CONFIG_MEDIA_TUNER_XC5000=m CONFIG_MEDIA_TUNER_MC44S803=m CONFIG_VIDEO_V4L2=m CONFIG_VIDEO_CAPTURE_DRIVERS=y CONFIG_VIDEO_FIXED_MINOR_RANGES=y CONFIG_V4L_USB_DRIVERS=y CONFIG_USB_VIDEO_CLASS=m CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y CONFIG_USB_GSPCA=m CONFIG_AGP=m CONFIG_VGA_ARB=y CONFIG_VGA_ARB_MAX_GPUS=16 CONFIG_VIDEO_OUTPUT_CONTROL=m CONFIG_FB=y CONFIG_FB_DDC=y CONFIG_FB_CFB_FILLRECT=y CONFIG_FB_CFB_COPYAREA=y CONFIG_FB_CFB_IMAGEBLIT=y CONFIG_FB_MACMODES=y CONFIG_FB_BACKLIGHT=y CONFIG_FB_MODE_HELPERS=y CONFIG_FB_TILEBLITTING=y CONFIG_FB_OF=y CONFIG_FB_RADEON=y CONFIG_FB_RADEON_I2C=y CONFIG_FB_RADEON_BACKLIGHT=y CONFIG_FB_ATY128=y CONFIG_FB_ATY128_BACKLIGHT=y CONFIG_FB_ATY=y CONFIG_FB_ATY_CT=y CONFIG_FB_ATY_GENERIC_LCD=y CONFIG_FB_ATY_GX=y CONFIG_FB_ATY_BACKLIGHT=y CONFIG_BACKLIGHT_LCD_SUPPORT=y CONFIG_LCD_CLASS_DEVICE=m CONFIG_BACKLIGHT_CLASS_DEVICE=y CONFIG_DISPLAY_SUPPORT=y CONFIG_VGA_CONSOLE=y CONFIG_DUMMY_CONSOLE=y CONFIG_FRAMEBUFFER_CONSOLE=y CONFIG_FONT_8x8=y CONFIG_FONT_8x16=y CONFIG_LOGO=y CONFIG_LOGO_LINUX_VGA16=y CONFIG_SOUND=m CONFIG_SOUND_OSS_CORE=y CONFIG_SOUND_OSS_CORE_PRECLAIM=y CONFIG_SOUND_PRIME=m CONFIG_SOUND_OSS=m CONFIG_SOUND_YM3812=m CONFIG_HID_SUPPORT=y CONFIG_HID=m CONFIG_HIDRAW=y CONFIG_USB_HID=m CONFIG_USB_HIDDEV=y CONFIG_USB_SUPPORT=y CONFIG_USB_ARCH_HAS_HCD=y CONFIG_USB_ARCH_HAS_OHCI=y CONFIG_USB_ARCH_HAS_EHCI=y CONFIG_USB=m CONFIG_USB_DEVICEFS=y CONFIG_USB_DEVICE_CLASS=y CONFIG_USB_EHCI_HCD=m CONFIG_USB_EHCI_ROOT_HUB_TT=y CONFIG_USB_EHCI_HCD_PPC_OF=y CONFIG_USB_OHCI_HCD=m CONFIG_USB_OHCI_HCD_PPC_OF_BE=y CONFIG_USB_OHCI_HCD_PPC_OF=y CONFIG_USB_OHCI_HCD_PCI=y CONFIG_USB_OHCI_BIG_ENDIAN_DESC=y CONFIG_USB_OHCI_BIG_ENDIAN_MMIO=y CONFIG_USB_OHCI_LITTLE_ENDIAN=y CONFIG_USB_UHCI_HCD=m CONFIG_USB_ACM=m CONFIG_USB_PRINTER=m CONFIG_USB_TMC=m CONFIG_USB_STORAGE=m CONFIG_USB_STORAGE_DATAFAB=m CONFIG_USB_STORAGE_FREECOM=m CONFIG_USB_STORAGE_ISD200=m CONFIG_USB_MDC800=m CONFIG_USB_MICROTEK=m CONFIG_USB_SERIAL=m CONFIG_USB_EZUSB=y CONFIG_USB_SERIAL_GENERIC=y CONFIG_USB_SERIAL_BELKIN=m CONFIG_USB_SERIAL_DIGI_ACCELEPORT=m CONFIG_USB_SERIAL_CP210X=m CONFIG_USB_SERIAL_CYPRESS_M8=m CONFIG_USB_SERIAL_EMPEG=m CONFIG_USB_SERIAL_FTDI_SIO=m CONFIG_USB_SERIAL_VISOR=m CONFIG_USB_SERIAL_IPAQ=m CONFIG_USB_SERIAL_IR=m CONFIG_USB_SERIAL_GARMIN=m CONFIG_USB_SERIAL_IPW=m CONFIG_USB_SERIAL_KLSI=m CONFIG_USB_SERIAL_KOBIL_SCT=m CONFIG_USB_SERIAL_MCT_U232=m CONFIG_USB_SERIAL_PL2303=m CONFIG_USB_SERIAL_HP4X=m CONFIG_USB_SERIAL_SAFE=m CONFIG_USB_SERIAL_SAFE_PADDED=y CONFIG_USB_SERIAL_CYBERJACK=m CONFIG_USB_SERIAL_OMNINET=m CONFIG_USB_RIO500=m CONFIG_USB_LEGOTOWER=m CONFIG_USB_LCD=m CONFIG_USB_LED=m CONFIG_USB_CYTHERM=m CONFIG_USB_IDMOUSE=m CONFIG_USB_TEST=m CONFIG_MMC=m CONFIG_MMC_BLOCK=m CONFIG_MMC_BLOCK_BOUNCE=y CONFIG_MMC_SDHCI=m CONFIG_AUXDISPLAY=y CONFIG_EXT2_FS=m CONFIG_EXT2_FS_XATTR=y CONFIG_EXT2_FS_POSIX_ACL=y CONFIG_EXT2_FS_SECURITY=y CONFIG_EXT3_FS=y CONFIG_EXT3_DEFAULTS_TO_ORDERED=y CONFIG_EXT3_FS_XATTR=y CONFIG_EXT3_FS_POSIX_ACL=y CONFIG_EXT3_FS_SECURITY=y CONFIG_EXT4_FS=m CONFIG_EXT4_FS_XATTR=y CONFIG_EXT4_FS_POSIX_ACL=y CONFIG_JBD=y CONFIG_JBD2=m CONFIG_FS_MBCACHE=y CONFIG_REISERFS_FS=m CONFIG_REISERFS_FS_XATTR=y CONFIG_REISERFS_FS_POSIX_ACL=y CONFIG_REISERFS_FS_SECURITY=y CONFIG_FS_POSIX_ACL=y CONFIG_BTRFS_FS=m CONFIG_BTRFS_FS_POSIX_ACL=y CONFIG_FILE_LOCKING=y CONFIG_FSNOTIFY=y CONFIG_INOTIFY=y CONFIG_INOTIFY_USER=y CONFIG_QUOTA=y CONFIG_PRINT_QUOTA_WARNING=y CONFIG_QUOTA_TREE=m CONFIG_QFMT_V2=m CONFIG_QUOTACTL=y CONFIG_FUSE_FS=m CONFIG_CUSE=m CONFIG_GENERIC_ACL=y CONFIG_FSCACHE=m CONFIG_FSCACHE_STATS=y CONFIG_FSCACHE_HISTOGRAM=y CONFIG_CACHEFILES=m CONFIG_CACHEFILES_HISTOGRAM=y CONFIG_ISO9660_FS=m CONFIG_ZISOFS=y CONFIG_UDF_FS=m CONFIG_UDF_NLS=y CONFIG_FAT_FS=m CONFIG_MSDOS_FS=m CONFIG_FAT_DEFAULT_CODEPAGE=437 CONFIG_PROC_FS=y CONFIG_PROC_KCORE=y CONFIG_PROC_SYSCTL=y CONFIG_PROC_PAGE_MONITOR=y CONFIG_SYSFS=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y CONFIG_MISC_FILESYSTEMS=y CONFIG_HFS_FS=m CONFIG_HFSPLUS_FS=m CONFIG_CRAMFS=y CONFIG_SQUASHFS=m CONFIG_SQUASHFS_XATTRS=y CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3 CONFIG_MINIX_FS=m CONFIG_ROMFS_FS=m CONFIG_ROMFS_BACKED_BY_BLOCK=y CONFIG_ROMFS_ON_BLOCK=y CONFIG_NETWORK_FILESYSTEMS=y CONFIG_NFS_FS=m CONFIG_NFS_V3=y CONFIG_NFS_V3_ACL=y CONFIG_NFS_V4=y CONFIG_NFS_FSCACHE=y CONFIG_LOCKD=m CONFIG_LOCKD_V4=y CONFIG_NFS_ACL_SUPPORT=m CONFIG_NFS_COMMON=y CONFIG_SUNRPC=m CONFIG_SUNRPC_GSS=m CONFIG_RPCSEC_GSS_KRB5=m CONFIG_CODA_FS=m CONFIG_PARTITION_ADVANCED=y CONFIG_AMIGA_PARTITION=y CONFIG_MAC_PARTITION=y CONFIG_MSDOS_PARTITION=y CONFIG_NLS=m CONFIG_NLS_DEFAULT="iso8859-1" CONFIG_NLS_CODEPAGE_437=m CONFIG_NLS_ASCII=m CONFIG_NLS_ISO8859_1=m CONFIG_NLS_UTF8=m CONFIG_BITREVERSE=y CONFIG_GENERIC_FIND_LAST_BIT=y CONFIG_CRC_CCITT=m CONFIG_CRC16=m CONFIG_CRC_ITU_T=m CONFIG_CRC32=y CONFIG_LIBCRC32C=m CONFIG_ZLIB_INFLATE=y CONFIG_ZLIB_DEFLATE=m CONFIG_LZO_COMPRESS=m CONFIG_LZO_DECOMPRESS=m CONFIG_TEXTSEARCH=y CONFIG_TEXTSEARCH_KMP=m CONFIG_TEXTSEARCH_BM=m CONFIG_TEXTSEARCH_FSM=m CONFIG_HAS_IOMEM=y CONFIG_HAS_IOPORT=y CONFIG_HAS_DMA=y CONFIG_NLATTR=y CONFIG_GENERIC_ATOMIC64=y CONFIG_ENABLE_WARN_DEPRECATED=y CONFIG_ENABLE_MUST_CHECK=y CONFIG_FRAME_WARN=1024 CONFIG_MAGIC_SYSRQ=y CONFIG_STRIP_ASM_SYMS=y CONFIG_DEBUG_KERNEL=y CONFIG_DETECT_SOFTLOCKUP=y CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0 CONFIG_DETECT_HUNG_TASK=y CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0 CONFIG_DEBUG_LIST=y CONFIG_SYSCTL_SYSCALL_CHECK=y CONFIG_HAVE_FUNCTION_TRACER=y CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y CONFIG_HAVE_DYNAMIC_FTRACE=y CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y CONFIG_TRACING_SUPPORT=y CONFIG_FTRACE=y CONFIG_BRANCH_PROFILE_NONE=y CONFIG_HAVE_ARCH_KGDB=y CONFIG_PPC_WERROR=y CONFIG_PRINT_STACK_DEPTH=64 CONFIG_BOOTX_TEXT=y CONFIG_KEYS=y CONFIG_DEFAULT_SECURITY_DAC=y CONFIG_DEFAULT_SECURITY="" CONFIG_XOR_BLOCKS=m CONFIG_ASYNC_CORE=m CONFIG_ASYNC_MEMCPY=m CONFIG_ASYNC_XOR=m CONFIG_ASYNC_PQ=m CONFIG_ASYNC_RAID6_RECOV=m CONFIG_CRYPTO=y CONFIG_CRYPTO_FIPS=y CONFIG_CRYPTO_ALGAPI=y CONFIG_CRYPTO_ALGAPI2=y CONFIG_CRYPTO_AEAD=m CONFIG_CRYPTO_AEAD2=y CONFIG_CRYPTO_BLKCIPHER=m CONFIG_CRYPTO_BLKCIPHER2=y CONFIG_CRYPTO_HASH=y CONFIG_CRYPTO_HASH2=y CONFIG_CRYPTO_RNG=m CONFIG_CRYPTO_RNG2=y CONFIG_CRYPTO_PCOMP=y CONFIG_CRYPTO_MANAGER=y CONFIG_CRYPTO_MANAGER2=y CONFIG_CRYPTO_MANAGER_TESTS=y CONFIG_CRYPTO_GF128MUL=m CONFIG_CRYPTO_NULL=m CONFIG_CRYPTO_WORKQUEUE=y CONFIG_CRYPTO_AUTHENC=m CONFIG_CRYPTO_TEST=m CONFIG_CRYPTO_CCM=m CONFIG_CRYPTO_GCM=m CONFIG_CRYPTO_SEQIV=m CONFIG_CRYPTO_CBC=m CONFIG_CRYPTO_CTR=m CONFIG_CRYPTO_CTS=m CONFIG_CRYPTO_ECB=m CONFIG_CRYPTO_PCBC=m CONFIG_CRYPTO_XTS=m CONFIG_CRYPTO_HMAC=y CONFIG_CRYPTO_XCBC=m CONFIG_CRYPTO_VMAC=m CONFIG_CRYPTO_CRC32C=m CONFIG_CRYPTO_GHASH=m CONFIG_CRYPTO_MD4=m CONFIG_CRYPTO_MD5=y CONFIG_CRYPTO_MICHAEL_MIC=m CONFIG_CRYPTO_RMD128=m CONFIG_CRYPTO_RMD160=m CONFIG_CRYPTO_RMD256=m CONFIG_CRYPTO_RMD320=m CONFIG_CRYPTO_SHA1=m CONFIG_CRYPTO_SHA256=m CONFIG_CRYPTO_SHA512=m CONFIG_CRYPTO_TGR192=m CONFIG_CRYPTO_WP512=m CONFIG_CRYPTO_AES=m CONFIG_CRYPTO_ANUBIS=m CONFIG_CRYPTO_ARC4=m CONFIG_CRYPTO_BLOWFISH=m CONFIG_CRYPTO_CAMELLIA=m CONFIG_CRYPTO_CAST5=m CONFIG_CRYPTO_CAST6=m CONFIG_CRYPTO_DES=m CONFIG_CRYPTO_FCRYPT=m CONFIG_CRYPTO_KHAZAD=m CONFIG_CRYPTO_SALSA20=m CONFIG_CRYPTO_SEED=m CONFIG_CRYPTO_SERPENT=m CONFIG_CRYPTO_TEA=m CONFIG_CRYPTO_TWOFISH=m CONFIG_CRYPTO_TWOFISH_COMMON=m CONFIG_CRYPTO_DEFLATE=m CONFIG_CRYPTO_ZLIB=m CONFIG_CRYPTO_LZO=m CONFIG_CRYPTO_ANSI_CPRNG=m -- Alan Curry -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-09 9:57 PROBLEM: memory corrupting bug, bisected to 6dda9d55 pacman @ 2010-10-11 12:52 ` Christoph Lameter 2010-10-11 14:30 ` Mel Gorman 1 sibling, 0 replies; 24+ messages in thread From: Christoph Lameter @ 2010-10-11 12:52 UTC (permalink / raw) To: pacman Cc: linux-mm, Andrew Morton, Mel Gorman, KOSAKI Motohiro, Yinghai Lu, linux-kernel The contents of those scribbles may reveal something. Are these 4 bytes a pointer? If so at what memory area are they pointing? A page struct? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-09 9:57 PROBLEM: memory corrupting bug, bisected to 6dda9d55 pacman 2010-10-11 12:52 ` Christoph Lameter @ 2010-10-11 14:30 ` Mel Gorman 2010-10-11 20:35 ` pacman 2010-10-11 21:00 ` Andrew Morton 1 sibling, 2 replies; 24+ messages in thread From: Mel Gorman @ 2010-10-11 14:30 UTC (permalink / raw) To: pacman Cc: linux-mm, Andrew Morton, Christoph Lameter, KOSAKI Motohiro, Yinghai Lu, linux-kernel On Sat, Oct 09, 2010 at 04:57:18AM -0500, pacman@kosh.dhis.org wrote: > (What a big Cc: list... scripts/get_maintainer.pl made me do it.) > > This will be a long story with a weak conclusion, sorry about that, but it's > been a long bug-hunt. > > With recent kernels I've seen a bug that appears to corrupt random 4-byte > chunks of memory. It's not easy to reproduce. It seems to happen only once > per boot, pretty quickly after userspace has gotten started, and sometimes it > doesn't happen at all. > A corruption of 4 bytes could be consistent with a pointer value being written to an incorrect location. > Symptoms that I have seen multiple times include: > #1. Oops during modprobe usbcore (in apply_relocate_add) > #2. (more frequent than #1) e2fsck dies of SIGSEGV or SIGILL > > I gdb'ed one of the e2fsck crashes and found that the SIGILL was indeed an > illegal instruction. A single instruction had been replaced by 4 seemingly > random bytes which did not form a valid instruction. So I began doing an md5 > check of e2fsck and its dependent libs on every boot. > > This made detection easier, as I found that about 50% of the time, booting a > bad kernel would cause an md5 mismatch in /lib/libe2p.so.2.3. None of this > corruption was actually present on disk. I was always able to boot my old > known-good kernel and md5 all the suspect files, and they were always fine. > > Using that test procedure, all the bad kernels showed the symptom on the > second boot, and all the good kernels had 6 consecutive boots without any > trouble. The git bisect ended here: > > commit 6dda9d55bf545013597724bf0cd79d01bd2bd944 > Author: Corrado Zoccolo <czoccolo@gmail.com> > > page allocator: reduce fragmentation in buddy allocator by adding buddies that are merging to the tail of the free lists > > mm/page_alloc.c | 30 +++++++++++++++++++++++++----- > 1 files changed, 25 insertions(+), 5 deletions(-) > > which is way back before 2.6.35-rc1. > > Since this is code that has obviously been tested by a lot of people and > hasn't hurt most of them, I figure it must be very sensitive to hardware > and/or kernel config options. I also considered the possibility of a compiler > bug. Most of my testing was done with gcc 4.3.2, but I also tried 4.4.2 and > that didn't make a difference. > > This is all happening on Pegasos2 (32-bit PPC). > > The latest kernel I've confirmed the bug on was 2.6.35.7. The bad commit > reverts cleanly on top of 2.6.35.7, and that results in a good kernel as > expected. (I can't test the latest Linus git tree until I solve the unrelated > bug that has apparently killed the keyboard driver.) > > Can someone familiar with the code take a fresh look at 6dda9d55 and spot a > bug? If not, what should I try next? > I think there is a slight bug but but not one that would cause corruption. if ((order < MAX_ORDER-1) && pfn_valid_within(page_to_pfn(buddy))) { That looks like it can result in checking the buddy for an order-(MAX_ORDER-1) page which is a bit bogus. Thing is, it should be harmless because there isn't an unusual write made. In case it's some weird compiler optimisation though, could you try this? diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 502a882..5b0eb8c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -530,7 +530,7 @@ static inline void __free_one_page(struct page *page, * so it's less likely to be used soon and more likely to be merged * as a higher order page */ - if ((order < MAX_ORDER-1) && pfn_valid_within(page_to_pfn(buddy))) { + if ((order < MAX_ORDER-2) && pfn_valid_within(page_to_pfn(buddy))) { struct page *higher_page, *higher_buddy; combined_idx = __find_combined_index(page_idx, order); higher_page = page + combined_idx - page_idx; -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-11 14:30 ` Mel Gorman @ 2010-10-11 20:35 ` pacman 2010-10-11 21:00 ` Andrew Morton 1 sibling, 0 replies; 24+ messages in thread From: pacman @ 2010-10-11 20:35 UTC (permalink / raw) To: Mel Gorman; +Cc: linux-mm, linux-kernel Mel Gorman writes: > > A corruption of 4 bytes could be consistent with a pointer value being > written to an incorrect location. The memory scribbles that I've looked at in detail and written down have been 0x86520000, 0xea5b0000, and 0x1d5f0000. They don't look very pointerish. The 2 low bytes being 0 in all 3 cases is an intriguing pattern though. That may not matter though because... > > I think there is a slight bug but but not one that would cause corruption. > > if ((order < MAX_ORDER-1) && pfn_valid_within(page_to_pfn(buddy))) { I think you found it. Think harder about how it might cause corruption. Applying your suggested patch really seems to have fixed it. Starting from v2.6.36-rc7-69-g6b0cd00 I applied your patch, booted 6 times, all clean. Reverted your patch, booted once, and /sbin/e2fsck failed its md5sum check. Sent a copy of the "bad" /sbin/e2fsck to another machine, rebooted with an old good kernel, reapplied your patch to the new kernel, and got 6 more good boots. The bad copy of e2fsck differs from the good one in 2 separate locations, each 4 bytes wide. The bogus values are the 0xea5b0000 and 0x1d5f0000 which I mentioned already. > That looks like it can result in checking the buddy for an order-(MAX_ORDER-1) > page which is a bit bogus. Thing is, it should be harmless because there > isn't an unusual write made. In case it's some weird compiler optimisation > though, could you try this? > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 502a882..5b0eb8c 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -530,7 +530,7 @@ static inline void __free_one_page(struct page *page, > * so it's less likely to be used soon and more likely to be merged > * as a higher order page > */ > - if ((order < MAX_ORDER-1) && pfn_valid_within(page_to_pfn(buddy))) { > + if ((order < MAX_ORDER-2) && pfn_valid_within(page_to_pfn(buddy))) { > struct page *higher_page, *higher_buddy; > combined_idx = __find_combined_index(page_idx, order); > higher_page = page + combined_idx - page_idx; > It doesn't look like there are any optimization tricks involved. I did a "make mm/page_alloc.s" before and after your patch, and the difference is simply this: --- mm/page_alloc.s.6b0cd00 2010-10-11 14:03:03.000000000 -0500 +++ mm/page_alloc.s.6b0cd00+mel 2010-10-11 14:03:49.000000000 -0500 @@ -3885,7 +3885,7 @@ .L523: mr 11,28 # page_idx, page_idx.2227 .L526: - cmplwi 7,29,9 #, tmp222, order + cmplwi 7,29,8 #, tmp222, order lwz 0,0(30) #* page, tmp220 stw 29,12(30) # <variable>.D.6650.D.6646.private, order oris 0,0,0x8 #, tmp221, tmp220, @@ -4337,7 +4337,7 @@ add 30,31,11 # buddy, page, tmp197 ble+ 7,.L578 # .L575: - cmplwi 7,27,9 #, tmp226, order + cmplwi 7,27,8 #, tmp226, order lwz 0,0(31) #* page, tmp224 stw 27,12(31) # <variable>.D.6650.D.6646.private, order oris 0,0,0x8 #, tmp225, tmp224, -- Alan Curry -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-11 14:30 ` Mel Gorman 2010-10-11 20:35 ` pacman @ 2010-10-11 21:00 ` Andrew Morton 2010-10-13 14:40 ` Mel Gorman 1 sibling, 1 reply; 24+ messages in thread From: Andrew Morton @ 2010-10-11 21:00 UTC (permalink / raw) To: Mel Gorman Cc: pacman, linux-mm, Christoph Lameter, KOSAKI Motohiro, Yinghai Lu, linux-kernel, linuxppc-dev (cc linuxppc-dev@lists.ozlabs.org) On Mon, 11 Oct 2010 15:30:22 +0100 Mel Gorman <mel@csn.ul.ie> wrote: > On Sat, Oct 09, 2010 at 04:57:18AM -0500, pacman@kosh.dhis.org wrote: > > (What a big Cc: list... scripts/get_maintainer.pl made me do it.) > > > > This will be a long story with a weak conclusion, sorry about that, but it's > > been a long bug-hunt. > > > > With recent kernels I've seen a bug that appears to corrupt random 4-byte > > chunks of memory. It's not easy to reproduce. It seems to happen only once > > per boot, pretty quickly after userspace has gotten started, and sometimes it > > doesn't happen at all. > > > > A corruption of 4 bytes could be consistent with a pointer value being > written to an incorrect location. It's corruption of user memory, which is unusual. I'd be wondering if there was a pre-existing bug which 6dda9d55bf545013597 has exposed - previously the corruption was hitting something harmless. Something like a missed CPU cache writeback or invalidate operation. How sensitive/vulnerable is PPC32 to such things? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-11 21:00 ` Andrew Morton @ 2010-10-13 14:40 ` Mel Gorman 2010-10-13 17:52 ` pacman 2010-10-18 20:59 ` Benjamin Herrenschmidt 0 siblings, 2 replies; 24+ messages in thread From: Mel Gorman @ 2010-10-13 14:40 UTC (permalink / raw) To: Andrew Morton Cc: pacman, linux-mm, Christoph Lameter, KOSAKI Motohiro, Yinghai Lu, linux-kernel, linuxppc-dev On Mon, Oct 11, 2010 at 02:00:39PM -0700, Andrew Morton wrote: > (cc linuxppc-dev@lists.ozlabs.org) > > On Mon, 11 Oct 2010 15:30:22 +0100 > Mel Gorman <mel@csn.ul.ie> wrote: > > > On Sat, Oct 09, 2010 at 04:57:18AM -0500, pacman@kosh.dhis.org wrote: > > > (What a big Cc: list... scripts/get_maintainer.pl made me do it.) > > > > > > This will be a long story with a weak conclusion, sorry about that, but it's > > > been a long bug-hunt. > > > > > > With recent kernels I've seen a bug that appears to corrupt random 4-byte > > > chunks of memory. It's not easy to reproduce. It seems to happen only once > > > per boot, pretty quickly after userspace has gotten started, and sometimes it > > > doesn't happen at all. > > > > > > > A corruption of 4 bytes could be consistent with a pointer value being > > written to an incorrect location. > > It's corruption of user memory, which is unusual. I'd be wondering if > there was a pre-existing bug which 6dda9d55bf545013597 has exposed - > previously the corruption was hitting something harmless. Something > like a missed CPU cache writeback or invalidate operation. > This seems somewhat plausible although it's hard to tell for sure. But lets say we had the following situation in memory [<----MAX_ORDER_NR_PAGES---->][<----MAX_ORDER_NR_PAGES---->] INITRD memmap array initrd gets freed and someone else very early in boot gets allocated in there. Lets further guess that the struct pages in the memmap area are managing the page frame where the INITRD was because it makes the situation slightly easier to trigger. As pages get freed in the memmap array, we could reference memory where initrd used to be but the physical memory is mapped at two virtual addresses. CPU A CPU B Reads kernelspace virtual (gets cache line) Writes userspace virtual (gets different cache line) IO, writes buffer destined for userspace (via cache line) Cache line eviction, writeback to memory This is somewhat contrived but I can see how it might happen even on one CPU particularly if the L1 cache is virtual and is loose about checking physical tags. > How sensitive/vulnerable is PPC32 to such things? > I can not tell you specifically but if the above scenario is in any way plausible, I believe it would depend on what sort of L1 cache the CPU has. Maybe this particular version has a virtual cache with no physical tagging and is depending on the OS not to make virtual aliasing mistakes. -- Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-13 14:40 ` Mel Gorman @ 2010-10-13 17:52 ` pacman 2010-10-18 11:33 ` Mel Gorman 2010-10-18 20:59 ` Benjamin Herrenschmidt 1 sibling, 1 reply; 24+ messages in thread From: pacman @ 2010-10-13 17:52 UTC (permalink / raw) To: Mel Gorman; +Cc: Andrew Morton, linux-mm, linux-kernel, linuxppc-dev Mel Gorman writes: > > On Mon, Oct 11, 2010 at 02:00:39PM -0700, Andrew Morton wrote: > > > > It's corruption of user memory, which is unusual. I'd be wondering if > > there was a pre-existing bug which 6dda9d55bf545013597 has exposed - > > previously the corruption was hitting something harmless. Something > > like a missed CPU cache writeback or invalidate operation. > > > > This seems somewhat plausible although it's hard to tell for sure. But > lets say we had the following situation in memory > > [<----MAX_ORDER_NR_PAGES---->][<----MAX_ORDER_NR_PAGES---->] > INITRD memmap array I don't use initrd, so this isn't exactly what happened here. But it could be close. Let me throw out some more information and see if it triggers any ideas. First, I tried a new test after seeing the corruption happen: # md5sum /sbin/e2fsck ; echo 1 > /proc/sys/vm/drop_caches ; md5sum /sbin/e2fsck And got 2 different answers. The second answer was the correct one. Since applying the suggested patch which changed MAX_ORDER-1 to MAX_ORDER-2, I've been trying to isolate exactly when the corruption happens. Since I don't know much about kernel code, my main method is stuffing the area full of printk's. First I duplicated the affected function __free_one_page, since it's inlined at 2 different places, so I could apply the patch to just one of them. This proved that the problem is happening when called from free_one_page. The patch which fixes (or at least covers up) the bug will only matter when order==MAX_ORDER-2, otherwise everything is the same. So I added a lot of printk's to show what's happening when order==MAX_ORDER-2. I found that, very repeatably, 126 such instances occur during boot, and 61 of them pass the page_is_buddy(higher_page, higher_buddy, order + 1) test, causing them to call list_add_tail. Next, since the bug appears when this code decides to call list_add_tail, I made my own wrapper for list_add_tail, which allowed me to force some of the calls to do list_add instead. Eventually I found that of the 61 calls, the last one makes the difference. Allowing the first 60 calls to go through to list_add_tail, and switching the last one to list_add, the symptom goes away. dump_stack() for that last call gave me a backtrace like this: [c0303e80] [c0008124] show_stack+0x4c/0x144 (unreliable) [c0303ec0] [c0068a84] free_one_page+0x28c/0x5b0 [c0303f20] [c0069588] __free_pages_ok+0xf8/0x120 [c0303f40] [c02d28c8] free_all_bootmem_core+0xf0/0x1f8 [c0303f70] [c02d29fc] free_all_bootmem+0x2c/0x6c [c0303f90] [c02cc7dc] mem_init+0x70/0x2ac [c0303fc0] [c02c66a4] start_kernel+0x150/0x27c [c0303ff0] [00003438] 0x3438 And this might be interesting: the PFN of the page being added in that critical 61st call is 130048, which exactly matches the number of available pages: free_area_init_node: node 0, pgdat c02fee6c, node_mem_map c0330000 DMA zone: 1024 pages used for memmap DMA zone: 0 pages reserved DMA zone: 130048 pages, LIFO batch:31 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048 Suspicious? If 130048 is added to the head of the order==MAX_ORDER-2 free list, there's no symptom. Add it to the tail, and the corruption appears. That's all I know so far. -- Alan Curry -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-13 17:52 ` pacman @ 2010-10-18 11:33 ` Mel Gorman 2010-10-18 19:10 ` pacman 2010-10-18 19:37 ` Andrew Morton 0 siblings, 2 replies; 24+ messages in thread From: Mel Gorman @ 2010-10-18 11:33 UTC (permalink / raw) To: pacman; +Cc: Andrew Morton, linux-mm, linux-kernel, linuxppc-dev On Wed, Oct 13, 2010 at 12:52:05PM -0500, pacman@kosh.dhis.org wrote: > Mel Gorman writes: > > > > On Mon, Oct 11, 2010 at 02:00:39PM -0700, Andrew Morton wrote: > > > > > > It's corruption of user memory, which is unusual. I'd be wondering if > > > there was a pre-existing bug which 6dda9d55bf545013597 has exposed - > > > previously the corruption was hitting something harmless. Something > > > like a missed CPU cache writeback or invalidate operation. > > > > > > > This seems somewhat plausible although it's hard to tell for sure. But > > lets say we had the following situation in memory > > > > [<----MAX_ORDER_NR_PAGES---->][<----MAX_ORDER_NR_PAGES---->] > > INITRD memmap array > > I don't use initrd, so this isn't exactly what happened here. But it could be > close. Let me throw out some more information and see if it triggers any > ideas. > Ok. > First, I tried a new test after seeing the corruption happen: > # md5sum /sbin/e2fsck ; echo 1 > /proc/sys/vm/drop_caches ; md5sum /sbin/e2fsck > And got 2 different answers. The second answer was the correct one. > > Since applying the suggested patch which changed MAX_ORDER-1 to MAX_ORDER-2, > I've been trying to isolate exactly when the corruption happens. Since I > don't know much about kernel code, my main method is stuffing the area full > of printk's. > > First I duplicated the affected function __free_one_page, since it's inlined > at 2 different places, so I could apply the patch to just one of them. This > proved that the problem is happening when called from free_one_page. > > The patch which fixes (or at least covers up) the bug will only matter when > order==MAX_ORDER-2, otherwise everything is the same. So I added a lot of > printk's to show what's happening when order==MAX_ORDER-2. I found that, very > repeatably, 126 such instances occur during boot, and 61 of them pass the > page_is_buddy(higher_page, higher_buddy, order + 1) test, causing them to > call list_add_tail. > > Next, since the bug appears when this code decides to call list_add_tail, > I made my own wrapper for list_add_tail, which allowed me to force some of > the calls to do list_add instead. Eventually I found that of the 61 calls, > the last one makes the difference. Allowing the first 60 calls to go through > to list_add_tail, and switching the last one to list_add, the symptom goes > away. > > dump_stack() for that last call gave me a backtrace like this: > [c0303e80] [c0008124] show_stack+0x4c/0x144 (unreliable) > [c0303ec0] [c0068a84] free_one_page+0x28c/0x5b0 > [c0303f20] [c0069588] __free_pages_ok+0xf8/0x120 > [c0303f40] [c02d28c8] free_all_bootmem_core+0xf0/0x1f8 > [c0303f70] [c02d29fc] free_all_bootmem+0x2c/0x6c > [c0303f90] [c02cc7dc] mem_init+0x70/0x2ac > [c0303fc0] [c02c66a4] start_kernel+0x150/0x27c > [c0303ff0] [00003438] 0x3438 > > And this might be interesting: the PFN of the page being added in that > critical 61st call is 130048, which exactly matches the number of available > pages: > > free_area_init_node: node 0, pgdat c02fee6c, node_mem_map c0330000 > DMA zone: 1024 pages used for memmap > DMA zone: 0 pages reserved > DMA zone: 130048 pages, LIFO batch:31 > Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048 > > Suspicious? > A bit but I still don't know why it would cause corruption. Maybe this is still a caching issue but the difference in timing between list_add and list_add_tail is enough to hide the bug. It's also possible there are some registers ioremapped after the memmap array and reading them is causing some problem. Andrew, what is the right thing to do here? We could flail around looking for explanations as to why the bug causes a user buffer corruption but never get an answer or do we go with this patch, preferably before 2.6.36 releases? ==== CUT HERE ==== mm, page-allocator: Do not check the state of a non-existant buddy during free There is a bug in commit [6dda9d55: page allocator: reduce fragmentation in buddy allocator by adding buddies that are merging to the tail of the free lists] that means a buddy at order MAX_ORDER is checked for merging. A page of this order never exists so at times, an effectively random piece of memory is being checked. Alan Curry has reported that this is causing memory corruption in userspace data on a PPC32 platform (http://lkml.org/lkml/2010/10/9/32). It is not clear why this is happening. It could be a cache coherency problem where pages mapped in both user and kernel space are getting different cache lines due to the bad read from kernel space (http://lkml.org/lkml/2010/10/13/179). It could also be that there are some special registers being io-remapped at the end of the memmap array and that a read has special meaning on them. Compiler bugs have been ruled out because the assembly before and after the patch looks relatively harmless. This patch fixes the problem by ensuring we are not reading a possibly invalid location of memory. It's not clear why the read causes corruption but one way or the other it is a buggy read. Signed-off-by: Mel Gorman <mel@csn.ul.ie> --- mm/page_alloc.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index a8cfa9c..93cef41 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -530,7 +530,7 @@ static inline void __free_one_page(struct page *page, * so it's less likely to be used soon and more likely to be merged * as a higher order page */ - if ((order < MAX_ORDER-1) && pfn_valid_within(page_to_pfn(buddy))) { + if ((order < MAX_ORDER-2) && pfn_valid_within(page_to_pfn(buddy))) { struct page *higher_page, *higher_buddy; combined_idx = __find_combined_index(page_idx, order); higher_page = page + combined_idx - page_idx; -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-18 11:33 ` Mel Gorman @ 2010-10-18 19:10 ` pacman 2010-10-18 21:10 ` Benjamin Herrenschmidt 2010-10-18 19:37 ` Andrew Morton 1 sibling, 1 reply; 24+ messages in thread From: pacman @ 2010-10-18 19:10 UTC (permalink / raw) To: Mel Gorman; +Cc: Andrew Morton, linux-mm, linux-kernel, linuxppc-dev Mel Gorman writes: > > A bit but I still don't know why it would cause corruption. Maybe this is still > a caching issue but the difference in timing between list_add and list_add_tail > is enough to hide the bug. It's also possible there are some registers > ioremapped after the memmap array and reading them is causing some > problem. I've been doing a lot more tests and I'm sure that 6dda9d55 is not really responsible. It just happens to provoke the bug in my particular setup. Whatever it is, it's very sensitive to small changes. At the end of free_all_bootmem, the free list for order 9 has 4 entries. Which one is at the head of the list depends on whether 6dda9d55 is applied or not. If page number 130048 is at the head of the list, it gets used fairly soon, and everything's fine. The alternative is that page number 64512 is at the head of the list, so it gets used fairly soon, and corruption occurs. > > Andrew, what is the right thing to do here? We could flail around looking > for explanations as to why the bug causes a user buffer corruption but never > get an answer or do we go with this patch, preferably before 2.6.36 releases? I've been flailing around quite a bit. Here's my latest result: Since I can view the corruption with md5sum /sbin/e2fsck, I know it's in a clean cached page. So I made an extra copy of /sbin/e2fsck, which won't be loaded into memory during boot. So now after the corruption happens, I can cmp -l /sbin/e2fsck good-e2fsck for a quick look at the changed bytes. Much easier than provoking a segfault under gdb. Then I got really creative and wrote a cmp replacement which mmaps the files and reports the physical addresses from /proc/self/pagemap of the pages that don't match. And the consistent result is that physical pages 64604 and 64609 (both in the range of the order=9 64512) have wrong contents. And the corruption is always a single word 128 bytes after the start of the page. Physical addresses 0x0fc5c080 and 0x0fc61080 are hit every time. The values of the corrupted words, observed in 5 consecutive boots, were: at 0fc5c080 at 0fc61080 ----------- ----------- c3540000 92510000 565c0000 23590000 c85b0000 97580000 d15f0000 9e5c0000 d95b0000 a8580000 The low 16 bits are all 0 and the upper 16 bits seem randomly distributed. But look at the differences: c3540000 - 92510000 = 31030000 565c0000 - 23590000 = 33030000 c85b0000 - 97580000 = 31030000 d15f0000 - 9e5c0000 = 33030000 d95b0000 - a8580000 = 31030000 This means something... but I don't know what. In a completely different method of investigation, I went back a few stable kernels, got 2.6.33.7 and applied 6dda9d55 to it, thinking that if 6dda9d55 only reveals a pre-existing bug, I could bisect it using 6dda9d55 as a bug-revealing assistant. The bug appeared when running 2.6.33.7 with 6dda9d55 applied. That was discouraging. >This patch fixes the problem by ensuring we are not reading a possibly >invalid location of memory. It's not clear why the read causes >corruption but one way or the other it is a buggy read. At least that part of the explanation is wrong. Where's the buggy read? The action taken by the 6dda9d55 version of __free_one_page looks perfectly legitimate to me. Page numbers: [129024 ] [130048 ] order=10 [129024 129536] [130048 130560] order=9 130048 is being freed. 130560 is not free. 129024 (the higher_buddy) is already free at order=10. So 130048 is being pushed to the tail of the free list, on the speculation that 130560 might soon be free and then the whole thing will form an order=11 free page, the only problem being that order=11 is too high so that later merge will never happen. It's not useful, and maybe not conceptually valid to say that 129024 is the buddy of 130048, but it is an existing page, and the only way it wouldn't be is if the total memory size was not a multiple of 1<<(MAX_ORDER-1) pages -- Alan Curry -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-18 19:10 ` pacman @ 2010-10-18 21:10 ` Benjamin Herrenschmidt 2010-10-18 21:33 ` pacman 0 siblings, 1 reply; 24+ messages in thread From: Benjamin Herrenschmidt @ 2010-10-18 21:10 UTC (permalink / raw) To: pacman; +Cc: Mel Gorman, linux-mm, Andrew Morton, linuxppc-dev, linux-kernel On Mon, 2010-10-18 at 14:10 -0500, pacman@kosh.dhis.org wrote: > I've been flailing around quite a bit. Here's my latest result: > > Since I can view the corruption with md5sum /sbin/e2fsck, I know it's in a > clean cached page. So I made an extra copy of /sbin/e2fsck, which won't be > loaded into memory during boot. So now after the corruption happens, I can > cmp -l /sbin/e2fsck good-e2fsck > for a quick look at the changed bytes. Much easier than provoking a segfault > under gdb. > > Then I got really creative and wrote a cmp replacement which mmaps the files > and reports the physical addresses from /proc/self/pagemap of the pages that > don't match. And the consistent result is that physical pages 64604 and 64609 > (both in the range of the order=9 64512) have wrong contents. And the > corruption is always a single word 128 bytes after the start of the page. > Physical addresses 0x0fc5c080 and 0x0fc61080 are hit every time. .../... You can do something fun... like a timer interrupt that peeks at those physical addresses from the linear mapping for example, and try to find out "when" they get set to the wrong value (you should observe the load from disk, then the corruption, unless they end up being loaded incorrectly (ie. dma coherency problem ?) ... >From there, you might be able to close onto the culprit a bit more, for example, try using the DABR register to set data access breakpoints shortly before the corruption spot. AFAIK, On those old 32-bit CPUs, you can set whether you want it to break on a real or a virtual address. You can also sprinkle tests for the page content through the code if that doesn't work to try to "close in" on the culprit (for example if it's a case of stray DMA, like a network driver bug or such). Cheers, Ben. > The values of the corrupted words, observed in 5 consecutive boots, were: > at 0fc5c080 at 0fc61080 > ----------- ----------- > c3540000 92510000 > 565c0000 23590000 > c85b0000 97580000 > d15f0000 9e5c0000 > d95b0000 a8580000 > > The low 16 bits are all 0 and the upper 16 bits seem randomly distributed. > But look at the differences: > > c3540000 - 92510000 = 31030000 > 565c0000 - 23590000 = 33030000 > c85b0000 - 97580000 = 31030000 > d15f0000 - 9e5c0000 = 33030000 > d95b0000 - a8580000 = 31030000 > > This means something... but I don't know what. > > In a completely different method of investigation, I went back a few stable > kernels, got 2.6.33.7 and applied 6dda9d55 to it, thinking that if 6dda9d55 > only reveals a pre-existing bug, I could bisect it using 6dda9d55 as a > bug-revealing assistant. The bug appeared when running 2.6.33.7 with 6dda9d55 > applied. That was discouraging. > > >This patch fixes the problem by ensuring we are not reading a possibly > >invalid location of memory. It's not clear why the read causes > >corruption but one way or the other it is a buggy read. > > At least that part of the explanation is wrong. Where's the buggy read? > The action taken by the 6dda9d55 version of __free_one_page looks perfectly > legitimate to me. Page numbers: > > [129024 ] [130048 ] order=10 > [129024 129536] [130048 130560] order=9 > > 130048 is being freed. 130560 is not free. 129024 (the higher_buddy) is > already free at order=10. So 130048 is being pushed to the tail of the free > list, on the speculation that 130560 might soon be free and then the whole > thing will form an order=11 free page, the only problem being that order=11 > is too high so that later merge will never happen. It's not useful, and maybe > not conceptually valid to say that 129024 is the buddy of 130048, but it is > an existing page, and the only way it wouldn't be is if the total memory size > was not a multiple of 1<<(MAX_ORDER-1) pages > > -- > Alan Curry > _______________________________________________ > Linuxppc-dev mailing list > Linuxppc-dev@lists.ozlabs.org > https://lists.ozlabs.org/listinfo/linuxppc-dev -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-18 21:10 ` Benjamin Herrenschmidt @ 2010-10-18 21:33 ` pacman 2010-10-19 10:16 ` Benjamin Herrenschmidt 0 siblings, 1 reply; 24+ messages in thread From: pacman @ 2010-10-18 21:33 UTC (permalink / raw) To: Benjamin Herrenschmidt Cc: Mel Gorman, linux-mm, Andrew Morton, linuxppc-dev, linux-kernel Benjamin Herrenschmidt writes: > > You can do something fun... like a timer interrupt that peeks at those > physical addresses from the linear mapping for example, and try to find > out "when" they get set to the wrong value (you should observe the load > from disk, then the corruption, unless they end up being loaded > incorrectly (ie. dma coherency problem ?) ... I'm headed toward something like that. Maybe not a timer, maybe a "check it every time the kernel is entered". But first I have to work out exactly when the disk load completes so I know when to start checking. > > >From there, you might be able to close onto the culprit a bit more, for > example, try using the DABR register to set data access breakpoints > shortly before the corruption spot. AFAIK, On those old 32-bit CPUs, you > can set whether you want it to break on a real or a virtual address. I thought of that, but as far as I can tell, this CPU doesn't have DABR. /proc/cpuinfo processor : 0 cpu : 7447/7457 clock : 999.999990MHz revision : 1.1 (pvr 8002 0101) bogomips : 66.66 timebase : 33333333 platform : CHRP model : Pegasos2 machine : CHRP Pegasos2 Memory : 512 MB My next thought was: right after the correct value appears in memory, unmap the page from the kernel and let it Oops when it tries to write there. Then I found out that the kernel is using BATs instead of page tables for its own view of memory. Booting with "nobats" completely changes the memory usage pattern (probably because it's allocating a lot of pages to hold PTEs that it didn't need before) > > You can also sprinkle tests for the page content through the code if > that doesn't work to try to "close in" on the culprit (for example if > it's a case of stray DMA, like a network driver bug or such). No network drivers are loaded when this happens. -- Alan Curry -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-18 21:33 ` pacman @ 2010-10-19 10:16 ` Benjamin Herrenschmidt 2010-10-19 18:10 ` pacman 0 siblings, 1 reply; 24+ messages in thread From: Benjamin Herrenschmidt @ 2010-10-19 10:16 UTC (permalink / raw) To: pacman; +Cc: Mel Gorman, linux-mm, Andrew Morton, linuxppc-dev, linux-kernel > > >From there, you might be able to close onto the culprit a bit more, for > > example, try using the DABR register to set data access breakpoints > > shortly before the corruption spot. AFAIK, On those old 32-bit CPUs, you > > can set whether you want it to break on a real or a virtual address. > > I thought of that, but as far as I can tell, this CPU doesn't have DABR. > /proc/cpuinfo > processor : 0 > cpu : 7447/7457 > clock : 999.999990MHz > revision : 1.1 (pvr 8002 0101) > bogomips : 66.66 > timebase : 33333333 > platform : CHRP > model : Pegasos2 > machine : CHRP Pegasos2 > Memory : 512 MB AFAIK, the 7447 is just a derivative of the 7450 design which -does- have a DABR ... Unless it's broken :-) > My next thought was: right after the correct value appears in memory, unmap > the page from the kernel and let it Oops when it tries to write there. Then I > found out that the kernel is using BATs instead of page tables for its own > view of memory. Booting with "nobats" completely changes the memory usage > pattern (probably because it's allocating a lot of pages to hold PTEs that it > didn't need before) Right. And that hides the problem I suppose ? > > You can also sprinkle tests for the page content through the code if > > that doesn't work to try to "close in" on the culprit (for example if > > it's a case of stray DMA, like a network driver bug or such). > > No network drivers are loaded when this happens. Ok. Cheers, Ben. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-19 10:16 ` Benjamin Herrenschmidt @ 2010-10-19 18:10 ` pacman 2010-10-19 20:47 ` Segher Boessenkool 2010-10-19 20:58 ` Benjamin Herrenschmidt 0 siblings, 2 replies; 24+ messages in thread From: pacman @ 2010-10-19 18:10 UTC (permalink / raw) To: Benjamin Herrenschmidt Cc: Mel Gorman, linux-mm, Andrew Morton, linuxppc-dev, linux-kernel Benjamin Herrenschmidt writes: > > > > I thought of that, but as far as I can tell, this CPU doesn't have DABR. > > AFAIK, the 7447 is just a derivative of the 7450 design which -does- > have a DABR ... Unless it's broken :-) Hmm. gdb resorts to single-stepping when I set a watchpoint while debugging some userspace program, which I assumed was caused by lack of hardware watchpoint support. But that's not important right now. I made a new discovery. During a test boot while looking at the usual symptom of a corrupted page cache, I run md5sum /sbin/e2fsck twice and got 2 different results, neither one of them correct. The third time, yet another different result. A few dozen more times, a few dozen more unique results. I had somehow managed to get a usable interactive shell while corruption was ongoing. So then I ran dd if=/dev/mem bs=4 count=1 skip=$((0xfc5c080/4)) | od -t x4 a few times very fast, plucking the first affected word directly out of memory by its physical address. The result: The low 16 bits are always zero as before. The high 16 bits are a counter, being incremented at about 1000Hz (as close as I could measure with a crude shell script. 1024Hz would also be within the margin of error). And it's little-endian. While I was watching this happen, there were only 5 or 6 userspace processes running, and 3 of them were shells. So I doubt that anything in userspace was doing it. It went on for a few minutes before I exited the interactive shell and allowed the boot to continue, while keeping an extra shell running on tty2 to continue making observations. It stopped incrementing almost immediately. So what type of driver, firmware, or hardware bug puts a 16-bit 1000Hz timer in memory, and does it in little-endian instead of the CPU's native byte order? And why does it stop doing it some time during the early init scripts, shortly after the root filesystem fsck? I have not yet attempted to repeat the experiment. If it is repeatable, I'll probe more deeply into those init scripts later. I'm looking hard at /etc/rcS.d/S11hwclock.sh -- Alan Curry -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-19 18:10 ` pacman @ 2010-10-19 20:47 ` Segher Boessenkool 2010-10-19 21:02 ` Benjamin Herrenschmidt 2010-10-19 20:58 ` Benjamin Herrenschmidt 1 sibling, 1 reply; 24+ messages in thread From: Segher Boessenkool @ 2010-10-19 20:47 UTC (permalink / raw) To: pacman Cc: Benjamin Herrenschmidt, Mel Gorman, linux-mm, Andrew Morton, linuxppc-dev, linux-kernel > I made a new discovery. And this nails it :-) > So then I ran > dd if=/dev/mem bs=4 count=1 skip=$((0xfc5c080/4)) | od -t x4 > a few times very fast, plucking the first affected word directly out of > memory by its physical address. The result: > > The low 16 bits are always zero as before. The high 16 bits are a counter, > being incremented at about 1000Hz (as close as I could measure with a > crude > shell script. 1024Hz would also be within the margin of error). And it's > little-endian. > So what type of driver, firmware, or hardware bug puts a 16-bit 1000Hz > timer > in memory, and does it in little-endian instead of the CPU's native byte > order? And why does it stop doing it some time during the early init > scripts, > shortly after the root filesystem fsck? It looks like it is the frame counter in an USB OHCI HCCA. 16-bit, 1kHz update, offset x'80 in a page. So either the kernel forgot to call quiesce on it, or the firmware doesn't implement that, or the firmware messed up some other way. Segher -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-19 20:47 ` Segher Boessenkool @ 2010-10-19 21:02 ` Benjamin Herrenschmidt 2010-10-20 3:23 ` pacman 0 siblings, 1 reply; 24+ messages in thread From: Benjamin Herrenschmidt @ 2010-10-19 21:02 UTC (permalink / raw) To: Segher Boessenkool Cc: pacman, Mel Gorman, linux-mm, Andrew Morton, linuxppc-dev, linux-kernel On Tue, 2010-10-19 at 22:47 +0200, Segher Boessenkool wrote: > > It looks like it is the frame counter in an USB OHCI HCCA. > 16-bit, 1kHz update, offset x'80 in a page. > > So either the kernel forgot to call quiesce on it, or the firmware > doesn't implement that, or the firmware messed up some other way. I vote for the FW being on crack. Wouldn't be the first time with Pegasos. It's an OHCI or an UHCI in there ? Can you try in prom_init.c changing the prom_close_stdin() function to also close "stdout" ? if (prom_getprop(_prom->chosen, "stdin", &val, sizeof(val)) > 0) call_prom("close", 1, 0, val); + if (prom_getprop(_prom->chosen, "stdout", &val, sizeof(val)) > 0) + call_prom("close", 1, 0, val); See if that makes a difference ? Last option would be to manually turn the thing off with MMIO in yet-another pegasos workaround in prom_init.c. Cheers, Ben. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-19 21:02 ` Benjamin Herrenschmidt @ 2010-10-20 3:23 ` pacman 2010-10-20 10:32 ` Benjamin Herrenschmidt 0 siblings, 1 reply; 24+ messages in thread From: pacman @ 2010-10-20 3:23 UTC (permalink / raw) To: Benjamin Herrenschmidt Cc: Segher Boessenkool, Mel Gorman, linux-mm, Andrew Morton, linuxppc-dev, linux-kernel Benjamin Herrenschmidt writes: > > On Tue, 2010-10-19 at 22:47 +0200, Segher Boessenkool wrote: > > > > It looks like it is the frame counter in an USB OHCI HCCA. > > 16-bit, 1kHz update, offset x'80 in a page. > > > > So either the kernel forgot to call quiesce on it, or the firmware > > doesn't implement that, or the firmware messed up some other way. > > I vote for the FW being on crack. Wouldn't be the first time with > Pegasos. > > It's an OHCI or an UHCI in there ? There's one of each... UHCI on the motherboard, OHCI on a card in a PCI expansion slot. They shipped the ODW with the extra controller on an expansion card since the on-board UHCI doesn't do USB2.0. And that OHCI controller does appear to be the culprit. The 2 affected addresses tick at 1000Hz until ohci-hcd is modprobe'd, then they stop. I think the mm people can consider this closed. 6dda9d55 didn't do anything but expose a problem which has been here all along. Will drop them from Cc list in any further messages. > > Can you try in prom_init.c changing the prom_close_stdin() function to > also close "stdout" ? > > if (prom_getprop(_prom->chosen, "stdin", &val, sizeof(val)) > 0) > call_prom("close", 1, 0, val); > + if (prom_getprop(_prom->chosen, "stdout", &val, sizeof(val)) > 0) > + call_prom("close", 1, 0, val); > > See if that makes a difference ? Huge difference. With no stdout to print to, the kernel seems to freeze up. Or at least it loses the console. The last message it prints is "Device tree struct 0x00933000 -> 0x00957000" then there's just nothing. I waited a while for the console to come on but it didn't. The diff fragment above applied inside prom_close_stdin, but there are some prom_printf calls after prom_close_stdin. Calling prom_printf after closing stdout sounds like it could be bad. If I moved it down below all the prom_printf's, it would be after the "quiesce" call. Would that be acceptable (or even interesting as an experiment)? Does a close need a quiesce after it? -- Alan Curry -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-20 3:23 ` pacman @ 2010-10-20 10:32 ` Benjamin Herrenschmidt 0 siblings, 0 replies; 24+ messages in thread From: Benjamin Herrenschmidt @ 2010-10-20 10:32 UTC (permalink / raw) To: pacman Cc: Segher Boessenkool, Mel Gorman, linux-mm, Andrew Morton, linuxppc-dev, linux-kernel On Tue, 2010-10-19 at 22:23 -0500, pacman@kosh.dhis.org wrote: > The diff fragment above applied inside prom_close_stdin, but there are > some > prom_printf calls after prom_close_stdin. Calling prom_printf after > closing > stdout sounds like it could be bad. If I moved it down below all the > prom_printf's, it would be after the "quiesce" call. Would that be > acceptable > (or even interesting as an experiment)? Does a close need a quiesce > after it? Just try :-) "quiesce" is something that afaik only apple ever implemented anyways. It uses hooks inside their OF to shut down all drivers that do bus master (among other HW sanitization tasks). Cheers, Ben. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-19 18:10 ` pacman 2010-10-19 20:47 ` Segher Boessenkool @ 2010-10-19 20:58 ` Benjamin Herrenschmidt 1 sibling, 0 replies; 24+ messages in thread From: Benjamin Herrenschmidt @ 2010-10-19 20:58 UTC (permalink / raw) To: pacman; +Cc: Mel Gorman, linux-mm, Andrew Morton, linuxppc-dev, linux-kernel On Tue, 2010-10-19 at 13:10 -0500, pacman@kosh.dhis.org wrote: > > So what type of driver, firmware, or hardware bug puts a 16-bit 1000Hz > timer > in memory, and does it in little-endian instead of the CPU's native > byte > order? And why does it stop doing it some time during the early init > scripts, > shortly after the root filesystem fsck? > > I have not yet attempted to repeat the experiment. If it is > repeatable, I'll > probe more deeply into those init scripts later. I'm looking hard at > /etc/rcS.d/S11hwclock.sh Stinks of USB... Ben. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-18 11:33 ` Mel Gorman 2010-10-18 19:10 ` pacman @ 2010-10-18 19:37 ` Andrew Morton 2010-10-18 21:02 ` Benjamin Herrenschmidt 2010-10-18 21:55 ` Thomas Gleixner 1 sibling, 2 replies; 24+ messages in thread From: Andrew Morton @ 2010-10-18 19:37 UTC (permalink / raw) To: Mel Gorman; +Cc: pacman, linux-mm, linux-kernel, linuxppc-dev On Mon, 18 Oct 2010 12:33:31 +0100 Mel Gorman <mel@csn.ul.ie> wrote: > A bit but I still don't know why it would cause corruption. Maybe this is still > a caching issue but the difference in timing between list_add and list_add_tail > is enough to hide the bug. It's also possible there are some registers > ioremapped after the memmap array and reading them is causing some > problem. > > Andrew, what is the right thing to do here? We could flail around looking > for explanations as to why the bug causes a user buffer corruption but never > get an answer or do we go with this patch, preferably before 2.6.36 releases? Well, you've spotted a bug so I'd say we fix it asap. It's a bit of a shame that we lose the only known way of reproducing a different bug, but presumably that will come back and bite someone else one day, and we'll fix it then :( -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-18 19:37 ` Andrew Morton @ 2010-10-18 21:02 ` Benjamin Herrenschmidt 2010-10-18 21:55 ` Thomas Gleixner 1 sibling, 0 replies; 24+ messages in thread From: Benjamin Herrenschmidt @ 2010-10-18 21:02 UTC (permalink / raw) To: Andrew Morton; +Cc: Mel Gorman, linux-mm, pacman, linuxppc-dev, linux-kernel On Mon, 2010-10-18 at 12:37 -0700, Andrew Morton wrote: > Well, you've spotted a bug so I'd say we fix it asap. > > It's a bit of a shame that we lose the only known way of reproducing a > different bug, but presumably that will come back and bite someone > else > one day, and we'll fix it then :( Well, I can always revert that and run some experiments here, provided I can reproduce the problem at all ... Cheers, Ben. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-18 19:37 ` Andrew Morton 2010-10-18 21:02 ` Benjamin Herrenschmidt @ 2010-10-18 21:55 ` Thomas Gleixner 2010-10-19 16:24 ` Helmut Grohne 1 sibling, 1 reply; 24+ messages in thread From: Thomas Gleixner @ 2010-10-18 21:55 UTC (permalink / raw) To: Andrew Morton Cc: Mel Gorman, pacman, linux-mm, LKML, linuxppc-dev, Helmut Grohne On Mon, 18 Oct 2010, Andrew Morton wrote: > On Mon, 18 Oct 2010 12:33:31 +0100 > Mel Gorman <mel@csn.ul.ie> wrote: > > > A bit but I still don't know why it would cause corruption. Maybe this is still > > a caching issue but the difference in timing between list_add and list_add_tail > > is enough to hide the bug. It's also possible there are some registers > > ioremapped after the memmap array and reading them is causing some > > problem. > > > > Andrew, what is the right thing to do here? We could flail around looking > > for explanations as to why the bug causes a user buffer corruption but never > > get an answer or do we go with this patch, preferably before 2.6.36 releases? > > Well, you've spotted a bug so I'd say we fix it asap. > > It's a bit of a shame that we lose the only known way of reproducing a > different bug, but presumably that will come back and bite someone else > one day, and we'll fix it then :( I might be completely one off as usual, but this thing reminds me of a bug I stared at yesterday night: http://permalink.gmane.org/gmane.linux.kernel/1049605 Reporter Cc'ed Thanks, tglx -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-18 21:55 ` Thomas Gleixner @ 2010-10-19 16:24 ` Helmut Grohne 2010-10-19 16:42 ` Thomas Gleixner 0 siblings, 1 reply; 24+ messages in thread From: Helmut Grohne @ 2010-10-19 16:24 UTC (permalink / raw) To: Thomas Gleixner Cc: Andrew Morton, Mel Gorman, pacman, linux-mm, LKML, linuxppc-dev On Mon, Oct 18, 2010 at 11:55:44PM +0200, Thomas Gleixner wrote: > I might be completely one off as usual, but this thing reminds me of a > bug I stared at yesterday night: This problem is completely unrelated. My problem was caused by using binutils-gold. Helmut -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-19 16:24 ` Helmut Grohne @ 2010-10-19 16:42 ` Thomas Gleixner 0 siblings, 0 replies; 24+ messages in thread From: Thomas Gleixner @ 2010-10-19 16:42 UTC (permalink / raw) To: Helmut Grohne Cc: Andrew Morton, Mel Gorman, pacman, linux-mm, LKML, linuxppc-dev On Tue, 19 Oct 2010, Helmut Grohne wrote: > On Mon, Oct 18, 2010 at 11:55:44PM +0200, Thomas Gleixner wrote: > > I might be completely one off as usual, but this thing reminds me of a > > bug I stared at yesterday night: > > This problem is completely unrelated. My problem was caused by using > binutils-gold. Ok, thanks for the update. One thing less to worry about :) Thanks, tglx -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55 2010-10-13 14:40 ` Mel Gorman 2010-10-13 17:52 ` pacman @ 2010-10-18 20:59 ` Benjamin Herrenschmidt 1 sibling, 0 replies; 24+ messages in thread From: Benjamin Herrenschmidt @ 2010-10-18 20:59 UTC (permalink / raw) To: Mel Gorman Cc: Andrew Morton, linuxppc-dev, linux-kernel, linux-mm, pacman, KOSAKI Motohiro, Christoph Lameter, Yinghai Lu On Wed, 2010-10-13 at 15:40 +0100, Mel Gorman wrote: > > This is somewhat contrived but I can see how it might happen even on one > CPU particularly if the L1 cache is virtual and is loose about checking > physical tags. > > > How sensitive/vulnerable is PPC32 to such things? > > > > I can not tell you specifically but if the above scenario is in any way > plausible, I believe it would depend on what sort of L1 cache the CPU > has. Maybe this particular version has a virtual cache with no physical > tagging and is depending on the OS not to make virtual aliasing mistakes. Nah, ppc doesn't have problems with cache aliases, it all looks physically tagged to the programmer (tho there's subtleties but none that explains the reported behaviour). Looks like real memory corruption to me. Cheers, Ben. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2010-10-20 10:36 UTC | newest] Thread overview: 24+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2010-10-09 9:57 PROBLEM: memory corrupting bug, bisected to 6dda9d55 pacman 2010-10-11 12:52 ` Christoph Lameter 2010-10-11 14:30 ` Mel Gorman 2010-10-11 20:35 ` pacman 2010-10-11 21:00 ` Andrew Morton 2010-10-13 14:40 ` Mel Gorman 2010-10-13 17:52 ` pacman 2010-10-18 11:33 ` Mel Gorman 2010-10-18 19:10 ` pacman 2010-10-18 21:10 ` Benjamin Herrenschmidt 2010-10-18 21:33 ` pacman 2010-10-19 10:16 ` Benjamin Herrenschmidt 2010-10-19 18:10 ` pacman 2010-10-19 20:47 ` Segher Boessenkool 2010-10-19 21:02 ` Benjamin Herrenschmidt 2010-10-20 3:23 ` pacman 2010-10-20 10:32 ` Benjamin Herrenschmidt 2010-10-19 20:58 ` Benjamin Herrenschmidt 2010-10-18 19:37 ` Andrew Morton 2010-10-18 21:02 ` Benjamin Herrenschmidt 2010-10-18 21:55 ` Thomas Gleixner 2010-10-19 16:24 ` Helmut Grohne 2010-10-19 16:42 ` Thomas Gleixner 2010-10-18 20:59 ` Benjamin Herrenschmidt
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).