* [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel @ 2012-06-30 11:41 Zdenek Kabelac 2012-06-30 19:55 ` Hugh Dickins 0 siblings, 1 reply; 13+ messages in thread From: Zdenek Kabelac @ 2012-06-30 11:41 UTC (permalink / raw) To: LVM general discussion and development, Hugh Dickins, amwang, Alasdair G Kergon [-- Attachment #1: Type: text/plain, Size: 6223 bytes --] Hi When I've used 3.5-rc kernels - I've noticed kernel deadlocks. Ooops log included. After some experimenting - reliable way to hit this oops is to run lvm test suite for 10 minutes. Since 3.5 merge window does not included anything related to this oops I've went for bisect. Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17 mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE My HW - t61, 4GB Zdenek BUG: unable to handle kernel NULL pointer dereference at 0000000000000178 IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] PGD 0 Oops: 0000 [#1] PREEMPT SMP CPU 1 Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 md_mod reiserfs xfs dm_mirror dm_region_hash dm_log dm_snapshot ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc ipv6 snd_hda_codec_analog arc4 hid_generic iwl3945 iwlegacy microcode psmouse serio_raw mac80211 i2c_i801 i2c_core snd_hda_intel snd_hda_codec cfg80211 lpc_ich mfd_core snd_pcm snd_page_alloc e1000e snd_timer wmi thinkpad_acpi nvram snd soundcore evdev usbhid hid binfmt_misc loop vhost_net tun kvm_intel nfsd kvm exportfs nfs_acl auth_rpcgss lockd sunrpc dm_mod autofs4 pcmcia sr_mod cdrom sdhci_pci sdhci mmc_core yenta_socket ehci_hcd uhci_hcd usbcore usb_common video backlight Pid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO 6464CTO/6464CTO RIP: 0010:[<ffffffffa031204d>] [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] RSP: 0018:ffff880126b77c68 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607 RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40 RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400 R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task ffff8801344ca310) Stack: ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0 ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000 0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0 Call Trace: [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror] [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror] [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod] [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod] [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod] [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod] [<ffffffff81060065>] process_one_work+0x1a5/0x6b0 [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0 [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80 [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod] [<ffffffff810608ee>] worker_thread+0x12e/0x2d0 [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210 [<ffffffff81069cf2>] kthread+0xa2/0xb0 [<ffffffff81533334>] kernel_thread_helper+0x4/0x10 [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120 [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60 [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0 [<ffffffff81533330>] ? gs_change+0xb/0xb Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8 c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 RIP [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] RSP <ffff880126b77c68> CR2: 0000000000000178 # bad: [cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2 # good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4 git bisect start 'v3.5-rc2' 'v3.4' # good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge' of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux git bisect good 3813d4024a75562baf77d3907fb6afbf8f9c8232 # good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new generic strnlen_user() function git bisect good 5723aa993d83803157c22327e90cd59e3dcbe879 # bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch 'for-3.5/drivers' of git://git.kernel.dk/linux-block git bisect bad a70f35af4e49f87ba4b6c4b30220fbb66cd74af6 # good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1' of git://git.linux-nfs.org/projects/trondmy/linux-nfs git bisect good 53f2c4a8fd882009a2a75c5b72d6898c0808616e # bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' of git://git.linaro.org/people/rmk/linux-arm git bisect bad 227d1e4319ffd8729781941d92f4ae4d85beecd9 # bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dump PG_compound_lock page flag git bisect bad be9cd873e2a706a688e37224d48e135efd8c0d26 # good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should not be user visible git bisect good 1cb3642a68c983ada0f4090a4dac1d70a96126ca # good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729] mm/vmstat.c: remove debug fs entries on failure of file creation and made extfrag_debug_root dentry local git bisect good bde8bd8a1d5242589ddcaef8e017b48b207c4729 # good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove redundant offset check when finally freeing bootmem git bisect good 549381e19cc7874bb15f0e4760c1004d4fe28d8a # good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if mapping excludes its zone git bisect good bde05d1ccd512696b09db9dd2e5f33ad19152605 # bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: remove truncate_range git bisect bad 17cf28afea2a1112f240a3a2da8af883be024811 # good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing when writing git bisect good ec9516fbc5fa814014991e1ae7f8860127122105 [-- Attachment #2: CONFIG --] [-- Type: text/plain, Size: 33663 bytes --] CONFIG_64BIT=y CONFIG_X86_64=y CONFIG_X86=y CONFIG_INSTRUCTION_DECODER=y CONFIG_OUTPUT_FORMAT="elf64-x86-64" CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig" CONFIG_LOCKDEP_SUPPORT=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_HAVE_LATENCYTOP_SUPPORT=y CONFIG_MMU=y CONFIG_NEED_DMA_MAP_STATE=y CONFIG_NEED_SG_DMA_LENGTH=y CONFIG_GENERIC_ISA_DMA=y CONFIG_GENERIC_BUG=y CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y CONFIG_GENERIC_HWEIGHT=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y CONFIG_RWSEM_XCHGADD_ALGORITHM=y CONFIG_GENERIC_CALIBRATE_DELAY=y CONFIG_ARCH_HAS_CPU_RELAX=y CONFIG_ARCH_HAS_DEFAULT_IDLE=y CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y CONFIG_ARCH_HAS_CPU_AUTOPROBE=y CONFIG_HAVE_SETUP_PER_CPU_AREA=y CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK=y CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK=y CONFIG_ARCH_HIBERNATION_POSSIBLE=y CONFIG_ARCH_SUSPEND_POSSIBLE=y CONFIG_ZONE_DMA32=y CONFIG_AUDIT_ARCH=y CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y CONFIG_X86_64_SMP=y CONFIG_X86_HT=y CONFIG_ARCH_HWEIGHT_CFLAGS="-fcall-saved-rdi -fcall-saved-rsi -fcall-saved-rdx -fcall-saved-rcx -fcall-saved-r8 -fcall-saved-r9 -fcall-saved-r10 -fcall-saved-r11" CONFIG_ARCH_CPU_PROBE_RELEASE=y CONFIG_ARCH_SUPPORTS_UPROBES=y CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config" CONFIG_HAVE_IRQ_WORK=y CONFIG_IRQ_WORK=y CONFIG_BUILDTIME_EXTABLE_SORT=y CONFIG_EXPERIMENTAL=y CONFIG_INIT_ENV_ARG_LIMIT=32 CONFIG_CROSS_COMPILE="" CONFIG_LOCALVERSION="" CONFIG_LOCALVERSION_AUTO=y CONFIG_HAVE_KERNEL_GZIP=y CONFIG_HAVE_KERNEL_BZIP2=y CONFIG_HAVE_KERNEL_LZMA=y CONFIG_HAVE_KERNEL_XZ=y CONFIG_HAVE_KERNEL_LZO=y CONFIG_KERNEL_LZMA=y CONFIG_DEFAULT_HOSTNAME="(none)" CONFIG_SWAP=y CONFIG_SYSVIPC=y CONFIG_SYSVIPC_SYSCTL=y CONFIG_POSIX_MQUEUE=y CONFIG_POSIX_MQUEUE_SYSCTL=y CONFIG_BSD_PROCESS_ACCT=y CONFIG_TASKSTATS=y CONFIG_TASK_DELAY_ACCT=y CONFIG_TASK_XACCT=y CONFIG_TASK_IO_ACCOUNTING=y CONFIG_AUDIT=y CONFIG_AUDITSYSCALL=y CONFIG_AUDIT_WATCH=y CONFIG_AUDIT_TREE=y CONFIG_HAVE_GENERIC_HARDIRQS=y CONFIG_GENERIC_HARDIRQS=y CONFIG_GENERIC_IRQ_PROBE=y CONFIG_GENERIC_IRQ_SHOW=y CONFIG_GENERIC_PENDING_IRQ=y CONFIG_IRQ_FORCED_THREADING=y CONFIG_SPARSE_IRQ=y CONFIG_CLOCKSOURCE_WATCHDOG=y CONFIG_ARCH_CLOCKSOURCE_DATA=y CONFIG_GENERIC_TIME_VSYSCALL=y CONFIG_GENERIC_CLOCKEVENTS=y CONFIG_GENERIC_CLOCKEVENTS_BUILD=y CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y CONFIG_GENERIC_CLOCKEVENTS_MIN_ADJUST=y CONFIG_GENERIC_CMOS_UPDATE=y CONFIG_TICK_ONESHOT=y CONFIG_NO_HZ=y CONFIG_HIGH_RES_TIMERS=y CONFIG_TREE_PREEMPT_RCU=y CONFIG_PREEMPT_RCU=y CONFIG_RCU_FANOUT=64 CONFIG_RCU_FANOUT_LEAF=16 CONFIG_RCU_FAST_NO_HZ=y CONFIG_TREE_RCU_TRACE=y CONFIG_IKCONFIG=m CONFIG_IKCONFIG_PROC=y CONFIG_LOG_BUF_SHIFT=17 CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y CONFIG_CGROUPS=y CONFIG_CGROUP_FREEZER=y CONFIG_CGROUP_DEVICE=y CONFIG_CPUSETS=y CONFIG_PROC_PID_CPUSET=y CONFIG_CGROUP_CPUACCT=y CONFIG_RESOURCE_COUNTERS=y CONFIG_CGROUP_SCHED=y CONFIG_FAIR_GROUP_SCHED=y CONFIG_CFS_BANDWIDTH=y CONFIG_RT_GROUP_SCHED=y CONFIG_BLK_CGROUP=y CONFIG_NAMESPACES=y CONFIG_UTS_NS=y CONFIG_IPC_NS=y CONFIG_PID_NS=y CONFIG_NET_NS=y CONFIG_SCHED_AUTOGROUP=y CONFIG_SYSFS_DEPRECATED=y CONFIG_RELAY=y CONFIG_BLK_DEV_INITRD=y CONFIG_INITRAMFS_SOURCE="" CONFIG_RD_GZIP=y CONFIG_RD_BZIP2=y CONFIG_RD_LZMA=y CONFIG_RD_XZ=y CONFIG_RD_LZO=y CONFIG_SYSCTL=y CONFIG_ANON_INODES=y CONFIG_UID16=y CONFIG_KALLSYMS=y CONFIG_KALLSYMS_ALL=y CONFIG_HOTPLUG=y CONFIG_PRINTK=y CONFIG_BUG=y CONFIG_ELF_CORE=y CONFIG_PCSPKR_PLATFORM=y CONFIG_HAVE_PCSPKR_PLATFORM=y CONFIG_BASE_FULL=y CONFIG_FUTEX=y CONFIG_EPOLL=y CONFIG_SIGNALFD=y CONFIG_TIMERFD=y CONFIG_EVENTFD=y CONFIG_SHMEM=y CONFIG_AIO=y CONFIG_HAVE_PERF_EVENTS=y CONFIG_PERF_EVENTS=y CONFIG_VM_EVENT_COUNTERS=y CONFIG_PCI_QUIRKS=y CONFIG_SLUB_DEBUG=y CONFIG_SLUB=y CONFIG_PROFILING=y CONFIG_TRACEPOINTS=y CONFIG_OPROFILE=m CONFIG_HAVE_OPROFILE=y CONFIG_OPROFILE_NMI_TIMER=y CONFIG_KPROBES=y CONFIG_JUMP_LABEL=y CONFIG_UPROBES=y CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y CONFIG_KRETPROBES=y CONFIG_USER_RETURN_NOTIFIER=y CONFIG_HAVE_IOREMAP_PROT=y CONFIG_HAVE_KPROBES=y CONFIG_HAVE_KRETPROBES=y CONFIG_HAVE_OPTPROBES=y CONFIG_HAVE_ARCH_TRACEHOOK=y CONFIG_HAVE_DMA_ATTRS=y CONFIG_USE_GENERIC_SMP_HELPERS=y CONFIG_GENERIC_SMP_IDLE_THREAD=y CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y CONFIG_HAVE_DMA_API_DEBUG=y CONFIG_HAVE_HW_BREAKPOINT=y CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y CONFIG_HAVE_USER_RETURN_NOTIFIER=y CONFIG_HAVE_PERF_EVENTS_NMI=y CONFIG_HAVE_ARCH_JUMP_LABEL=y CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y CONFIG_HAVE_CMPXCHG_LOCAL=y CONFIG_HAVE_CMPXCHG_DOUBLE=y CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y CONFIG_HAVE_ARCH_SECCOMP_FILTER=y CONFIG_SLABINFO=y CONFIG_RT_MUTEXES=y CONFIG_BASE_SMALL=0 CONFIG_MODULES=y CONFIG_MODULE_UNLOAD=y CONFIG_MODULE_FORCE_UNLOAD=y CONFIG_MODULE_SRCVERSION_ALL=y CONFIG_STOP_MACHINE=y CONFIG_BLOCK=y CONFIG_BLK_DEV_BSG=y CONFIG_BLK_DEV_INTEGRITY=y CONFIG_BLK_DEV_THROTTLING=y CONFIG_PARTITION_ADVANCED=y CONFIG_AMIGA_PARTITION=y CONFIG_MSDOS_PARTITION=y CONFIG_LDM_PARTITION=y CONFIG_EFI_PARTITION=y CONFIG_BLOCK_COMPAT=y CONFIG_IOSCHED_NOOP=y CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_CFQ=m CONFIG_DEFAULT_DEADLINE=y CONFIG_DEFAULT_IOSCHED="deadline" CONFIG_PREEMPT_NOTIFIERS=y CONFIG_PADATA=y CONFIG_UNINLINE_SPIN_UNLOCK=y CONFIG_FREEZER=y CONFIG_ZONE_DMA=y CONFIG_SMP=y CONFIG_X86_MPPARSE=y CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y CONFIG_SCHED_OMIT_FRAME_POINTER=y CONFIG_NO_BOOTMEM=y CONFIG_MEMTEST=y CONFIG_MCORE2=y CONFIG_X86_INTERNODE_CACHE_SHIFT=6 CONFIG_X86_CMPXCHG=y CONFIG_X86_L1_CACHE_SHIFT=6 CONFIG_X86_XADD=y CONFIG_X86_WP_WORKS_OK=y CONFIG_X86_INTEL_USERCOPY=y CONFIG_X86_USE_PPRO_CHECKSUM=y CONFIG_X86_P6_NOP=y CONFIG_X86_TSC=y CONFIG_X86_CMPXCHG64=y CONFIG_X86_CMOV=y CONFIG_X86_MINIMUM_CPU_FAMILY=64 CONFIG_X86_DEBUGCTLMSR=y CONFIG_CPU_SUP_INTEL=y CONFIG_CPU_SUP_AMD=y CONFIG_CPU_SUP_CENTAUR=y CONFIG_HPET_TIMER=y CONFIG_HPET_EMULATE_RTC=y CONFIG_DMI=y CONFIG_GART_IOMMU=y CONFIG_SWIOTLB=y CONFIG_IOMMU_HELPER=y CONFIG_NR_CPUS=2 CONFIG_PREEMPT=y CONFIG_PREEMPT_COUNT=y CONFIG_X86_LOCAL_APIC=y CONFIG_X86_IO_APIC=y CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y CONFIG_X86_MCE=y CONFIG_X86_MCE_INTEL=y CONFIG_X86_MCE_AMD=y CONFIG_X86_MCE_THRESHOLD=y CONFIG_X86_THERMAL_VECTOR=y CONFIG_MICROCODE=m CONFIG_MICROCODE_INTEL=y CONFIG_MICROCODE_AMD=y CONFIG_MICROCODE_OLD_INTERFACE=y CONFIG_X86_MSR=m CONFIG_X86_CPUID=m CONFIG_ARCH_PHYS_ADDR_T_64BIT=y CONFIG_ARCH_DMA_ADDR_T_64BIT=y CONFIG_DIRECT_GBPAGES=y CONFIG_ARCH_SPARSEMEM_ENABLE=y CONFIG_ARCH_SPARSEMEM_DEFAULT=y CONFIG_ARCH_SELECT_MEMORY_MODEL=y CONFIG_ARCH_PROC_KCORE_TEXT=y CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000 CONFIG_SELECT_MEMORY_MODEL=y CONFIG_SPARSEMEM_MANUAL=y CONFIG_SPARSEMEM=y CONFIG_HAVE_MEMORY_PRESENT=y CONFIG_SPARSEMEM_EXTREME=y CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y CONFIG_SPARSEMEM_ALLOC_MEM_MAP_TOGETHER=y CONFIG_SPARSEMEM_VMEMMAP=y CONFIG_HAVE_MEMBLOCK=y CONFIG_HAVE_MEMBLOCK_NODE_MAP=y CONFIG_ARCH_DISCARD_MEMBLOCK=y CONFIG_PAGEFLAGS_EXTENDED=y CONFIG_SPLIT_PTLOCK_CPUS=999999 CONFIG_COMPACTION=y CONFIG_MIGRATION=y CONFIG_PHYS_ADDR_T_64BIT=y CONFIG_ZONE_DMA_FLAG=1 CONFIG_BOUNCE=y CONFIG_VIRT_TO_BUS=y CONFIG_MMU_NOTIFIER=y CONFIG_KSM=y CONFIG_DEFAULT_MMAP_MIN_ADDR=4096 CONFIG_ARCH_SUPPORTS_MEMORY_FAILURE=y CONFIG_TRANSPARENT_HUGEPAGE=y CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y CONFIG_CROSS_MEMORY_ATTACH=y CONFIG_CLEANCACHE=y CONFIG_FRONTSWAP=y CONFIG_X86_CHECK_BIOS_CORRUPTION=y CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y CONFIG_X86_RESERVE_LOW=64 CONFIG_MTRR=y CONFIG_MTRR_SANITIZER=y CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1 CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1 CONFIG_X86_PAT=y CONFIG_ARCH_USES_PG_UNCACHED=y CONFIG_ARCH_RANDOM=y CONFIG_EFI=y CONFIG_HZ_300=y CONFIG_HZ=300 CONFIG_SCHED_HRTICK=y CONFIG_KEXEC=y CONFIG_KEXEC_JUMP=y CONFIG_PHYSICAL_START=0x1000000 CONFIG_PHYSICAL_ALIGN=0x1000000 CONFIG_HOTPLUG_CPU=y CONFIG_CMDLINE_BOOL=y CONFIG_CMDLINE="" CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y CONFIG_ARCH_HIBERNATION_HEADER=y CONFIG_SUSPEND=y CONFIG_SUSPEND_FREEZER=y CONFIG_HIBERNATE_CALLBACKS=y CONFIG_HIBERNATION=y CONFIG_PM_STD_PARTITION="" CONFIG_PM_SLEEP=y CONFIG_PM_SLEEP_SMP=y CONFIG_PM_WAKELOCKS=y CONFIG_PM_WAKELOCKS_LIMIT=100 CONFIG_PM_WAKELOCKS_GC=y CONFIG_PM_RUNTIME=y CONFIG_PM=y CONFIG_PM_DEBUG=y CONFIG_PM_ADVANCED_DEBUG=y CONFIG_CAN_PM_TRACE=y CONFIG_PM_TRACE=y CONFIG_PM_TRACE_RTC=y CONFIG_ACPI=y CONFIG_ACPI_SLEEP=y CONFIG_ACPI_PROCFS=y CONFIG_ACPI_PROCFS_POWER=y CONFIG_ACPI_AC=y CONFIG_ACPI_BATTERY=y CONFIG_ACPI_BUTTON=y CONFIG_ACPI_VIDEO=m CONFIG_ACPI_FAN=y CONFIG_ACPI_DOCK=y CONFIG_ACPI_PROCESSOR=y CONFIG_ACPI_HOTPLUG_CPU=y CONFIG_ACPI_PROCESSOR_AGGREGATOR=m CONFIG_ACPI_THERMAL=y CONFIG_ACPI_BLACKLIST_YEAR=0 CONFIG_ACPI_DEBUG=y CONFIG_ACPI_PCI_SLOT=y CONFIG_X86_PM_TIMER=y CONFIG_ACPI_CONTAINER=y CONFIG_ACPI_CUSTOM_METHOD=m CONFIG_CPU_FREQ=y CONFIG_CPU_FREQ_TABLE=y CONFIG_CPU_FREQ_STAT=y CONFIG_CPU_FREQ_STAT_DETAILS=y CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND=y CONFIG_CPU_FREQ_GOV_PERFORMANCE=y CONFIG_CPU_FREQ_GOV_POWERSAVE=y CONFIG_CPU_FREQ_GOV_USERSPACE=y CONFIG_CPU_FREQ_GOV_ONDEMAND=y CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y CONFIG_X86_PCC_CPUFREQ=m CONFIG_X86_ACPI_CPUFREQ=y CONFIG_X86_P4_CLOCKMOD=m CONFIG_X86_SPEEDSTEP_LIB=m CONFIG_CPU_IDLE=y CONFIG_CPU_IDLE_GOV_LADDER=y CONFIG_CPU_IDLE_GOV_MENU=y CONFIG_PCI=y CONFIG_PCI_DIRECT=y CONFIG_PCI_MMCONFIG=y CONFIG_PCI_DOMAINS=y CONFIG_PCIEPORTBUS=y CONFIG_PCIEAER=y CONFIG_PCIEASPM=y CONFIG_PCIEASPM_DEFAULT=y CONFIG_PCIE_PME=y CONFIG_ARCH_SUPPORTS_MSI=y CONFIG_PCI_MSI=y CONFIG_PCI_STUB=y CONFIG_HT_IRQ=y CONFIG_PCI_ATS=y CONFIG_PCI_IOV=y CONFIG_PCI_PRI=y CONFIG_PCI_PASID=y CONFIG_PCI_IOAPIC=y CONFIG_PCI_LABEL=y CONFIG_ISA_DMA_API=y CONFIG_AMD_NB=y CONFIG_PCCARD=y CONFIG_PCMCIA=m CONFIG_PCMCIA_LOAD_CIS=y CONFIG_CARDBUS=y CONFIG_YENTA=m CONFIG_YENTA_O2=y CONFIG_YENTA_RICOH=y CONFIG_YENTA_TI=y CONFIG_YENTA_ENE_TUNE=y CONFIG_YENTA_TOSHIBA=y CONFIG_PCCARD_NONSTATIC=y CONFIG_HOTPLUG_PCI=y CONFIG_BINFMT_ELF=y CONFIG_COMPAT_BINFMT_ELF=y CONFIG_ARCH_BINFMT_ELF_RANDOMIZE_PIE=y CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS=y CONFIG_BINFMT_MISC=m CONFIG_IA32_EMULATION=y CONFIG_IA32_AOUT=m CONFIG_X86_X32=y CONFIG_COMPAT=y CONFIG_COMPAT_FOR_U64_ALIGNMENT=y CONFIG_SYSVIPC_COMPAT=y CONFIG_KEYS_COMPAT=y CONFIG_HAVE_TEXT_POKE_SMP=y CONFIG_X86_DEV_DMA_OPS=y CONFIG_NET=y CONFIG_COMPAT_NETLINK_MESSAGES=y CONFIG_PACKET=y CONFIG_UNIX=y CONFIG_UNIX_DIAG=m CONFIG_XFRM=y CONFIG_INET=y CONFIG_IP_MULTICAST=y CONFIG_IP_ADVANCED_ROUTER=y CONFIG_IP_PNP=y CONFIG_IP_PNP_DHCP=y CONFIG_IP_PNP_BOOTP=y CONFIG_IP_MROUTE=y CONFIG_SYN_COOKIES=y CONFIG_INET_TUNNEL=m CONFIG_TCP_CONG_ADVANCED=y CONFIG_TCP_CONG_BIC=m CONFIG_TCP_CONG_CUBIC=y CONFIG_TCP_CONG_WESTWOOD=m CONFIG_TCP_CONG_HTCP=m CONFIG_DEFAULT_CUBIC=y CONFIG_DEFAULT_TCP_CONG="cubic" CONFIG_IPV6=m CONFIG_INET6_XFRM_MODE_TRANSPORT=m CONFIG_INET6_XFRM_MODE_TUNNEL=m CONFIG_INET6_XFRM_MODE_BEET=m CONFIG_IPV6_SIT=m CONFIG_IPV6_NDISC_NODETYPE=y CONFIG_NETWORK_SECMARK=y CONFIG_NETFILTER=y CONFIG_NETFILTER_ADVANCED=y CONFIG_BRIDGE_NETFILTER=y CONFIG_NETFILTER_NETLINK=m CONFIG_NETFILTER_NETLINK_QUEUE=m CONFIG_NETFILTER_NETLINK_LOG=m CONFIG_NF_CONNTRACK=m CONFIG_NF_CONNTRACK_MARK=y CONFIG_NF_CONNTRACK_SECMARK=y CONFIG_NF_CONNTRACK_PROCFS=y CONFIG_NF_CONNTRACK_EVENTS=y CONFIG_NF_CONNTRACK_FTP=m CONFIG_NF_CT_NETLINK=m CONFIG_NETFILTER_XTABLES=m CONFIG_NETFILTER_XT_MARK=m CONFIG_NETFILTER_XT_CONNMARK=m CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m CONFIG_NETFILTER_XT_TARGET_CONNMARK=m CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=m CONFIG_NETFILTER_XT_TARGET_DSCP=m CONFIG_NETFILTER_XT_TARGET_HL=m CONFIG_NETFILTER_XT_TARGET_LED=m CONFIG_NETFILTER_XT_TARGET_MARK=m CONFIG_NETFILTER_XT_TARGET_NFLOG=m CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m CONFIG_NETFILTER_XT_TARGET_TCPMSS=m CONFIG_NETFILTER_XT_MATCH_CONNBYTES=m CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=m CONFIG_NETFILTER_XT_MATCH_CONNMARK=m CONFIG_NETFILTER_XT_MATCH_CONNTRACK=m CONFIG_NETFILTER_XT_MATCH_DSCP=m CONFIG_NETFILTER_XT_MATCH_ECN=m CONFIG_NETFILTER_XT_MATCH_HL=m CONFIG_NETFILTER_XT_MATCH_IPRANGE=m CONFIG_NETFILTER_XT_MATCH_MARK=m CONFIG_NETFILTER_XT_MATCH_MULTIPORT=m CONFIG_NETFILTER_XT_MATCH_OSF=m CONFIG_NETFILTER_XT_MATCH_OWNER=m CONFIG_NETFILTER_XT_MATCH_POLICY=m CONFIG_NETFILTER_XT_MATCH_PHYSDEV=m CONFIG_NETFILTER_XT_MATCH_PKTTYPE=m CONFIG_NETFILTER_XT_MATCH_QUOTA=m CONFIG_NETFILTER_XT_MATCH_SCTP=m CONFIG_NETFILTER_XT_MATCH_STATE=m CONFIG_NETFILTER_XT_MATCH_STATISTIC=m CONFIG_NETFILTER_XT_MATCH_STRING=m CONFIG_NETFILTER_XT_MATCH_TIME=m CONFIG_NF_DEFRAG_IPV4=m CONFIG_NF_CONNTRACK_IPV4=m CONFIG_NF_CONNTRACK_PROC_COMPAT=y CONFIG_IP_NF_IPTABLES=m CONFIG_IP_NF_MATCH_AH=m CONFIG_IP_NF_MATCH_ECN=m CONFIG_IP_NF_MATCH_TTL=m CONFIG_IP_NF_FILTER=m CONFIG_IP_NF_TARGET_REJECT=m CONFIG_IP_NF_TARGET_ULOG=m CONFIG_NF_NAT=m CONFIG_NF_NAT_NEEDED=y CONFIG_IP_NF_TARGET_MASQUERADE=m CONFIG_IP_NF_TARGET_REDIRECT=m CONFIG_NF_NAT_FTP=m CONFIG_IP_NF_MANGLE=m CONFIG_IP_NF_TARGET_ECN=m CONFIG_IP_NF_TARGET_TTL=m CONFIG_IP_NF_ARPTABLES=m CONFIG_IP_NF_ARPFILTER=m CONFIG_IP_NF_ARP_MANGLE=m CONFIG_NF_DEFRAG_IPV6=m CONFIG_NF_CONNTRACK_IPV6=m CONFIG_IP6_NF_IPTABLES=m CONFIG_BRIDGE_NF_EBTABLES=m CONFIG_BRIDGE_EBT_T_NAT=m CONFIG_IP_SCTP=m CONFIG_SCTP_HMAC_NONE=y CONFIG_STP=m CONFIG_BRIDGE=m CONFIG_BRIDGE_IGMP_SNOOPING=y CONFIG_LLC=m CONFIG_DNS_RESOLVER=y CONFIG_RPS=y CONFIG_RFS_ACCEL=y CONFIG_XPS=y CONFIG_NETPRIO_CGROUP=m CONFIG_BQL=y CONFIG_NET_PKTGEN=m CONFIG_IRDA=m CONFIG_IRTTY_SIR=m CONFIG_BT=m CONFIG_BT_RFCOMM=m CONFIG_BT_RFCOMM_TTY=y CONFIG_BT_BNEP=m CONFIG_BT_HIDP=m CONFIG_BT_HCIBTUSB=m CONFIG_WIRELESS=y CONFIG_WEXT_CORE=y CONFIG_WEXT_PROC=y CONFIG_CFG80211=m CONFIG_CFG80211_DEVELOPER_WARNINGS=y CONFIG_CFG80211_DEFAULT_PS=y CONFIG_CFG80211_DEBUGFS=y CONFIG_CFG80211_WEXT=y CONFIG_WIRELESS_EXT_SYSFS=y CONFIG_LIB80211=m CONFIG_MAC80211=m CONFIG_MAC80211_HAS_RC=y CONFIG_MAC80211_RC_MINSTREL=y CONFIG_MAC80211_RC_MINSTREL_HT=y CONFIG_MAC80211_RC_DEFAULT_MINSTREL=y CONFIG_MAC80211_RC_DEFAULT="minstrel_ht" CONFIG_MAC80211_LEDS=y CONFIG_MAC80211_DEBUGFS=y CONFIG_RFKILL=y CONFIG_RFKILL_LEDS=y CONFIG_RFKILL_INPUT=y CONFIG_RFKILL_REGULATOR=m CONFIG_HAVE_BPF_JIT=y CONFIG_UEVENT_HELPER_PATH="" CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y CONFIG_STANDALONE=y CONFIG_PREVENT_FIRMWARE_BUILD=y CONFIG_FW_LOADER=y CONFIG_FIRMWARE_IN_KERNEL=y CONFIG_EXTRA_FIRMWARE="" CONFIG_DMA_SHARED_BUFFER=y CONFIG_CONNECTOR=m CONFIG_MTD=m CONFIG_MTD_REDBOOT_PARTS=m CONFIG_MTD_REDBOOT_DIRECTORY_BLOCK=-1 CONFIG_MTD_CHAR=m CONFIG_MTD_BLKDEVS=m CONFIG_MTD_BLOCK=m CONFIG_SM_FTL=m CONFIG_MTD_MAP_BANK_WIDTH_1=y CONFIG_MTD_MAP_BANK_WIDTH_2=y CONFIG_MTD_MAP_BANK_WIDTH_4=y CONFIG_MTD_CFI_I1=y CONFIG_MTD_CFI_I2=y CONFIG_MTD_MTDRAM=m CONFIG_MTDRAM_TOTAL_SIZE=4096 CONFIG_MTDRAM_ERASE_SIZE=128 CONFIG_MTD_BLOCK2MTD=m CONFIG_MTD_NAND_ECC=m CONFIG_MTD_NAND_ECC_SMC=y CONFIG_PNP=y CONFIG_PNPACPI=y CONFIG_BLK_DEV=y CONFIG_BLK_DEV_FD=m CONFIG_BLK_DEV_LOOP=m CONFIG_BLK_DEV_LOOP_MIN_COUNT=8 CONFIG_BLK_DEV_CRYPTOLOOP=m CONFIG_BLK_DEV_DRBD=m CONFIG_BLK_DEV_NBD=m CONFIG_BLK_DEV_RAM=y CONFIG_BLK_DEV_RAM_COUNT=16 CONFIG_BLK_DEV_RAM_SIZE=8192 CONFIG_CDROM_PKTCDVD=m CONFIG_CDROM_PKTCDVD_BUFFERS=8 CONFIG_VIRTIO_BLK=y CONFIG_INTEL_MEI=m CONFIG_HAVE_IDE=y CONFIG_SCSI_MOD=y CONFIG_RAID_ATTRS=m CONFIG_SCSI=y CONFIG_SCSI_DMA=y CONFIG_BLK_DEV_SD=y CONFIG_BLK_DEV_SR=m CONFIG_BLK_DEV_SR_VENDOR=y CONFIG_CHR_DEV_SG=y CONFIG_CHR_DEV_SCH=m CONFIG_SCSI_MULTI_LUN=y CONFIG_SCSI_CONSTANTS=y CONFIG_SCSI_LOGGING=y CONFIG_SCSI_SCAN_ASYNC=y CONFIG_SCSI_WAIT_SCAN=m CONFIG_SCSI_SPI_ATTRS=y CONFIG_SCSI_LOWLEVEL=y CONFIG_SCSI_DEBUG=m CONFIG_SCSI_VIRTIO=m CONFIG_ATA=y CONFIG_ATA_VERBOSE_ERROR=y CONFIG_ATA_ACPI=y CONFIG_SATA_PMP=y CONFIG_SATA_AHCI=y CONFIG_ATA_SFF=y CONFIG_ATA_BMDMA=y CONFIG_ATA_PIIX=y CONFIG_PATA_MPIIX=y CONFIG_MD=y CONFIG_BLK_DEV_MD=m CONFIG_MD_LINEAR=m CONFIG_MD_RAID0=m CONFIG_MD_RAID1=m CONFIG_MD_RAID10=m CONFIG_MD_RAID456=m CONFIG_MULTICORE_RAID456=y CONFIG_MD_MULTIPATH=m CONFIG_MD_FAULTY=m CONFIG_BLK_DEV_DM=m CONFIG_DM_DEBUG=y CONFIG_DM_BUFIO=m CONFIG_DM_PERSISTENT_DATA=m CONFIG_DM_CRYPT=m CONFIG_DM_SNAPSHOT=m CONFIG_DM_THIN_PROVISIONING=m CONFIG_DM_MIRROR=m CONFIG_DM_RAID=m CONFIG_DM_LOG_USERSPACE=m CONFIG_DM_ZERO=m CONFIG_DM_MULTIPATH=m CONFIG_DM_DELAY=m CONFIG_DM_UEVENT=y CONFIG_DM_FLAKEY=m CONFIG_DM_VERITY=m CONFIG_NETDEVICES=y CONFIG_NET_CORE=y CONFIG_DUMMY=m CONFIG_MII=m CONFIG_NETCONSOLE=m CONFIG_NETPOLL=y CONFIG_NETPOLL_TRAP=y CONFIG_NET_POLL_CONTROLLER=y CONFIG_TUN=m CONFIG_VETH=m CONFIG_VIRTIO_NET=m CONFIG_ETHERNET=y CONFIG_NET_VENDOR_INTEL=y CONFIG_E1000E=m CONFIG_IXGBEVF=m CONFIG_NET_VENDOR_REALTEK=y CONFIG_8139TOO=m CONFIG_8139TOO_PIO=y CONFIG_PPP=m CONFIG_PPP_BSDCOMP=m CONFIG_PPP_DEFLATE=m CONFIG_PPP_FILTER=y CONFIG_PPP_MPPE=m CONFIG_PPP_MULTILINK=y CONFIG_PPP_ASYNC=m CONFIG_SLHC=m CONFIG_WLAN=y CONFIG_IWLWIFI=m CONFIG_IWLEGACY=m CONFIG_IWL3945=m CONFIG_IWLEGACY_DEBUGFS=y CONFIG_INPUT=y CONFIG_INPUT_POLLDEV=y CONFIG_INPUT_SPARSEKMAP=m CONFIG_INPUT_MOUSEDEV=y CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024 CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768 CONFIG_INPUT_EVDEV=m CONFIG_INPUT_KEYBOARD=y CONFIG_KEYBOARD_ATKBD=y CONFIG_INPUT_MOUSE=y CONFIG_MOUSE_PS2=m CONFIG_MOUSE_PS2_ALPS=y CONFIG_MOUSE_PS2_LOGIPS2PP=y CONFIG_MOUSE_PS2_SYNAPTICS=y CONFIG_MOUSE_PS2_LIFEBOOK=y CONFIG_MOUSE_PS2_TRACKPOINT=y CONFIG_MOUSE_SERIAL=m CONFIG_INPUT_MISC=y CONFIG_INPUT_UINPUT=m CONFIG_SERIO=y CONFIG_SERIO_I8042=y CONFIG_SERIO_SERPORT=m CONFIG_SERIO_PCIPS2=m CONFIG_SERIO_LIBPS2=y CONFIG_SERIO_RAW=m CONFIG_VT=y CONFIG_CONSOLE_TRANSLATIONS=y CONFIG_VT_CONSOLE=y CONFIG_VT_CONSOLE_SLEEP=y CONFIG_HW_CONSOLE=y CONFIG_UNIX98_PTYS=y CONFIG_DEVKMEM=y CONFIG_SERIAL_8250=y CONFIG_SERIAL_8250_CONSOLE=y CONFIG_FIX_EARLYCON_MEM=y CONFIG_SERIAL_8250_PCI=y CONFIG_SERIAL_8250_PNP=y CONFIG_SERIAL_8250_NR_UARTS=16 CONFIG_SERIAL_8250_RUNTIME_UARTS=4 CONFIG_SERIAL_8250_EXTENDED=y CONFIG_SERIAL_8250_SHARE_IRQ=y CONFIG_SERIAL_8250_RSA=y CONFIG_SERIAL_CORE=y CONFIG_SERIAL_CORE_CONSOLE=y CONFIG_HVC_DRIVER=y CONFIG_VIRTIO_CONSOLE=m CONFIG_HW_RANDOM=m CONFIG_HW_RANDOM_INTEL=m CONFIG_HW_RANDOM_VIRTIO=m CONFIG_NVRAM=m CONFIG_HPET=y CONFIG_HPET_MMAP=y CONFIG_HANGCHECK_TIMER=m CONFIG_DEVPORT=y CONFIG_I2C=m CONFIG_I2C_BOARDINFO=y CONFIG_I2C_COMPAT=y CONFIG_I2C_CHARDEV=m CONFIG_I2C_HELPER_AUTO=y CONFIG_I2C_ALGOBIT=m CONFIG_I2C_I801=m CONFIG_I2C_ISCH=m CONFIG_I2C_PIIX4=m CONFIG_I2C_TINY_USB=m CONFIG_I2C_DEBUG_ALGO=y CONFIG_HSI=m CONFIG_HSI_BOARDINFO=y CONFIG_HSI_CHAR=m CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y CONFIG_POWER_SUPPLY=y CONFIG_HWMON=y CONFIG_SENSORS_ACPI_POWER=m CONFIG_THERMAL=y CONFIG_THERMAL_HWMON=y CONFIG_WATCHDOG=y CONFIG_WATCHDOG_CORE=y CONFIG_SOFT_WATCHDOG=m CONFIG_ITCO_WDT=m CONFIG_ITCO_VENDOR_SUPPORT=y CONFIG_SSB_POSSIBLE=y CONFIG_BCMA_POSSIBLE=y CONFIG_MFD_CORE=m CONFIG_LPC_SCH=m CONFIG_LPC_ICH=m CONFIG_REGULATOR=y CONFIG_REGULATOR_DEBUG=y CONFIG_REGULATOR_DUMMY=y CONFIG_REGULATOR_VIRTUAL_CONSUMER=m CONFIG_MEDIA_SUPPORT=m CONFIG_VIDEO_DEV=m CONFIG_VIDEO_V4L2_COMMON=m CONFIG_DVB_CORE=m CONFIG_VIDEO_MEDIA=m CONFIG_RC_CORE=m CONFIG_LIRC=m CONFIG_RC_MAP=m CONFIG_IR_SANYO_DECODER=m CONFIG_RC_LOOPBACK=m CONFIG_MEDIA_ATTACH=y CONFIG_MEDIA_TUNER=m CONFIG_MEDIA_TUNER_CUSTOMISE=y CONFIG_MEDIA_TUNER_SIMPLE=m CONFIG_MEDIA_TUNER_TDA9887=m CONFIG_MEDIA_TUNER_MT2060=m CONFIG_MEDIA_TUNER_MT2266=m CONFIG_MEDIA_TUNER_TDA18218=m CONFIG_MEDIA_TUNER_TDA18212=m CONFIG_VIDEO_V4L2=m CONFIG_VIDEOBUF_GEN=m CONFIG_VIDEOBUF_DMA_SG=m CONFIG_VIDEO_TVEEPROM=m CONFIG_VIDEO_TUNER=m CONFIG_V4L2_MEM2MEM_DEV=m CONFIG_VIDEOBUF2_CORE=m CONFIG_VIDEOBUF2_MEMOPS=m CONFIG_VIDEOBUF2_VMALLOC=m CONFIG_VIDEO_CAPTURE_DRIVERS=y CONFIG_VIDEO_ADV_DEBUG=y CONFIG_VIDEO_HELPER_CHIPS_AUTO=y CONFIG_VIDEO_IR_I2C=m CONFIG_VIDEO_MSP3400=m CONFIG_VIDEO_CS53L32A=m CONFIG_VIDEO_WM8775=m CONFIG_VIDEO_SAA6588=m CONFIG_VIDEO_SAA711X=m CONFIG_VIDEO_CX25840=m CONFIG_VIDEO_CX2341X=m CONFIG_VIDEO_VIVI=m CONFIG_V4L_USB_DRIVERS=y CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y CONFIG_VIDEO_PVRUSB2=m CONFIG_VIDEO_PVRUSB2_SYSFS=y CONFIG_VIDEO_PVRUSB2_DVB=y CONFIG_V4L_PCI_DRIVERS=y CONFIG_VIDEO_SAA7134=m CONFIG_VIDEO_SAA7134_ALSA=m CONFIG_VIDEO_SAA7134_RC=y CONFIG_V4L_MEM2MEM_DRIVERS=y CONFIG_VIDEO_MEM2MEM_TESTDEV=m CONFIG_DVB_MAX_ADAPTERS=8 CONFIG_DVB_DYNAMIC_MINORS=y CONFIG_DVB_CAPTURE_DRIVERS=y CONFIG_DVB_USB=m CONFIG_DVB_USB_DIB0700=m CONFIG_DVB_USB_CXUSB=m CONFIG_DVB_USB_NOVA_T_USB2=m CONFIG_DVB_USB_AF9015=m CONFIG_DVB_TTUSB_BUDGET=m CONFIG_DVB_FE_CUSTOMISE=y CONFIG_DVB_TDA8261=m CONFIG_DVB_TDA10071=m CONFIG_DVB_DIB3000MC=m CONFIG_DVB_DIB7000M=m CONFIG_DVB_DIB7000P=m CONFIG_DVB_AF9013=m CONFIG_DVB_DIB8000=m CONFIG_DVB_PLL=m CONFIG_DVB_TUNER_DIB0070=m CONFIG_DVB_DUMMY_FE=m CONFIG_AGP=y CONFIG_AGP_AMD64=y CONFIG_AGP_INTEL=y CONFIG_VGA_ARB=y CONFIG_VGA_ARB_MAX_GPUS=16 CONFIG_DRM=m CONFIG_DRM_KMS_HELPER=m CONFIG_DRM_TTM=m CONFIG_DRM_I915=m CONFIG_DRM_MGA=m CONFIG_DRM_CIRRUS_QEMU=m CONFIG_VIDEO_OUTPUT_CONTROL=m CONFIG_FB=y CONFIG_FIRMWARE_EDID=y CONFIG_FB_BOOT_VESA_SUPPORT=y CONFIG_FB_CFB_FILLRECT=y CONFIG_FB_CFB_COPYAREA=y CONFIG_FB_CFB_IMAGEBLIT=y CONFIG_FB_SYS_FILLRECT=m CONFIG_FB_SYS_COPYAREA=m CONFIG_FB_SYS_IMAGEBLIT=m CONFIG_FB_MODE_HELPERS=y CONFIG_FB_TILEBLITTING=y CONFIG_FB_VESA=y CONFIG_FB_EFI=y CONFIG_BACKLIGHT_LCD_SUPPORT=y CONFIG_LCD_CLASS_DEVICE=m CONFIG_LCD_PLATFORM=m CONFIG_BACKLIGHT_CLASS_DEVICE=m CONFIG_BACKLIGHT_GENERIC=m CONFIG_VGA_CONSOLE=y CONFIG_VGACON_SOFT_SCROLLBACK=y CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64 CONFIG_DUMMY_CONSOLE=y CONFIG_FRAMEBUFFER_CONSOLE=y CONFIG_FRAMEBUFFER_CONSOLE_DETECT_PRIMARY=y CONFIG_FONT_8x8=y CONFIG_FONT_8x16=y CONFIG_LOGO=y CONFIG_LOGO_LINUX_CLUT224=y CONFIG_SOUND=m CONFIG_SOUND_OSS_CORE=y CONFIG_SOUND_OSS_CORE_PRECLAIM=y CONFIG_SND=m CONFIG_SND_TIMER=m CONFIG_SND_PCM=m CONFIG_SND_HWDEP=m CONFIG_SND_RAWMIDI=m CONFIG_SND_JACK=y CONFIG_SND_SEQUENCER=m CONFIG_SND_OSSEMUL=y CONFIG_SND_MIXER_OSS=m CONFIG_SND_PCM_OSS=m CONFIG_SND_PCM_OSS_PLUGINS=y CONFIG_SND_SEQUENCER_OSS=y CONFIG_SND_HRTIMER=m CONFIG_SND_SEQ_HRTIMER_DEFAULT=y CONFIG_SND_DYNAMIC_MINORS=y CONFIG_SND_VERBOSE_PROCFS=y CONFIG_SND_VMASTER=y CONFIG_SND_KCTL_JACK=y CONFIG_SND_DMA_SGBUF=y CONFIG_SND_RAWMIDI_SEQ=m CONFIG_SND_EMU10K1_SEQ=m CONFIG_SND_MPU401_UART=m CONFIG_SND_AC97_CODEC=m CONFIG_SND_DRIVERS=y CONFIG_SND_DUMMY=m CONFIG_SND_SERIAL_U16550=m CONFIG_SND_MPU401=m CONFIG_SND_AC97_POWER_SAVE=y CONFIG_SND_AC97_POWER_SAVE_DEFAULT=0 CONFIG_SND_PCI=y CONFIG_SND_EMU10K1=m CONFIG_SND_HDA_INTEL=m CONFIG_SND_HDA_PREALLOC_SIZE=64 CONFIG_SND_HDA_INPUT_JACK=y CONFIG_SND_HDA_CODEC_REALTEK=y CONFIG_SND_HDA_CODEC_ANALOG=y CONFIG_SND_HDA_CODEC_SIGMATEL=y CONFIG_SND_HDA_CODEC_VIA=y CONFIG_SND_HDA_CODEC_HDMI=y CONFIG_SND_HDA_CODEC_CIRRUS=y CONFIG_SND_HDA_CODEC_CONEXANT=y CONFIG_SND_HDA_CODEC_CMEDIA=y CONFIG_SND_HDA_CODEC_SI3054=y CONFIG_SND_HDA_GENERIC=y CONFIG_SND_HDA_POWER_SAVE=y CONFIG_SND_HDA_POWER_SAVE_DEFAULT=500 CONFIG_SND_INTEL8X0=m CONFIG_AC97_BUS=m CONFIG_HID=m CONFIG_HID_GENERIC=m CONFIG_HID_A4TECH=m CONFIG_HID_APPLE=m CONFIG_HID_BELKIN=m CONFIG_HID_CHERRY=m CONFIG_HID_CHICONY=m CONFIG_HID_CYPRESS=m CONFIG_HID_EZKEY=m CONFIG_HID_KYE=m CONFIG_HID_KENSINGTON=m CONFIG_HID_LOGITECH=m CONFIG_HID_MICROSOFT=m CONFIG_HID_MONTEREY=m CONFIG_USB_HID=m CONFIG_USB_HIDDEV=y CONFIG_USB_ARCH_HAS_OHCI=y CONFIG_USB_ARCH_HAS_EHCI=y CONFIG_USB_ARCH_HAS_XHCI=y CONFIG_USB_SUPPORT=y CONFIG_USB_COMMON=m CONFIG_USB_ARCH_HAS_HCD=y CONFIG_USB=m CONFIG_USB_ANNOUNCE_NEW_DEVICES=y CONFIG_USB_SUSPEND=y CONFIG_USB_MON=m CONFIG_USB_XHCI_HCD=m CONFIG_USB_EHCI_HCD=m CONFIG_USB_EHCI_ROOT_HUB_TT=y CONFIG_USB_EHCI_TT_NEWSCHED=y CONFIG_USB_OHCI_HCD=m CONFIG_USB_OHCI_LITTLE_ENDIAN=y CONFIG_USB_UHCI_HCD=m CONFIG_USB_ACM=m CONFIG_USB_PRINTER=m CONFIG_USB_WDM=m CONFIG_USB_TMC=m CONFIG_USB_STORAGE=m CONFIG_USB_UAS=m CONFIG_USB_SERIAL=m CONFIG_USB_SERIAL_GENERIC=y CONFIG_USB_SERIAL_IR=m CONFIG_USB_SERIAL_WWAN=m CONFIG_USB_SERIAL_OPTION=m CONFIG_USB_TEST=m CONFIG_MMC=m CONFIG_MMC_BLOCK=m CONFIG_MMC_BLOCK_MINORS=8 CONFIG_MMC_BLOCK_BOUNCE=y CONFIG_MMC_SDHCI=m CONFIG_MMC_SDHCI_PCI=m CONFIG_MMC_SDHCI_PLTFM=m CONFIG_MMC_SDRICOH_CS=m CONFIG_NEW_LEDS=y CONFIG_LEDS_CLASS=y CONFIG_LEDS_TRIGGERS=y CONFIG_LEDS_TRIGGER_TIMER=m CONFIG_LEDS_TRIGGER_HEARTBEAT=m CONFIG_EDAC=y CONFIG_EDAC_DEBUG=y CONFIG_EDAC_DECODE_MCE=y CONFIG_EDAC_MM_EDAC=m CONFIG_EDAC_E752X=m CONFIG_RTC_LIB=y CONFIG_RTC_CLASS=y CONFIG_RTC_HCTOSYS=y CONFIG_RTC_HCTOSYS_DEVICE="rtc0" CONFIG_RTC_INTF_SYSFS=y CONFIG_RTC_INTF_PROC=y CONFIG_RTC_INTF_DEV=y CONFIG_RTC_DRV_CMOS=y CONFIG_DMADEVICES=y CONFIG_INTEL_IOATDMA=m CONFIG_DMA_ENGINE=y CONFIG_NET_DMA=y CONFIG_DCA=m CONFIG_VIRTIO=y CONFIG_VIRTIO_RING=y CONFIG_VIRTIO_PCI=y CONFIG_VIRTIO_BALLOON=y CONFIG_VIRTIO_MMIO=m CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y CONFIG_STAGING=y CONFIG_ZRAM=m CONFIG_ZRAM_DEBUG=y CONFIG_ZCACHE=y CONFIG_ZSMALLOC=y CONFIG_USB_WPAN_HCD=m CONFIG_X86_PLATFORM_DEVICES=y CONFIG_THINKPAD_ACPI=m CONFIG_THINKPAD_ACPI_ALSA_SUPPORT=y CONFIG_THINKPAD_ACPI_DEBUG=y CONFIG_THINKPAD_ACPI_VIDEO=y CONFIG_SENSORS_HDAPS=y CONFIG_ACPI_WMI=m CONFIG_ACPI_TOSHIBA=m CONFIG_INTEL_IPS=m CONFIG_CLKEVT_I8253=y CONFIG_I8253_LOCK=y CONFIG_CLKBLD_I8253=y CONFIG_IOMMU_SUPPORT=y CONFIG_VIRT_DRIVERS=y CONFIG_PM_DEVFREQ=y CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=y CONFIG_DEVFREQ_GOV_PERFORMANCE=y CONFIG_DEVFREQ_GOV_POWERSAVE=y CONFIG_DEVFREQ_GOV_USERSPACE=y CONFIG_EDD=m CONFIG_FIRMWARE_MEMMAP=y CONFIG_EFI_VARS=m CONFIG_DMIID=y CONFIG_DMI_SYSFS=m CONFIG_DCACHE_WORD_ACCESS=y CONFIG_EXT2_FS=y CONFIG_EXT2_FS_XATTR=y CONFIG_EXT2_FS_POSIX_ACL=y CONFIG_EXT2_FS_SECURITY=y CONFIG_EXT3_FS=y CONFIG_EXT3_DEFAULTS_TO_ORDERED=y CONFIG_EXT3_FS_XATTR=y CONFIG_EXT3_FS_POSIX_ACL=y CONFIG_EXT3_FS_SECURITY=y CONFIG_EXT4_FS=y CONFIG_EXT4_FS_XATTR=y CONFIG_EXT4_FS_POSIX_ACL=y CONFIG_EXT4_FS_SECURITY=y CONFIG_JBD=y CONFIG_JBD2=y CONFIG_FS_MBCACHE=y CONFIG_REISERFS_FS=m CONFIG_REISERFS_PROC_INFO=y CONFIG_REISERFS_FS_XATTR=y CONFIG_REISERFS_FS_POSIX_ACL=y CONFIG_REISERFS_FS_SECURITY=y CONFIG_XFS_FS=m CONFIG_XFS_QUOTA=y CONFIG_XFS_POSIX_ACL=y CONFIG_XFS_RT=y CONFIG_BTRFS_FS=m CONFIG_BTRFS_FS_POSIX_ACL=y CONFIG_BTRFS_FS_CHECK_INTEGRITY=y CONFIG_FS_POSIX_ACL=y CONFIG_EXPORTFS=m CONFIG_FILE_LOCKING=y CONFIG_FSNOTIFY=y CONFIG_DNOTIFY=y CONFIG_INOTIFY_USER=y CONFIG_FANOTIFY=y CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y CONFIG_QUOTACTL=y CONFIG_QUOTACTL_COMPAT=y CONFIG_AUTOFS4_FS=m CONFIG_FUSE_FS=m CONFIG_CUSE=m CONFIG_GENERIC_ACL=y CONFIG_FSCACHE=m CONFIG_FSCACHE_STATS=y CONFIG_FSCACHE_HISTOGRAM=y CONFIG_FSCACHE_DEBUG=y CONFIG_CACHEFILES=m CONFIG_CACHEFILES_DEBUG=y CONFIG_ISO9660_FS=m CONFIG_JOLIET=y CONFIG_ZISOFS=y CONFIG_UDF_FS=m CONFIG_UDF_NLS=y CONFIG_FAT_FS=m CONFIG_VFAT_FS=m CONFIG_FAT_DEFAULT_CODEPAGE=852 CONFIG_FAT_DEFAULT_IOCHARSET="cp1250" CONFIG_NTFS_FS=m CONFIG_NTFS_RW=y CONFIG_PROC_FS=y CONFIG_PROC_KCORE=y CONFIG_PROC_SYSCTL=y CONFIG_PROC_PAGE_MONITOR=y CONFIG_SYSFS=y CONFIG_TMPFS=y CONFIG_TMPFS_POSIX_ACL=y CONFIG_TMPFS_XATTR=y CONFIG_HUGETLBFS=y CONFIG_HUGETLB_PAGE=y CONFIG_CONFIGFS_FS=m CONFIG_MISC_FILESYSTEMS=y CONFIG_JFFS2_FS=m CONFIG_JFFS2_FS_DEBUG=0 CONFIG_JFFS2_FS_WRITEBUFFER=y CONFIG_JFFS2_COMPRESSION_OPTIONS=y CONFIG_JFFS2_ZLIB=y CONFIG_JFFS2_LZO=y CONFIG_JFFS2_RTIME=y CONFIG_JFFS2_RUBIN=y CONFIG_JFFS2_CMODE_PRIORITY=y CONFIG_CRAMFS=m CONFIG_SQUASHFS=m CONFIG_SQUASHFS_ZLIB=y CONFIG_SQUASHFS_XZ=y CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE=3 CONFIG_NETWORK_FILESYSTEMS=y CONFIG_NFS_FS=m CONFIG_NFS_V2=y CONFIG_NFS_V3=y CONFIG_NFS_V3_ACL=y CONFIG_NFS_V4=y CONFIG_NFS_V4_1=y CONFIG_PNFS_FILE_LAYOUT=m CONFIG_PNFS_BLOCK=m CONFIG_NFS_V4_1_IMPLEMENTATION_ID_DOMAIN="kernel.org" CONFIG_NFS_USE_KERNEL_DNS=y CONFIG_NFSD=m CONFIG_NFSD_V2_ACL=y CONFIG_NFSD_V3=y CONFIG_NFSD_V3_ACL=y CONFIG_NFSD_V4=y CONFIG_LOCKD=m CONFIG_LOCKD_V4=y CONFIG_NFS_ACL_SUPPORT=m CONFIG_NFS_COMMON=y CONFIG_SUNRPC=m CONFIG_SUNRPC_GSS=m CONFIG_SUNRPC_BACKCHANNEL=y CONFIG_CIFS=m CONFIG_CIFS_XATTR=y CONFIG_CIFS_POSIX=y CONFIG_NLS=y CONFIG_NLS_DEFAULT="iso8859-1" CONFIG_NLS_CODEPAGE_852=m CONFIG_NLS_CODEPAGE_1250=m CONFIG_NLS_ASCII=y CONFIG_NLS_ISO8859_1=m CONFIG_NLS_ISO8859_2=m CONFIG_NLS_UTF8=m CONFIG_DLM=m CONFIG_TRACE_IRQFLAGS_SUPPORT=y CONFIG_PRINTK_TIME=y CONFIG_DEFAULT_MESSAGE_LOGLEVEL=4 CONFIG_ENABLE_WARN_DEPRECATED=y CONFIG_ENABLE_MUST_CHECK=y CONFIG_FRAME_WARN=2048 CONFIG_MAGIC_SYSRQ=y CONFIG_DEBUG_FS=y CONFIG_DEBUG_KERNEL=y CONFIG_LOCKUP_DETECTOR=y CONFIG_HARDLOCKUP_DETECTOR=y CONFIG_BOOTPARAM_HARDLOCKUP_PANIC_VALUE=0 CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0 CONFIG_PANIC_ON_OOPS_VALUE=0 CONFIG_DETECT_HUNG_TASK=y CONFIG_DEFAULT_HUNG_TASK_TIMEOUT=120 CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0 CONFIG_SCHED_DEBUG=y CONFIG_SCHEDSTATS=y CONFIG_TIMER_STATS=y CONFIG_SLUB_STATS=y CONFIG_DEBUG_KMEMLEAK=y CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=500 CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y CONFIG_DEBUG_PREEMPT=y CONFIG_DEBUG_SPINLOCK=y CONFIG_DEBUG_MUTEXES=y CONFIG_DEBUG_LOCK_ALLOC=y CONFIG_PROVE_LOCKING=y CONFIG_PROVE_RCU=y CONFIG_LOCKDEP=y CONFIG_LOCK_STAT=y CONFIG_TRACE_IRQFLAGS=y CONFIG_DEBUG_ATOMIC_SLEEP=y CONFIG_STACKTRACE=y CONFIG_DEBUG_BUGVERBOSE=y CONFIG_DEBUG_INFO=y CONFIG_DEBUG_INFO_REDUCED=y CONFIG_DEBUG_MEMORY_INIT=y CONFIG_DEBUG_NOTIFIERS=y CONFIG_ARCH_WANT_FRAME_POINTERS=y CONFIG_FRAME_POINTER=y CONFIG_RCU_CPU_STALL_TIMEOUT=60 CONFIG_RCU_CPU_STALL_VERBOSE=y CONFIG_RCU_CPU_STALL_INFO=y CONFIG_RCU_TRACE=y CONFIG_LKDTM=m CONFIG_LATENCYTOP=y CONFIG_USER_STACKTRACE_SUPPORT=y CONFIG_NOP_TRACER=y CONFIG_HAVE_FUNCTION_TRACER=y CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y CONFIG_HAVE_DYNAMIC_FTRACE=y CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y CONFIG_HAVE_SYSCALL_TRACEPOINTS=y CONFIG_HAVE_C_RECORDMCOUNT=y CONFIG_TRACER_MAX_TRACE=y CONFIG_RING_BUFFER=y CONFIG_EVENT_TRACING=y CONFIG_CONTEXT_SWITCH_TRACER=y CONFIG_RING_BUFFER_ALLOW_SWAP=y CONFIG_TRACING=y CONFIG_GENERIC_TRACER=y CONFIG_TRACING_SUPPORT=y CONFIG_FTRACE=y CONFIG_FUNCTION_TRACER=y CONFIG_FUNCTION_GRAPH_TRACER=y CONFIG_IRQSOFF_TRACER=y CONFIG_PREEMPT_TRACER=y CONFIG_SCHED_TRACER=y CONFIG_FTRACE_SYSCALLS=y CONFIG_BRANCH_PROFILE_NONE=y CONFIG_STACK_TRACER=y CONFIG_BLK_DEV_IO_TRACE=y CONFIG_KPROBE_EVENT=y CONFIG_UPROBE_EVENT=y CONFIG_PROBE_EVENTS=y CONFIG_DYNAMIC_FTRACE=y CONFIG_FUNCTION_PROFILER=y CONFIG_FTRACE_MCOUNT_RECORD=y CONFIG_MMIOTRACE=y CONFIG_MMIOTRACE_TEST=m CONFIG_DYNAMIC_DEBUG=y CONFIG_DMA_API_DEBUG=y CONFIG_HAVE_ARCH_KGDB=y CONFIG_HAVE_ARCH_KMEMCHECK=y CONFIG_X86_VERBOSE_BOOTUP=y CONFIG_EARLY_PRINTK=y CONFIG_DEBUG_RODATA=y CONFIG_HAVE_MMIOTRACE_SUPPORT=y CONFIG_IO_DELAY_TYPE_0X80=0 CONFIG_IO_DELAY_TYPE_0XED=1 CONFIG_IO_DELAY_TYPE_UDELAY=2 CONFIG_IO_DELAY_TYPE_NONE=3 CONFIG_IO_DELAY_0X80=y CONFIG_DEFAULT_IO_DELAY_TYPE=0 CONFIG_DEBUG_BOOT_PARAMS=y CONFIG_OPTIMIZE_INLINING=y CONFIG_KEYS=y CONFIG_ENCRYPTED_KEYS=y CONFIG_SECURITY=y CONFIG_SECURITY_NETWORK=y CONFIG_LSM_MMAP_MIN_ADDR=65535 CONFIG_SECURITY_SELINUX=y CONFIG_SECURITY_SELINUX_BOOTPARAM=y CONFIG_SECURITY_SELINUX_BOOTPARAM_VALUE=1 CONFIG_SECURITY_SELINUX_DISABLE=y CONFIG_SECURITY_SELINUX_DEVELOP=y CONFIG_SECURITY_SELINUX_AVC_STATS=y CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1 CONFIG_INTEGRITY=y CONFIG_EVM=y CONFIG_DEFAULT_SECURITY_SELINUX=y CONFIG_DEFAULT_SECURITY="selinux" CONFIG_XOR_BLOCKS=m CONFIG_ASYNC_CORE=m CONFIG_ASYNC_MEMCPY=m CONFIG_ASYNC_XOR=m CONFIG_ASYNC_PQ=m CONFIG_ASYNC_RAID6_RECOV=m CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y CONFIG_CRYPTO=y CONFIG_CRYPTO_ALGAPI=y CONFIG_CRYPTO_ALGAPI2=y CONFIG_CRYPTO_AEAD=m CONFIG_CRYPTO_AEAD2=y CONFIG_CRYPTO_BLKCIPHER=y CONFIG_CRYPTO_BLKCIPHER2=y CONFIG_CRYPTO_HASH=y CONFIG_CRYPTO_HASH2=y CONFIG_CRYPTO_RNG=y CONFIG_CRYPTO_RNG2=y CONFIG_CRYPTO_PCOMP2=y CONFIG_CRYPTO_MANAGER=y CONFIG_CRYPTO_MANAGER2=y CONFIG_CRYPTO_USER=m CONFIG_CRYPTO_MANAGER_DISABLE_TESTS=y CONFIG_CRYPTO_GF128MUL=m CONFIG_CRYPTO_PCRYPT=m CONFIG_CRYPTO_WORKQUEUE=y CONFIG_CRYPTO_CRYPTD=m CONFIG_CRYPTO_CBC=y CONFIG_CRYPTO_ECB=m CONFIG_CRYPTO_LRW=m CONFIG_CRYPTO_XTS=m CONFIG_CRYPTO_HMAC=y CONFIG_CRYPTO_CRC32C=y CONFIG_CRYPTO_MD4=m CONFIG_CRYPTO_MD5=y CONFIG_CRYPTO_MICHAEL_MIC=m CONFIG_CRYPTO_SHA1=y CONFIG_CRYPTO_SHA1_SSSE3=y CONFIG_CRYPTO_SHA256=y CONFIG_CRYPTO_AES=y CONFIG_CRYPTO_AES_X86_64=m CONFIG_CRYPTO_AES_NI_INTEL=m CONFIG_CRYPTO_ARC4=m CONFIG_CRYPTO_BLOWFISH_COMMON=m CONFIG_CRYPTO_BLOWFISH_X86_64=m CONFIG_CRYPTO_DES=m CONFIG_CRYPTO_TWOFISH_COMMON=m CONFIG_CRYPTO_TWOFISH_X86_64=m CONFIG_CRYPTO_TWOFISH_X86_64_3WAY=m CONFIG_CRYPTO_LZO=y CONFIG_CRYPTO_HW=y CONFIG_HAVE_KVM=y CONFIG_HAVE_KVM_IRQCHIP=y CONFIG_HAVE_KVM_EVENTFD=y CONFIG_KVM_APIC_ARCHITECTURE=y CONFIG_KVM_MMIO=y CONFIG_KVM_ASYNC_PF=y CONFIG_HAVE_KVM_MSI=y CONFIG_VIRTUALIZATION=y CONFIG_KVM=m CONFIG_KVM_INTEL=m CONFIG_KVM_AMD=m CONFIG_KVM_MMU_AUDIT=y CONFIG_VHOST_NET=m CONFIG_BINARY_PRINTF=y CONFIG_RAID6_PQ=m CONFIG_BITREVERSE=y CONFIG_GENERIC_STRNCPY_FROM_USER=y CONFIG_GENERIC_STRNLEN_USER=y CONFIG_GENERIC_FIND_FIRST_BIT=y CONFIG_GENERIC_PCI_IOMAP=y CONFIG_GENERIC_IOMAP=y CONFIG_GENERIC_IO=y CONFIG_CRC_CCITT=m CONFIG_CRC16=y CONFIG_CRC_T10DIF=y CONFIG_CRC_ITU_T=m CONFIG_CRC32=y CONFIG_CRC32_SLICEBY8=y CONFIG_LIBCRC32C=m CONFIG_ZLIB_INFLATE=y CONFIG_ZLIB_DEFLATE=m CONFIG_LZO_COMPRESS=y CONFIG_LZO_DECOMPRESS=y CONFIG_XZ_DEC=y CONFIG_XZ_DEC_X86=y CONFIG_XZ_DEC_POWERPC=y CONFIG_XZ_DEC_IA64=y CONFIG_XZ_DEC_ARM=y CONFIG_XZ_DEC_ARMTHUMB=y CONFIG_XZ_DEC_SPARC=y CONFIG_XZ_DEC_BCJ=y CONFIG_DECOMPRESS_GZIP=y CONFIG_DECOMPRESS_BZIP2=y CONFIG_DECOMPRESS_LZMA=y CONFIG_DECOMPRESS_XZ=y CONFIG_DECOMPRESS_LZO=y CONFIG_TEXTSEARCH=y CONFIG_TEXTSEARCH_KMP=m CONFIG_TEXTSEARCH_BM=m CONFIG_TEXTSEARCH_FSM=m CONFIG_HAS_IOMEM=y CONFIG_HAS_IOPORT=y CONFIG_HAS_DMA=y CONFIG_CHECK_SIGNATURE=y CONFIG_CPU_RMAP=y CONFIG_DQL=y CONFIG_NLATTR=y CONFIG_LRU_CACHE=m CONFIG_AVERAGE=y CONFIG_DDR=y ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-06-30 11:41 [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel Zdenek Kabelac @ 2012-06-30 19:55 ` Hugh Dickins 2012-06-30 20:51 ` Zdenek Kabelac 0 siblings, 1 reply; 13+ messages in thread From: Hugh Dickins @ 2012-06-30 19:55 UTC (permalink / raw) To: Zdenek Kabelac Cc: linux-kernel, amwang, Alasdair G Kergon, LVM general discussion and development On Sat, 30 Jun 2012, Zdenek Kabelac wrote: > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks. > Ooops log included. After some experimenting - reliable way to hit this oops > is to run lvm test suite for 10 minutes. Since 3.5 merge window does not > included anything related to this oops I've went for bisect. Thanks a lot for reporting, and going to such effort to find a reproducible testcase that you could bisect on. > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17 > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE But this leaves me very puzzled. Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git under tests/ ? I see no mention of madvise or MADV_REMOVE or fallocate or anything related in that git tree. If you have something else running at the same time, which happens to use madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables it on (I guess ext4 from the =y in your config), then I suppose we should start searching for improper memory freeing or scribbling in its holepunch support: something that might be corrupting the dm_region in your oops. I'll be surprised if that is the case, but it's something that you can easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove(): that should tell us what process is using it. I'm not an LVM user, so I doubt I'll be able to reproduce your setup. Any ideas from the DM guys? Has anyone else seen anything like this? Do all your oopses look like this one? Thanks, Hugh > > My HW - t61, 4GB > > Zdenek > > BUG: unable to handle kernel NULL pointer dereference at 0000000000000178 > IP: [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] > PGD 0 > Oops: 0000 [#1] PREEMPT SMP > CPU 1 > Modules linked in: dm_raid raid456 async_raid6_recov async_memcpy async_pq > async_xor xor async_tx raid6_pq raid1 md_mod reiserfs xfs dm_mirror > dm_region_hash dm_log dm_snapshot ip6_tables ebtable_nat ebtables > ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state > nf_conntrack ipt_REJECT iptable_mangle xt_tcpudp iptable_filter ip_tables > x_tables bridge stp llc ipv6 snd_hda_codec_analog arc4 hid_generic iwl3945 > iwlegacy microcode psmouse serio_raw mac80211 i2c_i801 i2c_core snd_hda_intel > snd_hda_codec cfg80211 lpc_ich mfd_core snd_pcm snd_page_alloc e1000e > snd_timer wmi thinkpad_acpi nvram snd soundcore evdev usbhid hid binfmt_misc > loop vhost_net tun kvm_intel nfsd kvm exportfs nfs_acl auth_rpcgss lockd > sunrpc dm_mod autofs4 pcmcia sr_mod cdrom sdhci_pci sdhci mmc_core > yenta_socket ehci_hcd uhci_hcd usbcore usb_common video backlight > > Pid: 8691, comm: kworker/1:0 Not tainted 3.5.0-rc4-00006-g6fe139e #73 LENOVO > 6464CTO/6464CTO > RIP: 0010:[<ffffffffa031204d>] [<ffffffffa031204d>] > dm_rh_region_context+0xd/0x20 [dm_region_hash] > RSP: 0018:ffff880126b77c68 EFLAGS: 00010282 > RAX: 0000000000000000 RBX: ffff880121423450 RCX: 0000000000002607 > RDX: ffff880121676e40 RSI: 0000000000000000 RDI: ffff880121676e40 > RBP: ffff880126b77cb0 R08: 000060fec8800fe0 R09: 0000000000000000 > R10: 0000000000000001 R11: 0000000000000000 R12: ffff880121626400 > R13: ffff880121676e40 R14: 0000000000000000 R15: 0000000000000000 > FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: 0000000000000178 CR3: 0000000134a40000 CR4: 00000000000007e0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process kworker/1:0 (pid: 8691, threadinfo ffff880126b76000, task > ffff8801344ca310) > Stack: > ffffffffa0332ac3 ffff880126b77c80 0000000000000000 ffff880126b77cb0 > ffff880121423450 ffff880121626400 ffffffffa0332a90 0000000000000000 > 0000000000000000 ffff880126b77d00 ffffffffa00eca04 ffff880126b77ce0 > Call Trace: > [<ffffffffa0332ac3>] ? recovery_complete+0x33/0x170 [dm_mirror] > [<ffffffffa0332a90>] ? recover+0x1a0/0x1a0 [dm_mirror] > [<ffffffffa00eca04>] run_complete_job+0x74/0xd0 [dm_mod] > [<ffffffffa00ed525>] process_jobs+0x95/0x120 [dm_mod] > [<ffffffffa00ec990>] ? wake+0x20/0x20 [dm_mod] > [<ffffffffa00ed5e9>] do_work+0x39/0x80 [dm_mod] > [<ffffffff81060065>] process_one_work+0x1a5/0x6b0 > [<ffffffff8105ffff>] ? process_one_work+0x13f/0x6b0 > [<ffffffff815299c9>] ? _raw_spin_lock_irq+0x19/0x80 > [<ffffffffa00ed5b0>] ? process_jobs+0x120/0x120 [dm_mod] > [<ffffffff810608ee>] worker_thread+0x12e/0x2d0 > [<ffffffff810607c0>] ? rescuer_thread+0x210/0x210 > [<ffffffff81069cf2>] kthread+0xa2/0xb0 > [<ffffffff81533334>] kernel_thread_helper+0x4/0x10 > [<ffffffff81076e3c>] ? finish_task_switch+0x7c/0x120 > [<ffffffff8152a54b>] ? _raw_spin_unlock_irq+0x3b/0x60 > [<ffffffff8152a85d>] ? retint_restore_args+0xe/0xe > [<ffffffff81069c50>] ? kthread_worker_fn+0x1a0/0x1a0 > [<ffffffff81533330>] ? gs_change+0xb/0xb > Code: 89 e5 66 66 66 66 90 48 8b 06 8b 4f 04 5d 48 2b 87 80 01 00 00 48 d3 e8 > c3 0f 1f 44 00 00 55 48 89 e5 66 66 66 66 90 48 8b 07 5d <48> 8b 80 78 01 00 > 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 > RIP [<ffffffffa031204d>] dm_rh_region_context+0xd/0x20 [dm_region_hash] > RSP <ffff880126b77c68> > CR2: 0000000000000178 > > > # bad: [cfaf025112d3856637ff34a767ef785ef5cf2ca9] Linux 3.5-rc2 > # good: [76e10d158efb6d4516018846f60c2ab5501900bc] Linux 3.4 > git bisect start 'v3.5-rc2' 'v3.4' > # good: [3813d4024a75562baf77d3907fb6afbf8f9c8232] Merge tag 'ia64-3.5-merge' > of git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux > git bisect good 3813d4024a75562baf77d3907fb6afbf8f9c8232 > # good: [5723aa993d83803157c22327e90cd59e3dcbe879] x86: use the new generic > strnlen_user() function > git bisect good 5723aa993d83803157c22327e90cd59e3dcbe879 > # bad: [a70f35af4e49f87ba4b6c4b30220fbb66cd74af6] Merge branch > 'for-3.5/drivers' of git://git.kernel.dk/linux-block > git bisect bad a70f35af4e49f87ba4b6c4b30220fbb66cd74af6 > # good: [53f2c4a8fd882009a2a75c5b72d6898c0808616e] Merge tag 'nfs-for-3.5-1' > of git://git.linux-nfs.org/projects/trondmy/linux-nfs > git bisect good 53f2c4a8fd882009a2a75c5b72d6898c0808616e > # bad: [227d1e4319ffd8729781941d92f4ae4d85beecd9] Merge branch 'for-linus' of > git://git.linaro.org/people/rmk/linux-arm > git bisect bad 227d1e4319ffd8729781941d92f4ae4d85beecd9 > # bad: [be9cd873e2a706a688e37224d48e135efd8c0d26] mm/buddy: dump > PG_compound_lock page flag > git bisect bad be9cd873e2a706a688e37224d48e135efd8c0d26 > # good: [1cb3642a68c983ada0f4090a4dac1d70a96126ca] mfd: mc13xxx core should > not be user visible > git bisect good 1cb3642a68c983ada0f4090a4dac1d70a96126ca > # good: [bde8bd8a1d5242589ddcaef8e017b48b207c4729] mm/vmstat.c: remove debug > fs entries on failure of file creation and made extfrag_debug_root dentry > local > git bisect good bde8bd8a1d5242589ddcaef8e017b48b207c4729 > # good: [549381e19cc7874bb15f0e4760c1004d4fe28d8a] mm: bootmem: remove > redundant offset check when finally freeing bootmem > git bisect good 549381e19cc7874bb15f0e4760c1004d4fe28d8a > # good: [bde05d1ccd512696b09db9dd2e5f33ad19152605] shmem: replace page if > mapping excludes its zone > git bisect good bde05d1ccd512696b09db9dd2e5f33ad19152605 > # bad: [17cf28afea2a1112f240a3a2da8af883be024811] mm/fs: remove > truncate_range > git bisect bad 17cf28afea2a1112f240a3a2da8af883be024811 > # good: [ec9516fbc5fa814014991e1ae7f8860127122105] tmpfs: optimize clearing > when writing > git bisect good ec9516fbc5fa814014991e1ae7f8860127122105 ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-06-30 19:55 ` Hugh Dickins @ 2012-06-30 20:51 ` Zdenek Kabelac 2012-06-30 23:10 ` Hugh Dickins 0 siblings, 1 reply; 13+ messages in thread From: Zdenek Kabelac @ 2012-06-30 20:51 UTC (permalink / raw) To: Hugh Dickins Cc: linux-kernel, amwang, Alasdair G Kergon, LVM general discussion and development Dne 30.6.2012 21:55, Hugh Dickins napsal(a): > On Sat, 30 Jun 2012, Zdenek Kabelac wrote: >> >> When I've used 3.5-rc kernels - I've noticed kernel deadlocks. >> Ooops log included. After some experimenting - reliable way to hit this oops >> is to run lvm test suite for 10 minutes. Since 3.5 merge window does not >> included anything related to this oops I've went for bisect. > > Thanks a lot for reporting, and going to such effort to find > a reproducible testcase that you could bisect on. > >> >> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17 >> >> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE > > But this leaves me very puzzled. > > Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git > under tests/ ? Yes - that's it - make as root: cd test make check_local (inside test subdirectory should be enough, if not - just report any problem) > > I see no mention of madvise or MADV_REMOVE or fallocate or anything > related in that git tree. > > If you have something else running at the same time, which happens to use > madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables > it on (I guess ext4 from the =y in your config), then I suppose we should > start searching for improper memory freeing or scribbling in its holepunch > support: something that might be corrupting the dm_region in your oops. What the test is doing - it creates file in LVM_TEST_DIR (default is /tmp) and using loop device to simulate device (small size - it should fit bellow 200MB) Within this file second layer through virtual DM devices is created and simulates various numbers of PV devices to play with. So since everything now support TRIM - such operations should be passed down to the backend file - which probably triggers the path. > I'll be surprised if that is the case, but it's something that you can > easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove(): > that should tell us what process is using it. I could try that if that will help. > I'm not an LVM user, so I doubt I'll be able to reproduce your setup. Shouldn't be hard to run - unsure if every config setup is influnenced or just mine config. > > Any ideas from the DM guys? Has anyone else seen anything like this? > > Do all your oopses look like this one? I think I've get yet another one - but also within dm_rh_region It could be that your patch exposed problem of some different part of stack - not really sure - it's just now with 3.5 this crash will not allow to pass whole test suite - I've tried also in kvm machine and it's been reproducible (so in the worst case I could eventually send you 2GB image) The problem is - there is not a 'single test case' to trigger the oops (at least I've not figured out one) - it's the combination of multiple tests running after each other - but for simplication this should be enough: make check_local T=shell/lvconvert Which usually dies on shell/lvconvert-repair-transient.sh Zdenek ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-06-30 20:51 ` Zdenek Kabelac @ 2012-06-30 23:10 ` Hugh Dickins 2012-07-01 12:57 ` Zdenek Kabelac 0 siblings, 1 reply; 13+ messages in thread From: Hugh Dickins @ 2012-06-30 23:10 UTC (permalink / raw) To: Zdenek Kabelac Cc: linux-kernel, amwang, Alasdair G Kergon, LVM general discussion and development On Sat, 30 Jun 2012, Zdenek Kabelac wrote: > Dne 30.6.2012 21:55, Hugh Dickins napsal(a): > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote: > > > > > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks. > > > Ooops log included. After some experimenting - reliable way to hit this > > > oops > > > is to run lvm test suite for 10 minutes. Since 3.5 merge window does not > > > included anything related to this oops I've went for bisect. > > > > Thanks a lot for reporting, and going to such effort to find > > a reproducible testcase that you could bisect on. > > > > > > > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17 > > > > > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE > > > > But this leaves me very puzzled. > > > > Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git > > under tests/ ? > > Yes - that's it - > > make > as root: > cd test > make check_local > > (inside test subdirectory should be enough, if not - just report any problem) > > > > > I see no mention of madvise or MADV_REMOVE or fallocate or anything > > related in that git tree. > > > > If you have something else running at the same time, which happens to use > > madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables > > it on (I guess ext4 from the =y in your config), then I suppose we should > > start searching for improper memory freeing or scribbling in its holepunch > > support: something that might be corrupting the dm_region in your oops. > > What the test is doing - it creates file in LVM_TEST_DIR (default is /tmp) > and using loop device to simulate device (small size - it should fit bellow > 200MB) > > Within this file second layer through virtual DM devices is created and > simulates various numbers of PV devices to play with. This sounds much easier to set up than I was expecting: thanks for the info, I'll try it later on today. > > So since everything now support TRIM - such operations should be passed > down to the backend file - which probably triggers the path. What filesystem do you have for /tmp? If tmpfs, then it will make much more sense if we assume your bisection endpoint was off by one. Your bisection log was not quite complete; and even if it did appear to converge on the commit you cite, you might have got (un)lucky when testing the commit before it, and concluded "good" when more attempts would have said "bad". The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a much more likely first bad commit if your /tmp is on tmpfs: that does indeed wire up loop to pass TRIM down to tmpfs by fallocate - that indeed played a part in my own testing. Whereas if your /tmp is on ext4, loop has been passing TRIM down with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) should be completely irrelevant. > > > I'll be surprised if that is the case, but it's something that you can > > easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove(): > > that should tell us what process is using it. > > I could try that if that will help. That would help, if you're very sure of your bisection endpoint; but if your /tmp is on tmpfs, then I do think it's more likely that you've actually found a bug in the commit before. > > > I'm not an LVM user, so I doubt I'll be able to reproduce your setup. > > Shouldn't be hard to run - unsure if every config setup is influnenced > or just mine config. I'll start from your config. > > > > > Any ideas from the DM guys? Has anyone else seen anything like this? > > > > Do all your oopses look like this one? > > I think I've get yet another one - but also within dm_rh_region > > It could be that your patch exposed problem of some different part of stack - > not really sure - it's just now with 3.5 this crash will not allow to pass > whole test suite - I've tried also in kvm machine and it's been > reproducible (so in the worst case I could eventually send you 2GB image) > > The problem is - there is not a 'single test case' to trigger the oops (at > least I've not figured out one) - it's the combination of multiple tests > running after each other - but for simplication this should be enough: > > make check_local T=shell/lvconvert > > Which usually dies on shell/lvconvert-repair-transient.sh Thanks again, I'll report back later. Hugh ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-06-30 23:10 ` Hugh Dickins @ 2012-07-01 12:57 ` Zdenek Kabelac 2012-07-01 18:45 ` Hugh Dickins 0 siblings, 1 reply; 13+ messages in thread From: Zdenek Kabelac @ 2012-07-01 12:57 UTC (permalink / raw) To: Hugh Dickins Cc: linux-kernel, Lukas Czerner, amwang, Alasdair G Kergon, LVM general discussion and development Dne 1.7.2012 01:10, Hugh Dickins napsal(a): > On Sat, 30 Jun 2012, Zdenek Kabelac wrote: >> Dne 30.6.2012 21:55, Hugh Dickins napsal(a): >>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote: >>>> >>>> When I've used 3.5-rc kernels - I've noticed kernel deadlocks. >>>> Ooops log included. After some experimenting - reliable way to hit this >>>> oops >>>> is to run lvm test suite for 10 minutes. Since 3.5 merge window does not >>>> included anything related to this oops I've went for bisect. >>> >>> Thanks a lot for reporting, and going to such effort to find >>> a reproducible testcase that you could bisect on. >>> >>>> >>>> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17 >>>> >>>> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE >>> >>> But this leaves me very puzzled. >>> >>> Is the "lvm test suite" what I find at git.fedorahosted.org/git/lvm2.git >>> under tests/ ? >> >> Yes - that's it - >> >> make >> as root: >> cd test >> make check_local >> >> (inside test subdirectory should be enough, if not - just report any problem) >> >>> If you have something else running at the same time, which happens to use >>> madvise(,,MADV_REMOVE) on a filesystem which the commit above now enables >>> it on (I guess ext4 from the =y in your config), then I suppose we should >>> start searching for improper memory freeing or scribbling in its holepunch >>> support: something that might be corrupting the dm_region in your oops. >> >> What the test is doing - it creates file in LVM_TEST_DIR (default is /tmp) >> and using loop device to simulate device (small size - it should fit bellow >> 200MB) >> >> Within this file second layer through virtual DM devices is created and >> simulates various numbers of PV devices to play with. > > This sounds much easier to set up than I was expecting: > thanks for the info, I'll try it later on today. > >> >> So since everything now support TRIM - such operations should be passed >> down to the backend file - which probably triggers the path. > > What filesystem do you have for /tmp? > > If tmpfs, then it will make much more sense if we assume your bisection > endpoint was off by one. Your bisection log was not quite complete; > and even if it did appear to converge on the commit you cite, you might > have got (un)lucky when testing the commit before it, and concluded > "good" when more attempts would have said "bad". > > The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe > "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a > much more likely first bad commit if your /tmp is on tmpfs: > that does indeed wire up loop to pass TRIM down to tmpfs by > fallocate - that indeed played a part in my own testing. > > Whereas if your /tmp is on ext4, loop has been passing TRIM down > with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) > should be completely irrelevant. While I've been aware of the fact that tmpfs was enhanced with trim support - I've not tried to run on real ext4 filesystem since for my tests I'm using tmpfs for quite some time to safe rewrites of SSD :) So now I've checked with real ext4 - and the bug is there as well so I've went back - it crashes on 3.4, 3.3 and 3.2 as well. 3.1 is the first kernel which does survive (checked 5 repeated runs) And you are correct, the first commit which causes crash really is 83e4fa9c16e4af when I use tmpfs as backend storage - the problem why I've missed to properly identify this commit in my bisect is that crash usually happens on the second pass of the lvm test suite 'make check_local' execution - and I've been running test just once. To be sure I've run 5 run passes on 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing usually on second run, with commit 3f31d07571e the crash always happens in the first pass. I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 and it's crashing as well - so it's probably not uniqueness of my config. So is there any primary suspect in 3.2 which is worth to check - or I need another day to play another bisect game ? > >> >>> I'll be surprised if that is the case, but it's something that you can >>> easily check by inserting a WARN_ON(1) in mm/madvise.c madvise_remove(): >>> that should tell us what process is using it. >> >> I could try that if that will help. > > That would help, if you're very sure of your bisection endpoint; > but if your /tmp is on tmpfs, then I do think it's more likely > that you've actually found a bug in the commit before. the only thing which could be tricky is udev support (by default it's not enabled ---enable-udev_sync) However Debian based distros are distributing their own rules, which are not 100% compatible with upstream and create some unpredictable issues, where the slowness is the least problem.) If you have Fedora Rawhide with latest lvm2 installed - you should get pretty well configured system for running test dir (unfortunately there is no way to virtualize udev...) Zdenek ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-07-01 12:57 ` Zdenek Kabelac @ 2012-07-01 18:45 ` Hugh Dickins 2012-07-01 20:10 ` Zdenek Kabelac 0 siblings, 1 reply; 13+ messages in thread From: Hugh Dickins @ 2012-07-01 18:45 UTC (permalink / raw) To: Zdenek Kabelac Cc: amwang, Mike Snitzer, linux-kernel, Joe Thornber, LVM general discussion and development, Lukas Czerner, Alasdair G Kergon On Sun, 1 Jul 2012, Zdenek Kabelac wrote: > Dne 1.7.2012 01:10, Hugh Dickins napsal(a): > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote: > > > Dne 30.6.2012 21:55, Hugh Dickins napsal(a): > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote: > > > > > > > > > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks. > > > > > Ooops log included. After some experimenting - reliable way to hit > > > > > this > > > > > oops > > > > > is to run lvm test suite for 10 minutes. Since 3.5 merge window does > > > > > not > > > > > included anything related to this oops I've went for bisect. > > > > > > > > Thanks a lot for reporting, and going to such effort to find > > > > a reproducible testcase that you could bisect on. > > > > > > > > > > > > > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17 > > > > > > > > > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE > > > > > > > > But this leaves me very puzzled. > > > > > > > > Is the "lvm test suite" what I find at > > > > git.fedorahosted.org/git/lvm2.git > > > > under tests/ ? > > > > > > Yes - that's it - > > > > > > make > > > as root: > > > cd test > > > make check_local > > > > > > (inside test subdirectory should be enough, if not - just report any > > > problem) > > > > > > > If you have something else running at the same time, which happens to > > > > use > > > > madvise(,,MADV_REMOVE) on a filesystem which the commit above now > > > > enables > > > > it on (I guess ext4 from the =y in your config), then I suppose we > > > > should > > > > start searching for improper memory freeing or scribbling in its > > > > holepunch > > > > support: something that might be corrupting the dm_region in your oops. > > > > > > What the test is doing - it creates file in LVM_TEST_DIR (default is > > > /tmp) I ran "LVM_TEST_DIR=/tmp make check_local": without that it appeared to be using a subdirectory made under test/. And being a year or two out of date in my userspace, and unfamiliar with the syntax and whereabouts of lvm.conf, it was easiest for me to hack lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1 after I spotted a warning message about issue_discards. > > > and using loop device to simulate device (small size - it should fit > > > bellow > > > 200MB) > > > > > > Within this file second layer through virtual DM devices is created and > > > simulates various numbers of PV devices to play with. > > > > This sounds much easier to set up than I was expecting: > > thanks for the info, I'll try it later on today. Sorry, I never reached it yesterday, but arrived there this morning. > > > > > > > > So since everything now support TRIM - such operations should be passed > > > down to the backend file - which probably triggers the path. > > > > What filesystem do you have for /tmp? From your later remarks, I inferred tmpfs. > > > > If tmpfs, then it will make much more sense if we assume your bisection > > endpoint was off by one. Your bisection log was not quite complete; > > and even if it did appear to converge on the commit you cite, you might > > have got (un)lucky when testing the commit before it, and concluded > > "good" when more attempts would have said "bad". > > > > The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe > > "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a > > much more likely first bad commit if your /tmp is on tmpfs: > > that does indeed wire up loop to pass TRIM down to tmpfs by > > fallocate - that indeed played a part in my own testing. > > > > Whereas if your /tmp is on ext4, loop has been passing TRIM down > > with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) > > should be completely irrelevant. > > While I've been aware of the fact that tmpfs was enhanced with trim support - > I've not tried to run on real ext4 filesystem since for my tests I'm using > tmpfs for quite some time to safe rewrites of SSD :) > > So now I've checked with real ext4 - and the bug is there as well > so I've went back - it crashes on 3.4, 3.3 and 3.2 as well. > > 3.1 is the first kernel which does survive (checked 5 repeated runs) Very useful research, thank you. > > And you are correct, the first commit which causes crash really is > 83e4fa9c16e4af when I use tmpfs as backend storage - the problem why I've > missed to properly identify this commit in my bisect is that crash usually > happens on the second pass of the lvm test suite 'make check_local' execution > - and I've been running test just once. To be sure I've run 5 run passes on > 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing > usually on second run, with commit 3f31d07571e the crash always happens in > the first pass. > > I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 > and it's crashing as well - so it's probably not uniqueness of my config. > > So is there any primary suspect in 3.2 which is worth to check - or I need > another day to play another bisect game ? No need for a further bisect: 3.2 is when the disard/trim/fallocate support went into drivers/block/loop.c, so these tests would be unable to show if DM was right or wrong before then. > > > > > > > > > > I'll be surprised if that is the case, but it's something that you can > > > > easily check by inserting a WARN_ON(1) in mm/madvise.c > > > > madvise_remove(): > > > > that should tell us what process is using it. > > > > > > I could try that if that will help. > > > > That would help, if you're very sure of your bisection endpoint; > > but if your /tmp is on tmpfs, then I do think it's more likely > > that you've actually found a bug in the commit before. > > the only thing which could be tricky is udev support > (by default it's not enabled ---enable-udev_sync) > However Debian based distros are distributing their own rules, which are not > 100% compatible with upstream and create some unpredictable issues, > where the slowness is the least problem.) > > If you have Fedora Rawhide with latest lvm2 installed - you should get pretty > well configured system for running test dir (unfortunately there is no way > to virtualize udev...) I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS I did quickly crash around where you expected; though in my case it was in dm_rh_dec() called from mirror_end_io(). But I've not taken it any further than that. You've shown that it's as much a problem with ext4 as with tmpfs, and has been a problem ever since these tests' use of discard reached DM. I think it's fair to assume that there's something wrong with DM's handling of REQ_DISCARD. (Perhaps it was all correct when written, but I think there has been a series of modifications to REQ_DISCARD handling in the block layer, it's been rather troublesome.) I've added a few people to the Cc, innocents fingered by git blame on some REQ_DISCARD lines in drivers/md. But I'll back out myself: I'm not short of other things to look at, and have little to offer in drivers/md. Hugh ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-07-01 18:45 ` Hugh Dickins @ 2012-07-01 20:10 ` Zdenek Kabelac 2012-07-01 22:03 ` Zdenek Kabelac 0 siblings, 1 reply; 13+ messages in thread From: Zdenek Kabelac @ 2012-07-01 20:10 UTC (permalink / raw) To: Hugh Dickins Cc: amwang, Mike Snitzer, linux-kernel, Joe Thornber, LVM general discussion and development, Lukas Czerner, Alasdair G Kergon Dne 1.7.2012 20:45, Hugh Dickins napsal(a): > On Sun, 1 Jul 2012, Zdenek Kabelac wrote: >> Dne 1.7.2012 01:10, Hugh Dickins napsal(a): >>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote: >>>> Dne 30.6.2012 21:55, Hugh Dickins napsal(a): >>>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote: >>>>>> >>>>>> When I've used 3.5-rc kernels - I've noticed kernel deadlocks. >>>>>> Ooops log included. After some experimenting - reliable way to hit >>>>>> this >>>>>> oops >>>>>> is to run lvm test suite for 10 minutes. Since 3.5 merge window does >>>>>> not >>>>>> included anything related to this oops I've went for bisect. >>>>> >>>>> Thanks a lot for reporting, and going to such effort to find >>>>> a reproducible testcase that you could bisect on. >>>>> >>>>>> >>>>>> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17 >>>>>> >>>>>> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE >>>>> >>>>> But this leaves me very puzzled. >>>>> >>>>> Is the "lvm test suite" what I find at >>>>> git.fedorahosted.org/git/lvm2.git >>>>> under tests/ ? >>>> >>>> Yes - that's it - >>>> >>>> make >>>> as root: >>>> cd test >>>> make check_local >>>> >>>> (inside test subdirectory should be enough, if not - just report any >>>> problem) >>>> >>>>> If you have something else running at the same time, which happens to >>>>> use >>>>> madvise(,,MADV_REMOVE) on a filesystem which the commit above now >>>>> enables >>>>> it on (I guess ext4 from the =y in your config), then I suppose we >>>>> should >>>>> start searching for improper memory freeing or scribbling in its >>>>> holepunch >>>>> support: something that might be corrupting the dm_region in your oops. >>>> >>>> What the test is doing - it creates file in LVM_TEST_DIR (default is >>>> /tmp) > > I ran "LVM_TEST_DIR=/tmp make check_local": > without that it appeared to be using a subdirectory made under test/. > > And being a year or two out of date in my userspace, and unfamiliar with > the syntax and whereabouts of lvm.conf, it was easiest for me to hack > lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1 > after I spotted a warning message about issue_discards. > >>>> and using loop device to simulate device (small size - it should fit >>>> bellow >>>> 200MB) >>>> >>>> Within this file second layer through virtual DM devices is created and >>>> simulates various numbers of PV devices to play with. >>> >>> This sounds much easier to set up than I was expecting: >>> thanks for the info, I'll try it later on today. > > Sorry, I never reached it yesterday, but arrived there this morning. > >>> >>>> >>>> So since everything now support TRIM - such operations should be passed >>>> down to the backend file - which probably triggers the path. >>> >>> What filesystem do you have for /tmp? > > From your later remarks, I inferred tmpfs. > >>> >>> If tmpfs, then it will make much more sense if we assume your bisection >>> endpoint was off by one. Your bisection log was not quite complete; >>> and even if it did appear to converge on the commit you cite, you might >>> have got (un)lucky when testing the commit before it, and concluded >>> "good" when more attempts would have said "bad". >>> >>> The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe >>> "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a >>> much more likely first bad commit if your /tmp is on tmpfs: >>> that does indeed wire up loop to pass TRIM down to tmpfs by >>> fallocate - that indeed played a part in my own testing. >>> >>> Whereas if your /tmp is on ext4, loop has been passing TRIM down >>> with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) >>> should be completely irrelevant. >> >> While I've been aware of the fact that tmpfs was enhanced with trim support - >> I've not tried to run on real ext4 filesystem since for my tests I'm using >> tmpfs for quite some time to safe rewrites of SSD :) >> >> So now I've checked with real ext4 - and the bug is there as well >> so I've went back - it crashes on 3.4, 3.3 and 3.2 as well. >> >> 3.1 is the first kernel which does survive (checked 5 repeated runs) > > Very useful research, thank you. > >> >> And you are correct, the first commit which causes crash really is >> 83e4fa9c16e4af when I use tmpfs as backend storage - the problem why I've >> missed to properly identify this commit in my bisect is that crash usually >> happens on the second pass of the lvm test suite 'make check_local' execution >> - and I've been running test just once. To be sure I've run 5 run passes on >> 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing >> usually on second run, with commit 3f31d07571e the crash always happens in >> the first pass. >> >> I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 >> and it's crashing as well - so it's probably not uniqueness of my config. >> >> So is there any primary suspect in 3.2 which is worth to check - or I need >> another day to play another bisect game ? > > No need for a further bisect: 3.2 is when the disard/trim/fallocate > support went into drivers/block/loop.c, so these tests would be unable > to show if DM was right or wrong before then. Well I've played meanwhile the game with minimized kernel config and the outcome is: last working kernel is: 3.1.0-rc1-00008-g548ef6c first broken: 3.1.0-rc1-00009-gdfaa2ef dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef loop: add discard support for loop devices > I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS > I did quickly crash around where you expected; though in my case > it was in dm_rh_dec() called from mirror_end_io(). Change of Issue Discards option in lvm.conf is not needed. I'm able to get these oopses with this setting turned off. > But I've not taken it any further than that. You've shown that it's > as much a problem with ext4 as with tmpfs, and has been a problem > ever since these tests' use of discard reached DM. > > I think it's fair to assume that there's something wrong with DM's > handling of REQ_DISCARD. (Perhaps it was all correct when written, > but I think there has been a series of modifications to REQ_DISCARD > handling in the block layer, it's been rather troublesome.) > So does anyone has some idea what should be checked next ? Zdenek ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-07-01 20:10 ` Zdenek Kabelac @ 2012-07-01 22:03 ` Zdenek Kabelac 2012-07-02 9:00 ` Lukáš Czerner 2012-07-02 13:33 ` Mike Snitzer 0 siblings, 2 replies; 13+ messages in thread From: Zdenek Kabelac @ 2012-07-01 22:03 UTC (permalink / raw) To: Hugh Dickins Cc: amwang, Mike Snitzer, linux-kernel, Joe Thornber, LVM general discussion and development, Lukas Czerner, Alasdair G Kergon Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a): > Dne 1.7.2012 20:45, Hugh Dickins napsal(a): >> On Sun, 1 Jul 2012, Zdenek Kabelac wrote: >>> Dne 1.7.2012 01:10, Hugh Dickins napsal(a): >>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote: >>>>> Dne 30.6.2012 21:55, Hugh Dickins napsal(a): >>>>>> On Sat, 30 Jun 2012, Zdenek Kabelac wrote: >>>>>>> >>>>>>> When I've used 3.5-rc kernels - I've noticed kernel deadlocks. >>>>>>> Ooops log included. After some experimenting - reliable way to hit >>>>>>> this >>>>>>> oops >>>>>>> is to run lvm test suite for 10 minutes. Since 3.5 merge window does >>>>>>> not >>>>>>> included anything related to this oops I've went for bisect. >>>>>> >>>>>> Thanks a lot for reporting, and going to such effort to find >>>>>> a reproducible testcase that you could bisect on. >>>>>> >>>>>>> >>>>>>> Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17 >>>>>>> >>>>>>> mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE >>>>>> >>>>>> But this leaves me very puzzled. >>>>>> >>>>>> Is the "lvm test suite" what I find at >>>>>> git.fedorahosted.org/git/lvm2.git >>>>>> under tests/ ? >>>>> >>>>> Yes - that's it - >>>>> >>>>> make >>>>> as root: >>>>> cd test >>>>> make check_local >>>>> >>>>> (inside test subdirectory should be enough, if not - just report any >>>>> problem) >>>>> >>>>>> If you have something else running at the same time, which happens to >>>>>> use >>>>>> madvise(,,MADV_REMOVE) on a filesystem which the commit above now >>>>>> enables >>>>>> it on (I guess ext4 from the =y in your config), then I suppose we >>>>>> should >>>>>> start searching for improper memory freeing or scribbling in its >>>>>> holepunch >>>>>> support: something that might be corrupting the dm_region in your oops. >>>>> >>>>> What the test is doing - it creates file in LVM_TEST_DIR (default is >>>>> /tmp) >> >> I ran "LVM_TEST_DIR=/tmp make check_local": >> without that it appeared to be using a subdirectory made under test/. >> >> And being a year or two out of date in my userspace, and unfamiliar with >> the syntax and whereabouts of lvm.conf, it was easiest for me to hack >> lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1 >> after I spotted a warning message about issue_discards. >> >>>>> and using loop device to simulate device (small size - it should fit >>>>> bellow >>>>> 200MB) >>>>> >>>>> Within this file second layer through virtual DM devices is created and >>>>> simulates various numbers of PV devices to play with. >>>> >>>> This sounds much easier to set up than I was expecting: >>>> thanks for the info, I'll try it later on today. >> >> Sorry, I never reached it yesterday, but arrived there this morning. >> >>>> >>>>> >>>>> So since everything now support TRIM - such operations should be passed >>>>> down to the backend file - which probably triggers the path. >>>> >>>> What filesystem do you have for /tmp? >> >> From your later remarks, I inferred tmpfs. >> >>>> >>>> If tmpfs, then it will make much more sense if we assume your bisection >>>> endpoint was off by one. Your bisection log was not quite complete; >>>> and even if it did appear to converge on the commit you cite, you might >>>> have got (un)lucky when testing the commit before it, and concluded >>>> "good" when more attempts would have said "bad". >>>> >>>> The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe >>>> "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a >>>> much more likely first bad commit if your /tmp is on tmpfs: >>>> that does indeed wire up loop to pass TRIM down to tmpfs by >>>> fallocate - that indeed played a part in my own testing. >>>> >>>> Whereas if your /tmp is on ext4, loop has been passing TRIM down >>>> with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) >>>> should be completely irrelevant. >>> >>> While I've been aware of the fact that tmpfs was enhanced with trim support - >>> I've not tried to run on real ext4 filesystem since for my tests I'm using >>> tmpfs for quite some time to safe rewrites of SSD :) >>> >>> So now I've checked with real ext4 - and the bug is there as well >>> so I've went back - it crashes on 3.4, 3.3 and 3.2 as well. >>> >>> 3.1 is the first kernel which does survive (checked 5 repeated runs) >> >> Very useful research, thank you. >> >>> >>> And you are correct, the first commit which causes crash really is >>> 83e4fa9c16e4af when I use tmpfs as backend storage - the problem why I've >>> missed to properly identify this commit in my bisect is that crash usually >>> happens on the second pass of the lvm test suite 'make check_local' execution >>> - and I've been running test just once. To be sure I've run 5 run passes on >>> 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing >>> usually on second run, with commit 3f31d07571e the crash always happens in >>> the first pass. >>> >>> I've also checked some rawhide kernel vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 >>> and it's crashing as well - so it's probably not uniqueness of my config. >>> >>> So is there any primary suspect in 3.2 which is worth to check - or I need >>> another day to play another bisect game ? >> >> No need for a further bisect: 3.2 is when the disard/trim/fallocate >> support went into drivers/block/loop.c, so these tests would be unable >> to show if DM was right or wrong before then. > > Well I've played meanwhile the game with minimized kernel config > and the outcome is: > > last working kernel is: 3.1.0-rc1-00008-g548ef6c > first broken: 3.1.0-rc1-00009-gdfaa2ef > > dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef > loop: add discard support for loop devices > >> I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS >> I did quickly crash around where you expected; though in my case >> it was in dm_rh_dec() called from mirror_end_io(). > > Change of Issue Discards option in lvm.conf is not needed. > I'm able to get these oopses with this setting turned off. > >> But I've not taken it any further than that. You've shown that it's >> as much a problem with ext4 as with tmpfs, and has been a problem >> ever since these tests' use of discard reached DM. >> >> I think it's fair to assume that there's something wrong with DM's >> handling of REQ_DISCARD. (Perhaps it was all correct when written, >> but I think there has been a series of modifications to REQ_DISCARD >> handling in the block layer, it's been rather troublesome.) >> > > So does anyone has some idea what should be checked next ? I've enabled more kernel debugging tools - and this came out when running test suite: Unsure if this could be source of troubles? kernel BUG at arch/x86/mm/physaddr.c:18! invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC CPU 1 Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945 iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4 Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO 6464CTO/6464CTO RIP: 0010:[<ffffffff81021f00>] [<ffffffff81021f00>] __phys_addr+0x60/0x70 RSP: 0018:ffff880136e41aa0 EFLAGS: 00010213 RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000 RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40 R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40 FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090) Stack: ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0 ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000 Call Trace: [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod] [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod] [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod] [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror] [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror] [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror] [<ffffffff81050177>] process_one_work+0x197/0x460 [<ffffffff81050116>] ? process_one_work+0x136/0x460 [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror] [<ffffffff81050986>] worker_thread+0x126/0x2d0 [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0 [<ffffffff81056066>] kthread+0x96/0xa0 [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10 [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70 [<ffffffff8133b610>] ? gs_change+0xb/0xb Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba RIP [<ffffffff81021f00>] __phys_addr+0x60/0x70 RSP <ffff880136e41aa0> ---[ end trace 040dfd9d68fccabc ]--- Zdenek ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-07-01 22:03 ` Zdenek Kabelac @ 2012-07-02 9:00 ` Lukáš Czerner 2012-07-02 10:35 ` Lukáš Czerner 2012-07-02 13:33 ` Mike Snitzer 1 sibling, 1 reply; 13+ messages in thread From: Lukáš Czerner @ 2012-07-02 9:00 UTC (permalink / raw) To: Zdenek Kabelac Cc: amwang, Mike Snitzer, Hugh Dickins, linux-kernel, Joe Thornber, LVM general discussion and development, Lukas Czerner, Alasdair G Kergon On Mon, 2 Jul 2012, Zdenek Kabelac wrote: > Date: Mon, 02 Jul 2012 00:03:07 +0200 > From: Zdenek Kabelac <zkabelac@redhat.com> > To: Hugh Dickins <hughd@google.com> > Cc: Mike Snitzer <snitzer@redhat.com>, Mikulas Patocka <mpatocka@redhat.com>, > Joe Thornber <ejt@redhat.com>, > LVM general discussion and development <linux-lvm@redhat.com>, > amwang@redhat.com, Alasdair G Kergon <agk@redhat.com>, > linux-kernel@vger.kernel.org, Lukas Czerner <lczerner@redhat.com> > Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel > > Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a): > > Dne 1.7.2012 20:45, Hugh Dickins napsal(a): > > > On Sun, 1 Jul 2012, Zdenek Kabelac wrote: > > > > Dne 1.7.2012 01:10, Hugh Dickins napsal(a): > > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote: > > > > > > Dne 30.6.2012 21:55, Hugh Dickins napsal(a): > > > > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote: > > > > > > > > > > > > > > > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks. > > > > > > > > Ooops log included. After some experimenting - reliable way to > > > > > > > > hit > > > > > > > > this > > > > > > > > oops > > > > > > > > is to run lvm test suite for 10 minutes. Since 3.5 merge window > > > > > > > > does > > > > > > > > not > > > > > > > > included anything related to this oops I've went for bisect. > > > > > > > > > > > > > > Thanks a lot for reporting, and going to such effort to find > > > > > > > a reproducible testcase that you could bisect on. > > > > > > > > > > > > > > > > > > > > > > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17 > > > > > > > > > > > > > > > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE > > > > > > > > > > > > > > But this leaves me very puzzled. > > > > > > > > > > > > > > Is the "lvm test suite" what I find at > > > > > > > git.fedorahosted.org/git/lvm2.git > > > > > > > under tests/ ? > > > > > > > > > > > > Yes - that's it - > > > > > > > > > > > > make > > > > > > as root: > > > > > > cd test > > > > > > make check_local > > > > > > > > > > > > (inside test subdirectory should be enough, if not - just report any > > > > > > problem) > > > > > > > > > > > > > If you have something else running at the same time, which happens > > > > > > > to > > > > > > > use > > > > > > > madvise(,,MADV_REMOVE) on a filesystem which the commit above now > > > > > > > enables > > > > > > > it on (I guess ext4 from the =y in your config), then I suppose we > > > > > > > should > > > > > > > start searching for improper memory freeing or scribbling in its > > > > > > > holepunch > > > > > > > support: something that might be corrupting the dm_region in your > > > > > > > oops. > > > > > > > > > > > > What the test is doing - it creates file in LVM_TEST_DIR (default > > > > > > is > > > > > > /tmp) > > > > > > I ran "LVM_TEST_DIR=/tmp make check_local": > > > without that it appeared to be using a subdirectory made under test/. > > > > > > And being a year or two out of date in my userspace, and unfamiliar with > > > the syntax and whereabouts of lvm.conf, it was easiest for me to hack > > > lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1 > > > after I spotted a warning message about issue_discards. > > > > > > > > > and using loop device to simulate device (small size - it should fit > > > > > > bellow > > > > > > 200MB) > > > > > > > > > > > > Within this file second layer through virtual DM devices is created > > > > > > and > > > > > > simulates various numbers of PV devices to play with. > > > > > > > > > > This sounds much easier to set up than I was expecting: > > > > > thanks for the info, I'll try it later on today. > > > > > > Sorry, I never reached it yesterday, but arrived there this morning. > > > > > > > > > > > > > > > > > > > > So since everything now support TRIM - such operations should be > > > > > > passed > > > > > > down to the backend file - which probably triggers the path. > > > > > > > > > > What filesystem do you have for /tmp? > > > > > > From your later remarks, I inferred tmpfs. > > > > > > > > > > > > > If tmpfs, then it will make much more sense if we assume your > > > > > bisection > > > > > endpoint was off by one. Your bisection log was not quite complete; > > > > > and even if it did appear to converge on the commit you cite, you > > > > > might > > > > > have got (un)lucky when testing the commit before it, and concluded > > > > > "good" when more attempts would have said "bad". > > > > > > > > > > The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe > > > > > "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a > > > > > much more likely first bad commit if your /tmp is on tmpfs: > > > > > that does indeed wire up loop to pass TRIM down to tmpfs by > > > > > fallocate - that indeed played a part in my own testing. > > > > > > > > > > Whereas if your /tmp is on ext4, loop has been passing TRIM down > > > > > with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) > > > > > should be completely irrelevant. > > > > > > > > While I've been aware of the fact that tmpfs was enhanced with trim > > > > support - > > > > I've not tried to run on real ext4 filesystem since for my tests I'm > > > > using > > > > tmpfs for quite some time to safe rewrites of SSD :) > > > > > > > > So now I've checked with real ext4 - and the bug is there as well > > > > so I've went back - it crashes on 3.4, 3.3 and 3.2 as well. > > > > > > > > 3.1 is the first kernel which does survive (checked 5 repeated runs) > > > > > > Very useful research, thank you. > > > > > > > > > > > And you are correct, the first commit which causes crash really is > > > > 83e4fa9c16e4af when I use tmpfs as backend storage - the problem why > > > > I've > > > > missed to properly identify this commit in my bisect is that crash > > > > usually > > > > happens on the second pass of the lvm test suite 'make check_local' > > > > execution > > > > - and I've been running test just once. To be sure I've run 5 run passes > > > > on > > > > 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing > > > > usually on second run, with commit 3f31d07571e the crash always happens > > > > in > > > > the first pass. > > > > > > > > I've also checked some rawhide kernel > > > > vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 > > > > and it's crashing as well - so it's probably not uniqueness of my > > > > config. > > > > > > > > So is there any primary suspect in 3.2 which is worth to check - or I > > > > need > > > > another day to play another bisect game ? > > > > > > No need for a further bisect: 3.2 is when the disard/trim/fallocate > > > support went into drivers/block/loop.c, so these tests would be unable > > > to show if DM was right or wrong before then. > > > > Well I've played meanwhile the game with minimized kernel config > > and the outcome is: > > > > last working kernel is: 3.1.0-rc1-00008-g548ef6c > > first broken: 3.1.0-rc1-00009-gdfaa2ef > > > > dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef > > loop: add discard support for loop devices > > > > > I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS > > > I did quickly crash around where you expected; though in my case > > > it was in dm_rh_dec() called from mirror_end_io(). > > > > Change of Issue Discards option in lvm.conf is not needed. > > I'm able to get these oopses with this setting turned off. > > > > > But I've not taken it any further than that. You've shown that it's > > > as much a problem with ext4 as with tmpfs, and has been a problem > > > ever since these tests' use of discard reached DM. > > > > > > I think it's fair to assume that there's something wrong with DM's > > > handling of REQ_DISCARD. (Perhaps it was all correct when written, > > > but I think there has been a series of modifications to REQ_DISCARD > > > handling in the block layer, it's been rather troublesome.) > > > Hi Zdenek, > > > > So does anyone has some idea what should be checked next ? I've missed the information in the thread, but since you're the most familiar with the lvm test suite, could you distil the reproducer which does not require to run lvm test suit ? > > > I've enabled more kernel debugging tools - and this came out when running test > suite: > > Unsure if this could be source of troubles? > > kernel BUG at arch/x86/mm/physaddr.c:18! > invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC > CPU 1 > Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp > llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945 > iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4 > > Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO So you're testing rather old kernel so you might be missing some fixes there. Could you rerun the test with the recent kernel ? Also it appears that the bug here happens because dm requested a destination page which is within the kernel space. It seems that this has been initiated by the write request from the mirror target. So I do not immediately see how punch hole (discard) is involved at all. You might have been lucky enough to hit a different bug probably ? Looking at git log, this commit has been brought to my attention: 0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support seems related to this crash. Please retest with recent kernel. Thanks! -Lukas > 6464CTO/6464CTO > RIP: 0010:[<ffffffff81021f00>] [<ffffffff81021f00>] __phys_addr+0x60/0x70 > RSP: 0018:ffff880136e41aa0 EFLAGS: 00010213 > RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c > RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000 > RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40 > R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40 > FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task > ffff880136e3e090) > Stack: > ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0 > ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e > 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000 > Call Trace: > [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod] > [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod] > [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] > [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] > [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] > [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod] > [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] > [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] > [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror] > [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror] > [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] > [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror] > [<ffffffff81050177>] process_one_work+0x197/0x460 > [<ffffffff81050116>] ? process_one_work+0x136/0x460 > [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror] > [<ffffffff81050986>] worker_thread+0x126/0x2d0 > [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0 > [<ffffffff81056066>] kthread+0x96/0xa0 > [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10 > [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe > [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70 > [<ffffffff8133b610>] ? gs_change+0xb/0xb > Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00 > 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e > 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba > RIP [<ffffffff81021f00>] __phys_addr+0x60/0x70 > RSP <ffff880136e41aa0> > ---[ end trace 040dfd9d68fccabc ]--- > > > > Zdenek > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-07-02 9:00 ` LukᚠCzerner @ 2012-07-02 10:35 ` LukᚠCzerner 2012-07-02 13:41 ` Mike Snitzer 0 siblings, 1 reply; 13+ messages in thread From: LukᚠCzerner @ 2012-07-02 10:35 UTC (permalink / raw) To: LukᚠCzerner Cc: amwang, Mike Snitzer, Zdenek Kabelac, Hugh Dickins, linux-kernel, Joe Thornber, LVM general discussion and development, Alasdair G Kergon [-- Attachment #1: Type: TEXT/PLAIN, Size: 14864 bytes --] On Mon, 2 Jul 2012, Lukￜ Czerner wrote: > Date: Mon, 2 Jul 2012 11:00:06 +0200 (CEST) > From: Lukￜ Czerner <lczerner@redhat.com> > To: Zdenek Kabelac <zkabelac@redhat.com> > Cc: Hugh Dickins <hughd@google.com>, Mike Snitzer <snitzer@redhat.com>, > Mikulas Patocka <mpatocka@redhat.com>, Joe Thornber <ejt@redhat.com>, > LVM general discussion and development <linux-lvm@redhat.com>, > amwang@redhat.com, Alasdair G Kergon <agk@redhat.com>, > linux-kernel@vger.kernel.org, Lukas Czerner <lczerner@redhat.com> > Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel > > On Mon, 2 Jul 2012, Zdenek Kabelac wrote: > > > Date: Mon, 02 Jul 2012 00:03:07 +0200 > > From: Zdenek Kabelac <zkabelac@redhat.com> > > To: Hugh Dickins <hughd@google.com> > > Cc: Mike Snitzer <snitzer@redhat.com>, Mikulas Patocka <mpatocka@redhat.com>, > > Joe Thornber <ejt@redhat.com>, > > LVM general discussion and development <linux-lvm@redhat.com>, > > amwang@redhat.com, Alasdair G Kergon <agk@redhat.com>, > > linux-kernel@vger.kernel.org, Lukas Czerner <lczerner@redhat.com> > > Subject: Re: Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel > > > > Dne 1.7.2012 22:10, Zdenek Kabelac napsal(a): > > > Dne 1.7.2012 20:45, Hugh Dickins napsal(a): > > > > On Sun, 1 Jul 2012, Zdenek Kabelac wrote: > > > > > Dne 1.7.2012 01:10, Hugh Dickins napsal(a): > > > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote: > > > > > > > Dne 30.6.2012 21:55, Hugh Dickins napsal(a): > > > > > > > > On Sat, 30 Jun 2012, Zdenek Kabelac wrote: > > > > > > > > > > > > > > > > > > When I've used 3.5-rc kernels - I've noticed kernel deadlocks. > > > > > > > > > Ooops log included. After some experimenting - reliable way to > > > > > > > > > hit > > > > > > > > > this > > > > > > > > > oops > > > > > > > > > is to run lvm test suite for 10 minutes. Since 3.5 merge window > > > > > > > > > does > > > > > > > > > not > > > > > > > > > included anything related to this oops I've went for bisect. > > > > > > > > > > > > > > > > Thanks a lot for reporting, and going to such effort to find > > > > > > > > a reproducible testcase that you could bisect on. > > > > > > > > > > > > > > > > > > > > > > > > > > Game result is commit: 3f31d07571eeea18a7d34db9af21d2285b807a17 > > > > > > > > > > > > > > > > > > mm/fs: route MADV_REMOVE to FALLOC_FL_PUNCH_HOLE > > > > > > > > > > > > > > > > But this leaves me very puzzled. > > > > > > > > > > > > > > > > Is the "lvm test suite" what I find at > > > > > > > > git.fedorahosted.org/git/lvm2.git > > > > > > > > under tests/ ? > > > > > > > > > > > > > > Yes - that's it - > > > > > > > > > > > > > > make > > > > > > > as root: > > > > > > > cd test > > > > > > > make check_local > > > > > > > > > > > > > > (inside test subdirectory should be enough, if not - just report any > > > > > > > problem) > > > > > > > > > > > > > > > If you have something else running at the same time, which happens > > > > > > > > to > > > > > > > > use > > > > > > > > madvise(,,MADV_REMOVE) on a filesystem which the commit above now > > > > > > > > enables > > > > > > > > it on (I guess ext4 from the =y in your config), then I suppose we > > > > > > > > should > > > > > > > > start searching for improper memory freeing or scribbling in its > > > > > > > > holepunch > > > > > > > > support: something that might be corrupting the dm_region in your > > > > > > > > oops. > > > > > > > > > > > > > > What the test is doing - it creates file in LVM_TEST_DIR (default > > > > > > > is > > > > > > > /tmp) > > > > > > > > I ran "LVM_TEST_DIR=/tmp make check_local": > > > > without that it appeared to be using a subdirectory made under test/. > > > > > > > > And being a year or two out of date in my userspace, and unfamiliar with > > > > the syntax and whereabouts of lvm.conf, it was easiest for me to hack > > > > lib/config/defaults.h to #define DEFAULT_ISSUE_DISCARDS 1 > > > > after I spotted a warning message about issue_discards. > > > > > > > > > > > and using loop device to simulate device (small size - it should fit > > > > > > > bellow > > > > > > > 200MB) > > > > > > > > > > > > > > Within this file second layer through virtual DM devices is created > > > > > > > and > > > > > > > simulates various numbers of PV devices to play with. > > > > > > > > > > > > This sounds much easier to set up than I was expecting: > > > > > > thanks for the info, I'll try it later on today. > > > > > > > > Sorry, I never reached it yesterday, but arrived there this morning. > > > > > > > > > > > > > > > > > > > > > > > > So since everything now support TRIM - such operations should be > > > > > > > passed > > > > > > > down to the backend file - which probably triggers the path. > > > > > > > > > > > > What filesystem do you have for /tmp? > > > > > > > > From your later remarks, I inferred tmpfs. > > > > > > > > > > > > > > > > If tmpfs, then it will make much more sense if we assume your > > > > > > bisection > > > > > > endpoint was off by one. Your bisection log was not quite complete; > > > > > > and even if it did appear to converge on the commit you cite, you > > > > > > might > > > > > > have got (un)lucky when testing the commit before it, and concluded > > > > > > "good" when more attempts would have said "bad". > > > > > > > > > > > > The commit before, 83e4fa9c16e4af7122e31be3eca5d57881d236fe > > > > > > "tmpfs: support fallocate FALLOC_FL_PUNCH_HOLE", would be a > > > > > > much more likely first bad commit if your /tmp is on tmpfs: > > > > > > that does indeed wire up loop to pass TRIM down to tmpfs by > > > > > > fallocate - that indeed played a part in my own testing. > > > > > > > > > > > > Whereas if your /tmp is on ext4, loop has been passing TRIM down > > > > > > with fallocate since v3.0. And whichever, madvise(,,MADV_REMOVE) > > > > > > should be completely irrelevant. > > > > > > > > > > While I've been aware of the fact that tmpfs was enhanced with trim > > > > > support - > > > > > I've not tried to run on real ext4 filesystem since for my tests I'm > > > > > using > > > > > tmpfs for quite some time to safe rewrites of SSD :) > > > > > > > > > > So now I've checked with real ext4 - and the bug is there as well > > > > > so I've went back - it crashes on 3.4, 3.3 and 3.2 as well. > > > > > > > > > > 3.1 is the first kernel which does survive (checked 5 repeated runs) > > > > > > > > Very useful research, thank you. > > > > > > > > > > > > > > And you are correct, the first commit which causes crash really is > > > > > 83e4fa9c16e4af when I use tmpfs as backend storage - the problem why > > > > > I've > > > > > missed to properly identify this commit in my bisect is that crash > > > > > usually > > > > > happens on the second pass of the lvm test suite 'make check_local' > > > > > execution > > > > > - and I've been running test just once. To be sure I've run 5 run passes > > > > > on > > > > > 3.4.0-08568-gec9516f - which is OK, but 3.4.0-08569-g83e4fa9 is crashing > > > > > usually on second run, with commit 3f31d07571e the crash always happens > > > > > in > > > > > the first pass. > > > > > > > > > > I've also checked some rawhide kernel > > > > > vmlinuz-3.5.0-0.rc2.git0.1.fc18.x86_64 > > > > > and it's crashing as well - so it's probably not uniqueness of my > > > > > config. > > > > > > > > > > So is there any primary suspect in 3.2 which is worth to check - or I > > > > > need > > > > > another day to play another bisect game ? > > > > > > > > No need for a further bisect: 3.2 is when the disard/trim/fallocate > > > > support went into drivers/block/loop.c, so these tests would be unable > > > > to show if DM was right or wrong before then. > > > > > > Well I've played meanwhile the game with minimized kernel config > > > and the outcome is: > > > > > > last working kernel is: 3.1.0-rc1-00008-g548ef6c > > > first broken: 3.1.0-rc1-00009-gdfaa2ef > > > > > > dfaa2ef68e80c378e610e3c8c536f1c239e8d3ef > > > loop: add discard support for loop devices > > > > > > > I don't have Fedora Rawhide on, but after hacking ISSUE_DISCARDS > > > > I did quickly crash around where you expected; though in my case > > > > it was in dm_rh_dec() called from mirror_end_io(). > > > > > > Change of Issue Discards option in lvm.conf is not needed. > > > I'm able to get these oopses with this setting turned off. > > > > > > > But I've not taken it any further than that. You've shown that it's > > > > as much a problem with ext4 as with tmpfs, and has been a problem > > > > ever since these tests' use of discard reached DM. > > > > > > > > I think it's fair to assume that there's something wrong with DM's > > > > handling of REQ_DISCARD. (Perhaps it was all correct when written, > > > > but I think there has been a series of modifications to REQ_DISCARD > > > > handling in the block layer, it's been rather troublesome.) > > > > > > Hi Zdenek, > > > > > > > So does anyone has some idea what should be checked next ? > > I've missed the information in the thread, but since you're the most > familiar with the lvm test suite, could you distil the reproducer > which does not require to run lvm test suit ? > > > > > > > > I've enabled more kernel debugging tools - and this came out when running test > > suite: > > > > Unsure if this could be source of troubles? > > > > kernel BUG at arch/x86/mm/physaddr.c:18! > > invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC > > CPU 1 > > Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun bridge stp > > llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi sr_mod cdrom iwl3945 > > iwl_legacy mac80211 cfg80211 crypto e1000e evdev loop dm_mod autofs4 > > > > Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 LENOVO > > So you're testing rather old kernel so you might be missing some > fixes there. Could you rerun the test with the recent kernel ? > > Also it appears that the bug here happens because dm requested a > destination page which is within the kernel space. It seems that > this has been initiated by the write request from the mirror target. > So I do not immediately see how punch hole (discard) is involved at > all. You might have been lucky enough to hit a different bug > probably ? > > Looking at git log, this commit has been brought to my attention: > > 0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support > > seems related to this crash. > > Please retest with recent kernel. > > Thanks! > -Lukas > > > 6464CTO/6464CTO > > RIP: 0010:[<ffffffff81021f00>] [<ffffffff81021f00>] __phys_addr+0x60/0x70 > > RSP: 0018:ffff880136e41aa0 EFLAGS: 00010213 > > RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c > > RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000 > > RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000 > > R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40 > > R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40 > > FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 > > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > > CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0 > > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > > Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task > > ffff880136e3e090) > > Stack: > > ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0 > > ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e > > 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000 > > Call Trace: > > [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod] > > [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod] > > [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] > > [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] > > [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] > > [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod] > > [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] > > [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] > > [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror] > > [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror] > > [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] > > [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror] > > [<ffffffff81050177>] process_one_work+0x197/0x460 > > [<ffffffff81050116>] ? process_one_work+0x136/0x460 > > [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror] > > [<ffffffff81050986>] worker_thread+0x126/0x2d0 > > [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0 > > [<ffffffff81056066>] kthread+0x96/0xa0 > > [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10 > > [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe > > [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70 > > [<ffffffff8133b610>] ? gs_change+0xb/0xb > > Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 00 00 00 > > 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b <0f> 0b 0f 0b 66 2e > > 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba > > RIP [<ffffffff81021f00>] __phys_addr+0x60/0x70 > > RSP <ffff880136e41aa0> > > ---[ end trace 040dfd9d68fccabc ]--- > > > > > > > > Zdenek So from the original backtrace for the problem Zdenek is seeing on 3.5.0-rc4 (https://lkml.org/lkml/2012/6/30/98) I think that this is problem in the device mapper itself. I do not think it has anything to do with tmpfs or mm. According to bisects from Zdenek it clearly shows that the problem appear when the discard support for the loop device is added, so it is most likely related to the dm discard support. Anyway, the backtrace points to the NULL pointed dereference in dm_rh_region_context() which is simple function: void *dm_rh_region_context(struct dm_region *reg) { return reg->rh->context; } so either reg, or reg-rh is NULL. Now the only place this is used is from recovery_complete() in dm-raid1.c. So this is somewhat related to raid recovery. I am not familiar with the dm code, but can someone from the dm team look at this ? But just to be sure to rule out the punch hole thing Zdenek can you run your tests on the "real" discard capable device ? Or at least on the device which does not convert discard requests into punch hole ? You can use scsi_debug to create such device: modprobe scsi_debug dev_size_mb=16 sector_size=512 num_tgts=1 lbpu=1 Thanks! -Lukas ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-07-02 10:35 ` Lukáš Czerner @ 2012-07-02 13:41 ` Mike Snitzer 0 siblings, 0 replies; 13+ messages in thread From: Mike Snitzer @ 2012-07-02 13:41 UTC (permalink / raw) To: Lukáš Czerner Cc: amwang, Zdenek Kabelac, Hugh Dickins, linux-kernel, Joe Thornber, LVM general discussion and development, Alasdair G Kergon On Mon, Jul 02 2012 at 6:35am -0400, Lukáš Czerner <lczerner@redhat.com> wrote: > > > > So you're testing rather old kernel so you might be missing some > > fixes there. Could you rerun the test with the recent kernel ? > > > > Also it appears that the bug here happens because dm requested a > > destination page which is within the kernel space. It seems that > > this has been initiated by the write request from the mirror target. > > So I do not immediately see how punch hole (discard) is involved at > > all. You might have been lucky enough to hit a different bug > > probably ? > > > > Looking at git log, this commit has been brought to my attention: > > > > 0c535e0d6f463365c29623350dbd91642363c39b dm io: fix discard support > > > > seems related to this crash. > > > > Please retest with recent kernel. Ah, you beat me to recommending that fix ;) > So from the original backtrace for the problem Zdenek is seeing on 3.5.0-rc4 > (https://lkml.org/lkml/2012/6/30/98) I think that this is > problem in the device mapper itself. I do not think it has anything > to do with tmpfs or mm. According to bisects from Zdenek it clearly > shows that the problem appear when the discard support for the loop > device is added, so it is most likely related to the dm discard support. What about using scsi_debug with the dm-mirror target? Never say never, DM-mirror and/or dm-io code could still have an issue, but the commit referenced above did fix discard with the mirror target back in 3.3. > Anyway, the backtrace points to the NULL pointed dereference in > dm_rh_region_context() which is simple function: > > void *dm_rh_region_context(struct dm_region *reg) > { > return reg->rh->context; > } > > so either reg, or reg-rh is NULL. Now the only place this is used is > from recovery_complete() in dm-raid1.c. So this is somewhat related > to raid recovery. I am not familiar with the dm code, but can > someone from the dm team look at this ? I'll coordiinate with Zdenek. > But just to be sure to rule out the punch hole thing Zdenek can you > run your tests on the "real" discard capable device ? Or at least on > the device which does not convert discard requests into punch hole ? > You can use scsi_debug to create such device: > > modprobe scsi_debug dev_size_mb=16 sector_size=512 num_tgts=1 lbpu=1 Great minds think alike ;) ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-07-01 22:03 ` Zdenek Kabelac 2012-07-02 9:00 ` Lukáš Czerner @ 2012-07-02 13:33 ` Mike Snitzer 2012-07-04 11:51 ` Zdenek Kabelac 1 sibling, 1 reply; 13+ messages in thread From: Mike Snitzer @ 2012-07-02 13:33 UTC (permalink / raw) To: Zdenek Kabelac Cc: amwang, Hugh Dickins, linux-kernel, Joe Thornber, LVM general discussion and development, Lukas Czerner, Alasdair G Kergon Not sure why you're testing such old kernels without regard for fixes (likely just a hopeful bisect?). In any case, you definitely need this commit (was also marked for stable): 0c535e0 v3.3-rc7 dm io: fix discard support On Sun, Jul 01 2012 at 6:03pm -0400, Zdenek Kabelac <zkabelac@redhat.com> wrote: > > I've enabled more kernel debugging tools - and this came out when > running test suite: > > Unsure if this could be source of troubles? > > kernel BUG at arch/x86/mm/physaddr.c:18! > invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC > CPU 1 > Modules linked in: dm_mirror dm_region_hash dm_log aes_generic tun > bridge stp llc psmouse serio_raw cryptomgr aead arc4 crypto_algapi > sr_mod cdrom iwl3945 iwl_legacy mac80211 cfg80211 crypto e1000e > evdev loop dm_mod autofs4 > > Pid: 8, comm: kworker/1:0 Not tainted 3.1.0-rc1-00009-gdfaa2ef #30 > LENOVO 6464CTO/6464CTO > RIP: 0010:[<ffffffff81021f00>] [<ffffffff81021f00>] __phys_addr+0x60/0x70 > RSP: 0018:ffff880136e41aa0 EFLAGS: 00010213 > RAX: ffff87ffffffffff RBX: ffff880136e41ba0 RCX: ffff880136e41b3c > RDX: ffff880136e41b48 RSI: ffff880136e41b40 RDI: 0000000000000000 > RBP: ffff880136e41aa0 R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000001 R12: ffff880136e41b40 > R13: ffff880136e41b48 R14: ffff880136e41b3c R15: ffff88011e9bec40 > FS: 0000000000000000(0000) GS:ffff880137400000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > CR2: 00007f34760df0c0 CR3: 0000000001604000 CR4: 00000000000006a0 > DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Process kworker/1:0 (pid: 8, threadinfo ffff880136e40000, task ffff880136e3e090) > Stack: > ffff880136e41ad0 ffffffffa0013f3d 0000000000000800 ffff880136e41ba0 > ffff88011eae6800 ffff880136e41c10 ffff880136e41b80 ffffffffa001437e > 00000000239a96d0 0000000000000000 0000000000000000 0000000000000000 > Call Trace: > [<ffffffffa0013f3d>] km_get_page+0x2d/0xa0 [dm_mod] > [<ffffffffa001437e>] dispatch_io+0x1ae/0x250 [dm_mod] > [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] > [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] > [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] > [<ffffffffa0014627>] dm_io+0xe7/0x1f0 [dm_mod] > [<ffffffffa0013f10>] ? vm_get_page+0x60/0x60 [dm_mod] > [<ffffffffa0013e90>] ? vm_next_page+0x20/0x20 [dm_mod] > [<ffffffffa013e204>] do_writes+0x324/0x650 [dm_mirror] > [<ffffffffa013e085>] ? do_writes+0x1a5/0x650 [dm_mirror] > [<ffffffffa013dab0>] ? mirror_flush+0x120/0x120 [dm_mirror] > [<ffffffffa013e9f4>] do_mirror+0x104/0x230 [dm_mirror] > [<ffffffff81050177>] process_one_work+0x197/0x460 > [<ffffffff81050116>] ? process_one_work+0x136/0x460 > [<ffffffffa013e8f0>] ? do_reads+0x180/0x180 [dm_mirror] > [<ffffffff81050986>] worker_thread+0x126/0x2d0 > [<ffffffff81050860>] ? manage_workers.isra.28+0x1f0/0x1f0 > [<ffffffff81056066>] kthread+0x96/0xa0 > [<ffffffff8133b614>] kernel_thread_helper+0x4/0x10 > [<ffffffff8133405d>] ? retint_restore_args+0xe/0xe > [<ffffffff81055fd0>] ? __init_kthread_worker+0x70/0x70 > [<ffffffff8133b610>] ? gs_change+0xb/0xb > Code: ff ff 87 ff ff 48 39 c7 76 21 0f b6 0d a3 61 62 00 48 b8 00 00 > 00 00 00 78 00 00 48 01 f8 48 89 c2 48 d3 ea 48 85 d2 74 d0 0f 0b > <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 66 90 83 f9 01 49 ba > RIP [<ffffffff81021f00>] __phys_addr+0x60/0x70 > RSP <ffff880136e41aa0> > ---[ end trace 040dfd9d68fccabc ]--- > > > > Zdenek ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel 2012-07-02 13:33 ` Mike Snitzer @ 2012-07-04 11:51 ` Zdenek Kabelac 0 siblings, 0 replies; 13+ messages in thread From: Zdenek Kabelac @ 2012-07-04 11:51 UTC (permalink / raw) To: Mike Snitzer Cc: amwang, Hugh Dickins, linux-kernel, Joe Thornber, LVM general discussion and development, Lukas Czerner, Alasdair G Kergon Dne 2.7.2012 15:33, Mike Snitzer napsal(a): > Not sure why you're testing such old kernels without regard for fixes > (likely just a hopeful bisect?). > > In any case, you definitely need this commit (was also marked for > stable): 0c535e0 v3.3-rc7 dm io: fix discard support > > After some more crashed filesystems here is the outcome with trial patch from Mikulas included: https://bugzilla.redhat.com/show_bug.cgi?id=837607 Zdenek ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2012-07-04 11:51 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-06-30 11:41 [linux-lvm] Regression with FALLOC_FL_PUNCH_HOLE in 3.5-rc kernel Zdenek Kabelac 2012-06-30 19:55 ` Hugh Dickins 2012-06-30 20:51 ` Zdenek Kabelac 2012-06-30 23:10 ` Hugh Dickins 2012-07-01 12:57 ` Zdenek Kabelac 2012-07-01 18:45 ` Hugh Dickins 2012-07-01 20:10 ` Zdenek Kabelac 2012-07-01 22:03 ` Zdenek Kabelac 2012-07-02 9:00 ` Lukáš Czerner 2012-07-02 10:35 ` Lukáš Czerner 2012-07-02 13:41 ` Mike Snitzer 2012-07-02 13:33 ` Mike Snitzer 2012-07-04 11:51 ` Zdenek Kabelac
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).