From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756159Ab1HEBBS (ORCPT ); Thu, 4 Aug 2011 21:01:18 -0400 Received: from mx1.redhat.com ([209.132.183.28]:7117 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754627Ab1HEBBQ (ORCPT ); Thu, 4 Aug 2011 21:01:16 -0400 Date: Thu, 4 Aug 2011 21:00:38 -0400 From: Dave Jones To: Linux Kernel Cc: Linus Torvalds , Andrew Morton Subject: list corruption in the last few days. (block ? crypto ?) Message-ID: <20110805010038.GA18148@redhat.com> Mail-Followup-To: Dave Jones , Linux Kernel , Linus Torvalds , Andrew Morton MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Sometime in the last week, something was merged which causes my laptop to lock up occasionally. I can trigger it most of the time just by doing a kernel build. When it gets to the final linking stage, it locks up hard. I finally managed to coax something out of usb console to get the traces below, which seem to implicate something in the block layer ? my root device is an lvm volume on an dmcrypt'd block dev, which might be relevant, as I don't see this happening on other machines with simpler setups. I'm going to try bisecting, but it might take me a few days, because it's such a pain in the ass to reproduce this reliably. Dave [ 5913.233035] ------------[ cut here ]------------ [ 5913.233097] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98() [ 5913.233101] Hardware name: Adamo 13 [ 5913.233105] list_del corruption. prev->next should be ffffea000434fd20, but was ffffea0004199520 [ 5913.233108] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth snd_hwdep snd_seq snd_seq_device snd_pcm usb_debug dell_wmi sparse_keymap cdc_ether usbnet cdc_acm uvcvideo cdc_wdm mii cfg80211 snd_timer dell_laptop videodev dcdbas snd microcode v4l2_compat_ioctl32 soundcore joydev tg3 pcspkr snd_page_alloc iTCO_wdt i2c_i801 rfkill iTCO_vendor_support wmi virtio_net kvm_intel kvm ipv6 xts gf128mul dm_crypt i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 5913.233213] Pid: 0, comm: swapper Not tainted 3.0.0+ #127 [ 5913.233213] Call Trace: [ 5913.233213] [] warn_slowpath_common+0x83/0x9b [ 5913.233213] [] warn_slowpath_fmt+0x46/0x48 [ 5913.233213] [] __list_del_entry+0x8d/0x98 [ 5913.233213] [] list_del+0xe/0x2d [ 5913.233213] [] __slab_free+0x1db/0x235 [ 5913.233213] [] ? bvec_free_bs+0x35/0x37 [ 5913.233213] [] ? bvec_free_bs+0x35/0x37 [ 5913.233213] [] ? bvec_free_bs+0x35/0x37 [ 5913.233213] [] kmem_cache_free+0x88/0x102 [ 5913.233213] [] bvec_free_bs+0x35/0x37 [ 5913.233213] [] bio_free+0x34/0x64 [ 5913.233213] [] dm_bio_destructor+0x12/0x14 [ 5913.233213] [] bio_put+0x2b/0x2d [ 5913.233213] [] clone_endio+0x9e/0xb4 [ 5913.233213] [] bio_endio+0x2d/0x2f [ 5913.233213] [] crypt_dec_pending+0x5c/0x8b [dm_crypt] [ 5913.233213] [] crypt_endio+0x78/0x81 [dm_crypt] [ 5913.233213] [] bio_endio+0x2d/0x2f [ 5913.233213] [] req_bio_endio+0x91/0x99 [ 5913.233213] [] blk_update_request+0x1f2/0x395 [ 5913.233213] [] blk_update_bidi_request+0x21/0x6d [ 5913.233213] [] blk_end_bidi_request+0x1f/0x5d [ 5913.233213] [] blk_end_request+0x10/0x12 [ 5913.233213] [] scsi_io_completion+0x1dc/0x4d7 [ 5913.233213] [] scsi_finish_command+0xe4/0xed [ 5913.233213] [] scsi_softirq_done+0x109/0x112 [ 5913.233213] [] blk_done_softirq+0x7f/0x93 [ 5913.233213] [] __do_softirq+0x10f/0x257 [ 5913.233213] [] ? time_hardirqs_off+0x1b/0x2f [ 5913.233213] [] call_softirq+0x1c/0x30 [ 5913.233213] [] do_softirq+0x4b/0xa1 [ 5913.233213] [] irq_exit+0x5d/0xcd [ 5913.233213] [] do_IRQ+0x8d/0xa4 [ 5913.233213] [] common_interrupt+0x73/0x73 [ 5913.233213] [] ? __cycles_2_ns+0xe/0x3a [ 5913.233213] [] ? arch_local_irq_enable+0x8/0xd [ 5913.233213] [] ? trace_hardirqs_on+0xd/0xf [ 5913.233213] [] acpi_idle_enter_bm+0x235/0x26d [ 5913.233213] [] cpuidle_idle_call+0x11c/0x1fe [ 5913.233213] [] cpu_idle+0xe2/0x14c [ 5913.233213] [] ? start_kernel+0x3ec/0x3f7 [ 5913.233213] [] rest_init+0xda/0xe1 [ 5913.233213] [] ? csum_partial_copy_generic+0x16c/0x16c [ 5913.233213] [] start_kernel+0x3ec/0x3f7 [ 5913.233213] [] x86_64_start_reservations+0xaf/0[ 5913.239397] ------------[ cut here ]------------ [ 5913.239397] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98() [ 5913.239397] Hardware name: Adamo 13 [ 5913.239397] list_del corruption. prev->next should be ffffea00041a82a0, but was ffffea0004677e60 [ 5913.239397] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ cdc_eth[ 5913.317698] ------------[ cut here ]------------ [ 5913.317761] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98() [ 5913.317766] Hardware name: Adamo 13 [ 5913.317769] list_del corruption. prev->next should be ffffea0004145720, but was ffffea000415ee20 [ 5913.317773] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth snd_hwdep snd_seq snd_seq_device snd_pcm usb_debug dell_wmi sparse_keymap cdc_ether usbnet cdc_acm uvcvideo cdc_wdm mii cfg80211 snd_timer dell_laptop videodev dcdbas snd microcode v4l2_compat_ioctl32 soundcore joydev tg3 pcspkr snd_page_alloc iTCO_wdt i2c_i801 rfkill iTCO_vendor_support wmi virtio_net kvm_intel kvm ipv6 xts gf128mul dm_crypt i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 5913.317926] Pid: 0, comm: kworker/0:0 Tainted: G W 3.0.0+ #127 [ 5913.317930] Call Trace: [ 5913.317933] [] warn_slowpath_common+0x83/0x9b [ 5913.317944] [] warn_slowpath_fmt+0x46/0x48 [ 5913.317950] [] __list_del_entry+0x8d/0x98 [ 5913.317955] [] list_del+0xe/0x2d [ 5913.317961] [] __slab_free+0x1db/0x235 [ 5913.317968] [] ? bvec_free_bs+0x35/0x37 [ 5913.317973] [] ? bvec_free_bs+0x35/0x37 [ 5913.317978] [] ? bvec_free_bs+0x35/0x37 [ 5913.317985] [] kmem_cache_free+0x88/0x102 [ 5913.317990] [] bvec_free_bs+0x35/0x37 [ 5913.317995] [] bio_free+0x34/0x64 [ 5913.318002] [] dm_bio_destructor+0x12/0x14 [ 5913.318007] [] bio_put+0x2b/0x2d [ 5913.318012] [] clone_endio+0x9e/0xb4 [ 5913.318017] [] bio_endio+0x2d/0x2f [ 5913.318030] [] crypt_dec_pending+0x5c/0x8b [dm_crypt] [ 5913.318041] [] crypt_endio+0x78/0x81 [dm_crypt] [ 5913.318047] [] bio_endio+0x2d/0x2f [ 5913.318053] [] req_bio_endio+0x91/0x99 [ 5913.318059] [] blk_update_request+0x1f2/0x395 [ 5913.318064] [] blk_update_bidi_request+0x21/0x6d [ 5913.318070] [] blk_end_bidi_request+0x1f/0x5d [ 5913.318075] [] blk_end_request+0x10/0x12 [ 5913.318081] [] scsi_io_completion+0x1dc/0x4d7 [ 5913.318087] [] scsi_finish_command+0xe4/0xed [ 5913.318092] [] scsi_softirq_done+0x109/0x112 [ 5913.318098] [] blk_done_softirq+0x7f/0x93 [ 5913.318104] [] __do_softirq+0x10f/0x257 [ 5913.318110] [] ? time_hardirqs_off+0x1b/0x2f [ 5913.318116] [] call_softirq+0x1c/0x30 [ 5913.318123] [] do_softirq+0x4b/0xa1 [ 5913.318127] [] irq_exit+0x5d/0xcd [ 5913.318133] [] do_IRQ+0x8d/0xa4 [ 5913.318138] [] common_interrupt+0x73/0x73 [ 5913.318142] [] ? __cycles_2_ns+0xe/0x3a [ 5913.318152] [] ? arch_local_irq_enable+0x8/0xd [ 5913.318158] [] ? trace_hardirqs_on+0xd/0xf [ 5913.318164] [] acpi_idle_enter_simple+0xde/0x114 [ 5913.318170] [] cpuidle_idle_call+0x11c/0x1fe [ 5913.318176] [] cpu_idle+0xe2/0x14c [ 5913.318181] [] start_secondary+0x20d/0x218 [ 5913.318186] ---[ end trace ff2d5fa66f3f892e ]--- [ 5913.422416] ------------[ cut here ]------------ [ 5913.422501] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98() [ 5913.422505] Hardware name: Adamo 13 [ 5913.422509] list_del corruption. prev->next should be ffffea00045acda0, but was dead000000100100 [ 5913.422513] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth snd_hwdep snd_seq snd_seq_device snd_pcm usb_debug dell_wmi sparse_keymap cdc_ether usbnet cdc_acm uvcvideo cdc_wdm mii cfg80211 snd_timer dell_laptop videodev dcdbas snd microcode v4l2_compat_ioctl32 soundcore joydev tg3 pcspkr snd_page_alloc iTCO_wdt i2c_i801 rfkill iTCO_vendor_support wmi virtio_net kvm_intel kvm ipv6 xts gf128mul dm_crypt i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 5913.422667] Pid: 17555, comm: kworker/1:3 Tainted: G W 3.0.0+ #127 [ 5913.422670] Call Trace: [ 5913.422674] [] warn_slowpath_common+0x83/0x9b [ 5913.422685] [] warn_slowpath_fmt+0x46/0x48 [ 5913.422691] [] __list_del_entry+0x8d/0x98 [ 5913.422696] [] list_del+0xe/0x2d [ 5913.422702] [] __slab_free+0x1db/0x235 [ 5913.422709] [] ? bvec_free_bs+0x35/0x37 [ 5913.422715] [] ? bvec_free_bs+0x35/0x37 [ 5913.422720] [] ? bvec_free_bs+0x35/0x37 [ 5913.422726] [] kmem_cache_free+0x88/0x102 [ 5913.422732] [] bvec_free_bs+0x35/0x37 [ 5913.422737] [] bio_free+0x34/0x64 [ 5913.422743] [] dm_bio_destructor+0x12/0x14 [ 5913.422749] [] bio_put+0x2b/0x2d [ 5913.422754] [] clone_endio+0x9e/0xb4 [ 5913.422759] [] bio_endio+0x2d/0x2f [ 5913.422773] [] crypt_dec_pending+0x5c/0x8b [dm_crypt] [ 5913.422786] [] crypt_endio+0x78/0x81 [dm_crypt] [ 5913.422791] [] bio_endio+0x2d/0x2f [ 5913.422797] [] req_bio_endio+0x91/0x99 [ 5913.422803] [] blk_update_request+0x1f2/0x395 [ 5913.422808] [] blk_update_bidi_request+0x21/0x6d [ 5913.422814] [] blk_end_bidi_request+0x1f/0x5d [ 5913.422819] [] blk_end_request+0x10/0x12 [ 5913.422825] [] scsi_io_completion+0x1dc/0x4d7 [ 5913.422831] [] scsi_finish_command+0xe4/0xed [ 5913.422836] [] scsi_softirq_done+0x109/0x112 [ 5913.422842] [] blk_done_softirq+0x7f/0x93 [ 5913.422848] [] __do_softirq+0x10f/0x257 [ 5913.422854] [] ? time_hardirqs_off+0x1b/0x2f [ 5913.422867] [] ? crypt_convert+0x2b7/0x2b7 [dm_crypt] [ 5913.422873] [] call_softirq+0x1c/0x30 [ 5913.422879] [] do_softirq+0x4b/0xa1 [ 5913.422884] [] irq_exit+0x5d/0xcd [ 5913.422889] [] do_IRQ+0x8d/0xa4 [ 5913.422895] [] common_interrupt+0x73/0x73 [ 5913.422898] [] ? debug_mutex_init+0x3e/0x3e [ 5913.422909] [] ? arch_local_irq_save+0x9/0x18 [ 5913.422915] [] lock_is_held+0x2c/0x95 [ 5913.422930] [] rcu_read_lock_held+0x25/0x27 [ 5913.422935] [] __module_address+0x61/0xa0 [ 5913.422949] [] is_module_address+0x20/0x59 [ 5913.422960] [] static_obj+0x3c/0x54 [ 5913.422970] [] lockdep_init_map+0xa0/0x115 [ 5913.422982] [] __init_waitqueue_head+0x3a/0x4b [ [ 5913.555121] ------------[ cut here ]------------ [ 5913.555121] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98() [ 5913.555121] Hardware name: Adamo 13 [ 5913.555121] list_del corruption. prev->next should be ffffea000451a320, but was ffffea000480eda0 [ 5913.555121] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth snd_hwdep snd_seq snd_seq_device snd_pcm usb_debug dell_wmi sparse_keymap cdc_ether usbnet cdc_acm uvcvideo cdc_wdm mii cfg80211 snd_timer dell_laptop videodev dcdbas snd microcode v4l2_compat_ioctl32 soundcore joydev tg3 pcspkr snd_page_alloc iTCO_wdt i2c_i801 rfkill iTCO_vendor_support wmi virtio_net kvm_intel kvm ipv6 xts gf128mul dm_crypt i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 5913.555121] Pid: 0, comm: swapper Tainted: G W 3.0.0+ #127 [ 5913.555121] Call Trace: [ 5913.555121] [] warn_slowpath_common+0x83/0x9b [ 5913.555121] [] warn_slowpath_fmt+0x46/0x48 [ 5913.555121] [] __list_del_entry+0x8d/0x98 [ 5913.555121] [] list_del+0xe/0x2d [ 5913.555121] [] __slab_free+0x1db/0x235 [ 5913.555121] [] ? bvec_free_bs+0x35/0x37 [ 5913.555121] [] ? bvec_free_bs+0x35/0x37 [ 5913.555121] [] ? bvec_free_bs+0x35/0x37 [ 5913.555121] [] kmem_cache_free+0x88/0x102 [ 5913.555121] [] bvec_free_bs+0x35/0x37 [ 5913.555121] [] bio_free+0x34/0x64 [ 5913.555121] [] dm_bio_destructor+0x12/0x14 [ 5913.555121] [] bio_put+0x2b/0x2d [ 5913.555121] [] clone_endio+0x9e/0xb4 [ 5913.555121] [] bio_endio+0x2d/0x2f [ 5913.555121] [] crypt_dec_pending+0x5c/0x8b [dm_crypt] [ 5913.555121] [] crypt_endio+0x78/0x81 [dm_crypt] [ 5913.555121] [] bio_endio+0x2d/0x2f [ 5913.555121] [] req_bio_endio+0x91/0x99 [ 5913.555121] [] blk_update_request+0x1f2/0x395 [ 5913.555121] [] blk_update_bidi_request+0x21/0x6d [ 5913.555121] [] blk_end_bidi_request+0x1f/0x5d [ 5913.555121] [] blk_end_request+0x10/0x12 [ 5913.555121] [] scsi_io_completion+0x1dc/0x4d7 [ 5913.555121] [] scsi_finish_command+0xe4/0xed [ 5913.555121] [] scsi_softirq_done+0x109/0x112 [ 5913.555121] [] blk_done_softirq+0x7f/0x93 [ 5913.555121] [] __do_softirq+0x10f/0x257 [ 5913.555121] [] ? time_hardirqs_off+0x1b/0x2f [ 5913.555121] [] call_softirq+0x1c/0x30 [ 5913.555121] [] do_softirq+0x4b/0xa1 [ 5913.555121] [] irq_exit+0x5d/0xcd [ 5913.555121] [] do_IRQ+0x8d/0xa4 [ 5913.555121] [] common_interrupt+0x73/0x73 [ 5913.555121] [] ? __cycles_2_ns+0xe/0x3a [ 5913.555121] [] ? arch_local_irq_enable+0x8/0xd [ 5913.555121] [] ? trace_hardirqs_on+0xd/0xf [ 5913.555121] [] acpi_idle_enter_simple+0xde/0x114 [ 5913.555121] [] cpuidle_idle_call+0x11c/0x1fe [ 5913.555121] [] cpu_idle+0xe2/0x14c [ 5913.555121] [] ? start_kernel+0x3ec/0x3f7 [ 5913.555121] [] rest_init+0xda/0xe1 [ 5913.555121] [] ? csum_partial_copy_generic+0x16c/0x16c [ 5913.555121] [] start_kernel+0x3ec/0x3f7 [ 5913.555121] [ 5913.555121] ------------[ cut here ]------------ [ 5913.555121] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98() [ 5913.555121] Hardware name: Adamo 13 [ 5913.555121] list_del corruption. prev->next should be ffffea00046e6c20, but was ffffea00043ae720 [ 5913.555121] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth wmi vir[ 5913.5[ 5913.5[ 5913.5[ 5913.593105] ------------[ cut here ]------------ [ 5913.593114] WARNING: at lib/list_debug.c:26 __list_add+0x3d/0x7f() [ 5913.593118] Hardware name: Adamo 13 [ 5913.593122] list_add corruption. next->prev should be prev (ffff88013825fe70), but was ffffea000439a120. (next=ffffea00045873e0). [ 5913.593127] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth snd_hwdep snd_seq snd_seq_device snd_pcm usb_debug dell_wmi sparse_keymap cdc_ether usbnet cdc_acm uvcvideo cdc_wdm mii cfg80211 snd_timer dell_laptop videodev dcdbas snd microcode v4l2_compat_ioctl32 soundcore joydev tg3 pcspkr snd_page_alloc iTCO_wdt i2c_i801 rfkill iTCO_vendor_support wmi virtio_net kvm_intel kvm ipv6 xts gf128mul dm_crypt i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [ 5913.593285] Pid: 900, comm: flush-253:3 Tainted: G W 3.0.0+ #127 [ 5913.593289] Call Trace: [ 5913.593295] [] warn_slowpath_common+0x83/0x9b [ 5913.593301] [] warn_slowpath_fmt+0x46/0x48 [ 5913.593307] [] ? deactivate_slab+0x1b4/0x2b9 [ 5913.593313] [] __list_add+0x3d/0x7f [ 5913.593318] [] deactivate_slab+0x24f/0x2b9 [ 5913.593325] [] ? bvec_[ 5914.004821] -