From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755989AbbI3Nfz (ORCPT ); Wed, 30 Sep 2015 09:35:55 -0400 Received: from mail-wi0-f178.google.com ([209.85.212.178]:35656 "EHLO mail-wi0-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750902AbbI3Nfx (ORCPT ); Wed, 30 Sep 2015 09:35:53 -0400 To: linux-fsdevel@vger.kernel.org Cc: "Linux-Kernel@Vger. Kernel. Org" , Jan Kara , darrick.wong@oracle.com, SiteGround Operations From: Nikolay Borisov Subject: Crash in jbd2_chksum due to null journal->j_chksum_driver Message-ID: <560BE535.9080604@kyup.com> Date: Wed, 30 Sep 2015 16:35:49 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hello, Today a colleague was testing something and while doing so he observed the following crash: jbd2_journal_bmap: journal block not found at offset 67 on dm-26-8 Aborting journal on device dm-26-8. BUG: unable to handle kernel NULL pointer dereference at (null) IP: [] jbd2_superblock_csum+0x2b/0x80 PGD 3fcef54067 PUD 3fce84e067 PMD 0 Oops: 0000 [#1] SMP Modules linked in: act_police cls_basic sch_ingress veth dm_snapshot openvswitch gre vxlan ip_tunnel xt_owner xt_conntrack iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_CT nf_conntrack iptable_raw ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 ext2 dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mirror dm_region_hash dm_log ses enclosure igb i2c_algo_bit x86_pkg_temp_thermal crc32_pclmul i2c_i801 lpc_ich mfd_core ioapic ioatdma dca shpchp ipmi_devintf ipmi_si ipmi_msghandler CPU: 0 PID: 12059 Comm: jbd2/dm-26-8 Not tainted 3.12.47-clouder1 #1 Hardware name: Supermicro X10DRi/X10DRi, BIOS 1.1 04/14/2015 task: ffff883f904958b0 ti: ffff883fce4d8000 task.ti: ffff883fce4d8000 RIP: 0010:[] [] jbd2_superblock_csum+0x2b/0x80 RSP: 0018:ffff883fce4d9a58 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff883f8dd77000 RCX: 0000000000000006 RDX: 0000000000000000 RSI: ffff883f8dd77000 RDI: ffff883fa0fc6800 RBP: ffff883fce4d9a88 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000000 R12: 00000000f0459c0b R13: 0000000000000411 R14: ffff883f8dd77000 R15: 00000000560bb55d FS: 0000000000000000(0000) GS:ffff881fffa00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000003fd145d000 CR4: 00000000001407f0 Stack: ffffffff81e07402 ffff883fa0fc6800 00000000fffffffb ffff883fce4d9b90 ffff883f8dd77000 ffff883fa0fc6800 ffff883fce4d9aa8 ffffffff812b1369 0000000000000010 ffff883f90c772d8 ffff883fce4d9ae8 ffffffff812b1455 Call Trace: [] jbd2_superblock_csum_set+0x29/0x40 [] jbd2_write_superblock+0x85/0x1b0 [] jbd2_journal_update_sb_errno+0x50/0x60 [] __journal_abort_soft+0x50/0x60 [] jbd2_journal_bmap+0x90/0xa0 [] jbd2_journal_next_log_block+0x77/0x80 [] jbd2_journal_get_descriptor_buffer+0x23/0xb0 [] journal_submit_commit_record+0x7c/0x1e0 [] jbd2_journal_commit_transaction+0x194e/0x1d20 [] kjournald2+0xef/0x2b0 [] ? wake_up_bit+0x40/0x40 [] ? commit_timeout+0x10/0x10 [] kthread+0xce/0xe0 [] ? kthread_freezable_should_stop+0x80/0x80 [] ret_from_fork+0x58/0x90 [] ? kthread_freezable_should_stop+0x80/0x80 Code: 55 48 89 e5 41 54 53 48 83 ec 20 0f 1f 44 00 00 44 8b a6 fc 00 00 00 48 89 f3 c7 86 fc 00 00 00 00 00 00 00 48 8b 87 d0 04 00 00 <83> 38 04 77 39 48 89 45 d0 c7 45 d8 00 00 00 00 48 8d 7d d0 c7 RIP [] jbd2_superblock_csum+0x2b/0x80 RSP CR2: 0000000000000000 ---[ end trace e1bd94031f410b71 ]--- The ffffffff812b12eb address actually is jbd2_chksum and the instruction where the deference is happening in crypto_shash_descsize(), essentially journal->j_chksum_driver is NULL. Now, how we got ourselves in this situation - we have an lvm thin volume with ext4 fs and a container started from it, then, while the container is running we invoke the following command to scrub its contents: openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt