* Crash in jbd2_chksum due to null journal->j_chksum_driver @ 2015-09-30 13:35 Nikolay Borisov 2015-09-30 17:12 ` Darrick J. Wong 0 siblings, 1 reply; 4+ messages in thread From: Nikolay Borisov @ 2015-09-30 13:35 UTC (permalink / raw) To: linux-fsdevel Cc: Linux-Kernel@Vger. Kernel. Org, Jan Kara, darrick.wong, SiteGround Operations Hello, Today a colleague was testing something and while doing so he observed the following crash: jbd2_journal_bmap: journal block not found at offset 67 on dm-26-8 Aborting journal on device dm-26-8. BUG: unable to handle kernel NULL pointer dereference at (null) IP: [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 PGD 3fcef54067 PUD 3fce84e067 PMD 0 Oops: 0000 [#1] SMP Modules linked in: act_police cls_basic sch_ingress veth dm_snapshot openvswitch gre vxlan ip_tunnel xt_owner xt_conntrack iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_CT nf_conntrack iptable_raw ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 ext2 dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mirror dm_region_hash dm_log ses enclosure igb i2c_algo_bit x86_pkg_temp_thermal crc32_pclmul i2c_i801 lpc_ich mfd_core ioapic ioatdma dca shpchp ipmi_devintf ipmi_si ipmi_msghandler CPU: 0 PID: 12059 Comm: jbd2/dm-26-8 Not tainted 3.12.47-clouder1 #1 Hardware name: Supermicro X10DRi/X10DRi, BIOS 1.1 04/14/2015 task: ffff883f904958b0 ti: ffff883fce4d8000 task.ti: ffff883fce4d8000 RIP: 0010:[<ffffffff812b12eb>] [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 RSP: 0018:ffff883fce4d9a58 EFLAGS: 00010282 RAX: 0000000000000000 RBX: ffff883f8dd77000 RCX: 0000000000000006 RDX: 0000000000000000 RSI: ffff883f8dd77000 RDI: ffff883fa0fc6800 RBP: ffff883fce4d9a88 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000000 R12: 00000000f0459c0b R13: 0000000000000411 R14: ffff883f8dd77000 R15: 00000000560bb55d FS: 0000000000000000(0000) GS:ffff881fffa00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000003fd145d000 CR4: 00000000001407f0 Stack: ffffffff81e07402 ffff883fa0fc6800 00000000fffffffb ffff883fce4d9b90 ffff883f8dd77000 ffff883fa0fc6800 ffff883fce4d9aa8 ffffffff812b1369 0000000000000010 ffff883f90c772d8 ffff883fce4d9ae8 ffffffff812b1455 Call Trace: [<ffffffff812b1369>] jbd2_superblock_csum_set+0x29/0x40 [<ffffffff812b1455>] jbd2_write_superblock+0x85/0x1b0 [<ffffffff812b1b70>] jbd2_journal_update_sb_errno+0x50/0x60 [<ffffffff812b1bd0>] __journal_abort_soft+0x50/0x60 [<ffffffff812b1c80>] jbd2_journal_bmap+0x90/0xa0 [<ffffffff812b1ec7>] jbd2_journal_next_log_block+0x77/0x80 [<ffffffff812b1ef3>] jbd2_journal_get_descriptor_buffer+0x23/0xb0 [<ffffffff812aa02c>] journal_submit_commit_record+0x7c/0x1e0 [<ffffffff812abade>] jbd2_journal_commit_transaction+0x194e/0x1d20 [<ffffffff812b062f>] kjournald2+0xef/0x2b0 [<ffffffff810aef00>] ? wake_up_bit+0x40/0x40 [<ffffffff812b0540>] ? commit_timeout+0x10/0x10 [<ffffffff810ae48e>] kthread+0xce/0xe0 [<ffffffff810ae3c0>] ? kthread_freezable_should_stop+0x80/0x80 [<ffffffff816571c8>] ret_from_fork+0x58/0x90 [<ffffffff810ae3c0>] ? kthread_freezable_should_stop+0x80/0x80 Code: 55 48 89 e5 41 54 53 48 83 ec 20 0f 1f 44 00 00 44 8b a6 fc 00 00 00 48 89 f3 c7 86 fc 00 00 00 00 00 00 00 48 8b 87 d0 04 00 00 <83> 38 04 77 39 48 89 45 d0 c7 45 d8 00 00 00 00 48 8d 7d d0 c7 RIP [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 RSP <ffff883fce4d9a58> CR2: 0000000000000000 ---[ end trace e1bd94031f410b71 ]--- The ffffffff812b12eb address actually is jbd2_chksum and the instruction where the deference is happening in crypto_shash_descsize(), essentially journal->j_chksum_driver is NULL. Now, how we got ourselves in this situation - we have an lvm thin volume with ext4 fs and a container started from it, then, while the container is running we invoke the following command to scrub its contents: openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt </dev/zero | dd bs=64K of=/dev/volumegroupname/volumename And then when we try to umount the volume we get the aforementioned crash. Naturally, because we overwrite the on-disk contents jbd2_journal_bmap fails which triggers the journal abort which wants to update the on-disk errno, which naturally triggers a superblock checksum regeneration and this goes BOOM. I looked around the code but couldn't figure out a code path which allows the checksum driver to become null at runtime. ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Crash in jbd2_chksum due to null journal->j_chksum_driver 2015-09-30 13:35 Crash in jbd2_chksum due to null journal->j_chksum_driver Nikolay Borisov @ 2015-09-30 17:12 ` Darrick J. Wong 2015-09-30 18:13 ` Nikolay Borisov 0 siblings, 1 reply; 4+ messages in thread From: Darrick J. Wong @ 2015-09-30 17:12 UTC (permalink / raw) To: Nikolay Borisov Cc: linux-fsdevel, Linux-Kernel@Vger. Kernel. Org, Jan Kara, SiteGround Operations On Wed, Sep 30, 2015 at 04:35:49PM +0300, Nikolay Borisov wrote: > Hello, > > Today a colleague was testing something and while doing so he observed > the following crash: > > jbd2_journal_bmap: journal block not found at offset 67 on dm-26-8 > Aborting journal on device dm-26-8. > BUG: unable to handle kernel NULL pointer dereference at (null) > IP: [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 > PGD 3fcef54067 PUD 3fce84e067 PMD 0 > Oops: 0000 [#1] SMP > Modules linked in: act_police cls_basic sch_ingress veth dm_snapshot openvswitch gre vxlan ip_tunnel xt_owner xt_conntrack iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_CT nf_conntrack iptable_raw ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 ext2 dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mirror dm_region_hash dm_log ses enclosure igb i2c_algo_bit x86_pkg_temp_thermal crc32_pclmul i2c_i801 lpc_ich mfd_core ioapic ioatdma dca shpchp ipmi_devintf ipmi_si ipmi_msghandler > CPU: 0 PID: 12059 Comm: jbd2/dm-26-8 Not tainted 3.12.47-clouder1 #1 > Hardware name: Supermicro X10DRi/X10DRi, BIOS 1.1 04/14/2015 > task: ffff883f904958b0 ti: ffff883fce4d8000 task.ti: ffff883fce4d8000 > RIP: 0010:[<ffffffff812b12eb>] [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 > RSP: 0018:ffff883fce4d9a58 EFLAGS: 00010282 > RAX: 0000000000000000 RBX: ffff883f8dd77000 RCX: 0000000000000006 > RDX: 0000000000000000 RSI: ffff883f8dd77000 RDI: ffff883fa0fc6800 > RBP: ffff883fce4d9a88 R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000000000001 R11: 0000000000000000 R12: 00000000f0459c0b > R13: 0000000000000411 R14: ffff883f8dd77000 R15: 00000000560bb55d > FS: 0000000000000000(0000) GS:ffff881fffa00000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 0000000000000000 CR3: 0000003fd145d000 CR4: 00000000001407f0 > Stack: > ffffffff81e07402 ffff883fa0fc6800 00000000fffffffb ffff883fce4d9b90 > ffff883f8dd77000 ffff883fa0fc6800 ffff883fce4d9aa8 ffffffff812b1369 > 0000000000000010 ffff883f90c772d8 ffff883fce4d9ae8 ffffffff812b1455 > Call Trace: > [<ffffffff812b1369>] jbd2_superblock_csum_set+0x29/0x40 > [<ffffffff812b1455>] jbd2_write_superblock+0x85/0x1b0 > [<ffffffff812b1b70>] jbd2_journal_update_sb_errno+0x50/0x60 > [<ffffffff812b1bd0>] __journal_abort_soft+0x50/0x60 > [<ffffffff812b1c80>] jbd2_journal_bmap+0x90/0xa0 > [<ffffffff812b1ec7>] jbd2_journal_next_log_block+0x77/0x80 > [<ffffffff812b1ef3>] jbd2_journal_get_descriptor_buffer+0x23/0xb0 > [<ffffffff812aa02c>] journal_submit_commit_record+0x7c/0x1e0 > [<ffffffff812abade>] jbd2_journal_commit_transaction+0x194e/0x1d20 > [<ffffffff812b062f>] kjournald2+0xef/0x2b0 > [<ffffffff810aef00>] ? wake_up_bit+0x40/0x40 > [<ffffffff812b0540>] ? commit_timeout+0x10/0x10 > [<ffffffff810ae48e>] kthread+0xce/0xe0 > [<ffffffff810ae3c0>] ? kthread_freezable_should_stop+0x80/0x80 > [<ffffffff816571c8>] ret_from_fork+0x58/0x90 > [<ffffffff810ae3c0>] ? kthread_freezable_should_stop+0x80/0x80 > Code: 55 48 89 e5 41 54 53 48 83 ec 20 0f 1f 44 00 00 44 8b a6 fc 00 00 00 48 89 f3 c7 86 fc 00 00 00 00 00 00 00 48 8b 87 d0 04 00 00 <83> 38 04 77 39 48 89 45 d0 c7 45 d8 00 00 00 00 48 8d 7d d0 c7 > RIP [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 > RSP <ffff883fce4d9a58> > CR2: 0000000000000000 > ---[ end trace e1bd94031f410b71 ]--- > > The ffffffff812b12eb address actually is jbd2_chksum and the > instruction where the deference is happening in > crypto_shash_descsize(), essentially journal->j_chksum_driver is NULL. > > Now, how we got ourselves in this situation - we have an lvm thin > volume with ext4 fs and a container started from it, > then, while the container is running we invoke the following > command to scrub its contents: > > openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt </dev/zero | dd bs=64K of=/dev/volumegroupname/volumename > > > And then when we try to umount the volume we get the aforementioned > crash. Naturally, because we overwrite the on-disk contents jbd2_journal_bmap > fails which triggers the journal abort which wants to update the on-disk > errno, which naturally triggers a superblock checksum regeneration > and this goes BOOM. > > I looked around the code but couldn't figure out a code path > which allows the checksum driver to become null at runtime. Most likely is that the journal wasn't started with the checksum driver turned on, and then your randomizing of the journal sb *while it was running* flipped the feature bit on, causing jbd2 to think checksumming was turned on. I guess the "proper" fix is to set j_chksum_driver at journal load time if the superblock flags are set properly and then gate all other accesses on the status of j_chksum_driver just in case someone obliterates the journal sb. OTOH, why can't you unmount the FS and /then/ randomize the disk? --D > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Crash in jbd2_chksum due to null journal->j_chksum_driver 2015-09-30 17:12 ` Darrick J. Wong @ 2015-09-30 18:13 ` Nikolay Borisov 2015-09-30 18:43 ` Darrick J. Wong 0 siblings, 1 reply; 4+ messages in thread From: Nikolay Borisov @ 2015-09-30 18:13 UTC (permalink / raw) To: Darrick J. Wong Cc: Nikolay Borisov, linux-fsdevel, Linux-Kernel@Vger. Kernel. Org, Jan Kara, SiteGround Operations Hello, Well, I guess I can do that the thing is the current scenario was like that. Anyway, I thought something like what you describe could be happening. I saw your patch and I'm going to test it tomorrow. But I think the patch needs to be tagged for stable since there is going to be effort to make filesystems mountable in non-init user namespace and an arbitrary user could potentially cause instability on the system? Regards, Nikolay On Wed, Sep 30, 2015 at 8:12 PM, Darrick J. Wong <darrick.wong@oracle.com> wrote: > On Wed, Sep 30, 2015 at 04:35:49PM +0300, Nikolay Borisov wrote: >> Hello, >> >> Today a colleague was testing something and while doing so he observed >> the following crash: >> >> jbd2_journal_bmap: journal block not found at offset 67 on dm-26-8 >> Aborting journal on device dm-26-8. >> BUG: unable to handle kernel NULL pointer dereference at (null) >> IP: [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 >> PGD 3fcef54067 PUD 3fce84e067 PMD 0 >> Oops: 0000 [#1] SMP >> Modules linked in: act_police cls_basic sch_ingress veth dm_snapshot openvswitch gre vxlan ip_tunnel xt_owner xt_conntrack iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_CT nf_conntrack iptable_raw ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 ext2 dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mirror dm_region_hash dm_log ses enclosure igb i2c_algo_bit x86_pkg_temp_thermal crc32_pclmul i2c_i801 lpc_ich mfd_core ioapic ioatdma dca shpchp ipmi_devintf ipmi_si ipmi_msghandler >> CPU: 0 PID: 12059 Comm: jbd2/dm-26-8 Not tainted 3.12.47-clouder1 #1 >> Hardware name: Supermicro X10DRi/X10DRi, BIOS 1.1 04/14/2015 >> task: ffff883f904958b0 ti: ffff883fce4d8000 task.ti: ffff883fce4d8000 >> RIP: 0010:[<ffffffff812b12eb>] [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 >> RSP: 0018:ffff883fce4d9a58 EFLAGS: 00010282 >> RAX: 0000000000000000 RBX: ffff883f8dd77000 RCX: 0000000000000006 >> RDX: 0000000000000000 RSI: ffff883f8dd77000 RDI: ffff883fa0fc6800 >> RBP: ffff883fce4d9a88 R08: 0000000000000000 R09: 0000000000000000 >> R10: 0000000000000001 R11: 0000000000000000 R12: 00000000f0459c0b >> R13: 0000000000000411 R14: ffff883f8dd77000 R15: 00000000560bb55d >> FS: 0000000000000000(0000) GS:ffff881fffa00000(0000) knlGS:0000000000000000 >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> CR2: 0000000000000000 CR3: 0000003fd145d000 CR4: 00000000001407f0 >> Stack: >> ffffffff81e07402 ffff883fa0fc6800 00000000fffffffb ffff883fce4d9b90 >> ffff883f8dd77000 ffff883fa0fc6800 ffff883fce4d9aa8 ffffffff812b1369 >> 0000000000000010 ffff883f90c772d8 ffff883fce4d9ae8 ffffffff812b1455 >> Call Trace: >> [<ffffffff812b1369>] jbd2_superblock_csum_set+0x29/0x40 >> [<ffffffff812b1455>] jbd2_write_superblock+0x85/0x1b0 >> [<ffffffff812b1b70>] jbd2_journal_update_sb_errno+0x50/0x60 >> [<ffffffff812b1bd0>] __journal_abort_soft+0x50/0x60 >> [<ffffffff812b1c80>] jbd2_journal_bmap+0x90/0xa0 >> [<ffffffff812b1ec7>] jbd2_journal_next_log_block+0x77/0x80 >> [<ffffffff812b1ef3>] jbd2_journal_get_descriptor_buffer+0x23/0xb0 >> [<ffffffff812aa02c>] journal_submit_commit_record+0x7c/0x1e0 >> [<ffffffff812abade>] jbd2_journal_commit_transaction+0x194e/0x1d20 >> [<ffffffff812b062f>] kjournald2+0xef/0x2b0 >> [<ffffffff810aef00>] ? wake_up_bit+0x40/0x40 >> [<ffffffff812b0540>] ? commit_timeout+0x10/0x10 >> [<ffffffff810ae48e>] kthread+0xce/0xe0 >> [<ffffffff810ae3c0>] ? kthread_freezable_should_stop+0x80/0x80 >> [<ffffffff816571c8>] ret_from_fork+0x58/0x90 >> [<ffffffff810ae3c0>] ? kthread_freezable_should_stop+0x80/0x80 >> Code: 55 48 89 e5 41 54 53 48 83 ec 20 0f 1f 44 00 00 44 8b a6 fc 00 00 00 48 89 f3 c7 86 fc 00 00 00 00 00 00 00 48 8b 87 d0 04 00 00 <83> 38 04 77 39 48 89 45 d0 c7 45 d8 00 00 00 00 48 8d 7d d0 c7 >> RIP [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 >> RSP <ffff883fce4d9a58> >> CR2: 0000000000000000 >> ---[ end trace e1bd94031f410b71 ]--- >> >> The ffffffff812b12eb address actually is jbd2_chksum and the >> instruction where the deference is happening in >> crypto_shash_descsize(), essentially journal->j_chksum_driver is NULL. >> >> Now, how we got ourselves in this situation - we have an lvm thin >> volume with ext4 fs and a container started from it, >> then, while the container is running we invoke the following >> command to scrub its contents: >> >> openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt </dev/zero | dd bs=64K of=/dev/volumegroupname/volumename >> >> >> And then when we try to umount the volume we get the aforementioned >> crash. Naturally, because we overwrite the on-disk contents jbd2_journal_bmap >> fails which triggers the journal abort which wants to update the on-disk >> errno, which naturally triggers a superblock checksum regeneration >> and this goes BOOM. >> >> I looked around the code but couldn't figure out a code path >> which allows the checksum driver to become null at runtime. > > Most likely is that the journal wasn't started with the checksum driver > turned on, and then your randomizing of the journal sb *while it was running* > flipped the feature bit on, causing jbd2 to think checksumming was turned on. > > I guess the "proper" fix is to set j_chksum_driver at journal load time if > the superblock flags are set properly and then gate all other accesses on > the status of j_chksum_driver just in case someone obliterates the journal sb. > > OTOH, why can't you unmount the FS and /then/ randomize the disk? > > --D > >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Crash in jbd2_chksum due to null journal->j_chksum_driver 2015-09-30 18:13 ` Nikolay Borisov @ 2015-09-30 18:43 ` Darrick J. Wong 0 siblings, 0 replies; 4+ messages in thread From: Darrick J. Wong @ 2015-09-30 18:43 UTC (permalink / raw) To: Nikolay Borisov Cc: Nikolay Borisov, linux-fsdevel, Linux-Kernel@Vger. Kernel. Org, Jan Kara, SiteGround Operations On Wed, Sep 30, 2015 at 09:13:38PM +0300, Nikolay Borisov wrote: > Hello, > > > Well, I guess I can do that the thing is the current scenario was like > that. Anyway, > I thought something like what you describe could be happening. I saw > your patch and I'm > going to test it tomorrow. But I think the patch needs to be tagged > for stable since there > is going to be effort to make filesystems mountable in non-init user > namespace and > an arbitrary user could potentially cause instability on the system? <shrug> If non-root users can write arbitrarily to block devices, I'm sure a /lot/ more bad things can happen. But you're right, we could at least avoid crashing. --D > > Regards, > Nikolay > > On Wed, Sep 30, 2015 at 8:12 PM, Darrick J. Wong > <darrick.wong@oracle.com> wrote: > > On Wed, Sep 30, 2015 at 04:35:49PM +0300, Nikolay Borisov wrote: > >> Hello, > >> > >> Today a colleague was testing something and while doing so he observed > >> the following crash: > >> > >> jbd2_journal_bmap: journal block not found at offset 67 on dm-26-8 > >> Aborting journal on device dm-26-8. > >> BUG: unable to handle kernel NULL pointer dereference at (null) > >> IP: [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 > >> PGD 3fcef54067 PUD 3fce84e067 PMD 0 > >> Oops: 0000 [#1] SMP > >> Modules linked in: act_police cls_basic sch_ingress veth dm_snapshot openvswitch gre vxlan ip_tunnel xt_owner xt_conntrack iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_CT nf_conntrack iptable_raw ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr ipv6 ext2 dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mirror dm_region_hash dm_log ses enclosure igb i2c_algo_bit x86_pkg_temp_thermal crc32_pclmul i2c_i801 lpc_ich mfd_core ioapic ioatdma dca shpchp ipmi_devintf ipmi_si ipmi_msghandler > >> CPU: 0 PID: 12059 Comm: jbd2/dm-26-8 Not tainted 3.12.47-clouder1 #1 > >> Hardware name: Supermicro X10DRi/X10DRi, BIOS 1.1 04/14/2015 > >> task: ffff883f904958b0 ti: ffff883fce4d8000 task.ti: ffff883fce4d8000 > >> RIP: 0010:[<ffffffff812b12eb>] [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 > >> RSP: 0018:ffff883fce4d9a58 EFLAGS: 00010282 > >> RAX: 0000000000000000 RBX: ffff883f8dd77000 RCX: 0000000000000006 > >> RDX: 0000000000000000 RSI: ffff883f8dd77000 RDI: ffff883fa0fc6800 > >> RBP: ffff883fce4d9a88 R08: 0000000000000000 R09: 0000000000000000 > >> R10: 0000000000000001 R11: 0000000000000000 R12: 00000000f0459c0b > >> R13: 0000000000000411 R14: ffff883f8dd77000 R15: 00000000560bb55d > >> FS: 0000000000000000(0000) GS:ffff881fffa00000(0000) knlGS:0000000000000000 > >> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > >> CR2: 0000000000000000 CR3: 0000003fd145d000 CR4: 00000000001407f0 > >> Stack: > >> ffffffff81e07402 ffff883fa0fc6800 00000000fffffffb ffff883fce4d9b90 > >> ffff883f8dd77000 ffff883fa0fc6800 ffff883fce4d9aa8 ffffffff812b1369 > >> 0000000000000010 ffff883f90c772d8 ffff883fce4d9ae8 ffffffff812b1455 > >> Call Trace: > >> [<ffffffff812b1369>] jbd2_superblock_csum_set+0x29/0x40 > >> [<ffffffff812b1455>] jbd2_write_superblock+0x85/0x1b0 > >> [<ffffffff812b1b70>] jbd2_journal_update_sb_errno+0x50/0x60 > >> [<ffffffff812b1bd0>] __journal_abort_soft+0x50/0x60 > >> [<ffffffff812b1c80>] jbd2_journal_bmap+0x90/0xa0 > >> [<ffffffff812b1ec7>] jbd2_journal_next_log_block+0x77/0x80 > >> [<ffffffff812b1ef3>] jbd2_journal_get_descriptor_buffer+0x23/0xb0 > >> [<ffffffff812aa02c>] journal_submit_commit_record+0x7c/0x1e0 > >> [<ffffffff812abade>] jbd2_journal_commit_transaction+0x194e/0x1d20 > >> [<ffffffff812b062f>] kjournald2+0xef/0x2b0 > >> [<ffffffff810aef00>] ? wake_up_bit+0x40/0x40 > >> [<ffffffff812b0540>] ? commit_timeout+0x10/0x10 > >> [<ffffffff810ae48e>] kthread+0xce/0xe0 > >> [<ffffffff810ae3c0>] ? kthread_freezable_should_stop+0x80/0x80 > >> [<ffffffff816571c8>] ret_from_fork+0x58/0x90 > >> [<ffffffff810ae3c0>] ? kthread_freezable_should_stop+0x80/0x80 > >> Code: 55 48 89 e5 41 54 53 48 83 ec 20 0f 1f 44 00 00 44 8b a6 fc 00 00 00 48 89 f3 c7 86 fc 00 00 00 00 00 00 00 48 8b 87 d0 04 00 00 <83> 38 04 77 39 48 89 45 d0 c7 45 d8 00 00 00 00 48 8d 7d d0 c7 > >> RIP [<ffffffff812b12eb>] jbd2_superblock_csum+0x2b/0x80 > >> RSP <ffff883fce4d9a58> > >> CR2: 0000000000000000 > >> ---[ end trace e1bd94031f410b71 ]--- > >> > >> The ffffffff812b12eb address actually is jbd2_chksum and the > >> instruction where the deference is happening in > >> crypto_shash_descsize(), essentially journal->j_chksum_driver is NULL. > >> > >> Now, how we got ourselves in this situation - we have an lvm thin > >> volume with ext4 fs and a container started from it, > >> then, while the container is running we invoke the following > >> command to scrub its contents: > >> > >> openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt </dev/zero | dd bs=64K of=/dev/volumegroupname/volumename > >> > >> > >> And then when we try to umount the volume we get the aforementioned > >> crash. Naturally, because we overwrite the on-disk contents jbd2_journal_bmap > >> fails which triggers the journal abort which wants to update the on-disk > >> errno, which naturally triggers a superblock checksum regeneration > >> and this goes BOOM. > >> > >> I looked around the code but couldn't figure out a code path > >> which allows the checksum driver to become null at runtime. > > > > Most likely is that the journal wasn't started with the checksum driver > > turned on, and then your randomizing of the journal sb *while it was running* > > flipped the feature bit on, causing jbd2 to think checksumming was turned on. > > > > I guess the "proper" fix is to set j_chksum_driver at journal load time if > > the superblock flags are set properly and then gate all other accesses on > > the status of j_chksum_driver just in case someone obliterates the journal sb. > > > > OTOH, why can't you unmount the FS and /then/ randomize the disk? > > > > --D > > > >> -- > >> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > >> the body of a message to majordomo@vger.kernel.org > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2015-09-30 18:43 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-09-30 13:35 Crash in jbd2_chksum due to null journal->j_chksum_driver Nikolay Borisov 2015-09-30 17:12 ` Darrick J. Wong 2015-09-30 18:13 ` Nikolay Borisov 2015-09-30 18:43 ` Darrick J. Wong
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).