Mysterious Kernel Bug

Mysterious Kernel Bug

[Jue]

Hi!

I'm using btrfs on this system:
> Linux qmap02 3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC 
> 2014 x86_64 x86_64 x86_64 GNU/Linux

During a task with heavy I/O (it's a database server) suddenly every 
disk-accessing process freezes. Dmesg outputs the following when this 
happend:
> [38108.895509] ------------[ cut here ]------------
> [38108.895663] kernel BUG at 
> /build/buildd/linux-3.11.0/fs/btrfs/ctree.c:2964!
> [38108.895825] invalid opcode: 0000 [#1] SMP
> [38108.895970] Modules linked in: cirrus ttm drm_kms_helper drm 
> syscopyarea sysfillrect sysimgblt kvm_intel kvm virtio_balloon 
> i2c_piix4 mac_hid virtio_console psmouse serio_raw microcode lp 
> parport btrfs xor zlib_deflate raid6_pq libcrc32c floppy
> [38108.896013] CPU: 1 PID: 29950 Comm: btrfs-endio-wri Not tainted 
> 3.11.0-18-generic #32-Ubuntu
> [38108.896013] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007
> [38108.896013] task: ffff880dcf2c2ee0 ti: ffff8809b9230000 task.ti: 
> ffff8809b9230000
> [38108.896013] RIP: 0010:[<ffffffffa0058db1>] [<ffffffffa0058db1>] 
> btrfs_set_item_key_safe+0x161/0x170 [btrfs]
> [38108.896013] RSP: 0018:ffff8809b9231b20  EFLAGS: 00010286
> [38108.896013] RAX: 00000000ffffffff RBX: 0000000000000018 RCX: 
> 0000000017466000
> [38108.896013] RDX: 00000000ffffffff RSI: ffff8809b9231c26 RDI: 
> ffff8809b9231b3f
> [38108.896013] RBP: ffff8809b9231b78 R08: 0000000000017340 R09: 
> 0000000000000000
> [38108.896013] R10: 0000000017468000 R11: 0000000000000000 R12: 
> ffff8809b9231b2e
> [38108.896013] R13: ffff880dced2bd80 R14: ffff8809b9231c26 R15: 
> ffff880b95815680
> [38108.896013] FS:  0000000000000000(0000) GS:ffff880e1fc40000(0000) 
> knlGS:0000000000000000
> [38108.896013] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [38108.896013] CR2: 000000000085f001 CR3: 0000000c4dbcf000 CR4: 
> 00000000000006e0
> [38108.896013] Stack:
> [38108.896013]  ffff880cee276000 081a000000000000 006c000000000000 
> 1a00000000174640
> [38108.896013]  6c00000000000008 0000000017464000 ffff880b95815680 
> 0000000017466000
> [38108.896013]  0000000000000000 0000000000000ad3 ffff880dced2bd80 
> ffff8809b9231c70
> [38108.896013] Call Trace:
> [38108.896013]  [<ffffffffa008fba1>] __btrfs_drop_extents+0x421/0xae0 
> [btrfs]
> [38108.896013]  [<ffffffffa0090d50>] btrfs_drop_extents+0x60/0x90 [btrfs]
> [38108.896013]  [<ffffffffa007f6ec>] 
> insert_reserved_file_extent.constprop.58+0x6c/0x290 [btrfs]
> [38108.896013]  [<ffffffff816ef78e>] ? _raw_spin_lock+0xe/0x20
> [38108.896013]  [<ffffffffa0088aff>] 
> btrfs_finish_ordered_io+0x4ef/0x910 [btrfs]
> [38108.896013]  [<ffffffff81141f07>] ? mempool_free_slab+0x17/0x20
> [38108.896013]  [<ffffffff811422d9>] ? mempool_free+0x49/0x90
> [38108.896013]  [<ffffffff811ddcad>] ? bio_put+0x7d/0xa0
> [38108.896013]  [<ffffffffa00be498>] ? 
> end_compressed_bio_write+0xe8/0xf0 [btrfs]
> [38108.896013]  [<ffffffffa0089105>] finish_ordered_fn+0x15/0x20 [btrfs]
> [38108.896013]  [<ffffffffa00a954e>] worker_loop+0x15e/0x550 [btrfs]
> [38108.896013]  [<ffffffffa00a93f0>] ? btrfs_queue_worker+0x320/0x320 
> [btrfs]
> [38108.896013]  [<ffffffff810848d0>] kthread+0xc0/0xd0
> [38108.896013]  [<ffffffff81084810>] ? kthread_create_on_node+0x120/0x120
> [38108.896013]  [<ffffffff816f836c>] ret_from_fork+0x7c/0xb0
> [38108.896013]  [<ffffffff81084810>] ? kthread_create_on_node+0x120/0x120
> [38108.896013] Code: 48 8b 45 bf 48 8d 7d c7 4c 89 f6 48 89 45 d0 0f 
> b6 45 be 88 45 cf 48 8b 45 b6 48 89 45 c7 e8 c7 f2 ff ff 85 c0 0f 8f 
> 48 ff ff ff <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 
> 00 55
> [38108.896013] RIP  [<ffffffffa0058db1>] 
> btrfs_set_item_key_safe+0x161/0x170 [btrfs]
> [38108.896013]  RSP <ffff8809b9231b20>
> [38108.947225] ---[ end trace f5739e163e2811a8 ]---

Does anybody have any idea what this problem causes and how to solve it?

Thank you a lot!

Regards,
Juergen Fitschen

Re: Mysterious Kernel Bug

[Tomasz Chmielewski]

I had something similar with 3.14.0-rc7.

I saw it also with 3.14.0-rc3, as discussed in this thread:

http://thread.gmane.org/gmane.comp.file-systems.btrfs/32938/focus=32977

Here is the trace for 3.14.0-rc7:

[377944.904848] ------------[ cut here ]------------
[377944.904947] kernel BUG at fs/btrfs/ctree.c:3215!
[377944.905007] invalid opcode: 0000 [#1] SMP
[377944.906494] Modules linked in: cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop ehci_pci ehci_hcd battery tpm_infineon lpc_ich mfd_core pcspkr video tpm_tis tpm acpi_cpufreq button i2c_i801 parport_pc i2c_core parport ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata r8169 scsi_mod mii
[377944.906849] CPU: 4 PID: 10787 Comm: btrfs-endio-wri Not tainted 3.14.0-rc7 #1
[377944.906960] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
[377944.907075] task: ffff8807f276b000 ti: ffff880601fb8000 task.ti: ffff880601fb8000
[377944.907187] RIP: 0010:[<ffffffffa02a8361>]  [<ffffffffa02a8361>] btrfs_set_item_key_safe+0xb2/0x119 [btrfs]
[377944.907316] RSP: 0018:ffff880601fb9b18  EFLAGS: 00010286
[377944.907375] RAX: 00000000ffffffff RBX: 0000000000000005 RCX: 0000000000000000
[377944.907487] RDX: 00000000001ce000 RSI: ffff880601fb9c4f RDI: ffff880601fb9af7
[377944.907598] RBP: ffff880601fb9b68 R08: ffff880601fb9b38 R09: ffff880123f56d48
[377944.907708] R10: 0000000000001000 R11: 0000160000000000 R12: ffff880123f56c80
[377944.907820] R13: ffff8807eff09990 R14: ffff88043cdc7000 R15: ffff880601fb9c4f
[377944.907931] FS:  0000000000000000(0000) GS:ffff88081fb00000(0000) knlGS:0000000000000000
[377944.908045] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[377944.908105] CR2: 00007fb78313d9de CR3: 000000000160b000 CR4: 00000000001407e0
[377944.908215] Stack:
[377944.908269]  ffff880691594000 a200000000000000 6c00000000000926 00000000001cc000
[377944.908384]  ffff880601fb9b68 ffff8807eff09990 ffff880123f56c80 00000000001e0000
[377944.908499]  0000000000000000 0000000000000001 ffff880601fb9ca8 ffffffffa02d589d
[377944.908615] Call Trace:
[377944.908687]  [<ffffffffa02d589d>] __btrfs_drop_extents+0x6c1/0xb12 [btrfs]
[377944.908765]  [<ffffffffa02c9811>] insert_reserved_file_extent.constprop.63+0x9a/0x2a5 [btrfs]
[377944.908892]  [<ffffffffa02ccebb>] btrfs_finish_ordered_io+0x265/0x3ef [btrfs]
[377944.909009]  [<ffffffff81040e5d>] ? usleep_range+0x3d/0x3d
[377944.909080]  [<ffffffffa02cd055>] finish_ordered_fn+0x10/0x12 [btrfs]
[377944.909157]  [<ffffffffa02eb649>] worker_loop+0x15e/0x495 [btrfs]
[377944.909232]  [<ffffffffa02eb4eb>] ? btrfs_queue_worker+0x269/0x269 [btrfs]
[377944.909298]  [<ffffffff81050c92>] kthread+0xcd/0xd5
[377944.909360]  [<ffffffff81050bc5>] ? kthread_freezable_should_stop+0x43/0x43
[377944.909425]  [<ffffffff81399d7c>] ret_from_fork+0x7c/0xb0
[377944.909487]  [<ffffffff81050bc5>] ? kthread_freezable_should_stop+0x43/0x43
[377944.909549] Code: 8d 75 bf b9 11 00 00 00 4c 89 e7 48 63 d2 48 6b d2 19 48 83 c2 65 e8 b0 89 03 00 48 8d 7d bf 4c 89 fe e8 9e f5 ff ff 85 c0 7f 02 <0f> 0b 49 8b 47 09 48 63 d3 48 8d 75 bf 48 6b d2 19 b9 11 00 00
[377944.909794] RIP  [<ffffffffa02a8361>] btrfs_set_item_key_safe+0xb2/0x119 [btrfs]
[377944.909914]  RSP <ffff880601fb9b18>
[377944.910260] ---[ end trace e209399cfd0911cd ]---



-- 
Tomasz Chmielewski
http://wpkg.org