Re: 3.17: bcache_gc: BUG: soft lockup - CPU#2 stuck for 22s!

Zhu Yanhai
2014-10-31T09:20:53Z

Eric,
Is Cpu 0 busy at the moment you see the back trace? Could you please
check whether this patch helps?

http://thread.gmane.org/gmane.linux.kernel.bcache.devel/2713

Theoretically this livelock does exist all the way, however we can't
reproduce it on 3.17 kernel, or maybe it's just because the stress is
not big enough. Anyway it can be reproduced easily on RHEL6 kernel (we
have ported bcache back to the RHEL6 kernel). Dunno if it's caused by
the differences between the schedulers.

-zyh

2014-10-27 10:52 GMT+08:00 Eric Wheeler <bcache@lists.ewheeler.net>:
> Hello all,
>
> We are still getting bcache soft lockups from time to time. These are our
> current stacktraces in bcache_gc and rcu_sched. We recently re-formatted
> the bcache volumes so I believe it is unlikely to be data corruption (unless
> these bugs cause corruption).
>
> I'm happy to do testing or provide more information if you need it.
> Hopefully these four backtraces can shed light on the issue:
>
> CPU: 0 PID: 960 Comm: bcache_gc Tainted: G OEL 3.17.0 #2
> Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.10 01/09/2014
> task: ffff88080a37d220 ti: ffff880037a9c000 task.ti: ffff880037a9c000
> RIP: 0010:[<ffffffffa0392ab6>] [<ffffffffa0392ab6>]
> bch_extent_bad+0x196/0x1c0 [bcache]
> RSP: 0018:ffff880037a9fb68 EFLAGS: 00000202
> RAX: 0000000000000001 RBX: 000007ffffffffff RCX: 000000000000000a
> RDX: 0000000000000000 RSI: 00000004ce41a924 RDI: ffff88080c100000
> RBP: ffff880037a9fb78 R08: 0000000000000001 R09: 0000000000000001
> R10: 000007ffffffffff R11: 0000000000000080 R12: 000007ffffffffff
> R13: ffff880037a9fb58 R14: ffffffffa03928d5 R15: ffff880037a9fb48
> FS: 0000000000000000(0000) GS:ffff88082fc00000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007ff7b409aff0 CR3: 0000000001a14000 CR4: 00000000001407f0
> Stack:
> ffff880037a9fbc8 ffff880809d4f4d0 ffff880037a9fb88 ffffffffa038aa5a
> ffff880037a9fbb8 ffffffffa038aa29 000000000000222f ffff880809d4f4d0
> ffff880037a9fbc8 ffff880037a9fcb8 ffff880037a9fc38 ffffffffa038adc0
> Call Trace:
> [<ffffffffa038aa5a>] bch_ptr_bad+0xa/0x10 [bcache]
> [<ffffffffa038aa29>] bch_btree_iter_next_filter+0x39/0x50 [bcache]
> [<ffffffffa038adc0>] btree_gc_count_keys+0x50/0x70 [bcache]
> [<ffffffffa03910bf>] btree_gc_recurse+0x1bf/0x2e0 [bcache]
> [<ffffffffa038cd2c>] ? btree_gc_mark_node+0xdc/0x210 [bcache]
> [<ffffffffa03912e1>] bch_btree_gc_root+0x101/0x110 [bcache]
> [<ffffffffa03914db>] bch_btree_gc+0x1eb/0x460 [bcache]
> [<ffffffff81096f70>] ? bit_waitqueue+0xa0/0xa0
> [<ffffffffa0391788>] bch_gc_thread+0x38/0x120 [bcache]
> [<ffffffffa0391750>] ? bch_btree_gc+0x460/0x460 [bcache]
> [<ffffffff8107813e>] kthread+0xce/0xf0
> [<ffffffff81078070>] ? kthread_freezable_should_stop+0x70/0x70
> [<ffffffff815c517c>] ret_from_fork+0x7c/0xb0
> [<ffffffff81078070>] ? kthread_freezable_should_stop+0x70/0x70
> Code: 0f 84 a7 fe ff ff e8 cd d4 22 e1 b8 01 00 00 00 e9 9d fe ff ff 85 d2
> 0f 85 90 fe ff ff 0f 1f 84 00 00 00 00 00 41 ff c1 44 89 c8 <4c> 39 c0 0f 82
> 31 ff ff ff 31 c0 e9 77 fe ff ff e8 98 d4 22 e1
>
>
> INFO: rcu_sched self-detected stall on CPU { 0} (t=55337634 jiffies
> g=1978715 c=1978714 q=106641981)
> Task dump for CPU 0:
> bcache_gc R running task 0 960 2 0x00000008
> 0000000000000001 ffff88082fc03d18 ffffffff8107efa7 0000000000000000
> ffffffff81a4cc80 ffff88082fc03d38 ffffffff8107f04f ffff88082fc03d38
> ffffffff81a4cc80 ffff88082fc03d68 ffffffff810b5b01 ffffffff81a4cc80
> Call Trace:
> <IRQ> [<ffffffff8107efa7>] sched_show_task+0xa7/0x110
> [<ffffffff8107f04f>] dump_cpu_task+0x3f/0x50
> [<ffffffff810b5b01>] rcu_dump_cpu_stacks+0x91/0xd0
> [<ffffffff810b5c1e>] print_cpu_stall+0xde/0x140
> [<ffffffff810b60ce>] __rcu_pending+0x1fe/0x210
> [<ffffffff810b73bd>] rcu_check_callbacks+0xdd/0x190
> [<ffffffff810bace8>] update_process_times+0x48/0x80
> [<ffffffff810cc1a9>] tick_sched_handle+0x39/0x80
> [<ffffffff810cc3f4>] tick_sched_timer+0x54/0x90
> [<ffffffff810bd344>] __run_hrtimer+0x74/0x1d0
> [<ffffffff810cc3a0>] ? tick_nohz_handler+0xc0/0xc0
> [<ffffffff810bd6d6>] hrtimer_interrupt+0x106/0x280
> [<ffffffffa038aa50>] ? bch_ptr_invalid+0x10/0x10 [bcache]
> [<ffffffff8103884b>] local_apic_timer_interrupt+0x3b/0x60
> [<ffffffff815c7d85>] smp_apic_timer_interrupt+0x45/0x60
> [<ffffffff815c609d>] apic_timer_interrupt+0x6d/0x80
> <EOI> [<ffffffffa0392a26>] ? bch_extent_bad+0x106/0x1c0 [bcache]
> [<ffffffffa039294d>] ? bch_extent_bad+0x2d/0x1c0 [bcache]
> [<ffffffffa038aa5a>] bch_ptr_bad+0xa/0x10 [bcache]
> [<ffffffffa038aa29>] bch_btree_iter_next_filter+0x39/0x50 [bcache]
> [<ffffffffa038adc0>] btree_gc_count_keys+0x50/0x70 [bcache]
> [<ffffffffa03910bf>] btree_gc_recurse+0x1bf/0x2e0 [bcache]
> [<ffffffffa038cd2c>] ? btree_gc_mark_node+0xdc/0x210 [bcache]
> [<ffffffffa03912e1>] bch_btree_gc_root+0x101/0x110 [bcache]
> [<ffffffffa03914db>] bch_btree_gc+0x1eb/0x460 [bcache]
> [<ffffffff81096f70>] ? bit_waitqueue+0xa0/0xa0
> [<ffffffffa0391788>] bch_gc_thread+0x38/0x120 [bcache]
> [<ffffffffa0391750>] ? bch_btree_gc+0x460/0x460 [bcache]
> [<ffffffff8107813e>] kthread+0xce/0xf0
> [<ffffffff81078070>] ? kthread_freezable_should_stop+0x70/0x70
> [<ffffffff815c517c>] ret_from_fork+0x7c/0xb0
> [<ffffffff81078070>] ? kthread_freezable_should_stop+0x70/0x70
>
>
> NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [bcache_gc:960]
> Modules linked in: [...]
> CPU: 0 PID: 960 Comm: bcache_gc Tainted: G OEL 3.17.0 #2
> Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.10 01/09/2014
> task: ffff88080a37d220 ti: ffff880037a9c000 task.ti: ffff880037a9c000
> RIP: 0010:[<ffffffffa0391ed6>] [<ffffffffa0391ed6>] __ptr_invalid+0x6/0xd0
> [bcache]
> RSP: 0018:ffff880037a9fab0 EFLAGS: 00000287
> RAX: 0000000000000018 RBX: 000007ffffffffff RCX: 9000001003000001
> RDX: 0000000000000000 RSI: ffff8807ea24a168 RDI: ffff88080c100000
> RBP: ffff880037a9fab8 R08: ffff8807ea24a180 R09: 0000000000000001
> R10: 000007ffffffffff R11: 0000000000000030 R12: ffffffffffffff10
> R13: ffff880037a9fab8 R14: 0000000000000000 R15: ffff880037a9fe08
> FS: 0000000000000000(0000) GS:ffff88082fc00000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007ff7b409aff0 CR3: 0000000001a14000 CR4: 00000000001407f0
> Stack:
> ffff880809d4f4d0 ffff880037a9fb48 ffffffffa03928d5 ffff880037a9fb58
> 000007ffffffffff ffff880037a9fda8 ffff880037a9fe08 0000000000000080
> 000007ffffffffff 0000000100000000 ffff880037a9fcb8 0000000000003795
> Call Trace:
> [<ffffffffa03928d5>] __bch_extent_invalid+0xa5/0xd0 [bcache]
> [<ffffffffa0392912>] bch_extent_invalid+0x12/0x20 [bcache]
> [<ffffffffa039294d>] bch_extent_bad+0x2d/0x1c0 [bcache]
> [<ffffffffa038aa5a>] bch_ptr_bad+0xa/0x10 [bcache]
> [<ffffffffa038aa29>] bch_btree_iter_next_filter+0x39/0x50 [bcache]
> [<ffffffffa038adc0>] btree_gc_count_keys+0x50/0x70 [bcache]
> [<ffffffffa03910bf>] btree_gc_recurse+0x1bf/0x2e0 [bcache]
> [<ffffffffa038cd2c>] ? btree_gc_mark_node+0xdc/0x210 [bcache]
> [<ffffffffa03912e1>] bch_btree_gc_root+0x101/0x110 [bcache]
> [<ffffffffa03914db>] bch_btree_gc+0x1eb/0x460 [bcache]
> [<ffffffff81096f70>] ? bit_waitqueue+0xa0/0xa0
> [<ffffffffa0391788>] bch_gc_thread+0x38/0x120 [bcache]
> [<ffffffffa0391750>] ? bch_btree_gc+0x460/0x460 [bcache]
> [<ffffffff8107813e>] kthread+0xce/0xf0
> [<ffffffff81078070>] ? kthread_freezable_should_stop+0x70/0x70
> [<ffffffff815c517c>] ret_from_fork+0x7c/0xb0
> [<ffffffff81078070>] ? kthread_freezable_should_stop+0x70/0x70
> Code: 12 48 81 c4 88 00 00 00 5b 41 5c 41 5d 41 5e 41 5f c9 c3 e8 2d 8a cc
> e0 90 90 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5 41 54 <53> 0f 1f 44 00
> 00 4c 8b 1e 4d 89 d9 49 c1 e9 3c 41 83 e1 07 0f
>
>
> CPU: 0 PID: 960 Comm: bcache_gc Tainted: G OEL 3.17.0 #2
> Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS 2.10 01/09/2014
> task: ffff88080a37d220 ti: ffff880037a9c000 task.ti: ffff880037a9c000
> RIP: 0010:[<ffffffffa038a711>] [<ffffffffa038a711>]
> bch_btree_iter_next+0x21/0x300 [bcache]
> RSP: 0018:ffff880037a9fb78 EFLAGS: 00000202
> RAX: 9000000028000001 RBX: 000007ffffffffff RCX: 000000000000000a
> RDX: ffffffffa038aa50 RSI: ffff880809d4f4d0 RDI: ffff880037a9fbc8
> RBP: ffff880037a9fb88 R08: 0000000000000001 R09: 0000000000000001
> R10: 000007ffffffffff R11: 000000000000007d R12: ffff880037a9fe08
> R13: ffff880037a9fda8 R14: 000007ffffffffff R15: ffff880037a9fb58
> FS: 0000000000000000(0000) GS:ffff88082fc00000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007ff7b409aff0 CR3: 0000000001a14000 CR4: 00000000001407f0
> Stack:
> ffff880037a9fbc8 ffff880809d4f4d0 ffff880037a9fbb8 ffffffffa038aa18
> 00000000000007b6 ffff880809d4f4d0 ffff880037a9fbc8 ffff880037a9fcb8
> ffff880037a9fc38 ffffffffa038adc0 0000000000000004 0000000000000001
> Call Trace:
> [<ffffffffa038aa18>] bch_btree_iter_next_filter+0x28/0x50 [bcache]
> [<ffffffffa038adc0>] btree_gc_count_keys+0x50/0x70 [bcache]
> [<ffffffffa03910bf>] btree_gc_recurse+0x1bf/0x2e0 [bcache]
> [<ffffffffa038cd2c>] ? btree_gc_mark_node+0xdc/0x210 [bcache]
> [<ffffffffa03912e1>] bch_btree_gc_root+0x101/0x110 [bcache]
> [<ffffffffa03914db>] bch_btree_gc+0x1eb/0x460 [bcache]
> [<ffffffff81096f70>] ? bit_waitqueue+0xa0/0xa0
> [<ffffffffa0391788>] bch_gc_thread+0x38/0x120 [bcache]
> [<ffffffffa0391750>] ? bch_btree_gc+0x460/0x460 [bcache]
> [<ffffffff8107813e>] kthread+0xce/0xf0
> [<ffffffff81078070>] ? kthread_freezable_should_stop+0x70/0x70
> [<ffffffff815c517c>] ret_from_fork+0x7c/0xb0
> [<ffffffff81078070>] ? kthread_freezable_should_stop+0x70/0x70
> Code: 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 54 53 0f 1f 44 00 00 45
> 31 e4 48 89 fb 48 83 7f 08 00 74 57 4c 8b 67 10 49 8b 04 24 <48> c1 e8 3c 83
> e0 07 4d 8d 44 c4 10 48 8b 47 18 4c 89 47 10 49
>
>
>
> --
> Eric Wheeler, President eWheeler, Inc. dba Global Linux Security
> 888-LINUX26 (888-546-8926) Fax: 503-716-3878 PO Box 25107
> www.GlobalLinuxSecurity.pro Linux since 1996! Portland, OR 97298
>
> On Sun, 28 Sep 2014, Eric Wheeler wrote:
>
>> Hello Kent, Ross, all:
>>
>> We're getting bcache_gc backtraces and soft lockups; the system continues
>> to be responsive and eventually recovers. We are running 3.17-rc6. (This
>> appears to be a continuation of the thread from 2014-09-15)
>>
>> Please see the following two backtraces. The first shows up in
>> btree_gc_count_keys(), the other is triggered somehow by rcu_sched. We will
>> test with -rc7 this week, though I didn't see any bcache commits in rc7.
>>
>> The server is quite busy:
>> dd in userspace from dm-thinp snapshots to another server
>> two DRBD verify's active backed by dm-thinp volumes
>> note that, dd fills up the buffers so this could be operating with few
>> pages free. (Though we have min-mem set to 256MB.)
>>
>> I see we are hitting functions like bch_ptr_bad() and bch_extent_bad().
>> Could that indicate a cache corruption on our volume?
>>
>> I'm happy to test patches if you have any suggestions or tests that I
>> should run it through.
>>
>> -Eric
>>
>> (note that 5251 is the bcache_gc pid)
>>
>> [131110.478867] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s!
>> [bcache_gc:5251]
>> [131110.479313] Modules linked in: dm_thin_pool dm_bio_prison
>> dm_persistent_data dm_bufio sch_sfq nf_conntrack_ipv4 nf_defrag_ipv4
>> xt_conntrack ipt_REJECT zram xt_c omment iptable_filter ip_tables drbd(OE)
>> libcrc32c bnx2fc cnic uio fcoe libfcoe libfc bridge stp llc 8021q
>> ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_ conntrack
>> ip6table_filter ip6_tables ipv6 ext3 jbd vhost_net macvtap macvlan vhost tun
>> kvm_intel kvm crc32c_intel bcache aesni_intel ablk_helper cryptd lrw
>> gf128mul glue_helper aes_x86_64 iTCO_wdt iTCO_vendor_support serio_raw
>> pcspkr i2c_i801 i2ccore lpc_ich mfd_core e1000e ptp pps_core bna sg
>> sr_mod(E) cdrom(E) ext4(E) jbd2(E ) mbcache(E) usb_storage(E) sd_mod(E)
>> crc_t10dif(E) crct10dif_common(E) video(E) ahci(E) libahci(E) bfa(E)
>> scsi_transport_fc(E) arcmsr(E) dm_mirror(E) dm_region_has h(E) dm_log(E)
>> dm_mod(E)
>>
>> [131110.481865] CPU: 2 PID: 5251 Comm: bcache_gc Tainted: G OEL
>> 3.17.0-rc6 #1
>> [131110.482287] Hardware name: Supermicro X9SCL/X9SCM/X9SCL/X9SCM, BIOS
>> 2.10 01/09/2014
>> [131110.482714] task: ffff8807f4b08e20 ti: ffff8807e51f8000 task.ti:
>> ffff8807e51f8000
>> [131110.483148] RIP: 0010:[<ffffffffa03a7a13>] [<ffffffffa03a7a13>]
>> bch_extent_bad+0xf3/0x1c0 [bcache]
>> [131110.483576] RSP: 0018:ffff8807e51fbb68 EFLAGS: 00000246
>> [131110.483794] RAX: 00000004eb93f476 RBX: ffff8807e51fbe08 RCX:
>> 000000000000000a
>> [131110.484235] RDX: ffffc90017145000 RSI: 00000004eb93f476 RDI:
>> ffff8807ec8c0000
>> [131110.484657] RBP: ffff8807e51fbb78 R08: 0000000000000001 R09:
>> 0000000000000000
>> [131110.485096] R10: 000007ffffffffff R11: 000000000000000c R12:
>> 000007ffffffffff
>> [131110.485524] R13: ffff8807e51fbb58 R14: ffffffffa03a78d5 R15:
>> ffff8807e51fbb48
>> [131110.485969] FS: 0000000000000000(0000) GS:ffff88082fc80000(0000)
>> knlGS:0000000000000000
>> [131110.486393] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [131110.486609] CR2: 00007f5ff8e6c000 CR3: 0000000001a14000 CR4:
>> 00000000001427e0
>> [131110.487051] Stack:
>> [131110.487261] ffff8807e51fbbc8 ffff8807eccf64d0 ffff8807e51fbb88
>> ffffffffa039fa5a
>> [131110.487694] ffff8807e51fbbb8 ffffffffa039fa29 0000000000000dad
>> ffff8807eccf64d0
>> [131110.488140] ffff8807e51fbbc8 ffff8807e51fbcb8 ffff8807e51fbc38
>> ffffffffa039fdc0
>> [131110.488583] Call Trace:
>> [131110.488803] [<ffffffffa039fa5a>] bch_ptr_bad+0xa/0x10 [bcache]
>> [131110.489037] [<ffffffffa039fa29>] bch_btree_iter_next_filter+0x39/0x50
>> [bcache]
>> [131110.489453] [<ffffffffa039fdc0>] btree_gc_count_keys+0x50/0x70
>> [bcache]
>> [131110.489677] [<ffffffffa03a60bf>] btree_gc_recurse+0x1bf/0x2e0
>> [bcache]
>> [131110.489923] [<ffffffffa03a1d2c>] ? btree_gc_mark_node+0xdc/0x210
>> [bcache]
>> [131110.490146] [<ffffffffa03a62e1>] bch_btree_gc_root+0x101/0x110
>> [bcache]
>> [131110.490371] [<ffffffffa03a64db>] bch_btree_gc+0x1eb/0x460 [bcache]
>> [131110.490597] [<ffffffff81096f60>] ? bit_waitqueue+0xa0/0xa0
>> [131110.490821] [<ffffffffa03a6788>] bch_gc_thread+0x38/0x120 [bcache]
>> [131110.491062] [<ffffffffa03a6750>] ? bch_btree_gc+0x460/0x460 [bcache]
>> [131110.491284] [<ffffffff8107812e>] kthread+0xce/0xf0
>> [131110.491505] [<ffffffff81078060>] ?
>> kthread_freezable_should_stop+0x70/0x70
>> [131110.491732] [<ffffffff815c52bc>] ret_from_fork+0x7c/0xb0
>> [131110.491970] [<ffffffff81078060>] ?
>> kthread_freezable_should_stop+0x70/0x70
>> [131110.492194] Code: 0f 1f 84 00 00 00 00 00 48 8b 74 c3 10 48 89 f0 48
>> c1 e8 33 25 ff 0f 00 00 48 8b 94 c7 40 0c 00 00 48 89 f0 48 8b 92 d8 0a 00
>> 00 <48> c1 e8 08 4c 21 d0 48 d3 e8 48 8d 04 40 0f b6 54 82 06 40 28
>>
>>
>> Second backtrace:
>>
>> [131146.469270] INFO: rcu_sched self-detected stall on CPU { 2}
>> (t=11613552 jiffies g=3817860 c=3817859 q=24121607)
>> [131146.469724] Task dump for CPU 2:
>> [131146.469943] bcache_gc R running task 0 5251 2
>> 0x00000088
>> [131146.470168] 0000000000000003 ffff88082fc83d18 ffffffff8107ef97
>> 0000000000000002
>> [131146.470625] ffffffff81a4cc80 ffff88082fc83d38 ffffffff8107f03f
>> ffff88082fc83d38
>> [131146.471058] ffffffff81a4cc80 ffff88082fc83d68 ffffffff810b5b41
>> ffffffff81a4cc80
>> [131146.471507] Call Trace:
>> [131146.471718] <IRQ> [<ffffffff8107ef97>] sched_show_task+0xa7/0x110
>> [131146.471948] [<ffffffff8107f03f>] dump_cpu_task+0x3f/0x50
>> [131146.472167] [<ffffffff810b5b41>] rcu_dump_cpu_stacks+0x91/0xd0
>> [131146.472406] [<ffffffff810b5c5e>] print_cpu_stall+0xde/0x140
>> [131146.472630] [<ffffffff810b610e>] __rcu_pending+0x1fe/0x210
>> [131146.472855] [<ffffffff810b73fd>] rcu_check_callbacks+0xdd/0x190
>> [131146.473077] [<ffffffff810bad28>] update_process_times+0x48/0x80
>> [131146.473321] [<ffffffff810cc1e9>] tick_sched_handle+0x39/0x80
>> [131146.473540] [<ffffffff810cc434>] tick_sched_timer+0x54/0x90
>> [131146.473764] [<ffffffff810bd384>] __run_hrtimer+0x74/0x1d0
>> [131146.473986] [<ffffffff810cc3e0>] ? tick_nohz_handler+0xc0/0xc0
>> [131146.474208] [<ffffffff810bd716>] hrtimer_interrupt+0x106/0x280
>> [131146.474453] [<ffffffffa039fa50>] ? bch_ptr_invalid+0x10/0x10 [bcache]
>> [131146.474668] [<ffffffff810387cb>] local_apic_timer_interrupt+0x3b/0x60
>> [131146.474884] [<ffffffff815c7ec5>] smp_apic_timer_interrupt+0x45/0x60
>> [131146.475100] [<ffffffffa039fa50>] ? bch_ptr_invalid+0x10/0x10 [bcache]
>> [131146.475332] [<ffffffff815c61dd>] apic_timer_interrupt+0x6d/0x80
>> [131146.475544] <EOI> [<ffffffffa03a7951>] ? bch_extent_bad+0x31/0x1c0
>> [bcache]
>> [131146.475967] [<ffffffffa03a794d>] ? bch_extent_bad+0x2d/0x1c0 [bcache]
>> [131146.476183] [<ffffffffa039fa5a>] bch_ptr_bad+0xa/0x10 [bcache]
>> [131146.476418] [<ffffffffa039fa29>] bch_btree_iter_next_filter+0x39/0x50
>> [bcache]
>> [131146.476845] [<ffffffffa039fdc0>] btree_gc_count_keys+0x50/0x70
>> [bcache]
>> [131146.477064] [<ffffffffa03a60bf>] btree_gc_recurse+0x1bf/0x2e0
>> [bcache]
>> [131146.477302] [<ffffffffa03a1d2c>] ? btree_gc_mark_node+0xdc/0x210
>> [bcache]
>> [131146.477521] [<ffffffffa03a62e1>] bch_btree_gc_root+0x101/0x110
>> [bcache]
>> [131146.477747] [<ffffffffa03a64db>] bch_btree_gc+0x1eb/0x460 [bcache]
>> [131146.477964] [<ffffffff81096f60>] ? bit_waitqueue+0xa0/0xa0
>> [131146.478182] [<ffffffffa03a6788>] bch_gc_thread+0x38/0x120 [bcache]
>> [131146.478416] [<ffffffffa03a6750>] ? bch_btree_gc+0x460/0x460 [bcache]
>> [131146.478634] [<ffffffff8107812e>] kthread+0xce/0xf0
>> [131146.478847] [<ffffffff81078060>] ?
>> kthread_freezable_should_stop+0x70/0x70
>> [131146.479066] [<ffffffff815c52bc>] ret_from_fork+0x7c/0xb0
>> [131146.479302] [<ffffffff81078060>] ?
>> kthread_freezable_should_stop+0x70/0x70
>>
>>
>>
>> --
>> Eric Wheeler, President eWheeler, Inc. dba Global Linux Security
>> 888-LINUX26 (888-546-8926) Fax: 503-716-3878 PO Box 25107
>> www.GlobalLinuxSecurity.pro Linux since 1996! Portland, OR 97298
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html