Re: memory overflow or undeflow in free space tree / space_info?

[Stefan Priebe - Profihost AG <s.priebe@profihost.ag>]

Hi Josef,

this still hapens with current 4.8-rc* releases. Anything i can do to
debug this? May be insert some code to check for an under or overflow in
the code?

Stefan

Am 14.08.2016 um 17:22 schrieb Stefan Priebe - Profihost AG:
> Hi Josef,
> 
> anything i could do or test? Results with a vanilla next branch are the
> same.
> 
> Stefan
> 
> Am 11.08.2016 um 08:09 schrieb Stefan Priebe - Profihost AG:
>> Hello,
>>
>> the backtrace and info on umount looks the same:
>>
>> [241910.341124] ------------[ cut here ]------------
>> [241910.379991] WARNING: CPU: 1 PID: 26664 at
>> fs/btrfs/extent-tree.c:5701 btrfs_free_block_groups+0x370/0x410 [btrfs]
>> [241910.422099] Modules linked in: netconsole mpt3sas ipt_REJECT
>> raid_class nf_reject_ipv4 scsi_transport_sas xt_multiport 8021q garp
>> iptable_filter ip_tables x_tables bonding coretemp loop usbhid ehci_pci
>> i2c_i801 ehci_hcd usbcore i2c_core shpchp usb_common ipmi_si
>> ipmi_msghandler button btrfs dm_mod raid1 raid456 async_raid6_recov
>> async_memcpy async_pq async_xor async_tx xor raid6_pq md_mod sg sd_mod
>> ixgbe i40e mdio ptp ahci libahci pps_core megaraid_sas
>> [241910.616845] CPU: 1 PID: 26664 Comm: umount Not tainted
>> 4.7.0-rc6-29043-g8b8b08c #1
>> [241910.669646] Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 1.0c
>> 02/18/2015
>> [241910.723716]  0000000000000000 ffff8808d104bca8 ffffffffbd3d83cf
>> 0000000000000000
>> [241910.779309]  0000000000000000 ffff8808d104bcf8 ffffffffbd085615
>> ffff8808d104bd08
>> [241910.835143]  000016455a3410a8 00000047a0000000 0000000000000000
>> ffff8808469e2088
>> [241910.891882] Call Trace:
>> [241910.947624]  [<ffffffffbd3d83cf>] dump_stack+0x63/0x84
>> [241911.003714]  [<ffffffffbd085615>] __warn+0xe5/0x100
>> [241911.060167]  [<ffffffffbd08564d>] warn_slowpath_null+0x1d/0x20
>> [241911.117422]  [<ffffffffc058ca90>]
>> btrfs_free_block_groups+0x370/0x410 [btrfs]
>> [241911.175975]  [<ffffffffc059e7ab>] close_ctree+0x15b/0x330 [btrfs]
>> [241911.235170]  [<ffffffffc056f089>] btrfs_put_super+0x19/0x20 [btrfs]
>> [241911.294638]  [<ffffffffbd1deaff>] generic_shutdown_super+0x6f/0x100
>> [241911.353005]  [<ffffffffbd1df026>] kill_anon_super+0x16/0x30
>> [241911.409832]  [<ffffffffc05720fa>] btrfs_kill_super+0x1a/0xb0 [btrfs]
>> [241911.466467]  [<ffffffffbd1df1f1>] deactivate_locked_super+0x51/0x90
>> [241911.522602]  [<ffffffffbd1dfb8e>] deactivate_super+0x4e/0x70
>> [241911.577979]  [<ffffffffbd1fba73>] cleanup_mnt+0x43/0x90
>> [241911.633188]  [<ffffffffbd1fbb12>] __cleanup_mnt+0x12/0x20
>> [241911.688146]  [<ffffffffbd0a1f61>] task_work_run+0x81/0xb0
>> [241911.742740]  [<ffffffffbd07ffcd>] exit_to_usermode_loop+0x66/0x95
>> [241911.797039]  [<ffffffffbd002a7d>] do_syscall_64+0x10d/0x150
>> [241911.850750]  [<ffffffffbd6d9ca1>] entry_SYSCALL64_slow_path+0x25/0x25
>> [241911.903564] ---[ end trace fae017546778f2b0 ]---
>> [241911.955332] ------------[ cut here ]------------
>> [241912.006262] WARNING: CPU: 1 PID: 26664 at
>> fs/btrfs/extent-tree.c:5702 btrfs_free_block_groups+0x40a/0x410 [btrfs]
>> [241912.059326] Modules linked in: netconsole mpt3sas ipt_REJECT
>> raid_class nf_reject_ipv4 scsi_transport_sas xt_multiport 8021q garp
>> iptable_filter ip_tables x_tables bonding coretemp loop usbhid ehci_pci
>> i2c_i801 ehci_hcd usbcore i2c_core shpchp usb_common ipmi_si
>> ipmi_msghandler button btrfs dm_mod raid1 raid456 async_raid6_recov
>> async_memcpy async_pq async_xor async_tx xor raid6_pq md_mod sg sd_mod
>> ixgbe i40e mdio ptp ahci libahci pps_core megaraid_sas
>> [241912.298666] CPU: 1 PID: 26664 Comm: umount Tainted: G        W
>> 4.7.0-rc6-29043-g8b8b08c #1
>> [241912.363401] Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 1.0c
>> 02/18/2015
>> [241912.429395]  0000000000000000 ffff8808d104bca8 ffffffffbd3d83cf
>> 0000000000000000
>> [241912.497080]  0000000000000000 ffff8808d104bcf8 ffffffffbd085615
>> ffff8808d104bd08
>> [241912.565113]  000016465a3410a8 00000047a0000000 0000000000000000
>> ffff8808469e2088
>> [241912.634105] Call Trace:
>> [241912.702992]  [<ffffffffbd3d83cf>] dump_stack+0x63/0x84
>> [241912.773473]  [<ffffffffbd085615>] __warn+0xe5/0x100
>> [241912.844339]  [<ffffffffbd08564d>] warn_slowpath_null+0x1d/0x20
>> [241912.916083]  [<ffffffffc058cb2a>]
>> btrfs_free_block_groups+0x40a/0x410 [btrfs]
>> [241912.989103]  [<ffffffffc059e7ab>] close_ctree+0x15b/0x330 [btrfs]
>> [241913.062672]  [<ffffffffc056f089>] btrfs_put_super+0x19/0x20 [btrfs]
>> [241913.136364]  [<ffffffffbd1deaff>] generic_shutdown_super+0x6f/0x100
>> [241913.208701]  [<ffffffffbd1df026>] kill_anon_super+0x16/0x30
>> [241913.279194]  [<ffffffffc05720fa>] btrfs_kill_super+0x1a/0xb0 [btrfs]
>> [241913.348065]  [<ffffffffbd1df1f1>] deactivate_locked_super+0x51/0x90
>> [241913.415082]  [<ffffffffbd1dfb8e>] deactivate_super+0x4e/0x70
>> [241913.479841]  [<ffffffffbd1fba73>] cleanup_mnt+0x43/0x90
>> [241913.543353]  [<ffffffffbd1fbb12>] __cleanup_mnt+0x12/0x20
>> [241913.605959]  [<ffffffffbd0a1f61>] task_work_run+0x81/0xb0
>> [241913.667542]  [<ffffffffbd07ffcd>] exit_to_usermode_loop+0x66/0x95
>> [241913.729612]  [<ffffffffbd002a7d>] do_syscall_64+0x10d/0x150
>> [241913.791203]  [<ffffffffbd6d9ca1>] entry_SYSCALL64_slow_path+0x25/0x25
>> [241913.852485] ---[ end trace fae017546778f2b1 ]---
>> [241913.913638] ------------[ cut here ]------------
>> [241913.974871] WARNING: CPU: 1 PID: 26664 at
>> fs/btrfs/extent-tree.c:10013 btrfs_free_block_groups+0x2ba/0x410 [btrfs]
>> [241914.039315] Modules linked in: netconsole mpt3sas ipt_REJECT
>> raid_class nf_reject_ipv4 scsi_transport_sas xt_multiport 8021q garp
>> iptable_filter ip_tables x_tables bonding coretemp loop usbhid ehci_pci
>> i2c_i801 ehci_hcd usbcore i2c_core shpchp usb_common ipmi_si
>> ipmi_msghandler button btrfs dm_mod raid1 raid456 async_raid6_recov
>> async_memcpy async_pq async_xor async_tx xor raid6_pq md_mod sg sd_mod
>> ixgbe i40e mdio ptp ahci libahci pps_core megaraid_sas
>> [241914.315918] CPU: 1 PID: 26664 Comm: umount Tainted: G        W
>> 4.7.0-rc6-29043-g8b8b08c #1
>> [241914.388096] Hardware name: Supermicro X10DRH/X10DRH-iT, BIOS 1.0c
>> 02/18/2015
>> [241914.460679]  0000000000000000 ffff8808d104bca8 ffffffffbd3d83cf
>> 0000000000000000
>> [241914.534126]  0000000000000000 ffff8808d104bcf8 ffffffffbd085615
>> ffff8808d104bce8
>> [241914.607523]  0000271dbd3dac8c ffff88085184aac8 0000000000000038
>> 0000000000000000
>> [241914.681318] Call Trace:
>> [241914.754437]  [<ffffffffbd3d83cf>] dump_stack+0x63/0x84
>> [241914.828796]  [<ffffffffbd085615>] __warn+0xe5/0x100
>> [241914.902953]  [<ffffffffbd08564d>] warn_slowpath_null+0x1d/0x20
>> [241914.977271]  [<ffffffffc058c9da>]
>> btrfs_free_block_groups+0x2ba/0x410 [btrfs]
>> [241915.052041]  [<ffffffffc059e7ab>] close_ctree+0x15b/0x330 [btrfs]
>> [241915.126282]  [<ffffffffc056f089>] btrfs_put_super+0x19/0x20 [btrfs]
>> [241915.200758]  [<ffffffffbd1deaff>] generic_shutdown_super+0x6f/0x100
>> [241915.273872]  [<ffffffffbd1df026>] kill_anon_super+0x16/0x30
>> [241915.345132]  [<ffffffffc05720fa>] btrfs_kill_super+0x1a/0xb0 [btrfs]
>> [241915.414703]  [<ffffffffbd1df1f1>] deactivate_locked_super+0x51/0x90
>> [241915.482488]  [<ffffffffbd1dfb8e>] deactivate_super+0x4e/0x70
>> [241915.547994]  [<ffffffffbd1fba73>] cleanup_mnt+0x43/0x90
>> [241915.611962]  [<ffffffffbd1fbb12>] __cleanup_mnt+0x12/0x20
>> [241915.674717]  [<ffffffffbd0a1f61>] task_work_run+0x81/0xb0
>> [241915.736398]  [<ffffffffbd07ffcd>] exit_to_usermode_loop+0x66/0x95
>> [241915.798592]  [<ffffffffbd002a7d>] do_syscall_64+0x10d/0x150
>> [241915.860295]  [<ffffffffbd6d9ca1>] entry_SYSCALL64_slow_path+0x25/0x25
>> [241915.921642] ---[ end trace fae017546778f2b2 ]---
>> [241915.982893] BTRFS: space_info 4 has 114577997824 free, is not full
>> [241916.045103] BTRFS: space_info total=307627032576, used=193048903680,
>> pinned=0, reserved=0, may_use=688537059328, readonly=131072
>>
>> Greets,
>> Stefan
>>
>> Am 10.08.2016 um 23:31 schrieb Stefan Priebe - Profihost AG:
>>> Hi Josef,
>>>
>>> same again with chris next branch:
>>>
>>> ERROR: error during balancing '/vmbackup/': No space left on device
>>> There may be more info in syslog - try dmesg | tail
>>> Dumping filters: flags 0x7, state 0x0, force is off
>>>   DATA (flags 0x2): balancing, usage=5
>>>   METADATA (flags 0x2): balancing, usage=5
>>>   SYSTEM (flags 0x2): balancing, usage=5
>>>
>>> dmesg:
>>> [203784.411189] BTRFS info (device dm-0): 114 enospc errors during balance
>>>
>>> uname -r 4.7.0-rc6-29043-g8b8b08c
>>>
>>> Greets,
>>> Stefan
>>>
>>> Am 08.08.2016 um 08:17 schrieb Stefan Priebe - Profihost AG:
>>>> Am 04.08.2016 um 13:40 schrieb Stefan Priebe - Profihost AG:
>>>>> Am 29.07.2016 um 23:03 schrieb Josef Bacik:
>>>>>> On 07/29/2016 03:14 PM, Omar Sandoval wrote:
>>>>>>> On Fri, Jul 29, 2016 at 12:11:53PM -0700, Omar Sandoval wrote:
>>>>>>>> On Fri, Jul 29, 2016 at 08:40:26PM +0200, Stefan Priebe - Profihost
>>>>>>>> AG wrote:
>>>>>>>>> Dear list,
>>>>>>>>>
>>>>>>>>> i'm seeing btrfs no space messages frequently on big filesystems (>
>>>>>>>>> 30TB).
>>>>>>>>>
>>>>>>>>> In all cases i'm getting a trace like this one a space_info warning.
>>>>>>>>> (since commit [1]). Could someone please be so kind and help me
>>>>>>>>> debugging / fixing this bug? I'm using space_cache=v2 on all those
>>>>>>>>> systems.
>>>>>>>>
>>>>>>>> Hm, so I think this indicates a bug in space accounting somewhere else
>>>>>>>> rather than the free space tree itself. I haven't debugged one of these
>>>>>>>> issues before, I'll see if I can reproduce it. Cc'ing Josef, too.
>>>>>>>
>>>>>>> I should've asked, what sort of filesystem activity triggers this?
>>>>>>>
>>>>>>
>>>>>> Chris just fixed this I think, try his next branch from his git tree
>>>>>>
>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git
>>>>>
>>>>> Thanks now running a 4.4 with those patches backported. If that still
>>>>> shows an error i will try that vanilla tree.
>>>>
>>>> OK this didn't work. I'll start / try using the linux-btrfs next branch
>>>> and look if this helps.
>>>>
>>>> Greets,
>>>> Stefan
>>>>
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Stefan
>>>>>
>>>>>> and see if it still happens.  Thanks,
>>>>>>
>>>>>> Josef