Re: BTRFS constantly reports "No space left on device" even with a huge unallocated space

[Jeff Mahoney <jeffm@suse.com>]

[-- Attachment #1.1: Type: text/plain, Size: 4493 bytes --]

On 8/31/16 4:49 PM, Ronan Arraes Jardim Chagas wrote:
> Hi guys!
> 
> And the problem happened again. This time, I was only using Mozilla
> Firefox. I could get the very first message after the error. I hope it
> brings more information:

Ok, so I think this is a race that can happen when one thread is
starting a transaction and another thread is committing a transaction
that involves creating a snapshot.

We reserve blocks at the top of start_transaction and that reservation
stays with the root.  In: btrfs_commit_transaction->
create_pending_snapshots-> create_pending_snapshot->
qgroup_account_snapshot-> commit_fs_roots, we clear that reservation
from the root via btrfs_qgroup_free_meta_all, potentially while
start_transaction is waiting to join a new transaction.  Or not.  It can
happen asynchronously, which is the point of having the reservation
prior to that.

So the thing is that this error can only occur if start_transaction
fails after this race occurs.  That, combined with your report that you
were seeing ENOSPC instead of EDQUOT, leads me to believe that this is
just a side effect of whatever is causing you to not hit ENOSPC.  I
expect that you'll see it again -- you just won't see the WARN_ON
anymore since quotas are disabled.  I suspect it's probably the
btrfs_block_rsv_add call immediately after the reservation, but there's
no way to tell without tracing.

-Jeff


> [28039.672199] ------------[ cut here ]------------
> [28039.672253] WARNING: CPU: 3 PID: 31800 at ../fs/btrfs/qgroup.c:2667
> btrfs_qgroup_free_meta+0x88/0x90 [btrfs]
> [28039.672255] Modules linked in: fuse nf_log_ipv6 xt_pkttype
> nf_log_ipv4 nf_log_common xt_LOG xt_limit af_packet iscsi_ibft
> iscsi_boot_sysfs msr ip6t_REJECT nf_reject_ipv6 xt_tcpudp
> nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_raw ipt_REJECT nf_reject_ipv4
> iptable_raw xt_CT nvidia_drm(PO) nvidia_modeset(PO) iptable_filter
> nvidia(PO) ip6table_mangle nf_conntrack_netbios_ns
> nf_conntrack_broadcast drm_kms_helper nf_conntrack_ipv4 drm
> nf_defrag_ipv4 fb_sys_fops snd_hda_codec_hdmi joydev
> snd_hda_codec_realtek ip_tables syscopyarea snd_hda_codec_generic
> xt_conntrack snd_hda_intel sysfillrect intel_rapl sb_edac edac_core
> snd_hda_codec hp_wmi x86_pkg_temp_thermal intel_powerclamp snd_hda_core
> snd_hwdep nf_conntrack sparse_keymap sysimgblt coretemp kvm_intel kvm
> rfkill irqbypass snd_pcm snd_timer crct10dif_pclmul
> [28039.672305]  e1000e crc32_pclmul ghash_clmulni_intel snd aesni_intel
> ip6table_filter aes_x86_64 lrw gf128mul glue_helper ablk_helper
> iTCO_wdt iTCO_vendor_support mei_wdt ioatdma pcspkr cryptd ip6_tables
> ptp lpc_ich fjes i2c_i801 dca mfd_core soundcore pps_core shpchp
> tpm_infineon tpm_tis tpm mei_me mei x_tables btrfs xor raid6_pq
> hid_generic usbhid crc32c_intel serio_raw xhci_pci ehci_pci sr_mod
> firewire_ohci xhci_hcd ehci_hcd cdrom firewire_core crc_itu_t isci
> usbcore usb_common libsas ata_generic mpt3sas raid_class
> scsi_transport_sas wmi button sg
> [28039.672373] CPU: 3 PID: 31800 Comm: gnome-terminal- Tainted:
> P        W  O    4.7.1-1-default #1
> [28039.672375] Hardware name: Hewlett-Packard HP Z820 Workstation/158B,
> BIOS J63 v03.65 12/19/2013
> [28039.672378]  0000000000000000 ffffffff81393104 0000000000000000
> 0000000000000000
> [28039.672382]  ffffffff8107ca1e ffff881008780800 0000000000014000
> ffff881008780800
> [28039.672386]  ffffffffffffffe4 ffff88100b297c00 ffff88053b7e3540
> ffffffffa02c9f58
> [28039.672390] Call Trace:
> [28039.672406]  [<ffffffff8102ed5e>] dump_trace+0x5e/0x320
> [28039.672413]  [<ffffffff8102f12c>] show_stack_log_lvl+0x10c/0x180
> [28039.672419]  [<ffffffff8102fe41>] show_stack+0x21/0x40
> [28039.672425]  [<ffffffff81393104>] dump_stack+0x5c/0x78
> [28039.672430]  [<ffffffff8107ca1e>] __warn+0xbe/0xe0
> [28039.672461]  [<ffffffffa02c9f58>] btrfs_qgroup_free_meta+0x88/0x90
> [btrfs]
> [28039.672492]  [<ffffffffa0261023>] start_transaction+0x3c3/0x4f0
> [btrfs]
> [28039.672521]  [<ffffffffa0271078>] btrfs_create+0x38/0x1d0 [btrfs]
> [28039.672528]  [<ffffffff8121e8fb>] path_openat+0x139b/0x14a0
> [28039.672535]  [<ffffffff8121fb2e>] do_filp_open+0x7e/0xe0
> [28039.672541]  [<ffffffff8120e7a4>] do_sys_open+0x124/0x1f0
> [28039.672547]  [<ffffffff816bb4f6>]
> entry_SYSCALL_64_fastpath+0x1e/0xa8
> [28039.676186] DWARF2 unwinder stuck at
> entry_SYSCALL_64_fastpath+0x1e/0xa8


-- 
Jeff Mahoney
SUSE Labs


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 881 bytes --]