BTRFS: Transaction aborted (error -28)

BTRFS: Transaction aborted (error -28)

[Mordechay Kaganer]

B.H.

Hello!

While saving a backup (using rsync) the FS went read only and i've got
BTRFS: Transaction aborted (error -28) error. While writing a new
backup i was also doing btrfs send of another subvolume on the same FS
from this server out to another server.

After reboot, the device mounts OK. Does it mean something went wrong
with the FS? Need to run a full balance?

The device actually has more than 3TB free space:

# btrfs fi usage .
Overall:
    Device size:                  43.66TiB
    Device allocated:             40.43TiB
    Device unallocated:            3.23TiB
    Device missing:                  0.00B
    Used:                         40.42TiB
    Free (estimated):              3.24TiB      (min: 1.62TiB)
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 48.00KiB)

Data,single: Size:39.92TiB, Used:39.92TiB
   /dev/md1       10.19TiB
   /dev/md2       10.21TiB
   /dev/md3       10.41TiB
   /dev/md4        9.11TiB

Metadata,RAID1: Size:261.00GiB, Used:259.69GiB
   /dev/md1      196.00GiB
   /dev/md2      178.00GiB
   /dev/md3       65.00GiB
   /dev/md4       83.00GiB

System,RAID1: Size:8.00MiB, Used:4.47MiB
   /dev/md1        8.00MiB
   /dev/md2        8.00MiB

Unallocated:
   /dev/md1      545.79GiB
   /dev/md2      544.80GiB
   /dev/md3      457.81GiB
   /dev/md4        1.72TiB

btrfs-tools version: btrfs-progs v4.3.1

The kernel is based on Ubuntu mainline PPA 4.3.3
(http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.3.3-wily/) with a
patch to fix RAID10 issue.

The kernel log:
[5099580.930974] ------------[ cut here ]------------
[5099580.930989] WARNING: CPU: 1 PID: 6643 at
/home/yemot/build/ubuntu-mainline/fs/btrfs/extent-tree.c:6362
__btrfs_free_extent.isra.66+0x132/0xcb0 [btrfs]()
[5099580.930990] BTRFS: Transaction aborted (error -28)
[5099580.930991] Modules linked in: ipmi_ssif x86_pkg_temp_thermal
intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul
aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper ast
snd_hda_codec_realtek cryptd snd_hda_codec_generic ttm sb_edac
serio_raw drm_kms_helper edac_core joydev input_leds drm snd_hda_intel
snd_hda_codec fb_sys_fops syscopyarea lpc_ich snd_hda_core sysfillrect
sysimgblt snd_hwdep snd_pcm snd_timer mei_me snd soundcore mei ipmi_si
8250_fintek ipmi_msghandler wmi nfsd shpchp auth_rpcgss lp mac_hid
nfs_acl parport nfs lockd grace sunrpc fscache btrfs ses enclosure
raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq raid1 igb i2c_algo_bit hid_generic isci dca
firewire_ohci usbhid raid0 libsas ptp firewire_core ahci psmouse hid
multipath pps_core libahci scsi_transport_sas crc_itu_t aacraid fjes
linear
[5099580.931025] CPU: 1 PID: 6643 Comm: kworker/u16:19 Not tainted
4.3.3-040303.20160107-raid10-patch-generic #patch
[5099580.931026] Hardware name: To Be Filled By O.E.M. To Be Filled By
O.E.M./EPC602D8A, BIOS P1.20 04/16/2014
[5099580.931036] Workqueue: btrfs-extent-refs btrfs_extent_refs_helper [btrfs]
[5099580.931037]  ffffffffc02aa7b0 ffff88000bfd3ab0 ffffffff813a4524
ffff88000bfd3af8
[5099580.931039]  ffff88000bfd3ae8 ffffffff81079226 00000000ffffffe4
00000deed72a4000
[5099580.931040]  ffff8805fca9a1b0 ffff880857982000 0000000000000000
ffff88000bfd3b48
[5099580.931041] Call Trace:
[5099580.931045]  [<ffffffff813a4524>] dump_stack+0x44/0x60
[5099580.931047]  [<ffffffff81079226>] warn_slowpath_common+0x86/0xc0
[5099580.931049]  [<ffffffff810792ac>] warn_slowpath_fmt+0x4c/0x50
[5099580.931054]  [<ffffffffc020fe32>]
__btrfs_free_extent.isra.66+0x132/0xcb0 [btrfs]
[5099580.931059]  [<ffffffffc0213e2c>]
__btrfs_run_delayed_refs+0x71c/0x10e0 [btrfs]
[5099580.931065]  [<ffffffffc021751d>] btrfs_run_delayed_refs+0x7d/0x2b0 [btrfs]
[5099580.931070]  [<ffffffffc0217787>] delayed_ref_async_start+0x37/0x90 [btrfs]
[5099580.931078]  [<ffffffffc025c100>] normal_work_helper+0xc0/0x270 [btrfs]
[5099580.931080]  [<ffffffff8108ee9d>] ? pwq_activate_delayed_work+0x3d/0x80
[5099580.931087]  [<ffffffffc025c4e2>]
btrfs_extent_refs_helper+0x12/0x20 [btrfs]
[5099580.931088]  [<ffffffff81091057>] process_one_work+0x157/0x3e0
[5099580.931090]  [<ffffffff81091a3e>] worker_thread+0x4e/0x470
[5099580.931091]  [<ffffffff810919f0>] ? rescuer_thread+0x300/0x300
[5099580.931093]  [<ffffffff81096d09>] kthread+0xc9/0xe0
[5099580.931095]  [<ffffffff81096c40>] ? kthread_park+0x60/0x60
[5099580.931098]  [<ffffffff817b4acf>] ret_from_fork+0x3f/0x70
[5099580.931100]  [<ffffffff81096c40>] ? kthread_park+0x60/0x60
[5099580.931101] ---[ end trace 771ebb14ef31d548 ]---
[5099580.931102] BTRFS: error (device md1) in
__btrfs_free_extent:6362: errno=-28 No space left
[5099580.931126] BTRFS info (device md1): forced readonly
[5099580.931127] BTRFS: error (device md1) in
btrfs_run_delayed_refs:2858: errno=-28 No space left
[5099580.935938] BTRFS: error (device md1) in
__btrfs_free_extent:6362: errno=-28 No space left
[5099580.935939] BTRFS: error (device md1) in
__btrfs_free_extent:6362: errno=-28 No space left
[5099580.935941] BTRFS: error (device md1) in
btrfs_run_delayed_refs:2858: errno=-28 No space left
[5099580.936037] BTRFS: error (device md1) in
btrfs_run_delayed_refs:2858: errno=-28 No space left
[5099581.122804] BTRFS: error (device md1) in
__btrfs_free_extent:6362: errno=-28 No space left
[5099581.122830] BTRFS: error (device md1) in
btrfs_run_delayed_refs:2858: errno=-28 No space left

Thanks!


-- 
משיח NOW!
Moshiach is coming very soon, prepare yourself!
יחי אדוננו מורינו ורבינו מלך המשיח לעולם ועד!

Re: BTRFS: Transaction aborted (error -28)

[Duncan]

Mordechay Kaganer posted on Fri, 29 Jul 2016 16:37:05 +0300 as excerpted:

> While saving a backup (using rsync) the FS went read only and i've got
> BTRFS: Transaction aborted (error -28) error. While writing a new backup
> i was also doing btrfs send of another subvolume on the same FS from
> this server out to another server.
> 
> After reboot, the device mounts OK. Does it mean something went wrong
> with the FS? Need to run a full balance?

Because btrfs is designed to be copy-on-write, with atomic transactions, 
an aborted transaction /should/ leave you with a consistent filesystem, 
simply lacking whatever changes would have occurred in that transaction.

Of course the fact that the transaction aborted /does/ indicate some sort 
of problem, which may in fact have left the filesystem in an unknown 
state.  However, I'd recommend a couple other checks, not balance, to be 
sure you're good.  First, a btrfs scrub of the filesystem, just to be 
sure the checksums all come out fine (or are fixable and fixed if not).  
Second, a btrfs check, which will be read-only unless you add --repair or 
similar option.

If those both come out fine it was probably simply stress from the rsync 
and the send at the same time.  Rsync in particular is known to be 
intensive enough to be a reasonable real-world stress test for btrfs, and 
has in fact flushed out a number of bugs over the years that people 
didn't see in otherwise normal usage.  Combining it with send of course 
amplifies the stress on the filesystem, such that if bugs are going to 
show up, and btrfs isn't yet fully stable and mature so that's expected, 
combining rsync with pretty much anything else intensive as well is 
likely to flush them out.

> btrfs-tools version: btrfs-progs v4.3.1
> 
> The kernel is based on Ubuntu mainline PPA 4.3.3
> (http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.3.3-wily/) with a
> patch to fix RAID10 issue.

FWIW, kernel 4.3 is no longer really recommended or supported well on 
this list.  The list recommendation is the last two kernels series in 
either the LTS or current channels.  The last two upstream LTS series are 
4.4 and 4.1.  The latest current kernel is of course 4.7, so 4.7 and 4.6 
would be supported there.

Of course distros can and do choose other kernel series to support, and 
you may choose to go with them and their support instead, but if so, you 
really should be getting support from them, because they are tracking 
what patches they've backported, while this list is generally focused on 
mainline and won't know what distros have backported, and because the 
focus is on moving forward, we only generally support the two current 
kernel series, plus the two current LTS series for those who don't want 
to be upgrading to a new kernel series every couple months.

During normal operation, userspace (btrfs-progs) is less important as 
most commands simply call the kernel code to do the real work.  However, 
once something goes wrong and you're trying to recover, it's mostly 
userspace code, and the newer versions are better positioned to fix more 
problems, so they're recommended then.  Plus, as the commands and their 
output are still changing from time to time, get too old and the commands 
and output are different enough it's hard to compare.  So a good rule of 
thumb is to at least match your kernel x.y version, tho newer is fine.  
That way, provided you're following the kernel recommendations, you're 
assured of a reasonably current btrfs userland as well.

So I'd recommend upgrading to the latest kernel 4.4 if you want to stay 
with the stable series, or 4.6 or 4.7 if you want current, and then (less 
important) upgrading the btrfs userspace as well.  It's possible the 
newer kernel will handle the combined rsync and send stresses better, and 
if not, you're on a better base to provide bug reports, etc.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS: Transaction aborted (error -28)

[Mordechay Kaganer]

B.H.

Thanks for the detailed answer!

On Fri, Jul 29, 2016 at 8:23 PM, Duncan <1i5t5.duncan@cox.net> wrote:
> ...
> Of course the fact that the transaction aborted /does/ indicate some sort
> of problem, which may in fact have left the filesystem in an unknown
> state.  However, I'd recommend a couple other checks, not balance, to be
> sure you're good.  First, a btrfs scrub of the filesystem, just to be
> sure the checksums all come out fine (or are fixable and fixed if not).
> Second, a btrfs check, which will be read-only unless you add --repair or
> similar option.

OK, scrub finished with 0 errors so I assume the FS is probably in a good shape

Regarding 'btrfs check' - my FS is huge and with a lot of metadata so
btrfs check seems to consume a lot of RAM. I've added all of my system
disk's free space as a swap but the check was killed due to OOM after
consuming about 100GB of memory.

> ...
> So I'd recommend upgrading to the latest kernel 4.4 if you want to stay
> with the stable series, or 4.6 or 4.7 if you want current, and then (less
> important) upgrading the btrfs userspace as well.  It's possible the
> newer kernel will handle the combined rsync and send stresses better, and
> if not, you're on a better base to provide bug reports, etc.

OK, upgraded to 4.4 (Ubuntu 16.04 stock kernel) and the fresh
btrfs-progs 4.7. I'm assuming the error was due to some kind of bug or
race condition and the FS is clean. Let's see how it behaves. Thanks!

-- 
משיח NOW!
Moshiach is coming very soon, prepare yourself!
יחי אדוננו מורינו ורבינו מלך המשיח לעולם ועד!

Re: BTRFS: Transaction aborted (error -28)

[Mordechay Kaganer]

B.H.

> On Fri, Jul 29, 2016 at 8:23 PM, Duncan <1i5t5.duncan@cox.net> wrote:
>> So I'd recommend upgrading to the latest kernel 4.4 if you want to stay
>> with the stable series, or 4.6 or 4.7 if you want current, and then (less
>> important) upgrading the btrfs userspace as well.  It's possible the
>> newer kernel will handle the combined rsync and send stresses better, and
>> if not, you're on a better base to provide bug reports, etc.
>
> OK, upgraded to 4.4 (Ubuntu 16.04 stock kernel) and the fresh
> btrfs-progs 4.7. I'm assuming the error was due to some kind of bug or
> race condition and the FS is clean. Let's see how it behaves. Thanks!

Hello, i'm still getting ENOSPC errors. The latest time the log looks like this:

Aug  4 21:55:06 yemot-4u kernel: [304090.288927] ------------[ cut
here ]------------
Aug  4 21:55:06 yemot-4u kernel: [304090.288961] WARNING: CPU: 1 PID:
4531 at /build/linux-dcxD3m/linux-4.4.0/fs/btrfs/extent-tree.c:2927
btrfs_run_delayed_refs+0x26b/0
x2a0 [btrfs]()
Aug  4 21:55:06 yemot-4u kernel: [304090.288965] BTRFS: error (device
md1) in btrfs_run_delayed_refs:2927: errno=-28 No space left
Aug  4 21:55:06 yemot-4u kernel: [304090.288968] BTRFS info (device
md1): forced readonly
Aug  4 21:55:06 yemot-4u kernel: [304090.288972] BTRFS: error (device
md1) in btrfs_run_delayed_refs:2927: errno=-28 No space left
Aug  4 21:55:06 yemot-4u kernel: [304090.289129] BTRFS: Transaction
aborted (error -28)
Aug  4 21:55:06 yemot-4u kernel: [304090.289131] Modules linked in:
binfmt_misc ipmi_ssif btrfs x86_pkg_temp_thermal intel_powerclamp
coretemp kvm_intel kvm irqbypass c
rct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul
glue_helper ablk_helper cryptd input_leds sb_edac serio_raw joydev
edac_core lpc_ich snd_hda_codec_real
tek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core
snd_hwdep snd_pcm mei_me snd_timer mei snd soundcore shpchp ipmi_si
8250_fintek ipmi_msghandler mac_h
id nfsd auth_rpcgss nfs_acl lockd grace sunrpc lp parport autofs4
raid0 multipath linear raid456 async_raid6_recov async_memcpy async_pq
async_xor async_tx xor raid6_pq
 libcrc32c raid10 raid1 ses enclosure igb dca ast ttm drm_kms_helper
syscopyarea hid_generic sysfillrect firewire_ohci sysimgblt
fb_sys_fops ahci usbhid firewire_core p
tp psmouse libahci isci hid drm crc_itu_t libsas pps_core i2c_algo_bit
aacraid scsi_transport_sas wmi fjes
Aug  4 21:55:06 yemot-4u kernel: [304090.289201] CPU: 1 PID: 4531
Comm: kworker/u16:28 Not tainted 4.4.0-31-generic #50-Ubuntu
Aug  4 21:55:06 yemot-4u kernel: [304090.289203] Hardware name: To Be
Filled By O.E.M. To Be Filled By O.E.M./EPC602D8A, BIOS P1.20
04/16/2014
Aug  4 21:55:06 yemot-4u kernel: [304090.289226] Workqueue:
btrfs-extent-refs btrfs_extent_refs_helper [btrfs]
Aug  4 21:55:06 yemot-4u kernel: [304090.289229]  0000000000000286
00000000b56c494e ffff880744f4bc98 ffffffff813f1143
Aug  4 21:55:06 yemot-4u kernel: [304090.289232]  ffff880744f4bce0
ffffffffc06d8468 ffff880744f4bcd0 ffffffff81081102
Aug  4 21:55:06 yemot-4u kernel: [304090.289234]  ffff8807fbff3000
ffff880859b13000 ffff880729f30b90 0000000000000000
Aug  4 21:55:06 yemot-4u kernel: [304090.289237] Call Trace:
Aug  4 21:55:06 yemot-4u kernel: [304090.289244]  [<ffffffff813f1143>]
dump_stack+0x63/0x90
Aug  4 21:55:06 yemot-4u kernel: [304090.289249]  [<ffffffff81081102>]
warn_slowpath_common+0x82/0xc0
Aug  4 21:55:06 yemot-4u kernel: [304090.289252]  [<ffffffff8108119c>]
warn_slowpath_fmt+0x5c/0x80
Aug  4 21:55:06 yemot-4u kernel: [304090.289268]  [<ffffffffc063fbeb>]
btrfs_run_delayed_refs+0x26b/0x2a0 [btrfs]
Aug  4 21:55:06 yemot-4u kernel: [304090.289284]  [<ffffffffc063fc57>]
delayed_ref_async_start+0x37/0x90 [btrfs]
Aug  4 21:55:06 yemot-4u kernel: [304090.289303]  [<ffffffffc0686eaa>]
btrfs_scrubparity_helper+0xca/0x2f0 [btrfs]
Aug  4 21:55:06 yemot-4u kernel: [304090.289307]  [<ffffffff814f1906>]
? tty_ldisc_deref+0x16/0x20
Aug  4 21:55:06 yemot-4u kernel: [304090.289326]  [<ffffffffc068710e>]
btrfs_extent_refs_helper+0xe/0x10 [btrfs]
Aug  4 21:55:06 yemot-4u kernel: [304090.289330]  [<ffffffff8109a2c5>]
process_one_work+0x165/0x480
Aug  4 21:55:06 yemot-4u kernel: [304090.289333]  [<ffffffff8109a62b>]
worker_thread+0x4b/0x4c0
Aug  4 21:55:06 yemot-4u kernel: [304090.289336]  [<ffffffff8109a5e0>]
? process_one_work+0x480/0x480
Aug  4 21:55:06 yemot-4u kernel: [304090.289339]  [<ffffffff810a0808>]
kthread+0xd8/0xf0
Aug  4 21:55:06 yemot-4u kernel: [304090.289341]  [<ffffffff810a0730>]
? kthread_create_on_node+0x1e0/0x1e0
Aug  4 21:55:06 yemot-4u kernel: [304090.289345]  [<ffffffff8182decf>]
ret_from_fork+0x3f/0x70
Aug  4 21:55:06 yemot-4u kernel: [304090.289348]  [<ffffffff810a0730>]
? kthread_create_on_node+0x1e0/0x1e0
Aug  4 21:55:06 yemot-4u kernel: [304090.289350] ---[ end trace
90c37e7522254f86 ]---
Aug  4 21:55:06 yemot-4u kernel: [304090.289353] BTRFS: error (device
md1) in btrfs_run_delayed_refs:2927: errno=-28 No space left
Aug  4 21:55:06 yemot-4u kernel: [304090.328312] BTRFS: error (device
md1) in __btrfs_free_extent:6552: errno=-28 No space left
Aug  4 21:55:06 yemot-4u kernel: [304090.328344] BTRFS: error (device
md1) in btrfs_run_delayed_refs:2927: errno=-28 No space left

root@yemot-4u:~# uname -a
Linux yemot-4u 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC
2016 x86_64 x86_64 x86_64 GNU/Linux

root@yemot-4u:~# btrfs --version
btrfs-progs v4.7


-- 
משיח NOW!
Moshiach is coming very soon, prepare yourself!
יחי אדוננו מורינו ורבינו מלך המשיח לעולם ועד!