Re: No space left on device (28)

[Josef Bacik <jbacik@fusionio.com>]

On Tue, Mar 26, 2013 at 07:49:16AM -0600, Stefan Priebe - Profihost AG wrote:
> Hi Josef,
> 
> 
> Am 26.03.2013 14:30, schrieb Josef Bacik:
> > On Tue, Mar 26, 2013 at 06:55:23AM -0600, Stefan Priebe - Profihost AG wrote:
> >> Hi Josef,
> >>
> >> Am 26.03.2013 13:53, schrieb Josef Bacik:
> >>> On Tue, Mar 26, 2013 at 01:45:42AM -0600, Stefan Priebe wrote:
> >>>> Hi,
> >>>>
> >>>> output here:
> >>>> [  590.546162] returning enospc, space_info 3, size 0 reserved 0, flush 
> >>>> 2, flush_state 7  dumping space info
> >>>> [  590.548280] space_info 4 has 6439292928 free, is full
> >>>> [  590.548283] space_info total=25748307968, used=19308916736, pinned=0, 
> >>>> reserved=32768, may_use=6438354944, readonly=65536
> >>>> [  590.550147] returning enospc, space_info 3, size 0 reserved 0, flush 
> >>>> 2, flush_state 7  dumping space info
> >>>> [  590.552264] space_info 4 has 6439284736 free, is full
> >>>> [  590.552267] space_info total=25748307968, used=19308916736, pinned=0, 
> >>>> reserved=40960, may_use=6438354944, readonly=65536
> >>>> [  590.554141] returning enospc, space_info 3, size 0 reserved 0, flush 
> >>>> 2, flush_state 7  dumping space info
> >>>> [  590.556258] space_info 4 has 6439284736 free, is full
> >>>> [  590.556261] space_info total=25748307968, used=19308916736, pinned=0, 
> >>>> reserved=40960, may_use=6438354944, readonly=65536
> >>>> [  591.145255] returning enospc, space_info 3, size 0 reserved 0, flush 
> >>>> 2, flush_state 7  dumping space info
> >>>> [  591.147373] space_info 4 has 6439235584 free, is full
> >>>> [  591.147375] space_info total=25748307968, used=19308916736, pinned=0, 
> >>>> reserved=90112, may_use=6438354944, readonly=65536
> >>>> [  595.560257] returning enospc, space_info 3, size 0 reserved 0, flush 
> >>>> 2, flush_state 7  dumping space info
> >>>> [  595.562390] space_info 4 has 6439120896 free, is full
> >>>> [  595.562393] space_info total=25748307968, used=19309047808, pinned=0, 
> >>>> reserved=73728, may_use=6438297600, readonly=65536
> >>>>
> >>>
> >>> Weird, we have all the flushing stuff set and yet there is still a whole lot of
> >>> outstanding reservations.  Do you have compression enabled?  Thanks,
> >>
> >> no - it's just mounted with mount -o noatime
> >>
> >> :~# cat /proc/mounts | grep btrfs
> >> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
> >>
> > 
> > Ok I think I see what's going on.  Can you try this patch and see if it fixes
> > it?  Thanks,
> 
> It still does not fix the problem.
> 
> The rsync output looks like this so it does not work for file a but then
> continues on c d e, ...
> 
> sync -av --progress /backup/ /mnt/
> sending incremental file list
> .etc_openvpn/ipp.txt
>          229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
> .etc_openvpn/openvpn-status.log
>          360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> ".etc_openvpn/ipp.txt": No space left on device (28)
> .log/
> .log/UcliEvt.log
>       104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
> .log/auth.log
>     15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
> .log/auth.log.1
>     19431424  61%    7.35MB/s    0:00:01
> 
> the dmesg output looks like this:
> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
> 2, flush_state 7  dumping space info
> [  551.323694] space_info 4 has 6439526400 free, is full
> [  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
> reserved=49152, may_use=6438453248, readonly=65536
>

Ok so then this is probably it, let me know if it helps.  Thanks,

Josef
 
diff --git a/fs/btrfs/ordered-data.c b/fs/btrfs/ordered-data.c
index dc08d77..005c45d 100644
--- a/fs/btrfs/ordered-data.c
+++ b/fs/btrfs/ordered-data.c
@@ -557,6 +557,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root, int delay_iput)
 	INIT_LIST_HEAD(&splice);
 	INIT_LIST_HEAD(&works);
 
+	mutex_lock(&root->fs_info->ordered_operations_mutex);
 	spin_lock(&root->fs_info->ordered_extent_lock);
 	list_splice_init(&root->fs_info->ordered_extents, &splice);
 	while (!list_empty(&splice)) {
@@ -600,6 +601,7 @@ void btrfs_wait_ordered_extents(struct btrfs_root *root, int delay_iput)
 
 		cond_resched();
 	}
+	mutex_unlock(&root->fs_info->ordered_operations_mutex);
 }
 
 /*
-- 
1.7.7.6