Re: No space left on device (28)

[Josef Bacik <jbacik@fusionio.com>]

On Tue, Mar 26, 2013 at 01:05:36PM -0600, Stefan Priebe wrote:
> Hi Josef,
> 
> Am 26.03.2013 18:45, schrieb Josef Bacik:
> >> Am 26.03.2013 16:25, schrieb Josef Bacik:
> >>> On Tue, Mar 26, 2013 at 09:03:11AM -0600, Stefan Priebe - Profihost AG wrote:
> >>>> Hi,
> >>>> Am 26.03.2013 15:44, schrieb Josef Bacik:
> >>>>>>>> Am 26.03.2013 13:53, schrieb Josef Bacik:
> >>>>>>>> no - it's just mounted with mount -o noatime
> >>>>>>>>
> >>>>>>>> :~# cat /proc/mounts | grep btrfs
> >>>>>>>> /dev/mapper/raid54tb1 /mnt btrfs rw,noatime,space_cache 0 0
> >>>>>>>>
> >>>>>>>
> >>>>>>> Ok I think I see what's going on.  Can you try this patch and see if it fixes
> >>>>>>> it?  Thanks,
> >>>>>>
> >>>>>> It still does not fix the problem.
> >>>>>>
> >>>>>> The rsync output looks like this so it does not work for file a but then
> >>>>>> continues on c d e, ...
> >>>>>>
> >>>>>> sync -av --progress /backup/ /mnt/
> >>>>>> sending incremental file list
> >>>>>> .etc_openvpn/ipp.txt
> >>>>>>            229 100%    3.99kB/s    0:00:00 (xfer#2, to-check=1009/1196)
> >>>>>> .etc_openvpn/openvpn-status.log
> >>>>>>            360 100%    6.28kB/s    0:00:00 (xfer#3, to-check=1007/1196)
> >>>>>> rsync: rename "/mnt/.etc_openvpn/.ipp.txt.t9lucX" ->
> >>>>>> ".etc_openvpn/ipp.txt": No space left on device (28)
> >>>>>> .log/
> >>>>>> .log/UcliEvt.log
> >>>>>>         104188 100%  147.67kB/s    0:00:00 (xfer#4, to-check=1131/2700)
> >>>>>> .log/auth.log
> >>>>>>       15211522 100%    2.97MB/s    0:00:04 (xfer#5, to-check=1105/2700)
> >>>>>> .log/auth.log.1
> >>>>>>       19431424  61%    7.35MB/s    0:00:01
> >>>>>>
> >>>>>> the dmesg output looks like this:
> >>>>>> [  551.321576] returning enospc, space_info 3, size 0 reserved 0, flush
> >>>>>> 2, flush_state 7  dumping space info
> >>>>>> [  551.323694] space_info 4 has 6439526400 free, is full
> >>>>>> [  551.323696] space_info total=25748307968, used=19308666880, pinned=0,
> >>>>>> reserved=49152, may_use=6438453248, readonly=65536
> >>>>>>
> >>>>>
> >>>>> Ok so then this is probably it, let me know if it helps.  Thanks,
> >>>>
> >>>> OK it now has copied a lot of files (170) without an error all were very
> >>>> small.
> >>>>
> >>>
> >>> Welp progress is good.  Throw this into the mix and go again, it's just adding
> >>> some more debugging so I can make sure I'm going down the right rabbit hole.
> >>> Thanks,
> >>
> >> Output is now:
> >> [ 9587.445642] returning enospc, space_info 3, size 0 reserved 0, flush
> >> 2, flush_state 7  dumping space info
> >> [ 9587.527392] dumping block rsv 2, size 0 reserved 0
> >> [ 9587.567871] dumping block rsv 5, size 196608 reserved 196608
> >> [ 9587.607661] dumping block rsv 1, size 6438256640 reserved 6438256640
> >> [ 9587.646958] space_info 4 has 6439428096 free, is full
> >> [ 9587.646963] space_info total=25748307968, used=19308769280, pinned=0,
> >> reserved=45056, may_use=6438453248, readonly=65536
> >> [ 9587.649410] returning enospc, space_info 3, size 0 reserved 0, flush
> >> 2, flush_state 7  dumping space info
> >> [ 9587.727000] dumping block rsv 2, size 0 reserved 0
> >> [ 9587.765284] dumping block rsv 5, size 98304 reserved 98304
> >> [ 9587.802849] dumping block rsv 1, size 6438256640 reserved 6438256640
> >> [ 9587.839935] space_info 4 has 6439428096 free, is full
> >> [ 9587.839936] space_info total=25748307968, used=19308769280, pinned=0,
> >> reserved=45056, may_use=6438354944, readonly=65536
> >>
> >
> > Well then that looks like I was going down the wrong rabbit hole.  This should
> > fix you up, for real this time ;).  Thanks,
> 
> Yes - this works now. Which of the patches can i drop? Do i just need 
> the last one?
> Is it safe to add another 18TB raid via converting it to btrfs raid0?
> Will the fix be part of 3.9-rc5?
> 

So I'll put together all of the patches that actually need to go up for this and
post them, but basically its the mutex patch, the last patch I sent you and the
one that adjusts the reservations for rename and delete.  Thanks,

Josef