Re: ENOSPC at 94% full -- and causing BUGs elsewhere?

[Hugo Mills <hugo-lkml@carfax.org.uk>]

[-- Attachment #1: Type: text/plain, Size: 2900 bytes --]

On Tue, Oct 13, 2009 at 06:31:45AM -0400, Chris Mason wrote:
> On Mon, Oct 12, 2009 at 03:09:35PM +0100, Hugo Mills wrote:
> > On Sun, Oct 04, 2009 at 08:06:30AM -0400, Chris Mason wrote:
> > > On Sat, Oct 03, 2009 at 05:55:32PM -0400, Josef Bacik wrote:
> > > > On Sat, Oct 03, 2009 at 01:21:09PM +0100, Hugo Mills wrote:
> > > > >    I've just had the following on my home server. I believe that it's
> > > > > btrfs that's responsible, as the machine wasn't doing much other than
> > > > > reading/writing on a btrfs filesystem. The process that was doing so
> > > > > is now stuck in D+ state, and can't be killed. The timing of the oops
> > > > > at the end is also suggestive of being involved in the same incident.
> > > > > This is the only btrfs filesystem on the machine.
> > > > 
> > > > Patches have gone to Linus to fix the enospc problems.  You can try running the
> > > > enospc branch of Chris's git tree and it should behave better for you.  Thanks,
> > > 
> > > The right tree for this is the master branch of btrfs-unstable for
> > > 2.6.31.
> > 
> >    Thanks, Josef and Chris. I've now found the time to check out and
> > build the btrfs-unstable tree, and it is indeed handling the ENOSPC
> > condition much more cleanly.
> > 
> >    However, it seems to have got into a position where I have lots of
> > free space reported by df (over 10% of the size of the volume -- 185
> > GiB free of 1474 GiB total), but still refuses to write anything to
> > the filesystem. Do you have any suggestions for what I could try?
> 
> You've probably got most of that 10GB free allocated as metadata.  You
> could try btrfs-vol -b.

   I moved some 13 GiB of data off the filesystem, and ran 
btrfs-vol -b. As I reported on IRC, I then got this in my syslog:

Oct 13 13:16:19 vlad kernel: btrfs: relocating block group 1401224691712 flags 1
Oct 13 13:17:02 vlad kernel: btrfs: found 123 extents
Oct 13 13:17:10 vlad kernel: btrfs: found 123 extents
Oct 13 13:17:11 vlad kernel: btrfs: found 28 extents
Oct 13 13:17:21 vlad kernel: btrfs: found 28 extents
Oct 13 13:17:25 vlad kernel: btrfs: found 28 extents
Oct 13 13:17:26 vlad kernel: btrfs: found 27 extents
Oct 13 13:17:36 vlad kernel: btrfs: found 27 extents
Oct 13 13:17:39 vlad kernel: btrfs: found 27 extents
Oct 13 13:17:48 vlad kernel: btrfs: found 27 extents
... repeat forever (or at least for 50 minutes or so).

   The btrfs-vol -b process didn't respond to ^C, so on advice of
yanzheng on IRC I rebooted the machine. I'm currently running a
btrfsck on the filesystem, and will try btrfs-vol -b again when that's
done.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
   --- Do not meddle in the affairs of system administrators,  for ---   
                  they are subtle,  and quick to anger.                  

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]