ENOSPC on file deletion with 3.1.6

ENOSPC on file deletion with 3.1.6

[Arie Peterson]

Hi,


After upgrading my kernel from 2.6.38 (which has worked fine for months) to 
3.1.6, I got ENOSPC on recompiling gcc (even though df says there is 16G free 
of 50G; this is a raid1 setup, so in fact it's 8 of 25).

After this error, I tried to remove the compilation directory (with "rm -r"): 
this also gives ENOSPC. I am trying to work around this by first truncating 
files using "echo > $file", but this fails for some files, again with ENOSPC. 
Also, removal of files is very slow even if it succeeds.

Moreover, any write operation on the file system now fails with ENOSPC.

Reverting to my old kernel does not help: it now shows the same problem.


Is this a known issue? Is there a way to make this file system unstuck? (I have 
backups, but I'd like to preserve snapshot information if possible.) Should I 
try upgrading to an even newer kernel?


Kind regards,

Arie

Re: ENOSPC on file deletion with 3.1.6

[Sander]

Arie Peterson wrote (ao):
> After upgrading my kernel from 2.6.38 (which has worked fine for months) to 
> 3.1.6, I got ENOSPC on recompiling gcc (even though df says there is 16G free 
> of 50G; this is a raid1 setup, so in fact it's 8 of 25).
> 
> After this error, I tried to remove the compilation directory (with "rm -r"): 
> this also gives ENOSPC. I am trying to work around this by first truncating 
> files using "echo > $file", but this fails for some files, again with ENOSPC. 
> Also, removal of files is very slow even if it succeeds.
> 
> Moreover, any write operation on the file system now fails with ENOSPC.
> 
> Reverting to my old kernel does not help: it now shows the same problem.
> 
> Is this a known issue? Is there a way to make this file system unstuck? (I have 
> backups, but I'd like to preserve snapshot information if possible.) Should I 
> try upgrading to an even newer kernel?

Maybe your snapshots take up space. Can you show 'btrfs filesystem df /' ?

FWIW, I also had a disk full just a few days ago. Removed all snapshots
and some big files, but to no avail. Likely the background cleanup took
too much time. A reboot fixed this.

	Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: ENOSPC on file deletion with 3.1.6

[Arie Peterson]

On Tuesday 03 January 2012 15:06:43 Sander wrote:

> Maybe your snapshots take up space. Can you show 'btrfs filesystem df /' ?

Data, RAID1: total=22.72GB, used=14.73GB
Data: total=8.00MB, used=0.00
System, RAID1: total=8.00MB, used=12.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=2.25GB, used=1.88GB
Metadata: total=8.00MB, used=0.00

> FWIW, I also had a disk full just a few days ago. Removed all snapshots
> and some big files, but to no avail. Likely the background cleanup took
> too much time. A reboot fixed this.

OK, I'll keep this in mind. I'm a bit anxious to reboot, because I'm afraid 
booting will fail if the root file system cannot be written to.

> 	Sander

Thanks,

Arie

Re: ENOSPC on file deletion with 3.1.6

[Sander]

Arie Peterson wrote (ao):
> On Tuesday 03 January 2012 15:06:43 Sander wrote:
> > Maybe your snapshots take up space. Can you show 'btrfs filesystem df /' ?
> 
> Data, RAID1: total=22.72GB, used=14.73GB
> Data: total=8.00MB, used=0.00
> System, RAID1: total=8.00MB, used=12.00KB
> System: total=4.00MB, used=0.00
> Metadata, RAID1: total=2.25GB, used=1.88GB
> Metadata: total=8.00MB, used=0.00

Hm, not full.

> > FWIW, I also had a disk full just a few days ago. Removed all snapshots
> > and some big files, but to no avail. Likely the background cleanup took
> > too much time. A reboot fixed this.
> 
> OK, I'll keep this in mind. I'm a bit anxious to reboot, because I'm afraid 
> booting will fail if the root file system cannot be written to.

But you did already reboot as you said the old kernel exposed the same
behavior?

	Sander

-- 
Humilis IT Services and Solutions
http://www.humilis.net

Re: ENOSPC on file deletion with 3.1.6

[Arie Peterson]

On Tuesday 03 January 2012 15:22:58 Sander wrote:

> Hm, not full.
> 
> > OK, I'll keep this in mind. I'm a bit anxious to reboot, because I'm
> > afraid booting will fail if the root file system cannot be written to.
> 
> But you did already reboot as you said the old kernel exposed the same
> behavior?

You are right; the full history of events was:

(- compile new kernel (3.1.6))
- boot new kernel;
- recompile gcc: problem occurs;
- solve problem by removing compilation directory;
- boot old kernel;
- recompile gcc: problem occurs for this kernel as well.

After trying to remove the problematic files for some time, I took the chance 
and rebooted. After the reboot, the file system still gave ENOSPC on any write 
operation. However, it was able to boot anyway, and now the removal of the 
problematic files went much faster and without new ENOSPC. The compilation 
directory was completely removed, and immediately afterwards, the file system 
became writeable again.

Sander, thanks for your help.


I am still curious if this a known problem, and if upgrading to a newer kernel 
might prevent it from reoccurring. (I was planning to wait for 3.2 to be 
released and included in Gentoo's repository; maybe I shouldn't wait for 
this...)


Kind regards,

Arie

Re: ENOSPC on file deletion with 3.1.6

[cwillu]

On Tue, Jan 3, 2012 at 8:12 AM, Arie Peterson <ariep@xs4all.nl> wrote:
> On Tuesday 03 January 2012 15:06:43 Sander wrote:
>
>> Maybe your snapshots take up space. Can you show 'btrfs filesystem df /' ?
>
> Data, RAID1: total=22.72GB, used=14.73GB
> Data: total=8.00MB, used=0.00
> System, RAID1: total=8.00MB, used=12.00KB
> System: total=4.00MB, used=0.00
> Metadata, RAID1: total=2.25GB, used=1.88GB
> Metadata: total=8.00MB, used=0.00
>
>> FWIW, I also had a disk full just a few days ago. Removed all snapshots
>> and some big files, but to no avail. Likely the background cleanup took
>> too much time. A reboot fixed this.
>
> OK, I'll keep this in mind. I'm a bit anxious to reboot, because I'm afraid
> booting will fail if the root file system cannot be written to.

I'd probably run a "btrfs fi balance /", it should be able to recover
the space.  I'd typically be a little anxious if it was a large
filesystem as it's not interruptable except via power-button (in
principle it shouldn't matter, but...), but given that your filesystem
is quite small, it shouldn't take more than an hour or so.