Unexpected ENOSPC on a SSD-drive after day of uptime, kernel 2.6.32-rc5

Unexpected ENOSPC on a SSD-drive after day of uptime, kernel 2.6.32-rc5

[miyamoto moesasji]

I've just finished installing onto an OCZ Agilent v2 SSD with btrfs as
filesystem. However to my surprise I've hit an ENOSPC condition one
one of the partitions within less than a day of uptime, while the
filesystem on that partition only reported 50% to be in use, which is
far from the 75% limit people mention on the ML.

Note that this occurs using a vanilla 2.6.32-rc5 kernel on a 64-bit
gentoo system.

Error-message from logs:

2009-11-05T07:55:57.586574+00:00 PulsarX4 kernel: [  136.095961] no
space left, need 4096, 4440064 delalloc bytes, 10704142336 bytes_used,
0 bytes_reserved, 0 bytes_pinned, 0 bytes_readonly, 0 may use
10708582400 total
2009-11-05T07:55:57.645314+00:00 PulsarX4 kernel: [  136.154217] no
space left, need 4096, 4448256 delalloc bytes, 10704134144 bytes_used,
0 bytes_reserved, 0 bytes_pinned, 0 bytes_readonly, 0 may use
10708582400 total

Further details:

0) The partition that reports ENOSPC is mounted as:
/dev/sda3		/usr			btrfs		defaults,rw,nodev,noatime					

1) df -h reports : /dev/sda3              21G   11G  9.5G  53% /usr

2) btrfs-show :
Label: none  uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5
	Total devices 1 FS bytes used 10.60GB
	devid    1 size 20.00GB used 20.00GB path /dev/sda3

3) The other partitions using btrfs show a similar relatively large
difference between the space reported by df -h and the size being
taken up according to btrfs-show.

Although this potentially is a problem between screen and chair I
don't see what I am doing wrong. Note that the ENOSPC issue occurs on
a partition that still allows me to boot, so it is possible to run
tests if needed when this indeed turns out to be a bug.

Please let me know in case I need to provide further details from the logs.

Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel 2.6.32-rc5

[Josef Bacik]

On Thu, Nov 05, 2009 at 08:38:18PM +0000, miyamoto moesasji wrote:
> I've just finished installing onto an OCZ Agilent v2 SSD with btrfs as
> filesystem. However to my surprise I've hit an ENOSPC condition one
> one of the partitions within less than a day of uptime, while the
> filesystem on that partition only reported 50% to be in use, which is
> far from the 75% limit people mention on the ML.
> 
> Note that this occurs using a vanilla 2.6.32-rc5 kernel on a 64-bit
> gentoo system.
> 
> Error-message from logs:
> 
> 2009-11-05T07:55:57.586574+00:00 PulsarX4 kernel: [  136.095961] no
> space left, need 4096, 4440064 delalloc bytes, 10704142336 bytes_used,
> 0 bytes_reserved, 0 bytes_pinned, 0 bytes_readonly, 0 may use
> 10708582400 total
> 2009-11-05T07:55:57.645314+00:00 PulsarX4 kernel: [  136.154217] no
> space left, need 4096, 4448256 delalloc bytes, 10704134144 bytes_used,
> 0 bytes_reserved, 0 bytes_pinned, 0 bytes_readonly, 0 may use
> 10708582400 total
> 
> Further details:
> 
> 0) The partition that reports ENOSPC is mounted as:
> /dev/sda3		/usr			btrfs		defaults,rw,nodev,noatime					
> 
> 1) df -h reports : /dev/sda3              21G   11G  9.5G  53% /usr
> 
> 2) btrfs-show :
> Label: none  uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5
> 	Total devices 1 FS bytes used 10.60GB
> 	devid    1 size 20.00GB used 20.00GB path /dev/sda3
> 
> 3) The other partitions using btrfs show a similar relatively large
> difference between the space reported by df -h and the size being
> taken up according to btrfs-show.
> 
> Although this potentially is a problem between screen and chair I
> don't see what I am doing wrong. Note that the ENOSPC issue occurs on
> a partition that still allows me to boot, so it is possible to run
> tests if needed when this indeed turns out to be a bug.
> 
> Please let me know in case I need to provide further details from the logs.

Hmm looks like quite a bit of your fs got taken up for metadata.  Perhaps try
running btrfsctl -b /usr and see if that frees up some space for you.  Thanks,

Josef

Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel 2.6.32-rc5

[miyamoto moesasji]

> Hmm looks like quite a bit of your fs got taken up for metadata.  Perhaps try
> running btrfsctl -b /usr and see if that frees up some space for you. 
Thanks,

1) The btrfsctl does not have the -b option on my system, is that an option
that only is only enabled with the debug-utilities?

2) I would find it surprising if it is meta-data just looking at the numbers.
Below is the output of btrfs-show for all partitions with btrfs.

Label: none  uuid: a12ac0e9-cbea-4acf-bb26-181146940714
        Total devices 1 FS bytes used 89.53MB
        devid    1 size 8.00GB used 5.63GB path /dev/sda5

Label: none  uuid: 59997df9-c5a3-431f-b0a0-95b9e3b1afff
        Total devices 1 FS bytes used 150.14MB
        devid    1 size 4.00GB used 2.04GB path /dev/sda2

Label: none  uuid: 558766bb-5e0d-48dd-9a13-7117f3047710
        Total devices 1 FS bytes used 180.94MB
        devid    1 size 20.00GB used 5.04GB path /dev/sda6

Label: none  uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5
        Total devices 1 FS bytes used 10.60GB
        devid    1 size 20.00GB used 20.00GB path /dev/sda3

Label: none  uuid: 3cac73e5-e998-49bf-b8ee-ba953c92bc0b
        Total devices 1 FS bytes used 28.00KB
        devid    1 size 32.00GB used 2.04GB path /dev/sdb2

As an example the /var (dev/sda5) has only 89MB in use, while btrfs-show
demonstrates 5.63GB being used of the 8GB. It almost looks as if files that are
overwritten don't get removed from the space being in use. 

Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel?2.6.32-rc5

[Josef Bacik]

On Thu, Nov 05, 2009 at 08:55:12PM +0000, miyamoto moesasji wrote:
> 
> > Hmm looks like quite a bit of your fs got taken up for metadata.  Perhaps try
> > running btrfsctl -b /usr and see if that frees up some space for you. 
> Thanks,
> 
> 1) The btrfsctl does not have the -b option on my system, is that an option
> that only is only enabled with the debug-utilities?
> 

Sorry I always get that wrong, its btrfs-vol -b.

> 2) I would find it surprising if it is meta-data just looking at the numbers.
> Below is the output of btrfs-show for all partitions with btrfs.
> 
> Label: none  uuid: a12ac0e9-cbea-4acf-bb26-181146940714
>         Total devices 1 FS bytes used 89.53MB
>         devid    1 size 8.00GB used 5.63GB path /dev/sda5
> 
> Label: none  uuid: 59997df9-c5a3-431f-b0a0-95b9e3b1afff
>         Total devices 1 FS bytes used 150.14MB
>         devid    1 size 4.00GB used 2.04GB path /dev/sda2
> 
> Label: none  uuid: 558766bb-5e0d-48dd-9a13-7117f3047710
>         Total devices 1 FS bytes used 180.94MB
>         devid    1 size 20.00GB used 5.04GB path /dev/sda6
> 
> Label: none  uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5
>         Total devices 1 FS bytes used 10.60GB
>         devid    1 size 20.00GB used 20.00GB path /dev/sda3
> 
> Label: none  uuid: 3cac73e5-e998-49bf-b8ee-ba953c92bc0b
>         Total devices 1 FS bytes used 28.00KB
>         devid    1 size 32.00GB used 2.04GB path /dev/sdb2
> 
> As an example the /var (dev/sda5) has only 89MB in use, while btrfs-show
> demonstrates 5.63GB being used of the 8GB. It almost looks as if files that are
> overwritten don't get removed from the space being in use. 
> 

The used part is how much of the volume is allocated into chunks.  So when it
says 5.63 gb is being used, that means that 5.63 gbs of the volume has been
carved up into data/metadata chunks.  I hope to at some point make one of our
utilities tell you how much of that is for data and how much of that is
metadata.  Thanks,

Josef

Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel?2.6.32-rc5

[miyamoto moesasji]

1) Running btrfs-vol -b indeed does free up some space on the
completely full partition, but not much, just 1GB. However I can use
it again, so that is very helpful. Many thanks Josef!

For completeness: After running it on sda5 and sda3:
Label: none  uuid: a12ac0e9-cbea-4acf-bb26-181146940714
        Total devices 1 FS bytes used 90.31MB
        devid    1 size 8.00GB used 6.42GB path /dev/sda5

Label: none  uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5
        Total devices 1 FS bytes used 10.60GB
        devid    1 size 20.00GB used 18.99GB path /dev/sda3

2) However I would still like to point out that I find it very
surprising to see the amount of space taken up by data+meta-data,
which looks dangerous to me seeing how quickly I got into a disk full
situation while normal df indicated no problem whatsoever (if on root
I would basically have had a kernel panic). Is this really expected
behavior or is this a known problem already so no need to
trouble-shoot?

On 11/5/09, Josef Bacik <josef@redhat.com> wrote:
> On Thu, Nov 05, 2009 at 08:55:12PM +0000, miyamoto moesasji wrote:
>>
>> > Hmm looks like quite a bit of your fs got taken up for metadata.
>> > Perhaps try
>> > running btrfsctl -b /usr and see if that frees up some space for you.
>> Thanks,
>>
>> 1) The btrfsctl does not have the -b option on my system, is that an
>> option
>> that only is only enabled with the debug-utilities?
>>
>
> Sorry I always get that wrong, its btrfs-vol -b.
>
>> 2) I would find it surprising if it is meta-data just looking at the
>> numbers.
>> Below is the output of btrfs-show for all partitions with btrfs.
>>
>> Label: none  uuid: a12ac0e9-cbea-4acf-bb26-181146940714
>>         Total devices 1 FS bytes used 89.53MB
>>         devid    1 size 8.00GB used 5.63GB path /dev/sda5
>>
>> Label: none  uuid: 59997df9-c5a3-431f-b0a0-95b9e3b1afff
>>         Total devices 1 FS bytes used 150.14MB
>>         devid    1 size 4.00GB used 2.04GB path /dev/sda2
>>
>> Label: none  uuid: 558766bb-5e0d-48dd-9a13-7117f3047710
>>         Total devices 1 FS bytes used 180.94MB
>>         devid    1 size 20.00GB used 5.04GB path /dev/sda6
>>
>> Label: none  uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5
>>         Total devices 1 FS bytes used 10.60GB
>>         devid    1 size 20.00GB used 20.00GB path /dev/sda3
>>
>> Label: none  uuid: 3cac73e5-e998-49bf-b8ee-ba953c92bc0b
>>         Total devices 1 FS bytes used 28.00KB
>>         devid    1 size 32.00GB used 2.04GB path /dev/sdb2
>>
>> As an example the /var (dev/sda5) has only 89MB in use, while btrfs-show
>> demonstrates 5.63GB being used of the 8GB. It almost looks as if files
>> that are
>> overwritten don't get removed from the space being in use.
>>
>
> The used part is how much of the volume is allocated into chunks.  So when
> it
> says 5.63 gb is being used, that means that 5.63 gbs of the volume has been
> carved up into data/metadata chunks.  I hope to at some point make one of
> our
> utilities tell you how much of that is for data and how much of that is
> metadata.  Thanks,
>
> Josef
>

Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel?2.6.32-rc5

[Josef Bacik]

On Thu, Nov 05, 2009 at 10:37:10PM +0000, miyamoto moesasji wrote:
> 1) Running btrfs-vol -b indeed does free up some space on the
> completely full partition, but not much, just 1GB. However I can use
> it again, so that is very helpful. Many thanks Josef!
> 
> For completeness: After running it on sda5 and sda3:
> Label: none  uuid: a12ac0e9-cbea-4acf-bb26-181146940714
>         Total devices 1 FS bytes used 90.31MB
>         devid    1 size 8.00GB used 6.42GB path /dev/sda5
> 
> Label: none  uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5
>         Total devices 1 FS bytes used 10.60GB
>         devid    1 size 20.00GB used 18.99GB path /dev/sda3
> 
> 2) However I would still like to point out that I find it very
> surprising to see the amount of space taken up by data+meta-data,
> which looks dangerous to me seeing how quickly I got into a disk full
> situation while normal df indicated no problem whatsoever (if on root
> I would basically have had a kernel panic). Is this really expected
> behavior or is this a known problem already so no need to
> trouble-shoot?
> 

Have you been using btrfs since 2.6.32-rc3 or have you used it for a while now
and just recently gone to a 2.6.32-rc kernel?  The reason I ask is because we
did all sorts of things to try and make sure users didn't run out of space,
which included being overly agressive about making sure there was plenty of
metadata space.  The enospc patches that went into 2.6.32-rc3 (i think,
somewhere in there) has made this much better and you shouldn't be seeing this
bad of an imbalance towards metadata.  So, if this fs was created pre
2.6.32-rc3 then this is expected and unfortunate.  If this fs was post that time
then this is a problem and we need to figure out whats wrong.  Thanks,

Josef

Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel?2.6.32-rc5

[miyamoto moesasji]

I'm not 100% sure what is the correct answer.

This is a clean install, just installed this weekend so the system
itself has only been running with the 2.6.32-rc5 kernel, nothing older
than that and the drive itself was completely new/clean. However for
the install itself I've used SystemRescueCD which was using a 2.6.31.1
kernel. Hence the partitions have been formatted using the kernel
2.6.31.1 and that is also the kernel used during the install of the
system.

On 11/5/09, Josef Bacik <josef@redhat.com> wrote:
> On Thu, Nov 05, 2009 at 10:37:10PM +0000, miyamoto moesasji wrote:
>> 1) Running btrfs-vol -b indeed does free up some space on the
>> completely full partition, but not much, just 1GB. However I can use
>> it again, so that is very helpful. Many thanks Josef!
>>
>> For completeness: After running it on sda5 and sda3:
>> Label: none  uuid: a12ac0e9-cbea-4acf-bb26-181146940714
>>         Total devices 1 FS bytes used 90.31MB
>>         devid    1 size 8.00GB used 6.42GB path /dev/sda5
>>
>> Label: none  uuid: 0a89100d-096d-4c67-b3c7-745c9b7c3dc5
>>         Total devices 1 FS bytes used 10.60GB
>>         devid    1 size 20.00GB used 18.99GB path /dev/sda3
>>
>> 2) However I would still like to point out that I find it very
>> surprising to see the amount of space taken up by data+meta-data,
>> which looks dangerous to me seeing how quickly I got into a disk full
>> situation while normal df indicated no problem whatsoever (if on root
>> I would basically have had a kernel panic). Is this really expected
>> behavior or is this a known problem already so no need to
>> trouble-shoot?
>>
>
> Have you been using btrfs since 2.6.32-rc3 or have you used it for a while
> now
> and just recently gone to a 2.6.32-rc kernel?  The reason I ask is because
> we
> did all sorts of things to try and make sure users didn't run out of space,
> which included being overly agressive about making sure there was plenty of
> metadata space.  The enospc patches that went into 2.6.32-rc3 (i think,
> somewhere in there) has made this much better and you shouldn't be seeing
> this
> bad of an imbalance towards metadata.  So, if this fs was created pre
> 2.6.32-rc3 then this is expected and unfortunate.  If this fs was post that
> time
> then this is a problem and we need to figure out whats wrong.  Thanks,
>
> Josef
>

Re: Unexpected ENOSPC on a SSD-drive after day of uptime, kernel?2.6.32-rc5

[Josef Bacik]

On Thu, Nov 05, 2009 at 11:07:27PM +0000, miyamoto moesasji wrote:
> I'm not 100% sure what is the correct answer.
> 
> This is a clean install, just installed this weekend so the system
> itself has only been running with the 2.6.32-rc5 kernel, nothing older
> than that and the drive itself was completely new/clean. However for
> the install itself I've used SystemRescueCD which was using a 2.6.31.1
> kernel. Hence the partitions have been formatted using the kernel
> 2.6.31.1 and that is also the kernel used during the install of the
> system.
>

Yeah if most of the data was put on the disk under the old kernel then its
likely everything was skewed towards metadata and thats why you are having
problems.  Theres not much more I can say other than sorry :(.  If you can, try
copying the data from that volume to a new volume, format the old volume, and
copy it back under the new kernel and it should work out better.  Thanks,

Josef