kernel 4.1: no space left on device

kernel 4.1: no space left on device

[Stefan Priebe]

Hi,

while having some big btrfs volumes (44TB + 37TB).

I see on a regular basis the "no space left on device" message. I'm only 
able to "fix" this. By running btrfs balance AND unmounting and 
remounting the btrfs volume.

Is there any way to debug / workaround this one?

Greets,
Stefan

Re: kernel 4.1: no space left on device

[Moby]

On 06/27/2015 10:32 AM, Stefan Priebe wrote:
> Hi,
>
> while having some big btrfs volumes (44TB + 37TB).
>
> I see on a regular basis the "no space left on device" message. I'm 
> only able to "fix" this. By running btrfs balance AND unmounting and 
> remounting the btrfs volume.
>
> Is there any way to debug / workaround this one?
>
> Greets,
> Stefan
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
Assuming you ruled out accumulation of snapshots?

-- 
--Moby


First they came for the Jews and I did not speak out because I was not a Jew.
Then they came for the Communists and I did not speak out because I was not a Communist.
Then they came for the trade unionists and I did not speak out because I was not a trade unionist.
Then they came for me and there was no one left to speak out for me.  --  Pastor Martin Niemöller

Re: kernel 4.1: no space left on device

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

On Sat, 27 Jun 2015 17:32:04 +0200
Stefan Priebe <s.priebe@profihost.ag> wrote:

> Hi,
> 
> while having some big btrfs volumes (44TB + 37TB).
> 
> I see on a regular basis the "no space left on device" message. I'm only 
> able to "fix" this. By running btrfs balance AND unmounting and 
> remounting the btrfs volume.
> 
> Is there any way to debug / workaround this one?

You should post your 'df', 'btrfs fi df' and 'btrfs fi show' for both devices.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: kernel 4.1: no space left on device

[Stefan Priebe]

Am 27.06.2015 um 17:51 schrieb Roman Mamedov:
> On Sat, 27 Jun 2015 17:32:04 +0200
> Stefan Priebe <s.priebe@profihost.ag> wrote:
>
>> Hi,
>>
>> while having some big btrfs volumes (44TB + 37TB).
>>
>> I see on a regular basis the "no space left on device" message. I'm only
>> able to "fix" this. By running btrfs balance AND unmounting and
>> remounting the btrfs volume.
>>
>> Is there any way to debug / workaround this one?
>
> You should post your 'df', 'btrfs fi df' and 'btrfs fi show' for both devices.

sorry - here for one server:
[ ~]# btrfs filesystem df /vmbackup/
Data, single: total=7.58TiB, used=7.58TiB
System, DUP: total=8.00MiB, used=1.19MiB
System, single: total=4.00MiB, used=0.00B
Metadata, DUP: total=69.00GiB, used=38.25GiB
Metadata, single: total=8.00MiB, used=0.00B
GlobalReserve, single: total=512.00MiB, used=3.30MiB

[ ~]# btrfs fi show /vmbackup/
Label: none  uuid: 0df5ed87-07da-4c15-9a2b-1d0ff6ec208d
         Total devices 1 FS bytes used 7.62TiB
         devid    1 size 43.66TiB used 7.72TiB path /dev/md50

btrfs-progs v4.0.1

Stefan

Re: kernel 4.1: no space left on device

[Stefan Priebe]

Am 27.06.2015 um 23:50 schrieb Ruslanas Gžibovskis:
> Hi,
>
> df -i ?
> Maybe inode?

df -i for btrfs?
# df -i |grep vmbackup
/dev/md50              0      0        0     - /vmbackup

>
> Have a nice $day_time.
>
>
> On Sat, 27 Jun 2015 19:11 Stefan Priebe <s.priebe@profihost.ag
> <mailto:s.priebe@profihost.ag>> wrote:
>
>     Am 27.06.2015 um 17:51 schrieb Roman Mamedov:
>      > On Sat, 27 Jun 2015 17:32:04 +0200
>      > Stefan Priebe <s.priebe@profihost.ag
>     <mailto:s.priebe@profihost.ag>> wrote:
>      >
>      >> Hi,
>      >>
>      >> while having some big btrfs volumes (44TB + 37TB).
>      >>
>      >> I see on a regular basis the "no space left on device" message.
>     I'm only
>      >> able to "fix" this. By running btrfs balance AND unmounting and
>      >> remounting the btrfs volume.
>      >>
>      >> Is there any way to debug / workaround this one?
>      >
>      > You should post your 'df', 'btrfs fi df' and 'btrfs fi show' for
>     both devices.
>
>     sorry - here for one server:
>     [ ~]# btrfs filesystem df /vmbackup/
>     Data, single: total=7.58TiB, used=7.58TiB
>     System, DUP: total=8.00MiB, used=1.19MiB
>     System, single: total=4.00MiB, used=0.00B
>     Metadata, DUP: total=69.00GiB, used=38.25GiB
>     Metadata, single: total=8.00MiB, used=0.00B
>     GlobalReserve, single: total=512.00MiB, used=3.30MiB
>
>     [ ~]# btrfs fi show /vmbackup/
>     Label: none  uuid: 0df5ed87-07da-4c15-9a2b-1d0ff6ec208d
>               Total devices 1 FS bytes used 7.62TiB
>               devid    1 size 43.66TiB used 7.72TiB path /dev/md50
>
>     btrfs-progs v4.0.1
>
>     Stefan
>     --
>     To unsubscribe from this list: send the line "unsubscribe
>     linux-btrfs" in
>     the body of a message to majordomo@vger.kernel.org
>     <mailto:majordomo@vger.kernel.org>
>     More majordomo info at http://vger.kernel.org/majordomo-info.html
>

Re: kernel 4.1: no space left on device

[Chris Murphy]

On Sat, Jun 27, 2015 at 9:32 AM, Stefan Priebe <s.priebe@profihost.ag> wrote:
> Hi,
>
> while having some big btrfs volumes (44TB + 37TB).
>
> I see on a regular basis the "no space left on device" message. I'm only
> able to "fix" this. By running btrfs balance AND unmounting and remounting
> the btrfs volume.
>
> Is there any way to debug / workaround this one?

What kernel version?
Have you tried to reproduce with a newer kernel and an older kernel? I
suggest trying 4.1 for newer, and at oldest 3.18.16 for regression
testing.
You see the message where/when?
And what's the workload happening at the time?
 If you pause/stop that work load, can you reproduce it by by cp'ing
some files? Does it happen with small files (less than 16KB) or medium
(100MB) or large (1+GB) files, or always?
About how many snapshots or reflink'd large files?


If it were me, after the above upgrade and regression testing, I'd do
the following and add all information thus far (version of kernel, and
btrfs-progs, and the workflow description) to a bugzilla.kernel.org
bug and *attach* the long output results as files to that bug:

1. btrfs check, without --repair, attach complete output to bug
2. mount with option enospc_debug
3. strace cp to reproduce the problem, attach output to bug
4. dmesg from the time of mount at #2, through the end of the strace
in 3, attach output to bug
5. remount with option clear_cache and see if you can reproduce the problem



-- 
Chris Murphy

Re: kernel 4.1: no space left on device

[Stefan Priebe]

Still nobody? Upgraded to 4.2-rc2 and i still see the out of space 
situation on two 30TB und 40TB arrays every week.

Stefan

Am 27.06.2015 um 17:32 schrieb Stefan Priebe:
> Hi,
>
> while having some big btrfs volumes (44TB + 37TB).
>
> I see on a regular basis the "no space left on device" message. I'm only
> able to "fix" this. By running btrfs balance AND unmounting and
> remounting the btrfs volume.
>
> Is there any way to debug / workaround this one?
>
> Greets,
> Stefan