strange No space left on device issues

strange No space left on device issues

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1795 bytes --]

Hey.

The following happens on Debian stretch systems:
# uname -a
Linux lcg-lrz-admin 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux

What I have are VMs, which run with root fs as ext4 and which I want to
migrate to btrfs.
So I've added further disk images and then something like this:
- mkfs.btrfs --nodiscard --label system /dev/sdc2 (i.e. the new image)
- mounted that at /mnt
- created a subvol "root" in it
- stopped all services on the node
- remount,ro /
- mount --bind / /media
- cp -a /media/ /mnt/subvol/
- and then I'd go on move everything in place, install bootloader etc.

That used to always work, and does when I try the same with ext4
instead of btrfs on the new images.

But with btrfs I get spurious No space error like:
cp: cannot create regular file
'/mnt/root/X/media/usr/share/doc/openjdk-8-jre-
headless/api/java/security/PrivilegedExceptionAction.html': No space
left on device
cp: cannot create regular file
'/mnt/root/X/media/usr/share/doc/openjdk-8-jre-
headless/api/java/security/Provider.Service.html': No space left on
device
cp: cannot create regular file
'/mnt/root/X/media/usr/share/doc/openjdk-8-jre-
headless/api/javax/script/AbstractScriptEngine.html': No space left on
device

or:
cp: preserving permissions for
‘/mnt/root/X/usr/include/c++/6/gnu/javax/crypto/keyring/BaseKeyring.h’:
No space left on device
cp: preserving permissions for ‘/mnt/root/X/usr/share/doc/cmake-
data/html/variable/CMAKE_CXX_STANDARD_REQUIRED.html’: No space left on
device


All these happen always (when I create a fresh btrfs on the volume and
start over) with different files... and btrfs filesystem df shows
plenty of space left like in terms of >15GB left.


Any ideas?

Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: strange No space left on device issues

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 174 bytes --]

Oh and I should add:
After such error, cp goes on copying (with other files)...

Same issue occurs when I do something like tar -cf - /media | tar -xf


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: strange No space left on device issues

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

On Thu, 2017-07-20 at 15:00 +0000, Martin Raiber wrote:
> there are patches on this list/upstream which could fix this ( e.g.
> "fix
> delalloc accounting leak caused by u32 overflow"/"fix early ENOSPC
> due
> to delalloc").

mhh... it's a bit problematic to test these on that nodes...


> Do you use compression?

nope...


> It would be interesting if lowering the dirty ratio is a viable
> work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl
> vm.dirty_bytes=1258291200).

doesn't seem to change anything.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: strange No space left on device issues

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 727 bytes --]

On Thu, 2017-07-20 at 15:00 +0000, Martin Raiber wrote:
> It would be interesting if lowering the dirty ratio is a viable
> work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl
> vm.dirty_bytes=1258291200).
> 
> Regards,
> Martin

I took away a trailing 0 for each of them... and then it goes through
without error....

sysctl vm.dirty_bytes=125829120
vm.dirty_bytes = 125829120
sysctl vm.dirty_background_bytes=31457280
vm.dirty_background_bytes = 31457280


But what does that mean now... could there be still any corruptions?
And do you need to permanently set the value (until this is fixed in
stable), or is this just necessary when I had this large copying
operation?


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 05:20:13PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 15:00 +0000, Martin Raiber wrote:
> > It would be interesting if lowering the dirty ratio is a viable
> > work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl
> > vm.dirty_bytes=1258291200).
> > 
> > Regards,
> > Martin
> 
> I took away a trailing 0 for each of them... and then it goes through
> without error....
> 
> sysctl vm.dirty_bytes=125829120
> vm.dirty_bytes = 125829120
> sysctl vm.dirty_background_bytes=31457280
> vm.dirty_background_bytes = 31457280
> 
> 
> But what does that mean now... could there be still any corruptions?
> And do you need to permanently set the value (until this is fixed in
> stable), or is this just necessary when I had this large copying
> operation?
> 
> 
> Cheers,
> Chris.

Could you try 4.12? If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?

Re: strange No space left on device issues

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 354 bytes --]

On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> Could you try 4.12?
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18)
x86_64 GNU/Linux
from Debian experimental, doesn't fix the issue...


>  If that doesn't work, could you please also try
> https://patchwork.kernel.org/patch/9829593/?
Against 4.9?


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 07:48:24PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> > Could you try 4.12?
> Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18)
> x86_64 GNU/Linux
> from Debian experimental, doesn't fix the issue...

Okay, didn't think it would :)

> >  If that doesn't work, could you please also try
> > https://patchwork.kernel.org/patch/9829593/?
> Against 4.9?

Against 4.12 would be best, thanks!

Re: strange No space left on device issues

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 346 bytes --]

On Thu, 2017-07-20 at 10:55 -0700, Omar Sandoval wrote:
> Against 4.12 would be best, thanks!
okay,.. but that will take a while to compile...


in the meantime... do you know whether it's more or less safe to use
the 4.9 kernel without any fix, when I change the parameters mentioned
before, during the massive copying?

Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 08:06:52PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 10:55 -0700, Omar Sandoval wrote:
> > Against 4.12 would be best, thanks!
> okay,.. but that will take a while to compile...
> 
> 
> in the meantime... do you know whether it's more or less safe to use
> the 4.9 kernel without any fix, when I change the parameters mentioned
> before, during the massive copying?

Yes, that's a safe enough workaround. It's a good idea to change the
parameters back after the copy.

> Cheers,
> Chris.

Re: strange No space left on device issues

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 382 bytes --]

On Thu, 2017-07-20 at 11:14 -0700, Omar Sandoval wrote:
> Yes, that's a safe enough workaround. It's a good idea to change the
> parameters back after the copy.
you mean even without having the fix, right?

So AFAIU, the bug doesn't really cause FS corruption, but just "false"
ENOSPC and these happen during having meta-data creating (e.g. during
operations like mine) only?

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 10:28:15PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 11:14 -0700, Omar Sandoval wrote:
> > Yes, that's a safe enough workaround. It's a good idea to change the
> > parameters back after the copy.
> you mean even without having the fix, right?

Yes, even without the fix you should be okay.

> So AFAIU, the bug doesn't really cause FS corruption, but just "false"
> ENOSPC and these happen during having meta-data creating (e.g. during
> operations like mine) only?

Right.

Re: strange No space left on device issues

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1044 bytes --]

On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> If that doesn't work, could you please also try
> https://patchwork.kernel.org/patch/9829593/?

Okay, tried the patch now, applied upon:
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 GNU/Linux
(that is the Debian source package, with all their further patches and
their kernel config).

with the parameters at their defaults:
# sysctl vm.dirty_bytes
vm.dirty_bytes = 0
# sysctl vm.dirty_background_bytes
vm.dirty_background_bytes = 0



Tried copying the whole image three times (before I haven had a single
copy of the whole image with at least one error, so that should be
"proof" enough that it fixes the isse) upon the btrfs fs,... no errors
this time...

Looks good :-)


I assume you'll take care to get that patch into stable kernels?
Is this patch alone enough to recommend the Debian maintainers to
include it into their 4.9 long term stable kernels?

And would you recommend this as an "urgent" fix?


Cheers,
Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 11:33:56PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 10:32 -0700, Omar Sandoval wrote:
> > If that doesn't work, could you please also try
> > https://patchwork.kernel.org/patch/9829593/?
> 
> Okay, tried the patch now, applied upon:
> Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 GNU/Linux
> (that is the Debian source package, with all their further patches and
> their kernel config).
> 
> with the parameters at their defaults:
> # sysctl vm.dirty_bytes
> vm.dirty_bytes = 0
> # sysctl vm.dirty_background_bytes
> vm.dirty_background_bytes = 0

Just to be sure, did you explicitly write 0 to these? These sysctls are
really confusing, see https://www.kernel.org/doc/Documentation/sysctl/vm.txt.
Basically, there are two ways to specify these, either as a ratio of
system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a static
number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If you
set one, the other appears as 0, and the kernel sets the ratios by
default. But if you explicitly set them to 0, the kernel is going to
flush stuff extremely aggressively.

> Tried copying the whole image three times (before I haven had a single
> copy of the whole image with at least one error, so that should be
> "proof" enough that it fixes the isse) upon the btrfs fs,... no errors
> this time...
> 
> Looks good :-)

Awesome, glad to hear it! I hadn't been able to reproduce the issue
outside of Facebook. Can I add your tested-by?

> I assume you'll take care to get that patch into stable kernels?
> Is this patch alone enough to recommend the Debian maintainers to
> include it into their 4.9 long term stable kernels?

I'll mark it for stable, assuming Debian tracks the upstream LTS
releases it should get in.

> And would you recommend this as an "urgent" fix?

This bug has been around since 4.8, so it's not _that_ urgent, but it
sucks.

Thanks!

> Cheers,
> Chris.

Re: strange No space left on device issues

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1519 bytes --]

On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote:
> Just to be sure, did you explicitly write 0 to these?
Nope... that seemed to have been the default value, i.e. I used
sysctl(8) in read (and not set) mode here.



> These sysctls are
> really confusing, see https://www.kernel.org/doc/Documentation/sysctl
> /vm.txt.
> Basically, there are two ways to specify these, either as a ratio of
> system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a
> static
> number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If
> you
> set one, the other appears as 0, and the kernel sets the ratios by
> default. But if you explicitly set them to 0, the kernel is going to
> flush stuff extremely aggressively.
I see,... not sure why both are 0 here... at least I didn't change it
myself - must be something from the distro?


> Awesome, glad to hear it! I hadn't been able to reproduce the issue
> outside of Facebook. Can I add your tested-by?
Sure, but better use my other mail address for it, if you don't mind:
Christoph Anton Mitterer <mail@christoph.anton.mitterer.name>


> > I assume you'll take care to get that patch into stable kernels?
> > Is this patch alone enough to recommend the Debian maintainers to
> > include it into their 4.9 long term stable kernels?
> 
> I'll mark it for stable, assuming Debian tracks the upstream LTS
> releases it should get in.
Okay :-)

Nevertheless I'll open a bug at their BTS, just to be safe.


Thanks :)

Chris.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5930 bytes --]

Re: strange No space left on device issues

[Omar Sandoval]

On Thu, Jul 20, 2017 at 11:53:16PM +0200, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote:
> > Just to be sure, did you explicitly write 0 to these?
> Nope... that seemed to have been the default value, i.e. I used
> sysctl(8) in read (and not set) mode here.

Okay, all good then.

> > These sysctls are
> > really confusing, see https://www.kernel.org/doc/Documentation/sysctl
> > /vm.txt.
> > Basically, there are two ways to specify these, either as a ratio of
> > system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a
> > static
> > number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If
> > you
> > set one, the other appears as 0, and the kernel sets the ratios by
> > default. But if you explicitly set them to 0, the kernel is going to
> > flush stuff extremely aggressively.
> I see,... not sure why both are 0 here... at least I didn't change it
> myself - must be something from the distro?

That's normal, the default is to have the ratio set instead:

$ sysctl vm.dirty_{,background_}{bytes,ratio}
vm.dirty_bytes = 0
vm.dirty_ratio = 20
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10

> > Awesome, glad to hear it! I hadn't been able to reproduce the issue
> > outside of Facebook. Can I add your tested-by?
> Sure, but better use my other mail address for it, if you don't mind:
> Christoph Anton Mitterer <mail@christoph.anton.mitterer.name>

No problem. I'll resend the patch with that shortly.

> > > I assume you'll take care to get that patch into stable kernels?
> > > Is this patch alone enough to recommend the Debian maintainers to
> > > include it into their 4.9 long term stable kernels?
> > 
> > I'll mark it for stable, assuming Debian tracks the upstream LTS
> > releases it should get in.
> Okay :-)
> 
> Nevertheless I'll open a bug at their BTS, just to be safe.
> 
> 
> Thanks :)
> 
> Chris.

Re: strange No space left on device issues

[Hans van Kranenburg]

On 07/20/2017 11:53 PM, Christoph Anton Mitterer wrote:
> On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote:
> [...]
> 
>>> I assume you'll take care to get that patch into stable kernels?
>>> Is this patch alone enough to recommend the Debian maintainers to
>>> include it into their 4.9 long term stable kernels?
>>
>> I'll mark it for stable, assuming Debian tracks the upstream LTS
>> releases it should get in.

+1 \o/

> Okay :-)
> 
> Nevertheless I'll open a bug at their BTS, just to be safe.

The Debian kernel teams tries to minimize the amount of patches on top
of the original kernel code. Every patch added comes with a maintenance
burden.

So, the answer will probably be: Please get it in 4.9 itself.

-- 
Hans van Kranenburg