Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?

Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?

[Christian Theune]

[-- Attachment #1: Type: text/plain, Size: 1004 bytes --]

Hi,

not sure whether it’s possible, but we tried space_cache=v2 and obviously after working fine in staging it broke in production. Or rather: we upgraded from 4.4 to 4.9 and enabled the space_cache. Our production volume is around 50TiB usable (underlying HW Raid 6).

The machine crashes silently every 15 hours or so and takes _ages_ to reboot. It current is stuck trying to mount the local filesystems and I guess btrfs is doing something, but I don’t have shell access yet.

I’m wondering whether we can downgrade by booting back into 4.4 or will this break things even further? (We’ve had some unpleasant surprises with FS’ in the last months, so I thought I’d rather ask.)

Kind regards,
Christian

--
Christian Theune · ct@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?

[Austin S. Hemmelgarn]

On 2017-02-23 05:51, Christian Theune wrote:
> Hi,
>
> not sure whether it’s possible, but we tried space_cache=v2 and obviously after working fine in staging it broke in production. Or rather: we upgraded from 4.4 to 4.9 and enabled the space_cache. Our production volume is around 50TiB usable (underlying HW Raid 6).
>
> The machine crashes silently every 15 hours or so and takes _ages_ to reboot. It current is stuck trying to mount the local filesystems and I guess btrfs is doing something, but I don’t have shell access yet.
>
> I’m wondering whether we can downgrade by booting back into 4.4 or will this break things even further? (We’ve had some unpleasant surprises with FS’ in the last months, so I thought I’d rather ask.)
If it has a v2 free space cache and was mounted on a kernel newer than 
(I think) 4.8, it almost certainly won't mount on a 4.4 kernel.  The 
free-space-tree code got added in 4.5, and there was an incompatible 
filesystem flag added more recently to mark filesystems which are 
unaffected by a bug in the original implementation.  That flag gets set 
automatically on kernels that include the code to fix the bug's effects 
on the FS, which IIRC includes 4.9 kernels.

That said, if you can get it to mount at all on the 4.9 kernel, you can 
probably convert it back by mounting with -o clear_cache,space_cache=v1. 
  I'm saying probably in this case because it may be more than just the 
v2 space cache that's messed up, and if there's other corruption in the 
FS, you'll still see issues.  If it's just the free space cache though, 
adding those two options to the mount options (and removing 
space_cache=v2) should let it mount much more quickly, and you should 
see some above-average disk activity for a while, but afterwards you 
should be able to mount it on the 4.4 kernel again.

Re: Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?

[Christian Theune]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

Hi,

just for future reference if someone finds this thread: there is a bit of output I’m seeing with this crashing kernel (unclear whether related to btrfs or not):

  31 | 02/23/2017 | 09:51:22 | OS Stop/Shutdown #0x4f | Run-time critical stop | Asserted
  32 | Linux kernel panic: Out of memo
  33 | Linux kernel panic: ry and no k
  34 | Linux kernel panic: illable pro
  35 | Linux kernel panic: cesses...

Cheers,
Christian


--
Christian Theune · ct@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick


[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 496 bytes --]

Re: Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?

[Austin S. Hemmelgarn]

On 2017-02-23 08:19, Christian Theune wrote:
> Hi,
>
> just for future reference if someone finds this thread: there is a bit of output I’m seeing with this crashing kernel (unclear whether related to btrfs or not):
>
>   31 | 02/23/2017 | 09:51:22 | OS Stop/Shutdown #0x4f | Run-time critical stop | Asserted
>   32 | Linux kernel panic: Out of memo
>   33 | Linux kernel panic: ry and no k
>   34 | Linux kernel panic: illable pro
>   35 | Linux kernel panic: cesses...
>
OK, an OOM condition is (probably) not BTRFS related FWIW.  IIRC, there 
were a couple of issues with the OOM killer in 4.9, but I don't remember 
specifics.  To be able to determine anything useful though, you'd need a 
lot more context in the kernel logs than just that.

Re: Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?

[Qu Wenruo]

At 02/23/2017 06:51 PM, Christian Theune wrote:
> Hi,
>
> not sure whether it’s possible, but we tried space_cache=v2 and obviously after working fine in staging it broke in production. Or rather: we upgraded from 4.4 to 4.9 and enabled the space_cache. Our production volume is around 50TiB usable (underlying HW Raid 6).
>
> The machine crashes silently every 15 hours or so and takes _ages_ to reboot. It current is stuck trying to mount the local filesystems and I guess btrfs is doing something, but I don’t have shell access yet.
>
> I’m wondering whether we can downgrade by booting back into 4.4 or will this break things even further? (We’ve had some unpleasant surprises with FS’ in the last months, so I thought I’d rather ask.)

You could use "btrfs check --clear-space-cache" to completely cleanup 
space cache. Both v1(free space cache, file based) or v2(free space 
tree) are supported.

And specially for v2 space cache (space cache tree), "btrfs check 
--clear-space-cache" will also clear the ro_compat flag, so older kernel 
should mount the fs without problem.

In free-space-cache.c
------
         features = btrfs_super_compat_ro_flags(fs_info->super_copy);
         features &= ~(BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE_VALID |
                       BTRFS_FEATURE_COMPAT_RO_FREE_SPACE_TREE);
         btrfs_set_super_compat_ro_flags(fs_info->super_copy, features);
------

Thanks,
Qu

>
> Kind regards,
> Christian
>
> --
> Christian Theune · ct@flyingcircus.io · +49 345 219401 0
> Flying Circus Internet Operations GmbH · http://flyingcircus.io
> Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
> HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick
>

Re: Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?

[Austin S. Hemmelgarn]

On 2017-02-23 19:54, Qu Wenruo wrote:
>
>
> At 02/23/2017 06:51 PM, Christian Theune wrote:
>> Hi,
>>
>> not sure whether it’s possible, but we tried space_cache=v2 and
>> obviously after working fine in staging it broke in production. Or
>> rather: we upgraded from 4.4 to 4.9 and enabled the space_cache. Our
>> production volume is around 50TiB usable (underlying HW Raid 6).
>>
>> The machine crashes silently every 15 hours or so and takes _ages_ to
>> reboot. It current is stuck trying to mount the local filesystems and
>> I guess btrfs is doing something, but I don’t have shell access yet.
>>
>> I’m wondering whether we can downgrade by booting back into 4.4 or
>> will this break things even further? (We’ve had some unpleasant
>> surprises with FS’ in the last months, so I thought I’d rather ask.)
>
> You could use "btrfs check --clear-space-cache" to completely cleanup
> space cache. Both v1(free space cache, file based) or v2(free space
> tree) are supported.
>
> And specially for v2 space cache (space cache tree), "btrfs check
> --clear-space-cache" will also clear the ro_compat flag, so older kernel
> should mount the fs without problem.
That's really good to know.  I hadn't remembered that using mount 
options didn't clear the flag.

Re: Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?

[Omar Sandoval]

On Fri, Feb 24, 2017 at 07:23:42AM -0500, Austin S. Hemmelgarn wrote:
> On 2017-02-23 19:54, Qu Wenruo wrote:
> > 
> > 
> > At 02/23/2017 06:51 PM, Christian Theune wrote:
> > > Hi,
> > > 
> > > not sure whether it’s possible, but we tried space_cache=v2 and
> > > obviously after working fine in staging it broke in production. Or
> > > rather: we upgraded from 4.4 to 4.9 and enabled the space_cache. Our
> > > production volume is around 50TiB usable (underlying HW Raid 6).
> > > 
> > > The machine crashes silently every 15 hours or so and takes _ages_ to
> > > reboot. It current is stuck trying to mount the local filesystems and
> > > I guess btrfs is doing something, but I don’t have shell access yet.
> > > 
> > > I’m wondering whether we can downgrade by booting back into 4.4 or
> > > will this break things even further? (We’ve had some unpleasant
> > > surprises with FS’ in the last months, so I thought I’d rather ask.)
> > 
> > You could use "btrfs check --clear-space-cache" to completely cleanup
> > space cache. Both v1(free space cache, file based) or v2(free space
> > tree) are supported.
> > 
> > And specially for v2 space cache (space cache tree), "btrfs check
> > --clear-space-cache" will also clear the ro_compat flag, so older kernel
> > should mount the fs without problem.
> That's really good to know.  I hadn't remembered that using mount options
> didn't clear the flag.

Mounting with -oclear_cache,nospace_cache will clear the free space tree
and the ro_compat flag. Mounting with just -oclear_cache rebuilds the
free space tree (and keeps the ro_compat flag).

Re: Downgrading kernel 4.9 to 4.4 with space_cache=v2 enabled?

[Omar Sandoval]

On Thu, Feb 23, 2017 at 08:38:02AM -0500, Austin S. Hemmelgarn wrote:
> On 2017-02-23 08:19, Christian Theune wrote:
> > Hi,
> > 
> > just for future reference if someone finds this thread: there is a bit of output I’m seeing with this crashing kernel (unclear whether related to btrfs or not):
> > 
> >   31 | 02/23/2017 | 09:51:22 | OS Stop/Shutdown #0x4f | Run-time critical stop | Asserted
> >   32 | Linux kernel panic: Out of memo
> >   33 | Linux kernel panic: ry and no k
> >   34 | Linux kernel panic: illable pro
> >   35 | Linux kernel panic: cesses...
> > 
> OK, an OOM condition is (probably) not BTRFS related FWIW.  IIRC, there were
> a couple of issues with the OOM killer in 4.9, but I don't remember
> specifics.  To be able to determine anything useful though, you'd need a lot
> more context in the kernel logs than just that.

Yeah, if there is a problem with the free space tree, I'd be happy to
take a look, but it's impossible to tell from this.