BTRFS over LVM remounts read-only

BTRFS over LVM remounts read-only

[MegaBrutal]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

Hello,

I'm using BTRFS over LVM and after some time of usage (days or hours),
it just remounts itself, and I don't see the reason why, while it is
said to be fault-tolerant.

It is possible that I have bad sectors on the disk, though I don't
find it likely. Even then, I don't think bad sectors should cause that
much of a problem that the FS needs to be remounted because of it. Is
there a way to make BTRFS to detect and mark bad sectors like any
other file system does?

Here are the /etc/fstab entries those concern my BTRFS file system:
/dev/mapper/vmhost--vg-vmhost--rootfs /               btrfs
defaults,subvol=@ 0       1
/dev/mapper/vmhost--vg-vmhost--rootfs /home           btrfs
defaults,subvol=@home 0       2
/dev/mapper/vmhost--vg-vmhost--rootfs /media/btrfs    btrfs   noauto
         0       3

I've attached a dmesg output that may help developers to see what's the problem.

Do you have any ideas what causes the problem?

Also, if the disaster happens and the FS gets remounted read-only, is
there a way to force remount it in read-write mode? I've tried "mount
-o remount,rw" and "mount -o force,remount,rw" too, but it doesn't
work. Also I can't mount the BTRFS root or any subvolumes elsewhere
until I restart the system.

Probably my description of the problem wasn't detailed enough, but
since I'm totally clueless about the problem (not counting the
possible bad sectors), I can't tell more right now. But you can ask
more details and I'll try to answer.


MegaBrutal

[-- Attachment #2: btrfs-failure.dmesg.bz2 --]
[-- Type: application/x-bzip2, Size: 18971 bytes --]

Re: BTRFS over LVM remounts read-only

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2380 bytes --]

On Sat, Jan 11, 2014 at 03:32:26PM +0100, MegaBrutal wrote:
> Hello,
> 
> I'm using BTRFS over LVM and after some time of usage (days or hours),
> it just remounts itself, and I don't see the reason why, while it is
> said to be fault-tolerant.
> 
> It is possible that I have bad sectors on the disk, though I don't
> find it likely. Even then, I don't think bad sectors should cause that
> much of a problem that the FS needs to be remounted because of it. Is
> there a way to make BTRFS to detect and mark bad sectors like any
> other file system does?
> 
> Here are the /etc/fstab entries those concern my BTRFS file system:
> /dev/mapper/vmhost--vg-vmhost--rootfs /               btrfs
> defaults,subvol=@ 0       1
> /dev/mapper/vmhost--vg-vmhost--rootfs /home           btrfs
> defaults,subvol=@home 0       2
> /dev/mapper/vmhost--vg-vmhost--rootfs /media/btrfs    btrfs   noauto
>          0       3
> 
> I've attached a dmesg output that may help developers to see what's the problem.
> 
> Do you have any ideas what causes the problem?

[60631.481913] attempt to access beyond end of device
[60631.481935] dm-1: rw=1073, want=42917896, limit=42917888
[60631.481941] btrfs_dev_stat_print_on_error: 34 callbacks suppressed
[60631.481949] btrfs: bdev /dev/mapper/vmhost--vg-vmhost--rootfs errs: wr 311347, rd 0, flush 0, corrupt 0, gen 0

   This is the problem.

   It looks like you've shrunk the LV without first shrinking the
filesystem. Depending on how much you shrunk it by, the odds are
fairly good that significant chunks of the FS are now toast.

   Hugo.

> Also, if the disaster happens and the FS gets remounted read-only, is
> there a way to force remount it in read-write mode? I've tried "mount
> -o remount,rw" and "mount -o force,remount,rw" too, but it doesn't
> work. Also I can't mount the BTRFS root or any subvolumes elsewhere
> until I restart the system.
> 
> Probably my description of the problem wasn't detailed enough, but
> since I'm totally clueless about the problem (not counting the
> possible bad sectors), I can't tell more right now. But you can ask
> more details and I'll try to answer.
> 
> 
> MegaBrutal



-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
     --- Gomez, darling, don't torture yourself.  That's my job. ---     

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: BTRFS over LVM remounts read-only

[MegaBrutal]

2014/1/11 Hugo Mills <hugo@carfax.org.uk>:
>
> [60631.481913] attempt to access beyond end of device
> [60631.481935] dm-1: rw=1073, want=42917896, limit=42917888
> [60631.481941] btrfs_dev_stat_print_on_error: 34 callbacks suppressed
> [60631.481949] btrfs: bdev /dev/mapper/vmhost--vg-vmhost--rootfs errs: wr 311347, rd 0, flush 0, corrupt 0, gen 0
>
>    This is the problem.
>
>    It looks like you've shrunk the LV without first shrinking the
> filesystem. Depending on how much you shrunk it by, the odds are
> fairly good that significant chunks of the FS are now toast.
>
>    Hugo.
>

Thank you very much! Strange, I indeed shrinked the LV, but only by a
few megabytes to make space for /boot (as turned out, GRUB2 can't boot
from LVM), and I think I did shrink the FS too... But maybe not. As it
started out as a test system, I intentionally wanted to test some
obscure situations here. Thankfully I don't have considerable data
loss and I don't think any valuable data was on the space that was cut
off. Strange that the system was running like this for a month with
this problem being undetected.

How can I shrink the FS to the correct size right now, ensuring that I
really shrink it to the exact LV size?

Re: BTRFS over LVM remounts read-only

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2399 bytes --]

On Sat, Jan 11, 2014 at 06:10:35PM +0100, MegaBrutal wrote:
> 2014/1/11 Hugo Mills <hugo@carfax.org.uk>:
> >
> > [60631.481913] attempt to access beyond end of device
> > [60631.481935] dm-1: rw=1073, want=42917896, limit=42917888
> > [60631.481941] btrfs_dev_stat_print_on_error: 34 callbacks suppressed
> > [60631.481949] btrfs: bdev /dev/mapper/vmhost--vg-vmhost--rootfs errs: wr 311347, rd 0, flush 0, corrupt 0, gen 0
> >
> >    This is the problem.
> >
> >    It looks like you've shrunk the LV without first shrinking the
> > filesystem. Depending on how much you shrunk it by, the odds are
> > fairly good that significant chunks of the FS are now toast.
> >
> >    Hugo.
> >
> 
> Thank you very much! Strange, I indeed shrinked the LV, but only by a
> few megabytes to make space for /boot (as turned out, GRUB2 can't boot
> from LVM), and I think I did shrink the FS too... But maybe not. As it
> started out as a test system, I intentionally wanted to test some
> obscure situations here. Thankfully I don't have considerable data
> loss and I don't think any valuable data was on the space that was cut
> off. Strange that the system was running like this for a month with
> this problem being undetected.
> 
> How can I shrink the FS to the correct size right now, ensuring that I
> really shrink it to the exact LV size?

   In the current state, I'm not sure that you can -- the FS metadata
still thinks that there's some data stored at an address that's off
the end of the LV, and it'll try to read it if you attempt to shrink
the FS.

   You may be able to enlarge the LV up to the size it was before, and
then shrink, but you'll probably get EIO from the filesystem because
the checksums for the (un)dead area don't match any more (assuming
that the data in question has been overwritten by something else in
the mean time).

   I suspect that we need some kind of "yes, I know the data's
buggered, but carry on regardless" option for a lot of commands, like
resize(shrink), device delete, device replace. This would help in the
case where a lo-fi recovery is acceptable.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
     --- "Your problem is that you have a negative personality." ---     
                             "No,  I don't!"                             

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: BTRFS over LVM remounts read-only

[Chris Murphy]

On Jan 11, 2014, at 10:10 AM, MegaBrutal <megabrutal@gmail.com> wrote:

> 2014/1/11 Hugo Mills <hugo@carfax.org.uk>:
>> 
>>   It looks like you've shrunk the LV without first shrinking the
>> filesystem. Depending on how much you shrunk it by, the odds are
>> fairly good that significant chunks of the FS are now toast.
>> 
>>   Hugo.
>> 
> 
> Thank you very much! Strange, I indeed shrinked the LV, but only by a
> few megabytes to make space for /boot (as turned out, GRUB2 can't boot
> from LVM),

GRUB 2.00 (the final release) can definitely boot from LVs, I do it all the time.

> and I think I did shrink the FS too... But maybe not. As it
> started out as a test system, I intentionally wanted to test some
> obscure situations here. Thankfully I don't have considerable data
> loss and I don't think any valuable data was on the space that was cut
> off. Strange that the system was running like this for a month with
> this problem being undetected.

Only once the file system needed to write outside of the partition would there likely be an error, but from your dmesg "wr 311347" means several hundred thousand failed write attempts. There are probably other messages similar to this going way back.


> 
> How can I shrink the FS to the correct size right now, ensuring that I
> really shrink it to the exact LV size?

Right now it's a broken file system, just recreate it.

Chris Murphy

Re: BTRFS over LVM remounts read-only

[Szalma László]

2014-01-11 18:10 keltezéssel, MegaBrutal írta:
> How can I shrink the FS to the correct size right now, ensuring that I 
> really shrink it to the exact LV size?

btrfs fi re 10G /dev/mapper/vg-lv
lvresize -L 10G vg/lv

Grub can boot a root on lv (with appropriate initrd), but the grub 
itself has to be on a standard bios bootable partition (as far as I know)


DBLaci

Re: BTRFS over LVM remounts read-only

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 756 bytes --]

On Sat, 11 Jan 2014 19:48:55 +0100
Szalma László <dblaci@dblaci.hu> wrote:

> 2014-01-11 18:10 keltezéssel, MegaBrutal írta:
> > How can I shrink the FS to the correct size right now, ensuring that I 
> > really shrink it to the exact LV size?
> 
> btrfs fi re 10G /dev/mapper/vg-lv
> lvresize -L 10G vg/lv

That's setting yourself up for a failure if these tools happen to disagree on
what "G" means (GB vs GiB). Maybe that's what happened here with the thread
starter's FS?

To be completely safe, I would recommend to resize the FS in two steps, leaving
some safety margin to the partition resize first:

  btrfs fi re 8G /dev/mapper/vg-lv
  lvresize -L 10G vg/lv
  btrfs fi re max /dev/mapper/vg-lv

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: BTRFS over LVM remounts read-only

[Szalma László]

Yeah, I used to do the way you wrote before, but in fact, G means 
gigabyte (1024^3), and works the same with both tools. On smaller units 
You have to watch for the rounding to extent size (4M by default).

Moreover, some could use:

btrfs fi re -2G /mount          (well, I just noticed, I accidentally 
wrote the device last time, but btrfs resize should be used on mount point.)
lvresize -L -2G /device

  :)

DBLaci

And of course, this is what should be done to prevent data loss. This 
method won't fix the original problem of course...

Thanks for your comment!

2014-01-11 20:16 keltezéssel, Roman Mamedov írta:
> On Sat, 11 Jan 2014 19:48:55 +0100
> Szalma László <dblaci@dblaci.hu> wrote:
>
>> 2014-01-11 18:10 keltezéssel, MegaBrutal írta:
>>> How can I shrink the FS to the correct size right now, ensuring that I
>>> really shrink it to the exact LV size?
>> btrfs fi re 10G /dev/mapper/vg-lv
>> lvresize -L 10G vg/lv
> That's setting yourself up for a failure if these tools happen to disagree on
> what "G" means (GB vs GiB). Maybe that's what happened here with the thread
> starter's FS?
>
> To be completely safe, I would recommend to resize the FS in two steps, leaving
> some safety margin to the partition resize first:
>
>    btrfs fi re 8G /dev/mapper/vg-lv
>    lvresize -L 10G vg/lv
>    btrfs fi re max /dev/mapper/vg-lv
>

Re: BTRFS over LVM remounts read-only

[Chris Murphy]

On Jan 11, 2014, at 12:16 PM, Roman Mamedov <rm@romanrm.net> wrote:

> On Sat, 11 Jan 2014 19:48:55 +0100
> Szalma László <dblaci@dblaci.hu> wrote:
> 
>> 2014-01-11 18:10 keltezéssel, MegaBrutal írta:
>>> How can I shrink the FS to the correct size right now, ensuring that I 
>>> really shrink it to the exact LV size?
>> 
>> btrfs fi re 10G /dev/mapper/vg-lv
>> lvresize -L 10G vg/lv
> 
> That's setting yourself up for a failure if these tools happen to disagree on
> what "G" means (GB vs GiB). Maybe that's what happened here with the thread
> starter's FS?

Yes. I've suggested a feature enhancement for Btrfs resize to report its new logical end (relative to fs start which would be 0). That can be used to deduce the resize needed for either partitions or LVs. If this is an fs shrink, it could just do the shrink without a warn. For an fs grow, if the partition isn't the right size, it fails and also states the new logical end value needed for the previously failed command to work.


> To be completely safe, I would recommend to resize the FS in two steps, leaving
> some safety margin to the partition resize first:
> 
>  btrfs fi re 8G /dev/mapper/vg-lv
>  lvresize -L 10G vg/lv
>  btrfs fi re max /dev/mapper/vg-lv

That's also a reasonable.


Chris Murphy

Re: BTRFS over LVM remounts read-only

[Chris Murphy]

On Jan 11, 2014, at 11:48 AM, Szalma László <dblaci@dblaci.hu> wrote:

> but the grub itself has to be on a standard bios bootable partition (as far as I know)

Negative. GRUB2 has support for pretty much everything out there: ZFS, Btrfs, LVM, LUKS, md raid 0, 1, 10, 5, 6. It will boot Btrfs multiple devices, single, raid1, raid0, and raid10, and zlib or lzo.

The proper modules are baked into core.img during grub-install which determines the requirements for finding grub's root (which is typically /boot/grub). Core.img is embedded in the MBR gap (or into grubx64.efi). And that core.img is all that's needed to locate /boot/grub/ where even more modules can be arbitrarily loaded, as well as grub.cfg. At this point the kernel and initramfs could be anywhere, located, and booted.


Chris Murphy

Re: BTRFS over LVM remounts read-only

[Duncan]

Chris Murphy posted on Sat, 11 Jan 2014 16:03:20 -0700 as excerpted:

> On Jan 11, 2014, at 11:48 AM, Szalma László <dblaci@dblaci.hu> wrote:
> 
>> but the grub itself has to be on a standard bios bootable partition (as
>> far as I know)
> 
> Negative. GRUB2 has support for pretty much everything out there: ZFS,
> Btrfs, LVM, LUKS, md raid 0, 1, 10, 5, 6. It will boot Btrfs multiple
> devices, single, raid1, raid0, and raid10, and zlib or lzo.
> 
> The proper modules are baked into core.img during grub-install which
> determines the requirements for finding grub's root (which is typically
> /boot/grub). Core.img is embedded in the MBR gap (or into grubx64.efi).
> And that core.img is all that's needed to locate /boot/grub/ where even
> more modules can be arbitrarily loaded, as well as grub.cfg. At this
> point the kernel and initramfs could be anywhere, located, and booted.

Correct, but omitting one critical bit, GPT on legacy BIOS, with a BIOS-
reserved partition grub can install its core image into.

Recommended BIOS-reserved partition size is 1-2 MiB and grub can often 
work with as little as 32 KiB, often the gap size if it has to install to 
the gap, so 1-2 MiB should be plenty.  However, in ordered to align 
partitions properly, as my smallest partition, I generally start the BIOS 
reserved partition at 1 MiB and make it 3 MiB in size, so all further 
partitions are 4 MiB aligned.

With a full 3 MiB to work with grub-install can do a pretty fat core.img, 
including a bunch of modules it wouldn't otherwise have room for.  
Actually, looking on /boot, the entire grub/i386-pc subdir with ALL 
modules is only 1886013 bytes according to du, so a full grub core with 
all modules (for i386-pc anyway) builtin should be less than 2 MiB, and 
it shouldn't need the modules installed on /boot at all, tho I'm not sure 
whether it takes advantage of all that room to install everything in the 
core image since the room is available to do so, or only includes what it 
detects it needs, regardless.

Completing my under-1-gig layout is a 124 MiB efi-reserved partition, 
unused ATM but there to prevent a forced re-gdisk in case I need it later 
(leaving further partitions 128 MiB aligned), a 256 MiB /boot, and a 640 
MiB /var/log.  That's the first 1 GiB and all further partitions are 
whole-GiBs in size, leaving all partitions beyond 1 GiB, full-GiB aligned.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman