"bad metadata" not fixed by btrfs repair

"bad metadata" not fixed by btrfs repair

[Marc Haber]

Hi,

I have a btrfs which btrfs check --repair doesn't fix:

# btrfs check --repair /dev/mapper/fanbtr
bad metadata [4425377054720, 4425377071104) crossing stripe boundary
bad metadata [4425380134912, 4425380151296) crossing stripe boundary
bad metadata [4427532795904, 4427532812288) crossing stripe boundary
bad metadata [4568321753088, 4568321769472) crossing stripe boundary
bad metadata [4568489656320, 4568489672704) crossing stripe boundary
bad metadata [4571474493440, 4571474509824) crossing stripe boundary
bad metadata [4571946811392, 4571946827776) crossing stripe boundary
bad metadata [4572782919680, 4572782936064) crossing stripe boundary
bad metadata [4573086351360, 4573086367744) crossing stripe boundary
bad metadata [4574221041664, 4574221058048) crossing stripe boundary
bad metadata [4574373412864, 4574373429248) crossing stripe boundary
bad metadata [4574958649344, 4574958665728) crossing stripe boundary
bad metadata [4575996018688, 4575996035072) crossing stripe boundary
bad metadata [4580376772608, 4580376788992) crossing stripe boundary
repaired damaged extent references
Fixed 0 roots.
checking free space cache
checking fs roots
checking csums
checking root refs
enabling repair mode
Checking filesystem on /dev/mapper/fanbtr
UUID: 90f8d728-6bae-4fca-8cda-b368ba2c008e
cache and super generation don't match, space cache will be invalidated
found 97171628230 bytes used err is 0
total csum bytes: 91734220
total tree bytes: 3021848576
total fs tree bytes: 2762784768
total extent tree bytes: 148570112
btree space waste bytes: 545440822
file data blocks allocated: 308328280064
 referenced 177314340864
# btrfs check --repair /dev/mapper/fanbtr
checking extents
bad metadata [4425377054720, 4425377071104) crossing stripe boundary
bad metadata [4425380134912, 4425380151296) crossing stripe boundary
bad metadata [4427532795904, 4427532812288) crossing stripe boundary
bad metadata [4568321753088, 4568321769472) crossing stripe boundary
bad metadata [4568489656320, 4568489672704) crossing stripe boundary
bad metadata [4571474493440, 4571474509824) crossing stripe boundary
bad metadata [4571946811392, 4571946827776) crossing stripe boundary
bad metadata [4572782919680, 4572782936064) crossing stripe boundary
bad metadata [4573086351360, 4573086367744) crossing stripe boundary
bad metadata [4574221041664, 4574221058048) crossing stripe boundary
bad metadata [4574373412864, 4574373429248) crossing stripe boundary
bad metadata [4574958649344, 4574958665728) crossing stripe boundary
bad metadata [4575996018688, 4575996035072) crossing stripe boundary
bad metadata [4580376772608, 4580376788992) crossing stripe boundary
repaired damaged extent references
Fixed 0 roots.
checking free space cache
checking fs roots
checking csums
checking root refs
enabling repair mode
Checking filesystem on /dev/mapper/fanbtr
UUID: 90f8d728-6bae-4fca-8cda-b368ba2c008e
cache and super generation don't match, space cache will be invalidated
found 97171628230 bytes used err is 0
total csum bytes: 91734220
total tree bytes: 3021848576
total fs tree bytes: 2762784768
total extent tree bytes: 148570112
btree space waste bytes: 545440822
file data blocks allocated: 308328280064
 referenced 177314340864

How do I fix this?

Does the kernel play a role in btrfs check --repair, or is this all a
userspace matter?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: "bad metadata" not fixed by btrfs repair

[Marc Haber]

On Mon, Mar 28, 2016 at 04:37:14PM +0200, Marc Haber wrote:
> I have a btrfs which btrfs check --repair doesn't fix:
> 
> # btrfs check --repair /dev/mapper/fanbtr
> bad metadata [4425377054720, 4425377071104) crossing stripe boundary
> bad metadata [4425380134912, 4425380151296) crossing stripe boundary
> bad metadata [4427532795904, 4427532812288) crossing stripe boundary
> bad metadata [4568321753088, 4568321769472) crossing stripe boundary
> bad metadata [4568489656320, 4568489672704) crossing stripe boundary
> bad metadata [4571474493440, 4571474509824) crossing stripe boundary
> bad metadata [4571946811392, 4571946827776) crossing stripe boundary
> bad metadata [4572782919680, 4572782936064) crossing stripe boundary
> bad metadata [4573086351360, 4573086367744) crossing stripe boundary
> bad metadata [4574221041664, 4574221058048) crossing stripe boundary
> bad metadata [4574373412864, 4574373429248) crossing stripe boundary
> bad metadata [4574958649344, 4574958665728) crossing stripe boundary
> bad metadata [4575996018688, 4575996035072) crossing stripe boundary
> bad metadata [4580376772608, 4580376788992) crossing stripe boundary
> repaired damaged extent references
> Fixed 0 roots.
> checking free space cache
> checking fs roots
> checking csums
> checking root refs
> enabling repair mode
> Checking filesystem on /dev/mapper/fanbtr
> UUID: 90f8d728-6bae-4fca-8cda-b368ba2c008e
> cache and super generation don't match, space cache will be invalidated
> found 97171628230 bytes used err is 0
> total csum bytes: 91734220
> total tree bytes: 3021848576
> total fs tree bytes: 2762784768
> total extent tree bytes: 148570112
> btree space waste bytes: 545440822
> file data blocks allocated: 308328280064
>  referenced 177314340864

Mounting this filesystem gives:
Mar 28 20:25:18 fan kernel: [   20.979673] BTRFS error (device dm-16): could not find root 8
Mar 28 20:25:18 fan kernel: [   20.979739] BTRFS error (device dm-16): could not find root 8
Mar 28 20:25:18 fan kernel: [   20.980900] BTRFS error (device dm-16): could not find root 8
Mar 28 20:25:18 fan kernel: [   20.980948] BTRFS error (device dm-16): could not find root 8
Mar 28 20:25:18 fan kernel: [   20.981428] BTRFS error (device dm-16): could not find root 8
Mar 28 20:25:18 fan kernel: [   20.981472] BTRFS error (device dm-16): could not find root 8

which is not detected by btrfs check.

What is going on here?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: "bad metadata" not fixed by btrfs repair

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2923 bytes --]

On Mon, Mar 28, 2016 at 08:42:37PM +0200, Marc Haber wrote:
> On Mon, Mar 28, 2016 at 04:37:14PM +0200, Marc Haber wrote:
> > I have a btrfs which btrfs check --repair doesn't fix:
> > 
> > # btrfs check --repair /dev/mapper/fanbtr
> > bad metadata [4425377054720, 4425377071104) crossing stripe boundary
> > bad metadata [4425380134912, 4425380151296) crossing stripe boundary
> > bad metadata [4427532795904, 4427532812288) crossing stripe boundary
> > bad metadata [4568321753088, 4568321769472) crossing stripe boundary
> > bad metadata [4568489656320, 4568489672704) crossing stripe boundary
> > bad metadata [4571474493440, 4571474509824) crossing stripe boundary
> > bad metadata [4571946811392, 4571946827776) crossing stripe boundary
> > bad metadata [4572782919680, 4572782936064) crossing stripe boundary
> > bad metadata [4573086351360, 4573086367744) crossing stripe boundary
> > bad metadata [4574221041664, 4574221058048) crossing stripe boundary
> > bad metadata [4574373412864, 4574373429248) crossing stripe boundary
> > bad metadata [4574958649344, 4574958665728) crossing stripe boundary
> > bad metadata [4575996018688, 4575996035072) crossing stripe boundary
> > bad metadata [4580376772608, 4580376788992) crossing stripe boundary
> > repaired damaged extent references
> > Fixed 0 roots.
> > checking free space cache
> > checking fs roots
> > checking csums
> > checking root refs
> > enabling repair mode
> > Checking filesystem on /dev/mapper/fanbtr
> > UUID: 90f8d728-6bae-4fca-8cda-b368ba2c008e
> > cache and super generation don't match, space cache will be invalidated
> > found 97171628230 bytes used err is 0
> > total csum bytes: 91734220
> > total tree bytes: 3021848576
> > total fs tree bytes: 2762784768
> > total extent tree bytes: 148570112
> > btree space waste bytes: 545440822
> > file data blocks allocated: 308328280064
> >  referenced 177314340864
> 
> Mounting this filesystem gives:
> Mar 28 20:25:18 fan kernel: [   20.979673] BTRFS error (device dm-16): could not find root 8
> Mar 28 20:25:18 fan kernel: [   20.979739] BTRFS error (device dm-16): could not find root 8
> Mar 28 20:25:18 fan kernel: [   20.980900] BTRFS error (device dm-16): could not find root 8
> Mar 28 20:25:18 fan kernel: [   20.980948] BTRFS error (device dm-16): could not find root 8
> Mar 28 20:25:18 fan kernel: [   20.981428] BTRFS error (device dm-16): could not find root 8
> Mar 28 20:25:18 fan kernel: [   20.981472] BTRFS error (device dm-16): could not find root 8
> 
> which is not detected by btrfs check.
> 
> What is going on here?

   "Could not find root 8" is harmless (and will be going away as a
message soon). It just means that systemd is probing the FS for
quotas, and you don't have quotas enabled.

   Hugo.

-- 
Hugo Mills             | Hey, Virtual Memory! Now I can have a *really big*
hugo@... carfax.org.uk | ramdisk!
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: "bad metadata" not fixed by btrfs repair

[Marc Haber]

On Mon, Mar 28, 2016 at 06:51:02PM +0000, Hugo Mills wrote:
>    "Could not find root 8" is harmless (and will be going away as a
> message soon). It just means that systemd is probing the FS for
> quotas, and you don't have quotas enabled.

*phew* That message was not what I wanted to read on this filesystem.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: "bad metadata" not fixed by btrfs repair

[Nazar Mokrynskyi]

[-- Attachment #1: Type: text/plain, Size: 3041 bytes --]

I have the same thing with kernel 4.5 and btrfs-progs 4.4.

Wrote about it 2 weeks ago and didn't get any answer: 
https://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg51609.html

However, despite those messages everything seems to work fine.

Sincerely, Nazar Mokrynskyi
github.com/nazar-pc
Skype: nazar-pc
Diaspora: nazarpc@diaspora.mokrynskyi.com
Tox: A9D95C9AA5F7A3ED75D83D0292E22ACE84BA40E912185939414475AF28FD2B2A5C8EF5261249

On 28.03.16 21:42, Marc Haber wrote:
> On Mon, Mar 28, 2016 at 04:37:14PM +0200, Marc Haber wrote:
>> I have a btrfs which btrfs check --repair doesn't fix:
>>
>> # btrfs check --repair /dev/mapper/fanbtr
>> bad metadata [4425377054720, 4425377071104) crossing stripe boundary
>> bad metadata [4425380134912, 4425380151296) crossing stripe boundary
>> bad metadata [4427532795904, 4427532812288) crossing stripe boundary
>> bad metadata [4568321753088, 4568321769472) crossing stripe boundary
>> bad metadata [4568489656320, 4568489672704) crossing stripe boundary
>> bad metadata [4571474493440, 4571474509824) crossing stripe boundary
>> bad metadata [4571946811392, 4571946827776) crossing stripe boundary
>> bad metadata [4572782919680, 4572782936064) crossing stripe boundary
>> bad metadata [4573086351360, 4573086367744) crossing stripe boundary
>> bad metadata [4574221041664, 4574221058048) crossing stripe boundary
>> bad metadata [4574373412864, 4574373429248) crossing stripe boundary
>> bad metadata [4574958649344, 4574958665728) crossing stripe boundary
>> bad metadata [4575996018688, 4575996035072) crossing stripe boundary
>> bad metadata [4580376772608, 4580376788992) crossing stripe boundary
>> repaired damaged extent references
>> Fixed 0 roots.
>> checking free space cache
>> checking fs roots
>> checking csums
>> checking root refs
>> enabling repair mode
>> Checking filesystem on /dev/mapper/fanbtr
>> UUID: 90f8d728-6bae-4fca-8cda-b368ba2c008e
>> cache and super generation don't match, space cache will be invalidated
>> found 97171628230 bytes used err is 0
>> total csum bytes: 91734220
>> total tree bytes: 3021848576
>> total fs tree bytes: 2762784768
>> total extent tree bytes: 148570112
>> btree space waste bytes: 545440822
>> file data blocks allocated: 308328280064
>>   referenced 177314340864
> Mounting this filesystem gives:
> Mar 28 20:25:18 fan kernel: [   20.979673] BTRFS error (device dm-16): could not find root 8
> Mar 28 20:25:18 fan kernel: [   20.979739] BTRFS error (device dm-16): could not find root 8
> Mar 28 20:25:18 fan kernel: [   20.980900] BTRFS error (device dm-16): could not find root 8
> Mar 28 20:25:18 fan kernel: [   20.980948] BTRFS error (device dm-16): could not find root 8
> Mar 28 20:25:18 fan kernel: [   20.981428] BTRFS error (device dm-16): could not find root 8
> Mar 28 20:25:18 fan kernel: [   20.981472] BTRFS error (device dm-16): could not find root 8
>
> which is not detected by btrfs check.
>
> What is going on here?
>
> Greetings
> Marc
>



[-- Attachment #2: ÐÑÑпÑогÑаÑÑÑний пÑÐ´Ð¿Ð¸Ñ S/MIME --]
[-- Type: application/pkcs7-signature, Size: 3825 bytes --]

Re: "bad metadata" not fixed by btrfs repair

[Chris Murphy]

On Mon, Mar 28, 2016 at 12:51 PM, Nazar Mokrynskyi <nazar@mokrynskyi.com> wrote:

>>> # btrfs check --repair /dev/mapper/fanbtr
>>> bad metadata [4425377054720, 4425377071104) crossing stripe boundary
>>> bad metadata [4425380134912, 4425380151296) crossing stripe boundary
>>> bad metadata [4427532795904, 4427532812288) crossing stripe boundary
>>> bad metadata [4568321753088, 4568321769472) crossing stripe boundary
>>> bad metadata [4568489656320, 4568489672704) crossing stripe boundary
>>> bad metadata [4571474493440, 4571474509824) crossing stripe boundary
>>> bad metadata [4571946811392, 4571946827776) crossing stripe boundary
>>> bad metadata [4572782919680, 4572782936064) crossing stripe boundary
>>> bad metadata [4573086351360, 4573086367744) crossing stripe boundary
>>> bad metadata [4574221041664, 4574221058048) crossing stripe boundary
>>> bad metadata [4574373412864, 4574373429248) crossing stripe boundary
>>> bad metadata [4574958649344, 4574958665728) crossing stripe boundary
>>> bad metadata [4575996018688, 4575996035072) crossing stripe boundary
>>> bad metadata [4580376772608, 4580376788992) crossing stripe boundary


http://git.kernel.org/cgit/linux/kernel/git/kdave/btrfs-progs.git/tree/cmds-check.c
line 7722 discusses this error message and it looks like there's no
repair function for it yet; uncertain what problems can result from
this.

line 4576 has a possible clue, but I don't know what "can't handle"
means, if real problems just silently are permitted or what.



-- 
Chris Murphy

Re: "bad metadata" not fixed by btrfs repair

[Marc Haber]

On Mon, Mar 28, 2016 at 02:46:54PM -0600, Chris Murphy wrote:
> http://git.kernel.org/cgit/linux/kernel/git/kdave/btrfs-progs.git/tree/cmds-check.c
> line 7722 discusses this error message and it looks like there's no
> repair function for it yet; uncertain what problems can result from
> this.

This basically means that I am ripe for a new mkfs.btrfs, right?

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: "bad metadata" not fixed by btrfs repair

[Marc Haber]

On Mon, Mar 28, 2016 at 09:51:43PM +0300, Nazar Mokrynskyi wrote:
> However, despite those messages everything seems to work fine.

In my case, trying to balance the filesystem results in a the balance
not finishing in hours, I/O performance going way down during the
balance, and a plethora of kernel traces showing up in the syslog:

Mar 22 22:37:08 fan kernel: [  429.488415] ------------[ cut here ]------------
Mar 22 22:37:08 fan kernel: [  429.488455] WARNING: CPU: 5 PID: 3899 at fs/btrfs/extent-tree.c:7897 btrfs_alloc_tree_block+0xeb/0x3d6 [btrfs]()
Mar 22 22:37:08 fan kernel: [  429.488456] BTRFS: block rsv returned -28
Mar 22 22:37:08 fan kernel: [  429.488458] Modules linked in: dummy ebtable_filter ebtables ip6table_filter ip6_tables cpufreq_userspace cpufreq_stats cpufreq_powersave cpufreq_conservative kvm_amd kvm irqbypass snd_hda_codec_realtek snd_hda_codec_generic amd64_edac_mod input_leds edac_mce_amd pcspkr snd_hda_codec_hdmi snd_cmipci k10temp edac_core snd_mpu401_uart snd_opl3_lib snd_rawmidi snd_hda_intel snd_seq_device snd_hda_codec i2c_piix4 snd_hda_core acpi_cpufreq asus_atk0110 snd_hwdep tpm_tis evdev sg snd_pcm_oss tpm snd_mixer_oss snd_pcm snd_timer snd soundcore shpchp processor iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp iptable_filter ip_tables x_tables bridge stp llc hwmon_vid autofs4 crc32c_generic btrfs xor raid6_pq ext4 crc16 mbcache jbd2 hmac sha256_ssse3 sha256_generic drbg ansi_cprng xts gf128mul algif_skcipher af_alg dm_crypt dm_mod hid_generic usbhid hid usb_storage sr_mod sd_mod cdrom ohci_pci r8169 mii xhci_pci amdkfd xhci_hcd ohci_hcd ehci_pci ehci_hcd radeon ahci i2c_algo_bit libahci ttm libata drm_kms_helper sym53c8xx scsi_transport_spi drm usbcore scsi_mod usb_common i2c_core button
Mar 22 22:37:08 fan kernel: [  429.488507] CPU: 5 PID: 3899 Comm: btrfs Tainted: G        W       4.4.5-zgws1 #2
Mar 22 22:37:08 fan kernel: [  429.488508] Hardware name: System manufacturer System Product Name/M5A88-V EVO, BIOS 1603    10/12/2012
Mar 22 22:37:08 fan kernel: [  429.488510]  000000000000005b ffffffff811dd418 ffff8805f57fb890 0000000000000009
Mar 22 22:37:08 fan kernel: [  429.488512]  ffffffff81051e21 ffffffffa04cb29f ffff8805bbf46f00 ffff8805f57fb8e8
Mar 22 22:37:08 fan kernel: [  429.488514]  ffff8800b2390000 ffff88053c63d020 ffffffff81051e79 ffffffffa053e65b
Mar 22 22:37:08 fan kernel: [  429.488516] Call Trace:
Mar 22 22:37:08 fan kernel: [  429.488523]  [<ffffffff811dd418>] ? dump_stack+0x5a/0x6f
Mar 22 22:37:08 fan kernel: [  429.488526]  [<ffffffff81051e21>] ? warn_slowpath_common+0x8e/0xa3
Mar 22 22:37:08 fan kernel: [  429.488537]  [<ffffffffa04cb29f>] ? btrfs_alloc_tree_block+0xeb/0x3d6 [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488539]  [<ffffffff81051e79>] ? warn_slowpath_fmt+0x43/0x4b
Mar 22 22:37:08 fan kernel: [  429.488551]  [<ffffffffa04f50e7>] ? set_extent_buffer_dirty+0x76/0x82 [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488562]  [<ffffffffa04cb29f>] ? btrfs_alloc_tree_block+0xeb/0x3d6 [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488572]  [<ffffffffa04ba93c>] ? generic_bin_search.constprop.37+0xf0/0x128 [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488581]  [<ffffffffa04b967a>] ? __btrfs_cow_block+0xdf/0x443 [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488591]  [<ffffffffa04b9b0a>] ? btrfs_cow_block+0xd9/0x143 [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488602]  [<ffffffffa0517654>] ? do_relocation+0x255/0x3ad [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488612]  [<ffffffffa04bf02b>] ? block_rsv_add_bytes+0x43/0x4a [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488622]  [<ffffffffa04ca230>] ? btrfs_block_rsv_refill+0x6e/0x7f [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488634]  [<ffffffffa05187c7>] ? relocate_tree_blocks+0x2ee/0x42c [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488645]  [<ffffffffa0519e4d>] ? relocate_block_group+0x27a/0x4ab [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488656]  [<ffffffffa051a1b0>] ? btrfs_relocate_block_group+0x132/0x25a [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488668]  [<ffffffffa04f9180>] ? btrfs_relocate_chunk.isra.35+0x3c/0xad [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488680]  [<ffffffffa04fa96c>] ? btrfs_balance+0xd23/0xd8f [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488691]  [<ffffffffa05037a6>] ? btrfs_ioctl_balance+0x233/0x2ae [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488703]  [<ffffffffa0506152>] ? btrfs_ioctl+0x1234/0x2699 [btrfs]
Mar 22 22:37:08 fan kernel: [  429.488706]  [<ffffffff811329bc>] ? mem_cgroup_charge_statistics.isra.30+0x22/0x4f
Mar 22 22:37:08 fan kernel: [  429.488708]  [<ffffffff811363ea>] ? mem_cgroup_commit_charge+0x8a/0xaa
Mar 22 22:37:08 fan kernel: [  429.488711]  [<ffffffff810f56bb>] ? __lru_cache_add+0x19/0x38
Mar 22 22:37:08 fan kernel: [  429.488713]  [<ffffffff8110a1fb>] ? set_pte_at+0x5/0x8
Mar 22 22:37:08 fan kernel: [  429.488715]  [<ffffffff8110daf2>] ? handle_mm_fault+0x488/0xea5
Mar 22 22:37:08 fan kernel: [  429.488717]  [<ffffffff81149d98>] ? do_vfs_ioctl+0x3ad/0x424
Mar 22 22:37:08 fan kernel: [  429.488719]  [<ffffffff81149d98>] ? do_vfs_ioctl+0x3ad/0x424
Mar 22 22:37:08 fan kernel: [  429.488721]  [<ffffffff81149e5c>] ? SyS_ioctl+0x4d/0x6f
Mar 22 22:37:08 fan kernel: [  429.488724]  [<ffffffff8140dcae>] ? entry_SYSCALL_64_fastpath+0x12/0x6d
Mar 22 22:37:08 fan kernel: [  429.488725] ---[ end trace 63ee8891bd9de879 ]---

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: "bad metadata" not fixed by btrfs repair

[Austin S. Hemmelgarn]

On 2016-03-28 10:37, Marc Haber wrote:
> Hi,
>
> I have a btrfs which btrfs check --repair doesn't fix:
>
> # btrfs check --repair /dev/mapper/fanbtr
> bad metadata [4425377054720, 4425377071104) crossing stripe boundary
> bad metadata [4425380134912, 4425380151296) crossing stripe boundary
> bad metadata [4427532795904, 4427532812288) crossing stripe boundary
> bad metadata [4568321753088, 4568321769472) crossing stripe boundary
> bad metadata [4568489656320, 4568489672704) crossing stripe boundary
> bad metadata [4571474493440, 4571474509824) crossing stripe boundary
> bad metadata [4571946811392, 4571946827776) crossing stripe boundary
> bad metadata [4572782919680, 4572782936064) crossing stripe boundary
> bad metadata [4573086351360, 4573086367744) crossing stripe boundary
> bad metadata [4574221041664, 4574221058048) crossing stripe boundary
> bad metadata [4574373412864, 4574373429248) crossing stripe boundary
> bad metadata [4574958649344, 4574958665728) crossing stripe boundary
> bad metadata [4575996018688, 4575996035072) crossing stripe boundary
> bad metadata [4580376772608, 4580376788992) crossing stripe boundary
> repaired damaged extent references
> Fixed 0 roots.
> checking free space cache
> checking fs roots
> checking csums
> checking root refs
> enabling repair mode
> Checking filesystem on /dev/mapper/fanbtr
> UUID: 90f8d728-6bae-4fca-8cda-b368ba2c008e
> cache and super generation don't match, space cache will be invalidated
> found 97171628230 bytes used err is 0
> total csum bytes: 91734220
> total tree bytes: 3021848576
> total fs tree bytes: 2762784768
> total extent tree bytes: 148570112
> btree space waste bytes: 545440822
> file data blocks allocated: 308328280064
>   referenced 177314340864
> # btrfs check --repair /dev/mapper/fanbtr
> checking extents
> bad metadata [4425377054720, 4425377071104) crossing stripe boundary
> bad metadata [4425380134912, 4425380151296) crossing stripe boundary
> bad metadata [4427532795904, 4427532812288) crossing stripe boundary
> bad metadata [4568321753088, 4568321769472) crossing stripe boundary
> bad metadata [4568489656320, 4568489672704) crossing stripe boundary
> bad metadata [4571474493440, 4571474509824) crossing stripe boundary
> bad metadata [4571946811392, 4571946827776) crossing stripe boundary
> bad metadata [4572782919680, 4572782936064) crossing stripe boundary
> bad metadata [4573086351360, 4573086367744) crossing stripe boundary
> bad metadata [4574221041664, 4574221058048) crossing stripe boundary
> bad metadata [4574373412864, 4574373429248) crossing stripe boundary
> bad metadata [4574958649344, 4574958665728) crossing stripe boundary
> bad metadata [4575996018688, 4575996035072) crossing stripe boundary
> bad metadata [4580376772608, 4580376788992) crossing stripe boundary
> repaired damaged extent references
> Fixed 0 roots.
> checking free space cache
> checking fs roots
> checking csums
> checking root refs
> enabling repair mode
> Checking filesystem on /dev/mapper/fanbtr
> UUID: 90f8d728-6bae-4fca-8cda-b368ba2c008e
> cache and super generation don't match, space cache will be invalidated
> found 97171628230 bytes used err is 0
> total csum bytes: 91734220
> total tree bytes: 3021848576
> total fs tree bytes: 2762784768
> total extent tree bytes: 148570112
> btree space waste bytes: 545440822
> file data blocks allocated: 308328280064
>   referenced 177314340864
>
> How do I fix this?
>
> Does the kernel play a role in btrfs check --repair, or is this all a
> userspace matter?
>
> Greetings
> Marc
>
I had been hoping somebody with a bit more knowledge of this would 
answer, but seeing as that hasn't happened...

Did you convert this filesystem from ext4 (or ext3)?  If so, then you 
appear to have done so with a faulty version of btrfs-convert (I don't 
remember when btrfs-convert started having issues, but I'm relatively 
certain it's not completely fixed yet).  If that is the case, that's 
probably the ultimate cause of the 'bad metadata (xxxxxxxx, xxxxxxxx) 
crossing stripe boundary' thing.

You hadn't mentioned what version of btrfs-progs you're using, and that 
is somewhat important for recovery.  I'm not sure if current versions of 
btrfs check can fix this issue, but I know for a fact that older 
versions (prior to at least 4.1) can not fix it.

As far as what the kernel is involved with, the easy way to check is if 
it's operating on a mounted filesystem or not.  If it only operates on 
mounted filesystems, it almost certainly goes through the kernel, if it 
only operates on unmounted filesystems, it's almost certainly done in 
userspace (except dev scan and technically fi show).  The typical advice 
is that you worry about kernel version for normal operation, and 
userspace version for initial setup (mkfs), and recovery (check, 
restore, recover, etc).

Now, slightly higher level discussion:
1. If you converted this filesystem using an old version of 
btrfs-convert, I would suggest recreating it from backup if possible. 
convert often results in sub-optimal data layout, and converted 
filesystems have (from what I've seen on the list at least) historically 
been more prone to bugs than ones created normally.
2. Regarding general support:  If you're using an enterprise 
distribution (RHEL, SLES, CentOS, OEL, or something similar), you are 
almost certainly going to get better support from your vendor than from 
the mailing list or IRC.  They do crazy things with kernel versioning, 
and it's pretty much impossible to know what patches have been 
back-ported to their kernels, which means we can't know what known bugs 
may be present.  The same somewhat applies for Ubuntu (they have a very 
bad habit recently of picking kernel versions without long-term support 
upstream, which means they back-port all kinds of things too), although 
to nowhere near the same degree.  In both cases, they're essentially 
promising to support their own versions, so they should provide support 
for BTRFS on them too (if they include it officially).
3. Regarding versioning: If 2 doesn't apply to you, then it is advised 
to stay on the most recent stable upstream kernel version (currently 
4.5), or at least something close to it or marked by kernel.org for long 
term support (4.4 and 4.1 are the most recent LTS versions IIRC).  It 
isn't imperative that you run the same userspace version as a kernel 
version, but keeping the tools up to date will usually help with 
recovery (and let you use newer features as well).

Re: "bad metadata" not fixed by btrfs repair

[Marc Haber]

On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
> Did you convert this filesystem from ext4 (or ext3)?

No.

> You hadn't mentioned what version of btrfs-progs you're using, and that is
> somewhat important for recovery.  I'm not sure if current versions of btrfs
> check can fix this issue, but I know for a fact that older versions (prior
> to at least 4.1) can not fix it.

4.1 for creation and btrfs check.

> As far as what the kernel is involved with, the easy way to check is if it's
> operating on a mounted filesystem or not.  If it only operates on mounted
> filesystems, it almost certainly goes through the kernel, if it only
> operates on unmounted filesystems, it's almost certainly done in userspace
> (except dev scan and technically fi show).

Then btrfs check is a userspace-only matter, as it wants the fs
unmounted, and it is irrelevant that I did btrfs check from a rescue
system with an older kernel, 3.16 if I recall correctly.

> 2. Regarding general support:  If you're using an enterprise distribution
> (RHEL, SLES, CentOS, OEL, or something similar), you are almost certainly
> going to get better support from your vendor than from the mailing list or
> IRC.

My "productive" desktops (fan is one of them) run Debian unstable with
a current vanilla kernel. At the moment, I can't use 4.5 because it
acts up with KVM.  When I need a rescue system, I use grml, which
unfortunately hasn't released since November 2014 and is still with
kernel 3.16

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: "bad metadata" not fixed by btrfs repair

[Marc Haber]

On Tue, Mar 29, 2016 at 08:43:51AM +0200, Marc Haber wrote:
> On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
> > As far as what the kernel is involved with, the easy way to check is if it's
> > operating on a mounted filesystem or not.  If it only operates on mounted
> > filesystems, it almost certainly goes through the kernel, if it only
> > operates on unmounted filesystems, it's almost certainly done in userspace
> > (except dev scan and technically fi show).
> 
> Then btrfs check is a userspace-only matter, as it wants the fs
> unmounted, and it is irrelevant that I did btrfs check from a rescue
> system with an older kernel, 3.16 if I recall correctly.

And it also means that I should not try btrfs balance from grml
because btrfs balance goes through the kernel code.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: "bad metadata" not fixed by btrfs repair

[Qu Wenruo]

First of all.

The "crossing stripe boundary" error message itself is *HARMLESS* for 
recent kernels.

It only means, that metadata extent won't be checked by scrub on recent 
kernels.
Because scrub by its codes, has a limitation that, it can only check 
tree blocks which are inside a 64K block.

Old kernel won't have anything wrong, until that tree block is being 
scrubbed.
When scrubbed, old kernel just BUG_ON().

Now recent kernel will handle such limitation by checking extent 
allocation and avoid crossing boundary, so new created fs with new 
kernel won't cause such error message at all.

But for old created fs, the problem can't be avoided, but at least, new 
kernels will not BUG_ON() when you scrub these extents, they just get 
ignored (not that good, but at least no BUG_ON).

And new fsck will check such case, gives such warning.

Overall, you're OK if you are using recent kernels.

Marc Haber wrote on 2016/03/29 08:43 +0200:
> On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
>> Did you convert this filesystem from ext4 (or ext3)?
>
> No.
>
>> You hadn't mentioned what version of btrfs-progs you're using, and that is
>> somewhat important for recovery.  I'm not sure if current versions of btrfs
>> check can fix this issue, but I know for a fact that older versions (prior
>> to at least 4.1) can not fix it.
>
> 4.1 for creation and btrfs check.

I assume that you have run older kernel on it, like v4.1 or v4.2.

In those old kernels, it lacks the check to avoid such extent allocation 
check.

>
>> As far as what the kernel is involved with, the easy way to check is if it's
>> operating on a mounted filesystem or not.  If it only operates on mounted
>> filesystems, it almost certainly goes through the kernel, if it only
>> operates on unmounted filesystems, it's almost certainly done in userspace
>> (except dev scan and technically fi show).
>
> Then btrfs check is a userspace-only matter, as it wants the fs
> unmounted, and it is irrelevant that I did btrfs check from a rescue
> system with an older kernel, 3.16 if I recall correctly.

Not recommended to use older kernel to RW mount or use older fsck to do 
repair.
As it's possible that older kernel/btrfsck may allocate extent that 
cross the 64K boundary.

>
>> 2. Regarding general support:  If you're using an enterprise distribution
>> (RHEL, SLES, CentOS, OEL, or something similar), you are almost certainly
>> going to get better support from your vendor than from the mailing list or
>> IRC.
>
> My "productive" desktops (fan is one of them) run Debian unstable with
> a current vanilla kernel. At the moment, I can't use 4.5 because it
> acts up with KVM.  When I need a rescue system, I use grml, which
> unfortunately hasn't released since November 2014 and is still with
> kernel 3.16

To fix your problem(make these error message just disappear, even they 
are harmless on recent kernels), the most easy one, is to balance your 
metadata.

As I explained, the bug only lies in metadata, and balance will allocate 
new tree blocks, then copy old data into new locations.

In the allocation process of recent kernel, it will avoid such cross 
boundary, and to fix your problem.

But if you are using old kernels, don't scrub your metadata.

Thanks,
Qu
>
> Greetings
> Marc
>

Re: "bad metadata" not fixed by btrfs repair

[Marc Haber]

On Wed, Mar 30, 2016 at 03:00:19PM +0800, Qu Wenruo wrote:
> Marc Haber wrote on 2016/03/29 08:43 +0200:
> >On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
> >>Did you convert this filesystem from ext4 (or ext3)?
> >
> >No.
> >
> >>You hadn't mentioned what version of btrfs-progs you're using, and that is
> >>somewhat important for recovery.  I'm not sure if current versions of btrfs
> >>check can fix this issue, but I know for a fact that older versions (prior
> >>to at least 4.1) can not fix it.
> >
> >4.1 for creation and btrfs check.
> 
> I assume that you have run older kernel on it, like v4.1 or v4.2.

No, the productive system was always on a reasonably recent kernel. I
guess that this instance of btrfs has never been mounted on anything
older than 4.4.4. The rescue system I used to btrfs check (4.4-1 from
Debian unstable, I updated btrfs-tools on the rescue system before
going btrfs check) had kernel 3.16, but I have never actually mounted
the btrfs there.

> >Then btrfs check is a userspace-only matter, as it wants the fs
> >unmounted, and it is irrelevant that I did btrfs check from a rescue
> >system with an older kernel, 3.16 if I recall correctly.
> 
> Not recommended to use older kernel to RW mount or use older fsck to do
> repair.

Oldest kernel that has mounted this btrfs is 4.4.4, fsck that touched
the fs is 4.4. I'm trying to get hold of btrfs-tools 4.5.

> >My "productive" desktops (fan is one of them) run Debian unstable with
> >a current vanilla kernel. At the moment, I can't use 4.5 because it
> >acts up with KVM.  When I need a rescue system, I use grml, which
> >unfortunately hasn't released since November 2014 and is still with
> >kernel 3.16
> 
> To fix your problem(make these error message just disappear, even they are
> harmless on recent kernels), the most easy one, is to balance your metadata.

This does not work on kernel 4.4.6 with tools 4.4. Truckloads of
kernel traces, "WARNING: CPU: 5 PID: 31021 at
fs/btrfs/extent-tree.c:7897 btrfs_alloc_tree_block+0xeb/0x3d6
[btrfs]()", "BTRFS: block rsv returned -28", full trace is in this
thread.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: "bad metadata" not fixed by btrfs repair

[Qu Wenruo]

Marc Haber wrote on 2016/03/30 09:18 +0200:
> On Wed, Mar 30, 2016 at 03:00:19PM +0800, Qu Wenruo wrote:
>> Marc Haber wrote on 2016/03/29 08:43 +0200:
>>> On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
>>>> Did you convert this filesystem from ext4 (or ext3)?
>>>
>>> No.
>>>
>>>> You hadn't mentioned what version of btrfs-progs you're using, and that is
>>>> somewhat important for recovery.  I'm not sure if current versions of btrfs
>>>> check can fix this issue, but I know for a fact that older versions (prior
>>>> to at least 4.1) can not fix it.
>>>
>>> 4.1 for creation and btrfs check.
>>
>> I assume that you have run older kernel on it, like v4.1 or v4.2.
>
> No, the productive system was always on a reasonably recent kernel. I
> guess that this instance of btrfs has never been mounted on anything
> older than 4.4.4. The rescue system I used to btrfs check (4.4-1 from
> Debian unstable, I updated btrfs-tools on the rescue system before
> going btrfs check) had kernel 3.16, but I have never actually mounted
> the btrfs there.
>
>>> Then btrfs check is a userspace-only matter, as it wants the fs
>>> unmounted, and it is irrelevant that I did btrfs check from a rescue
>>> system with an older kernel, 3.16 if I recall correctly.
>>
>> Not recommended to use older kernel to RW mount or use older fsck to do
>> repair.
>
> Oldest kernel that has mounted this btrfs is 4.4.4, fsck that touched
> the fs is 4.4. I'm trying to get hold of btrfs-tools 4.5.

Oh, I just forgot to ask for the btrfs-progs version.

The stripe crossing boundary output used to be false alert, as I forgot 
the to "-1" when checking the extent end position.

Didn't remember the exact version, but updating to 4.5 will never be a 
bad idea.
>
>>> My "productive" desktops (fan is one of them) run Debian unstable with
>>> a current vanilla kernel. At the moment, I can't use 4.5 because it
>>> acts up with KVM.  When I need a rescue system, I use grml, which
>>> unfortunately hasn't released since November 2014 and is still with
>>> kernel 3.16
>>
>> To fix your problem(make these error message just disappear, even they are
>> harmless on recent kernels), the most easy one, is to balance your metadata.
>
> This does not work on kernel 4.4.6 with tools 4.4. Truckloads of
> kernel traces, "WARNING: CPU: 5 PID: 31021 at
> fs/btrfs/extent-tree.c:7897 btrfs_alloc_tree_block+0xeb/0x3d6
> [btrfs]()", "BTRFS: block rsv returned -28", full trace is in this
> thread.

That's ENOSPC, seems to be another problem.

Did your btrfs have enough *unallocated* space?

Thanks,
Qu
>
> Greetings
> Marc
>

Re: "bad metadata" not fixed by btrfs repair

[Marc Haber]

On Wed, Mar 30, 2016 at 04:03:17PM +0800, Qu Wenruo wrote:
> Did your btrfs have enough *unallocated* space?

87 Gig out of a total 200 Gig Device size. I guess that should be
enough for a rebalance of 2,8 Gig Metadata.

Greetings
Ma "please excuse my cynism" rc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421

Re: "bad metadata" not fixed by btrfs repair

[Henk Slager]

On Wed, Mar 30, 2016 at 9:00 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
> First of all.
>
> The "crossing stripe boundary" error message itself is *HARMLESS* for recent
> kernels.
>
> It only means, that metadata extent won't be checked by scrub on recent
> kernels.
> Because scrub by its codes, has a limitation that, it can only check tree
> blocks which are inside a 64K block.
>
> Old kernel won't have anything wrong, until that tree block is being
> scrubbed.
> When scrubbed, old kernel just BUG_ON().
>
> Now recent kernel will handle such limitation by checking extent allocation
> and avoid crossing boundary, so new created fs with new kernel won't cause
> such error message at all.
>
> But for old created fs, the problem can't be avoided, but at least, new
> kernels will not BUG_ON() when you scrub these extents, they just get
> ignored (not that good, but at least no BUG_ON).
>
> And new fsck will check such case, gives such warning.
>
> Overall, you're OK if you are using recent kernels.
>
> Marc Haber wrote on 2016/03/29 08:43 +0200:
>>
>> On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
>>>
>>> Did you convert this filesystem from ext4 (or ext3)?
>>
>>
>> No.
>>
>>> You hadn't mentioned what version of btrfs-progs you're using, and that
>>> is
>>> somewhat important for recovery.  I'm not sure if current versions of
>>> btrfs
>>> check can fix this issue, but I know for a fact that older versions
>>> (prior
>>> to at least 4.1) can not fix it.
>>
>>
>> 4.1 for creation and btrfs check.
>
>
> I assume that you have run older kernel on it, like v4.1 or v4.2.
>
> In those old kernels, it lacks the check to avoid such extent allocation
> check.
>
>>
>>> As far as what the kernel is involved with, the easy way to check is if
>>> it's
>>> operating on a mounted filesystem or not.  If it only operates on mounted
>>> filesystems, it almost certainly goes through the kernel, if it only
>>> operates on unmounted filesystems, it's almost certainly done in
>>> userspace
>>> (except dev scan and technically fi show).
>>
>>
>> Then btrfs check is a userspace-only matter, as it wants the fs
>> unmounted, and it is irrelevant that I did btrfs check from a rescue
>> system with an older kernel, 3.16 if I recall correctly.
>
>
> Not recommended to use older kernel to RW mount or use older fsck to do
> repair.
> As it's possible that older kernel/btrfsck may allocate extent that cross
> the 64K boundary.
>
>>
>>> 2. Regarding general support:  If you're using an enterprise distribution
>>> (RHEL, SLES, CentOS, OEL, or something similar), you are almost certainly
>>> going to get better support from your vendor than from the mailing list
>>> or
>>> IRC.
>>
>>
>> My "productive" desktops (fan is one of them) run Debian unstable with
>> a current vanilla kernel. At the moment, I can't use 4.5 because it
>> acts up with KVM.  When I need a rescue system, I use grml, which
>> unfortunately hasn't released since November 2014 and is still with
>> kernel 3.16
>
>
> To fix your problem(make these error message just disappear, even they are
> harmless on recent kernels), the most easy one, is to balance your metadata.

I did a balance with filter -musage=100  (kernel/tools 4.5/4.5) of the
filesystem mentioned in here:
http://www.spinics.net/lists/linux-btrfs/msg51405.html

but still   bad metadata [ ),  crossing stripe boundary   messages,
double amount compared to 2 months ago

Kernel operating this fs has always been maximum 1 month behind
'Latest Stable Kernel' (kernel.org terminology)

> As I explained, the bug only lies in metadata, and balance will allocate new
> tree blocks, then copy old data into new locations.
>
> In the allocation process of recent kernel, it will avoid such cross
> boundary, and to fix your problem.
>
> But if you are using old kernels, don't scrub your metadata.
>
> Thanks,
> Qu
>>
>>
>> Greetings
>> Marc
>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "bad metadata" not fixed by btrfs repair

[Qu Wenruo]

Henk Slager wrote on 2016/03/30 16:03 +0200:
> On Wed, Mar 30, 2016 at 9:00 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>> First of all.
>>
>> The "crossing stripe boundary" error message itself is *HARMLESS* for recent
>> kernels.
>>
>> It only means, that metadata extent won't be checked by scrub on recent
>> kernels.
>> Because scrub by its codes, has a limitation that, it can only check tree
>> blocks which are inside a 64K block.
>>
>> Old kernel won't have anything wrong, until that tree block is being
>> scrubbed.
>> When scrubbed, old kernel just BUG_ON().
>>
>> Now recent kernel will handle such limitation by checking extent allocation
>> and avoid crossing boundary, so new created fs with new kernel won't cause
>> such error message at all.
>>
>> But for old created fs, the problem can't be avoided, but at least, new
>> kernels will not BUG_ON() when you scrub these extents, they just get
>> ignored (not that good, but at least no BUG_ON).
>>
>> And new fsck will check such case, gives such warning.
>>
>> Overall, you're OK if you are using recent kernels.
>>
>> Marc Haber wrote on 2016/03/29 08:43 +0200:
>>>
>>> On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
>>>>
>>>> Did you convert this filesystem from ext4 (or ext3)?
>>>
>>>
>>> No.
>>>
>>>> You hadn't mentioned what version of btrfs-progs you're using, and that
>>>> is
>>>> somewhat important for recovery.  I'm not sure if current versions of
>>>> btrfs
>>>> check can fix this issue, but I know for a fact that older versions
>>>> (prior
>>>> to at least 4.1) can not fix it.
>>>
>>>
>>> 4.1 for creation and btrfs check.
>>
>>
>> I assume that you have run older kernel on it, like v4.1 or v4.2.
>>
>> In those old kernels, it lacks the check to avoid such extent allocation
>> check.
>>
>>>
>>>> As far as what the kernel is involved with, the easy way to check is if
>>>> it's
>>>> operating on a mounted filesystem or not.  If it only operates on mounted
>>>> filesystems, it almost certainly goes through the kernel, if it only
>>>> operates on unmounted filesystems, it's almost certainly done in
>>>> userspace
>>>> (except dev scan and technically fi show).
>>>
>>>
>>> Then btrfs check is a userspace-only matter, as it wants the fs
>>> unmounted, and it is irrelevant that I did btrfs check from a rescue
>>> system with an older kernel, 3.16 if I recall correctly.
>>
>>
>> Not recommended to use older kernel to RW mount or use older fsck to do
>> repair.
>> As it's possible that older kernel/btrfsck may allocate extent that cross
>> the 64K boundary.
>>
>>>
>>>> 2. Regarding general support:  If you're using an enterprise distribution
>>>> (RHEL, SLES, CentOS, OEL, or something similar), you are almost certainly
>>>> going to get better support from your vendor than from the mailing list
>>>> or
>>>> IRC.
>>>
>>>
>>> My "productive" desktops (fan is one of them) run Debian unstable with
>>> a current vanilla kernel. At the moment, I can't use 4.5 because it
>>> acts up with KVM.  When I need a rescue system, I use grml, which
>>> unfortunately hasn't released since November 2014 and is still with
>>> kernel 3.16
>>
>>
>> To fix your problem(make these error message just disappear, even they are
>> harmless on recent kernels), the most easy one, is to balance your metadata.
>
> I did a balance with filter -musage=100  (kernel/tools 4.5/4.5) of the
> filesystem mentioned in here:
> http://www.spinics.net/lists/linux-btrfs/msg51405.html
>
> but still   bad metadata [ ),  crossing stripe boundary   messages,
> double amount compared to 2 months ago

Would you please give an example of the output?
So I can check if it's really crossing the boundary.

Thanks,
Qu
>
> Kernel operating this fs has always been maximum 1 month behind
> 'Latest Stable Kernel' (kernel.org terminology)
>
>> As I explained, the bug only lies in metadata, and balance will allocate new
>> tree blocks, then copy old data into new locations.
>>
>> In the allocation process of recent kernel, it will avoid such cross
>> boundary, and to fix your problem.
>>
>> But if you are using old kernels, don't scrub your metadata.
>>
>> Thanks,
>> Qu
>>>
>>>
>>> Greetings
>>> Marc
>>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: "bad metadata" not fixed by btrfs repair

[Henk Slager]

On Thu, Mar 31, 2016 at 2:28 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> Henk Slager wrote on 2016/03/30 16:03 +0200:
>>
>> On Wed, Mar 30, 2016 at 9:00 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>> wrote:
>>>
>>> First of all.
>>>
>>> The "crossing stripe boundary" error message itself is *HARMLESS* for
>>> recent
>>> kernels.
>>>
>>> It only means, that metadata extent won't be checked by scrub on recent
>>> kernels.
>>> Because scrub by its codes, has a limitation that, it can only check tree
>>> blocks which are inside a 64K block.
>>>
>>> Old kernel won't have anything wrong, until that tree block is being
>>> scrubbed.
>>> When scrubbed, old kernel just BUG_ON().
>>>
>>> Now recent kernel will handle such limitation by checking extent
>>> allocation
>>> and avoid crossing boundary, so new created fs with new kernel won't
>>> cause
>>> such error message at all.
>>>
>>> But for old created fs, the problem can't be avoided, but at least, new
>>> kernels will not BUG_ON() when you scrub these extents, they just get
>>> ignored (not that good, but at least no BUG_ON).
>>>
>>> And new fsck will check such case, gives such warning.
>>>
>>> Overall, you're OK if you are using recent kernels.
>>>
>>> Marc Haber wrote on 2016/03/29 08:43 +0200:
>>>>
>>>>
>>>> On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
>>>>>
>>>>>
>>>>> Did you convert this filesystem from ext4 (or ext3)?
>>>>
>>>>
>>>>
>>>> No.
>>>>
>>>>> You hadn't mentioned what version of btrfs-progs you're using, and that
>>>>> is
>>>>> somewhat important for recovery.  I'm not sure if current versions of
>>>>> btrfs
>>>>> check can fix this issue, but I know for a fact that older versions
>>>>> (prior
>>>>> to at least 4.1) can not fix it.
>>>>
>>>>
>>>>
>>>> 4.1 for creation and btrfs check.
>>>
>>>
>>>
>>> I assume that you have run older kernel on it, like v4.1 or v4.2.
>>>
>>> In those old kernels, it lacks the check to avoid such extent allocation
>>> check.
>>>
>>>>
>>>>> As far as what the kernel is involved with, the easy way to check is if
>>>>> it's
>>>>> operating on a mounted filesystem or not.  If it only operates on
>>>>> mounted
>>>>> filesystems, it almost certainly goes through the kernel, if it only
>>>>> operates on unmounted filesystems, it's almost certainly done in
>>>>> userspace
>>>>> (except dev scan and technically fi show).
>>>>
>>>>
>>>>
>>>> Then btrfs check is a userspace-only matter, as it wants the fs
>>>> unmounted, and it is irrelevant that I did btrfs check from a rescue
>>>> system with an older kernel, 3.16 if I recall correctly.
>>>
>>>
>>>
>>> Not recommended to use older kernel to RW mount or use older fsck to do
>>> repair.
>>> As it's possible that older kernel/btrfsck may allocate extent that cross
>>> the 64K boundary.
>>>
>>>>
>>>>> 2. Regarding general support:  If you're using an enterprise
>>>>> distribution
>>>>> (RHEL, SLES, CentOS, OEL, or something similar), you are almost
>>>>> certainly
>>>>> going to get better support from your vendor than from the mailing list
>>>>> or
>>>>> IRC.
>>>>
>>>>
>>>>
>>>> My "productive" desktops (fan is one of them) run Debian unstable with
>>>> a current vanilla kernel. At the moment, I can't use 4.5 because it
>>>> acts up with KVM.  When I need a rescue system, I use grml, which
>>>> unfortunately hasn't released since November 2014 and is still with
>>>> kernel 3.16
>>>
>>>
>>>
>>> To fix your problem(make these error message just disappear, even they
>>> are
>>> harmless on recent kernels), the most easy one, is to balance your
>>> metadata.
>>
>>
>> I did a balance with filter -musage=100  (kernel/tools 4.5/4.5) of the
>> filesystem mentioned in here:
>> http://www.spinics.net/lists/linux-btrfs/msg51405.html
>>
>> but still   bad metadata [ ),  crossing stripe boundary   messages,
>> double amount compared to 2 months ago
>
>
> Would you please give an example of the output?
> So I can check if it's really crossing the boundary.

This is the 1st one of the 105 messages:
bad metadata [8263437058048, 8263437062144) crossing stripe boundary

For all ... [X,Y) ...
X is 64K aligned and Y - X = 4K

So in my case, all false alerts.

Re: "bad metadata" not fixed by btrfs repair

[Qu Wenruo]

Henk Slager wrote on 2016/03/30 16:03 +0200:
> On Wed, Mar 30, 2016 at 9:00 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>> First of all.
>>
>> The "crossing stripe boundary" error message itself is *HARMLESS* for recent
>> kernels.
>>
>> It only means, that metadata extent won't be checked by scrub on recent
>> kernels.
>> Because scrub by its codes, has a limitation that, it can only check tree
>> blocks which are inside a 64K block.
>>
>> Old kernel won't have anything wrong, until that tree block is being
>> scrubbed.
>> When scrubbed, old kernel just BUG_ON().
>>
>> Now recent kernel will handle such limitation by checking extent allocation
>> and avoid crossing boundary, so new created fs with new kernel won't cause
>> such error message at all.
>>
>> But for old created fs, the problem can't be avoided, but at least, new
>> kernels will not BUG_ON() when you scrub these extents, they just get
>> ignored (not that good, but at least no BUG_ON).
>>
>> And new fsck will check such case, gives such warning.
>>
>> Overall, you're OK if you are using recent kernels.
>>
>> Marc Haber wrote on 2016/03/29 08:43 +0200:
>>>
>>> On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
>>>>
>>>> Did you convert this filesystem from ext4 (or ext3)?
>>>
>>>
>>> No.
>>>
>>>> You hadn't mentioned what version of btrfs-progs you're using, and that
>>>> is
>>>> somewhat important for recovery.  I'm not sure if current versions of
>>>> btrfs
>>>> check can fix this issue, but I know for a fact that older versions
>>>> (prior
>>>> to at least 4.1) can not fix it.
>>>
>>>
>>> 4.1 for creation and btrfs check.
>>
>>
>> I assume that you have run older kernel on it, like v4.1 or v4.2.
>>
>> In those old kernels, it lacks the check to avoid such extent allocation
>> check.
>>
>>>
>>>> As far as what the kernel is involved with, the easy way to check is if
>>>> it's
>>>> operating on a mounted filesystem or not.  If it only operates on mounted
>>>> filesystems, it almost certainly goes through the kernel, if it only
>>>> operates on unmounted filesystems, it's almost certainly done in
>>>> userspace
>>>> (except dev scan and technically fi show).
>>>
>>>
>>> Then btrfs check is a userspace-only matter, as it wants the fs
>>> unmounted, and it is irrelevant that I did btrfs check from a rescue
>>> system with an older kernel, 3.16 if I recall correctly.
>>
>>
>> Not recommended to use older kernel to RW mount or use older fsck to do
>> repair.
>> As it's possible that older kernel/btrfsck may allocate extent that cross
>> the 64K boundary.
>>
>>>
>>>> 2. Regarding general support:  If you're using an enterprise distribution
>>>> (RHEL, SLES, CentOS, OEL, or something similar), you are almost certainly
>>>> going to get better support from your vendor than from the mailing list
>>>> or
>>>> IRC.
>>>
>>>
>>> My "productive" desktops (fan is one of them) run Debian unstable with
>>> a current vanilla kernel. At the moment, I can't use 4.5 because it
>>> acts up with KVM.  When I need a rescue system, I use grml, which
>>> unfortunately hasn't released since November 2014 and is still with
>>> kernel 3.16
>>
>>
>> To fix your problem(make these error message just disappear, even they are
>> harmless on recent kernels), the most easy one, is to balance your metadata.
>
> I did a balance with filter -musage=100  (kernel/tools 4.5/4.5) of the
> filesystem mentioned in here:
> http://www.spinics.net/lists/linux-btrfs/msg51405.html
>
> but still   bad metadata [ ),  crossing stripe boundary   messages,
> double amount compared to 2 months ago
>
> Kernel operating this fs has always been maximum 1 month behind
> 'Latest Stable Kernel' (kernel.org terminology)

Would you please try the following patch?
https://patchwork.kernel.org/patch/8706891/

It is based on v4.5 and I think it should fix the false alert.

Thanks,
Qu

>
>> As I explained, the bug only lies in metadata, and balance will allocate new
>> tree blocks, then copy old data into new locations.
>>
>> In the allocation process of recent kernel, it will avoid such cross
>> boundary, and to fix your problem.
>>
>> But if you are using old kernels, don't scrub your metadata.
>>
>> Thanks,
>> Qu
>>>
>>>
>>> Greetings
>>> Marc
>>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: "bad metadata" not fixed by btrfs repair

[Henk Slager]

On Thu, Mar 31, 2016 at 4:23 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>
>
> Henk Slager wrote on 2016/03/30 16:03 +0200:
>>
>> On Wed, Mar 30, 2016 at 9:00 AM, Qu Wenruo <quwenruo@cn.fujitsu.com>
>> wrote:
>>>
>>> First of all.
>>>
>>> The "crossing stripe boundary" error message itself is *HARMLESS* for
>>> recent
>>> kernels.
>>>
>>> It only means, that metadata extent won't be checked by scrub on recent
>>> kernels.
>>> Because scrub by its codes, has a limitation that, it can only check tree
>>> blocks which are inside a 64K block.
>>>
>>> Old kernel won't have anything wrong, until that tree block is being
>>> scrubbed.
>>> When scrubbed, old kernel just BUG_ON().
>>>
>>> Now recent kernel will handle such limitation by checking extent
>>> allocation
>>> and avoid crossing boundary, so new created fs with new kernel won't
>>> cause
>>> such error message at all.
>>>
>>> But for old created fs, the problem can't be avoided, but at least, new
>>> kernels will not BUG_ON() when you scrub these extents, they just get
>>> ignored (not that good, but at least no BUG_ON).
>>>
>>> And new fsck will check such case, gives such warning.
>>>
>>> Overall, you're OK if you are using recent kernels.
>>>
>>> Marc Haber wrote on 2016/03/29 08:43 +0200:
>>>>
>>>>
>>>> On Mon, Mar 28, 2016 at 03:35:32PM -0400, Austin S. Hemmelgarn wrote:
>>>>>
>>>>>
>>>>> Did you convert this filesystem from ext4 (or ext3)?
>>>>
>>>>
>>>>
>>>> No.
>>>>
>>>>> You hadn't mentioned what version of btrfs-progs you're using, and that
>>>>> is
>>>>> somewhat important for recovery.  I'm not sure if current versions of
>>>>> btrfs
>>>>> check can fix this issue, but I know for a fact that older versions
>>>>> (prior
>>>>> to at least 4.1) can not fix it.
>>>>
>>>>
>>>>
>>>> 4.1 for creation and btrfs check.
>>>
>>>
>>>
>>> I assume that you have run older kernel on it, like v4.1 or v4.2.
>>>
>>> In those old kernels, it lacks the check to avoid such extent allocation
>>> check.
>>>
>>>>
>>>>> As far as what the kernel is involved with, the easy way to check is if
>>>>> it's
>>>>> operating on a mounted filesystem or not.  If it only operates on
>>>>> mounted
>>>>> filesystems, it almost certainly goes through the kernel, if it only
>>>>> operates on unmounted filesystems, it's almost certainly done in
>>>>> userspace
>>>>> (except dev scan and technically fi show).
>>>>
>>>>
>>>>
>>>> Then btrfs check is a userspace-only matter, as it wants the fs
>>>> unmounted, and it is irrelevant that I did btrfs check from a rescue
>>>> system with an older kernel, 3.16 if I recall correctly.
>>>
>>>
>>>
>>> Not recommended to use older kernel to RW mount or use older fsck to do
>>> repair.
>>> As it's possible that older kernel/btrfsck may allocate extent that cross
>>> the 64K boundary.
>>>
>>>>
>>>>> 2. Regarding general support:  If you're using an enterprise
>>>>> distribution
>>>>> (RHEL, SLES, CentOS, OEL, or something similar), you are almost
>>>>> certainly
>>>>> going to get better support from your vendor than from the mailing list
>>>>> or
>>>>> IRC.
>>>>
>>>>
>>>>
>>>> My "productive" desktops (fan is one of them) run Debian unstable with
>>>> a current vanilla kernel. At the moment, I can't use 4.5 because it
>>>> acts up with KVM.  When I need a rescue system, I use grml, which
>>>> unfortunately hasn't released since November 2014 and is still with
>>>> kernel 3.16
>>>
>>>
>>>
>>> To fix your problem(make these error message just disappear, even they
>>> are
>>> harmless on recent kernels), the most easy one, is to balance your
>>> metadata.
>>
>>
>> I did a balance with filter -musage=100  (kernel/tools 4.5/4.5) of the
>> filesystem mentioned in here:
>> http://www.spinics.net/lists/linux-btrfs/msg51405.html
>>
>> but still   bad metadata [ ),  crossing stripe boundary   messages,
>> double amount compared to 2 months ago
>>
>> Kernel operating this fs has always been maximum 1 month behind
>> 'Latest Stable Kernel' (kernel.org terminology)
>
>
> Would you please try the following patch?
> https://patchwork.kernel.org/patch/8706891/
>
> It is based on v4.5 and I think it should fix the false alert.

I have applied the patch to 4.5 and ran  btrfs check  again and now no
    bad metadata [ ),  crossing stripe boundary   messages anymore
(and also no other errors).
Thanks.

> Thanks,
> Qu
>
>
>>
>>> As I explained, the bug only lies in metadata, and balance will allocate
>>> new
>>> tree blocks, then copy old data into new locations.
>>>
>>> In the allocation process of recent kernel, it will avoid such cross
>>> boundary, and to fix your problem.
>>>
>>> But if you are using old kernels, don't scrub your metadata.
>>>
>>> Thanks,
>>> Qu
>>>>
>>>>
>>>>
>>>> Greetings
>>>> Marc
>>>>
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>
>

Re: "bad metadata" not fixed by btrfs repair

[Henk Slager]

On Mon, Mar 28, 2016 at 4:37 PM, Marc Haber <mh+linux-btrfs@zugschlus.de> wrote:
> Hi,
>
> I have a btrfs which btrfs check --repair doesn't fix:
>
> # btrfs check --repair /dev/mapper/fanbtr
> bad metadata [4425377054720, 4425377071104) crossing stripe boundary
> bad metadata [4425380134912, 4425380151296) crossing stripe boundary
> bad metadata [4427532795904, 4427532812288) crossing stripe boundary
> bad metadata [4568321753088, 4568321769472) crossing stripe boundary
> bad metadata [4568489656320, 4568489672704) crossing stripe boundary
> bad metadata [4571474493440, 4571474509824) crossing stripe boundary
> bad metadata [4571946811392, 4571946827776) crossing stripe boundary
> bad metadata [4572782919680, 4572782936064) crossing stripe boundary
> bad metadata [4573086351360, 4573086367744) crossing stripe boundary
> bad metadata [4574221041664, 4574221058048) crossing stripe boundary
> bad metadata [4574373412864, 4574373429248) crossing stripe boundary
> bad metadata [4574958649344, 4574958665728) crossing stripe boundary
> bad metadata [4575996018688, 4575996035072) crossing stripe boundary
> bad metadata [4580376772608, 4580376788992) crossing stripe boundary

In this case, for all  ... [X,Y) ...

X is 64K aligned and Y - X = 16K

So also false alerts.

Re: "bad metadata" not fixed by btrfs repair

[Marc Haber]

On Thu, Mar 31, 2016 at 08:42:46PM +0200, Henk Slager wrote:
> So also false alerts.

btrfs-tools 4.5.1 with Qu's patch from patchwork doesnt show those
warnings any more.

Greetings
Marc

-- 
-----------------------------------------------------------------------------
Marc Haber         | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany    |  lose things."    Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature |  How to make an American Quilt | Fax: *49 6224 1600421