open_ctree failed after ATA errors

[Florian Bruhin <me@the-compiler.org>]

[-- Attachment #1: Type: text/plain, Size: 4642 bytes --]

I have the following setup:

- Two harddisks
    - Both individually encrypted using LUKS
    - Both combined into a btrfs using the btrfs raid1 feature

- The above duplicated twice:
    - /dev/mapper/data1 and /dev/mapper/data2 -> /mnt/data
    - /dev/mapper/secdata1 and /dev/mapper/secdata2 -> /mnt/secdata

Recently, I saw the following messages in my kernel logs all few days:

    ata6.00: exception Emask 0x10 SAct 0x40000 SErr 0x400000 action 0x6 frozen
    ata6.00: irq_stat 0x08000000, interface fatal error
    ata6: SError: { Handshk }
    ata6.00: failed command: WRITE FPDMA QUEUED
    ata6.00: cmd 61/08:90:e8:29:85/01:00:03:00:00/40 tag 18 ncq 135168 out
             res 40/00:94:e8:29:85/00:00:03:00:00/40 Emask 0x10 (ATA bus error)
    ata6.00: status: { DRDY }
    ata6: hard resetting link
    ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
    ata6.00: configured for UDMA/133
    ata6: EH complete
    ata6.00: exception Emask 0x10 SAct 0x800000 SErr 0x400000 action 0x6 frozen
    ata6.00: irq_stat 0x08000000, interface fatal error
    ata6: SError: { Handshk }
    ata6.00: failed command: WRITE FPDMA QUEUED
    ata6.00: cmd 61/00:b8:f0:2a:85/02:00:03:00:00/40 tag 23 ncq 262144 out
             res 40/00:bc:f0:2a:85/00:00:03:00:00/40 Emask 0x10 (ATA bus error)
    ata6.00: status: { DRDY }
    ata6: hard resetting link
    ata6: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
    ata6.00: configured for UDMA/133
    ata6: EH complete

I thought maybe it was just a temporary problem or related to
upgrading the kernel recently (3.17.1 -> 3.17.2) and not rebooting
yet, so I rebooted.

Since then, I could run cryptsetup luksOpen without problems, but
mounting the devices hanged for ~15 seconds and then returned without
error, but didn't mount anything.

When strace'ing mount, it hanged here:

    mount("/dev/mapper/data1", "/mnt/data", "btrfs", MS_MGC_VAL, NULL)

(which then returned 0). I didn't see anything in the kernel logs.

I then tried the following:

    # cryptsetup luksClose ... # for all 4 disks
    # cryptsetup luksOpen ...  # for all 4 disks
    # btrfs device scan --all-devices
    # mount /dev/mapper/data1 /mnt/data
    # mount /dev/mapper/secdata1 /mnt/data

The same thing happened, and I then saw this in the kernel logs:

    [Nov11 15:33] BTRFS info (device dm-3): disk space caching is enabled
    [Nov11 15:34] BTRFS info (device dm-3): disk space caching is enabled
    [Nov11 15:35] BTRFS info (device dm-3): disk space caching is enabled
    [Nov11 15:36] BTRFS info (device dm-3): disk space caching is enabled
    [Nov11 15:37] BTRFS info (device dm-3): disk space caching is enabled
    [ +16.054127] BTRFS: open_ctree failed
    [Nov11 15:38] BTRFS info (device dm-3): disk space caching is enabled
    [Nov11 16:02] BTRFS info (device dm-2): disk space caching is enabled

How could I mount these volumes again? Is it a good idea to use
btrfs-zero-log as described in [1]?

Some other information:

- Distribution: Archlinux
- uname -a: Linux moody 3.17.2-1-ARCH #1 SMP PREEMPT Thu Oct 30 20:49:39 CET 2014 x86_64 GNU/Linux
- btrfs --version: Btrfs v3.17
- btrfs fi show:

    Label: 'secdata2'  uuid: 38267260-b656-4c66-a123-5f9214066ae1
            Total devices 2 FS bytes used 2.06TiB
            devid    1 size 3.64TiB used 2.06TiB path /dev/mapper/secdata2
            devid    2 size 3.64TiB used 2.06TiB path /dev/mapper/secdata1

    Label: 'data2'  uuid: b67ca50d-dbde-445d-922a-3479849b5499
            Total devices 2 FS bytes used 2.35TiB
            devid    1 size 2.73TiB used 2.69TiB path /dev/mapper/data1
            devid    3 size 2.73TiB used 2.69TiB path /dev/mapper/data2

- btrfs fi df /mnt/data:

    Data, single: total=58.42GiB, used=14.48GiB
    System, single: total=4.00MiB, used=12.00KiB
    Metadata, single: total=1.01GiB, used=471.80MiB
    GlobalReserve, single: total=160.00MiB, used=0.00B

- btrfs fi df /mnt/secdata

    Data, single: total=58.42GiB, used=14.48GiB
    System, single: total=4.00MiB, used=12.00KiB
    Metadata, single: total=1.01GiB, used=471.80MiB
    GlobalReserve, single: total=160.00MiB, used=0.00B

If there's anything else I can provide please let me know. Please Cc
me on replies, as I'm not on the list. Thanks in advance!

Florian

[1] https://btrfs.wiki.kernel.org/index.php/Btrfs-zero-log

-- 
http://www.the-compiler.org | me@the-compiler.org (Mail/XMPP)
             GPG 0xFD55A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/

[-- Attachment #2: Type: application/pgp-signature, Size: 819 bytes --]