From: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
To: Qu Wenruo <quwenruo.btrfs@gmx.com>,
Matthias Bodenbinder <matthias@bodenbinder.de>,
linux-btrfs@vger.kernel.org
Subject: Re: Question: raid1 behaviour on failure
Date: Fri, 22 Apr 2016 11:21:47 +0900 [thread overview]
Message-ID: <57198ABB.1060406@jp.fujitsu.com> (raw)
In-Reply-To: <5718C068.8080508@gmx.com>
On 2016/04/21 20:58, Qu Wenruo wrote:
>
>
> On 04/21/2016 03:45 PM, Satoru Takeuchi wrote:
>> On 2016/04/21 15:23, Satoru Takeuchi wrote:
>>> On 2016/04/20 14:17, Matthias Bodenbinder wrote:
>>>> Am 18.04.2016 um 09:22 schrieb Qu Wenruo:
>>>>> BTW, it would be better to post the dmesg for better debug.
>>>>
>>>> So here we. I did the same test again. Here is a full log of what i
>>>> did. It seems to be mean like a bug in btrfs.
>>>> Sequenz of events:
>>>> 1. mount the raid1 (2 disc with different size)
>>>> 2. unplug the biggest drive (hotplug)
>>>> 3. try to copy something to the degraded raid1
>>>> 4. plugin the device again (hotplug)
>>>>
>>>> This scenario does not work. The disc array is NOT redundant! I can
>>>> not work with it while a drive is missing and I can not reattach the
>>>> device so that everything works again.
>>>>
>>>> The btrfs module crashes during the test.
>>>>
>>>> I am using LMDE2 with backports:
>>>> btrfs-tools 4.4-1~bpo8+1
>>>> linux-image-4.4.0-0.bpo.1-amd64
>>>>
>>>> Matthias
>>>>
>>>>
>>>> rakete - root - /root
>>>> 1# mount /mnt/raid1/
>>>>
>>>> Journal:
>>>>
>>>> Apr 20 07:01:16 rakete kernel: BTRFS info (device sdi): enabling auto
>>>> defrag
>>>> Apr 20 07:01:16 rakete kernel: BTRFS info (device sdi): disk space
>>>> caching is enabled
>>>> Apr 20 07:01:16 rakete kernel: BTRFS: has skinny extents
>>>>
>>>> rakete - root - /mnt/raid1
>>>> 3# ll
>>>> insgesamt 0
>>>> drwxrwxr-x 1 root root 36 Nov 14 2014 AfterShot2(64-bit)
>>>> drwxrwxr-x 1 root root 5082 Apr 17 09:06 etc
>>>> drwxr-xr-x 1 root root 108 Mär 24 07:31 var
>>>>
>>>> 4# btrfs fi show
>>>> Label: none uuid: 16d5891f-5d52-4b29-8591-588ddf11e73d
>>>> Total devices 3 FS bytes used 1.60GiB
>>>> devid 1 size 698.64GiB used 3.03GiB path /dev/sdg
>>>> devid 2 size 465.76GiB used 3.03GiB path /dev/sdh
>>>> devid 3 size 232.88GiB used 0.00B path /dev/sdi
>>>>
>>>> ####
>>>> unplug device sdg:
>>>>
>>>> Apr 20 07:03:05 rakete kernel: Buffer I/O error on dev sdf1, logical
>>>> block 243826688, lost sync page write
>>>> Apr 20 07:03:05 rakete kernel: JBD2: Error -5 detected when updating
>>>> journal superblock for sdf1-8.
>>>> Apr 20 07:03:05 rakete kernel: Aborting journal on device sdf1-8.
>>>> Apr 20 07:03:05 rakete kernel: Buffer I/O error on dev sdf1, logical
>>>> block 243826688, lost sync page write
>>>> Apr 20 07:03:05 rakete kernel: JBD2: Error -5 detected when updating
>>>> journal superblock for sdf1-8.
>>>> Apr 20 07:03:05 rakete umount[16405]: umount: /mnt/raid1: target is busy
>>>> Apr 20 07:03:05 rakete umount[16405]: (In some cases useful info
>>>> about processes that
>>>> Apr 20 07:03:05 rakete umount[16405]: use the device is found by
>>>> lsof(8) or fuser(1).)
>>>> Apr 20 07:03:05 rakete systemd[1]: mnt-raid1.mount mount process
>>>> exited, code=exited status=32
>>>> Apr 20 07:03:05 rakete systemd[1]: Failed unmounting /mnt/raid1.
>>>> Apr 20 07:03:24 rakete kernel: usb 3-1: new SuperSpeed USB device
>>>> number 3 using xhci_hcd
>>>> Apr 20 07:03:24 rakete kernel: usb 3-1: New USB device found,
>>>> idVendor=152d, idProduct=0567
>>>> Apr 20 07:03:24 rakete kernel: usb 3-1: New USB device strings:
>>>> Mfr=10, Product=11, SerialNumber=5
>>>> Apr 20 07:03:24 rakete kernel: usb 3-1: Product: USB to ATA/ATAPI Bridge
>>>> Apr 20 07:03:24 rakete kernel: usb 3-1: Manufacturer: JMicron
>>>> Apr 20 07:03:24 rakete kernel: usb 3-1: SerialNumber: 152D00539000
>>>> Apr 20 07:03:24 rakete kernel: usb-storage 3-1:1.0: USB Mass Storage
>>>> device detected
>>>> Apr 20 07:03:24 rakete kernel: usb-storage 3-1:1.0: Quirks match for
>>>> vid 152d pid 0567: 5000000
>>>> Apr 20 07:03:24 rakete kernel: scsi host9: usb-storage 3-1:1.0
>>>> Apr 20 07:03:24 rakete mtp-probe[16424]: checking bus 3, device 3:
>>>> "/sys/devices/pci0000:00/0000:00:1c.5/0000:04:00.0/usb3/3-1"
>>>> Apr 20 07:03:24 rakete mtp-probe[16424]: bus: 3, device: 3 was not an
>>>> MTP device
>>>> Apr 20 07:03:25 rakete kernel: scsi 9:0:0:0: Direct-Access WDC
>>>> WD20 02FAEX-007BA0 0125 PQ: 0 ANSI: 6
>>>> Apr 20 07:03:25 rakete kernel: scsi 9:0:0:1: Direct-Access WDC
>>>> WD50 01AALS-00L3B2 0125 PQ: 0 ANSI: 6
>>>> Apr 20 07:03:25 rakete kernel: scsi 9:0:0:2: Direct-Access
>>>> SAMSUNG SP2504C 0125 PQ: 0 ANSI: 6
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: Attached scsi generic sg6
>>>> type 0
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: Attached scsi generic sg7
>>>> type 0
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: [sdf] 3907029168 512-byte
>>>> logical blocks: (2.00 TB/1.82 TiB)
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: [sdf] Write Protect is off
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: [sdf] Mode Sense: 67 00 10 08
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:2: Attached scsi generic sg8
>>>> type 0
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: [sdj] 976773168 512-byte
>>>> logical blocks: (500 GB/466 GiB)
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: [sdf] No Caching mode page
>>>> found
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: [sdf] Assuming drive
>>>> cache: write through
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: [sdj] Write Protect is off
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: [sdj] Mode Sense: 67 00 10 08
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:2: [sdk] 488395055 512-byte
>>>> logical blocks: (250 GB/233 GiB)
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: [sdj] No Caching mode page
>>>> found
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: [sdj] Assuming drive
>>>> cache: write through
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:2: [sdk] Write Protect is off
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:2: [sdk] Mode Sense: 67 00 10 08
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:2: [sdk] No Caching mode page
>>>> found
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:2: [sdk] Assuming drive
>>>> cache: write through
>>>> Apr 20 07:03:25 rakete kernel: sdf: sdf1
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:0: [sdf] Attached SCSI disk
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:1: [sdj] Attached SCSI disk
>>>> Apr 20 07:03:25 rakete kernel: sd 9:0:0:2: [sdk] Attached SCSI disk
>>>> Apr 20 07:03:25 rakete kernel: EXT4-fs (sdf1): recovery complete
>>>> Apr 20 07:03:25 rakete kernel: EXT4-fs (sdf1): mounted filesystem
>>>> with ordered data mode. Opts: (null)
>>>> Apr 20 07:03:25 rakete udisksd[3671]: Error statting /dev/sdg: No
>>>> such file or directory
>>>>
>>>>
>>>> ####
>>>> 5# btrfs fi show
>>>> Label: none uuid: 16d5891f-5d52-4b29-8591-588ddf11e73d
>>>> Total devices 3 FS bytes used 1.60GiB
>>>> devid 2 size 465.76GiB used 3.03GiB path /dev/sdj
>>>> devid 3 size 232.88GiB used 0.00B path /dev/sdk
>>>> *** Some devices missing
>>>> ####
>>>
>>> Here the names of *online* devices are changed
>>> (/dev/sdh => /dev/sdj, /dev/sdi => /dev/sdk) after just
>>> offlining a device (/dev/sdf). It's odd regardless of
>>> whether Btrfs works fine or not.
>>>
>>> Can anyone explain this behavior?
>>
>> FYI,
>>
>> I tried to reproduce this problem on VM.
>> Here USB storages are /dev/sd{a,b,c}.
>>
>> Step to reproduce:
>>
>> 1. create a fs on /dev/sd{a,b,c}
>> 2. mount this fs
>> 3. Surprise unplug /dev/sdc
>> 4. Write to this fs till ENOSPC happens
>>
>> Then, although there are I/O errors about /dev/sdc,
>> device names didn't change and ro remount didn't happen.
>>
>> command log:
>> =================================
>> # mkfs.btrfs -f -m raid1 -d raid1 /dev/sd{a,b,c}
>> btrfs-progs v4.5.1-41-g8202204-dirty
>> See http://btrfs.wiki.kernel.org for more information.
>>
>> Label: (null)
>> UUID: 16a54915-c807-42cf-8365-82c0780c5ab5
>> Node size: 16384
>> Sector size: 4096
>> Filesystem size: 15.00GiB
>> Block group profiles:
>> Data: RAID1 1.01GiB
>> Metadata: RAID1 1.01GiB
>> System: RAID1 12.00MiB
>> SSD detected: no
>> Incompat features: extref, skinny-metadata
>> Number of devices: 3
>> Devices:
>> ID SIZE PATH
>> 1 5.00GiB /dev/sda
>> 2 5.00GiB /dev/sdb
>> 3 5.00GiB /dev/sdc
>>
>> # mount /dev/sda /scratch_mnt/
>> # btrfs fi show /scratch_mnt/
>> Label: none uuid: 16a54915-c807-42cf-8365-82c0780c5ab5
>> Total devices 3 FS bytes used 640.00KiB
>> devid 1 size 5.00GiB used 2.00GiB path /dev/sda
>> devid 2 size 5.00GiB used 1.01GiB path /dev/sdb
>> devid 3 size 5.00GiB used 1.01GiB path /dev/sdc
>>
>> #
>> # # *** surprise unplug happens here ***
>> #
>> # btrfs fi show /scratch_mnt/
>
> Would you please post the output of "btrfs-debug-tree -t 3"?
>
> I guess the case would be that, there is not raid1 stripe in device 3, so all data/metadata allocation/cow happens without problem.
> "btrfs-debug-tree -t 3" output would verify my guess.
OK, here it is.
btrfs-debug-tree -t 3 before cp:
===========================
btrfs-progs v4.5.1-41-g8202204-dirty
chunk tree
leaf 20987904 items 6 free space 15503 generation 5 owner 3
fs uuid 30771a06-e6a8-4cbc-a094-893049fa5060
chunk uuid 2325f1b9-1bf0-4247-8c29-7b179eabf1b2
item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
dev item devid 1 total_bytes 5368709120 bytes used 2147483648
dev uuid 06bc0993-39d3-4d9a-b484-760ae2150c3a
item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
dev item devid 2 total_bytes 5368709120 bytes used 1082130432
dev uuid 3868895f-295b-4a89-a01c-ad0f1c5ac758
item 2 key (DEV_ITEMS DEV_ITEM 3) itemoff 15989 itemsize 98
dev item devid 3 total_bytes 5368709120 bytes used 1082130432
dev uuid 911e8702-9428-4b8e-bc6d-d212e909a1ef
item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) itemoff 15877 itemsize 112
chunk length 8388608 owner 2 stripe_len 65536
type SYSTEM|RAID1 num_stripes 2
stripe 0 devid 3 offset 1048576
dev uuid: 911e8702-9428-4b8e-bc6d-d212e909a1ef
stripe 1 devid 2 offset 1048576
dev uuid: 3868895f-295b-4a89-a01c-ad0f1c5ac758
item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 29360128) itemoff 15765 itemsize 112
chunk length 1073741824 owner 2 stripe_len 65536
type METADATA|RAID1 num_stripes 2
stripe 0 devid 1 offset 20971520
dev uuid: 06bc0993-39d3-4d9a-b484-760ae2150c3a
stripe 1 devid 3 offset 9437184
dev uuid: 911e8702-9428-4b8e-bc6d-d212e909a1ef
item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 1103101952) itemoff 15653 itemsize 112
chunk length 1073741824 owner 2 stripe_len 65536
type DATA|RAID1 num_stripes 2
stripe 0 devid 2 offset 9437184
dev uuid: 3868895f-295b-4a89-a01c-ad0f1c5ac758
stripe 1 devid 1 offset 1094713344
dev uuid: 06bc0993-39d3-4d9a-b484-760ae2150c3a
total bytes 16106127360
bytes used 114688
uuid 30771a06-e6a8-4cbc-a094-893049fa5060
===========================
Here I hot unplug devid 2 (/dev/sdb).
btrfs-debug-tree -t 3 after cp (which cause ENOSPC):
===========================
btrfs-progs v4.5.1-41-g8202204-dirty
warning, device 2 is missing
chunk tree
leaf 20987904 items 11 free space 14818 generation 9 owner 3
fs uuid 30771a06-e6a8-4cbc-a094-893049fa5060
chunk uuid 2325f1b9-1bf0-4247-8c29-7b179eabf1b2
item 0 key (DEV_ITEMS DEV_ITEM 1) itemoff 16185 itemsize 98
dev item devid 1 total_bytes 5368709120 bytes used 4294967296
dev uuid 06bc0993-39d3-4d9a-b484-760ae2150c3a
item 1 key (DEV_ITEMS DEV_ITEM 2) itemoff 16087 itemsize 98
dev item devid 2 total_bytes 5368709120 bytes used 5367660544
dev uuid 3868895f-295b-4a89-a01c-ad0f1c5ac758
item 2 key (DEV_ITEMS DEV_ITEM 3) itemoff 15989 itemsize 98
dev item devid 3 total_bytes 5368709120 bytes used 5367660544
dev uuid 911e8702-9428-4b8e-bc6d-d212e909a1ef
item 3 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520) itemoff 15877 itemsize 112
chunk length 8388608 owner 2 stripe_len 65536
type SYSTEM|RAID1 num_stripes 2
stripe 0 devid 3 offset 1048576
dev uuid: 911e8702-9428-4b8e-bc6d-d212e909a1ef
stripe 1 devid 2 offset 1048576
dev uuid: 3868895f-295b-4a89-a01c-ad0f1c5ac758
item 4 key (FIRST_CHUNK_TREE CHUNK_ITEM 29360128) itemoff 15765 itemsize 112
chunk length 1073741824 owner 2 stripe_len 65536
type METADATA|RAID1 num_stripes 2
stripe 0 devid 1 offset 20971520
dev uuid: 06bc0993-39d3-4d9a-b484-760ae2150c3a
stripe 1 devid 3 offset 9437184
dev uuid: 911e8702-9428-4b8e-bc6d-d212e909a1ef
item 5 key (FIRST_CHUNK_TREE CHUNK_ITEM 1103101952) itemoff 15653 itemsize 112
chunk length 1073741824 owner 2 stripe_len 65536
type DATA|RAID1 num_stripes 2
stripe 0 devid 2 offset 9437184
dev uuid: 3868895f-295b-4a89-a01c-ad0f1c5ac758
stripe 1 devid 1 offset 1094713344
dev uuid: 06bc0993-39d3-4d9a-b484-760ae2150c3a
item 6 key (FIRST_CHUNK_TREE CHUNK_ITEM 2176843776) itemoff 15541 itemsize 112
chunk length 1073741824 owner 2 stripe_len 65536
type DATA|RAID1 num_stripes 2
stripe 0 devid 2 offset 1083179008
dev uuid: 3868895f-295b-4a89-a01c-ad0f1c5ac758
stripe 1 devid 3 offset 1083179008
dev uuid: 911e8702-9428-4b8e-bc6d-d212e909a1ef
item 7 key (FIRST_CHUNK_TREE CHUNK_ITEM 3250585600) itemoff 15429 itemsize 112
chunk length 1073741824 owner 2 stripe_len 65536
type DATA|RAID1 num_stripes 2
stripe 0 devid 1 offset 2168455168
dev uuid: 06bc0993-39d3-4d9a-b484-760ae2150c3a
stripe 1 devid 3 offset 2156920832
dev uuid: 911e8702-9428-4b8e-bc6d-d212e909a1ef
item 8 key (FIRST_CHUNK_TREE CHUNK_ITEM 4324327424) itemoff 15317 itemsize 112
chunk length 1073741824 owner 2 stripe_len 65536
type DATA|RAID1 num_stripes 2
stripe 0 devid 2 offset 2156920832
dev uuid: 3868895f-295b-4a89-a01c-ad0f1c5ac758
stripe 1 devid 1 offset 3242196992
dev uuid: 06bc0993-39d3-4d9a-b484-760ae2150c3a
item 9 key (FIRST_CHUNK_TREE CHUNK_ITEM 5398069248) itemoff 15205 itemsize 112
chunk length 1073741824 owner 2 stripe_len 65536
type DATA|RAID1 num_stripes 2
stripe 0 devid 2 offset 3230662656
dev uuid: 3868895f-295b-4a89-a01c-ad0f1c5ac758
stripe 1 devid 3 offset 3230662656
dev uuid: 911e8702-9428-4b8e-bc6d-d212e909a1ef
item 10 key (FIRST_CHUNK_TREE CHUNK_ITEM 6471811072) itemoff 15093 itemsize 112
chunk length 1064304640 owner 2 stripe_len 65536
type DATA|RAID1 num_stripes 2
stripe 0 devid 2 offset 4304404480
dev uuid: 3868895f-295b-4a89-a01c-ad0f1c5ac758
stripe 1 devid 3 offset 4304404480
dev uuid: 911e8702-9428-4b8e-bc6d-d212e909a1ef
total bytes 16106127360
bytes used 6711709696
uuid 30771a06-e6a8-4cbc-a094-893049fa5060
===========================
In both before cp and after cp, there are
chunks containing /dev/sdb (devid 2).
Thanks,
Satoru
>
> Thanks,
> Qu
>> Label: none uuid: 16a54915-c807-42cf-8365-82c0780c5ab5
>> Total devices 3 FS bytes used 1.81GiB
>> devid 1 size 5.00GiB used 2.00GiB path /dev/sda
>> devid 2 size 5.00GiB used 2.01GiB path /dev/sdb
>> *** Some devices missing
>>
>> # cp -a linux /scratch_mnt/
>> # cp -a linux /scratch_mnt/linux.2
>> # cp -a linux /scratch_mnt/linux.3
>> cp: error writing ‘/scratch_mnt/linux.3/drivers/scsi/lpfc/lpfc_els.c’:
>> No space left on device
>> ...
>> # mount | grep scratch
>> /dev/sda on /scratch_mnt type btrfs
>> (rw,relatime,seclabel,space_cache,subvolid=5,subvol=/)
>> # dmesg | tail
>> [ 1400.778705] BTRFS warning (device sdc): lost page write due to IO
>> error on /dev/sdc
>> [ 1438.604796] btrfs_dev_stat_print_on_error: 174 callbacks suppressed
>> [ 1438.604803] BTRFS error (device sdc): bdev /dev/sdc errs: wr 125633,
>> rd 1, flush 276, corrupt 0, gen 0
>> [ 1438.609782] BTRFS error (device sdc): bdev /dev/sdc errs: wr 125634,
>> rd 1, flush 276, corrupt 0, gen 0
>> [ 1438.613331] BTRFS error (device sdc): bdev /dev/sdc errs: wr 125634,
>> rd 1, flush 277, corrupt 0, gen 0
>> [ 1438.669090] btrfs_end_buffer_write_sync: 52 callbacks suppressed
>> [ 1438.669095] BTRFS warning (device sdc): lost page write due to IO
>> error on /dev/sdc
>> [ 1438.669098] BTRFS error (device sdc): bdev /dev/sdc errs: wr 125635,
>> rd 1, flush 277, corrupt 0, gen 0
>> [ 1438.672621] BTRFS warning (device sdc): lost page write due to IO
>> error on /dev/sdc
>> [ 1438.672626] BTRFS error (device sdc): bdev /dev/sdc errs: wr 125636,
>> rd 1, flush 277, corrupt 0, gen 0
>> =================================
>>
>> Thanks,
>> Satoru
>>
>>>
>>> Thanks,
>>> Satoru
>>>
>>>> still mounted in rw mode:
>>>> /dev/sdg on /mnt/raid1 type btrfs
>>>> (rw,noatime,space_cache,autodefrag,subvolid=5,subvol=/)
>>>> ####
>>>> 7# cp -r /root/ .
>>>> cp: das Verzeichnis „./root“ kann nicht angelegt werden:
>>>> Eingabe-/Ausgabefehler
>>>> Apr 20 07:05:37 rakete kernel: BTRFS error (device sdi): bdev
>>>> /dev/sdg errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
>>>> Apr 20 07:05:37 rakete kernel: BTRFS error (device sdi): bdev
>>>> /dev/sdg errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
>>>> Apr 20 07:05:37 rakete kernel: BTRFS error (device sdi): bdev
>>>> /dev/sdg errs: wr 0, rd 3, flush 0, corrupt 0, gen 0
>>>> Apr 20 07:05:37 rakete kernel: BTRFS error (device sdi): bdev
>>>> /dev/sdg errs: wr 0, rd 4, flush 0, corrupt 0, gen 0
>>>> Apr 20 07:05:37 rakete kernel: BTRFS error (device sdi): bdev
>>>> /dev/sdg errs: wr 0, rd 5, flush 0, corrupt 0, gen 0
>>>> Apr 20 07:05:37 rakete kernel: BTRFS error (device sdi): bdev
>>>> /dev/sdg errs: wr 0, rd 6, flush 0, corrupt 0, gen 0
>>>> Apr 20 07:05:37 rakete kernel: BTRFS error (device sdi): bdev
>>>> /dev/sdg errs: wr 0, rd 7, flush 0, corrupt 0, gen 0
>>>> Apr 20 07:05:37 rakete kernel: BTRFS error (device sdi): bdev
>>>> /dev/sdg errs: wr 0, rd 8, flush 0, corrupt 0, gen 0
>>>> Apr 20 07:05:37 rakete kernel: BTRFS error (device sdi): bdev
>>>> /dev/sdg errs: wr 0, rd 9, flush 0, corrupt 0, gen 0
>>>> Apr 20 07:05:37 rakete kernel: BTRFS error (device sdi): bdev
>>>> /dev/sdg errs: wr 0, rd 10, flush 0, corrupt 0, gen 0
>>>> Apr 20 07:05:37 rakete kernel: BTRFS error (device sdi): error
>>>> reading free space cache
>>>> Apr 20 07:05:37 rakete kernel: BTRFS warning (device sdi): failed to
>>>> load free space cache for block group 20497563648, rebuilding it now
>>>> Apr 20 07:05:37 rakete kernel: ------------[ cut here ]------------
>>>> Apr 20 07:05:37 rakete kernel: WARNING: CPU: 7 PID: 16738 at
>>>> /build/linux-H3jpF0/linux-4.4.6/fs/btrfs/ctree.c:1156
>>>> __btrfs_cow_block+0x56f/0x5e0 [btrfs]()
>>>> Apr 20 07:05:37 rakete kernel: BTRFS: Transaction aborted (error -5)
>>>> Apr 20 07:05:37 rakete kernel: Modules linked in: uas usb_storage
>>>> pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) binfmt_misc dvb_ttpci
>>>> saa7146_vv ttpci_eeprom saa7146 videobuf_dma_sg videobuf_core
>>>> dvb_core v4l2_common videodev media cfg80211 vboxdrv(O)
>>>> cpufreq_powersave cpufreq_conservative cpufreq_userspace
>>>> cpufreq_stats snd_hda_codec_hdmi intel_rapl iosf_mbi
>>>> x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm
>>>> irqbypass crct10dif_pclmul crc32_pclmul eeepc_wmi asus_wmi joydev
>>>> sparse_keymap drbg iTCO_wdt iTCO_vendor_support snd_hda_codec_realtek
>>>> rfkill ansi_cprng snd_hda_codec_generic nvidia(PO) aesni_intel
>>>> aes_x86_64 lrw gf128mul snd_hda_intel glue_helper ablk_helper
>>>> snd_hda_codec cryptd snd_hda_core serio_raw pcspkr snd_hwdep snd_pcm
>>>> i2c_i801 snd_timer snd lpc_ich soundcore 8250_fintek mei_me shpchp mei
>>>> Apr 20 07:05:37 rakete kernel: mfd_core battery tpm_tis tpm evdev
>>>> processor drm fuse ecryptfs cbc sha256_ssse3 sha256_generic hmac
>>>> encrypted_keys parport_pc ppdev lp parport autofs4 ext4 crc16 mbcache
>>>> jbd2 btrfs raid456 async_raid6_recov async_memcpy async_pq async_xor
>>>> async_tx xor hid_generic usbhid hid raid6_pq libcrc32c crc32c_generic
>>>> md_mod dm_mirror dm_region_hash dm_log dm_mod sr_mod sg cdrom sd_mod
>>>> ata_generic ahci libahci pata_via xhci_pci ehci_pci crc32c_intel
>>>> xhci_hcd ehci_hcd libata psmouse scsi_mod atl1c usbcore usb_common
>>>> fjes video wmi fan thermal button
>>>> Apr 20 07:05:37 rakete kernel: CPU: 7 PID: 16738 Comm: cp Tainted:
>>>> P O 4.4.0-0.bpo.1-amd64 #1 Debian 4.4.6-1~bpo8+1
>>>> Apr 20 07:05:37 rakete kernel: Hardware name: System manufacturer
>>>> System Product Name/P8H67-V, BIOS 3707 07/12/2013
>>>> Apr 20 07:05:37 rakete kernel: 0000000000000286 000000006a1407c8
>>>> ffffffff812ed425 ffff88016b6dfb90
>>>> Apr 20 07:05:37 rakete kernel: ffffffffa03817b8 ffffffff81077ea1
>>>> ffff88018e7fcd30 ffff88016b6dfbe8
>>>> Apr 20 07:05:37 rakete kernel: ffff88005d863e88 ffff8801cde7a980
>>>> ffff88018e7fce48 ffffffff81077f2c
>>>> Apr 20 07:05:37 rakete kernel: Call Trace:
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffff812ed425>] ?
>>>> dump_stack+0x5c/0x77
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffff81077ea1>] ?
>>>> warn_slowpath_common+0x81/0xb0
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffff81077f2c>] ?
>>>> warn_slowpath_fmt+0x5c/0x80
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffffa02d74af>] ?
>>>> __btrfs_cow_block+0x56f/0x5e0 [btrfs]
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffffa02d76af>] ?
>>>> btrfs_cow_block+0x10f/0x1d0 [btrfs]
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffffa02db2cd>] ?
>>>> btrfs_search_slot+0x1fd/0xa30 [btrfs]
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffffa02dd3f1>] ?
>>>> btrfs_insert_empty_items+0x71/0xc0 [btrfs]
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffff811f4d92>] ?
>>>> insert_inode_locked4+0xa2/0x1c0
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffffa030ee5d>] ?
>>>> btrfs_new_inode+0x1cd/0x590 [btrfs]
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffffa0310a77>] ?
>>>> btrfs_mkdir+0x107/0x1f0 [btrfs]
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffff811e80b0>] ?
>>>> vfs_mkdir+0xb0/0x140
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffff811e9d3e>] ?
>>>> SyS_mkdir+0xce/0x110
>>>> Apr 20 07:05:37 rakete kernel: [<ffffffff81592736>] ?
>>>> system_call_fast_compare_end+0xc/0x6b
>>>> Apr 20 07:05:37 rakete kernel: ---[ end trace 025eb0e83ffed96f ]---
>>>> Apr 20 07:05:37 rakete kernel: BTRFS: error (device sdi) in
>>>> __btrfs_cow_block:1156: errno=-5 IO failure
>>>> Apr 20 07:05:37 rakete kernel: BTRFS info (device sdi): forced readonly
>>>>
>>>> ####
>>>> Try to copy again:
>>>> 11# cp -r /root/ .
>>>> cp: cannot create directory './root': Read-only file system
>>>> ####
>>>> /dev/sdg on /mnt/raid1 type btrfs
>>>> (ro,noatime,space_cache,autodefrag,subvolid=5,subvol=/)
>>>> ####
>>>> plugin device sdg again:
>>>>
>>>> Apr 20 07:07:39 rakete udisksd[3671]: Cleaning up mount point
>>>> /media/matthias/BACKUP (device 8:81 no longer exist)
>>>> Apr 20 07:07:39 rakete kernel: usb 3-1: USB disconnect, device number 3
>>>> Apr 20 07:07:39 rakete udisksd[3671]: Error statting /dev/sdg: No
>>>> such file or directory
>>>> Apr 20 07:07:39 rakete umount[16807]: umount: /mnt/raid1: target is busy
>>>> Apr 20 07:07:39 rakete umount[16807]: (In some cases useful info
>>>> about processes that
>>>> Apr 20 07:07:39 rakete umount[16807]: use the device is found by
>>>> lsof(8) or fuser(1).)
>>>> Apr 20 07:07:39 rakete systemd[1]: mnt-raid1.mount mount process
>>>> exited, code=exited status=32
>>>> Apr 20 07:07:39 rakete systemd[1]: Failed unmounting /mnt/raid1.
>>>> Apr 20 07:08:01 rakete kernel: usb 3-1: new SuperSpeed USB device
>>>> number 4 using xhci_hcd
>>>> Apr 20 07:08:01 rakete kernel: usb 3-1: New USB device found,
>>>> idVendor=152d, idProduct=0567
>>>> Apr 20 07:08:01 rakete kernel: usb 3-1: New USB device strings:
>>>> Mfr=10, Product=11, SerialNumber=5
>>>> Apr 20 07:08:01 rakete kernel: usb 3-1: Product: USB to ATA/ATAPI Bridge
>>>> Apr 20 07:08:01 rakete kernel: usb 3-1: Manufacturer: JMicron
>>>> Apr 20 07:08:01 rakete kernel: usb 3-1: SerialNumber: 152D00539000
>>>> Apr 20 07:08:01 rakete kernel: usb-storage 3-1:1.0: USB Mass Storage
>>>> device detected
>>>> Apr 20 07:08:01 rakete kernel: usb-storage 3-1:1.0: Quirks match for
>>>> vid 152d pid 0567: 5000000
>>>> Apr 20 07:08:01 rakete kernel: scsi host10: usb-storage 3-1:1.0
>>>> Apr 20 07:08:01 rakete mtp-probe[16826]: checking bus 3, device 4:
>>>> "/sys/devices/pci0000:00/0000:00:1c.5/0000:04:00.0/usb3/3-1"
>>>> Apr 20 07:08:01 rakete mtp-probe[16826]: bus: 3, device: 4 was not an
>>>> MTP device
>>>> Apr 20 07:08:02 rakete kernel: scsi 10:0:0:0: Direct-Access WDC
>>>> WD20 02FAEX-007BA0 0125 PQ: 0 ANSI: 6
>>>> Apr 20 07:08:02 rakete kernel: scsi 10:0:0:1: Direct-Access WDC
>>>> WD75 00AACS-00C7B0 0125 PQ: 0 ANSI: 6
>>>> Apr 20 07:08:02 rakete kernel: scsi 10:0:0:2: Direct-Access WDC
>>>> WD50 01AALS-00L3B2 0125 PQ: 0 ANSI: 6
>>>> Apr 20 07:08:02 rakete kernel: scsi 10:0:0:3: Direct-Access
>>>> SAMSUNG SP2504C 0125 PQ: 0 ANSI: 6
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:0: Attached scsi generic sg6
>>>> type 0
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:0: [sdf] 3907029168 512-byte
>>>> logical blocks: (2.00 TB/1.82 TiB)
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:0: [sdf] Write Protect is off
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:0: [sdf] Mode Sense: 67 00
>>>> 10 08
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:1: Attached scsi generic sg7
>>>> type 0
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:0: [sdf] No Caching mode
>>>> page found
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:0: [sdf] Assuming drive
>>>> cache: write through
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:1: [sdj] 1465149168 512-byte
>>>> logical blocks: (750 GB/699 GiB)
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:2: Attached scsi generic sg8
>>>> type 0
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:1: [sdj] Write Protect is off
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:1: [sdj] Mode Sense: 67 00
>>>> 10 08
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:3: Attached scsi generic sg9
>>>> type 0
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:2: [sdk] 976773168 512-byte
>>>> logical blocks: (500 GB/466 GiB)
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:1: [sdj] No Caching mode
>>>> page found
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:1: [sdj] Assuming drive
>>>> cache: write through
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:2: [sdk] Write Protect is off
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:2: [sdk] Mode Sense: 67 00
>>>> 10 08
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:3: [sdl] 488395055 512-byte
>>>> logical blocks: (250 GB/233 GiB)
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:2: [sdk] No Caching mode
>>>> page found
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:2: [sdk] Assuming drive
>>>> cache: write through
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:3: [sdl] Write Protect is off
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:3: [sdl] Mode Sense: 67 00
>>>> 10 08
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:3: [sdl] No Caching mode
>>>> page found
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:3: [sdl] Assuming drive
>>>> cache: write through
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:0: [sdf] Attached SCSI disk
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:1: [sdj] Attached SCSI disk
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:2: [sdk] Attached SCSI disk
>>>> Apr 20 07:08:02 rakete kernel: sd 10:0:0:3: [sdl] Attached SCSI disk
>>>> Apr 20 07:08:02 rakete kernel: EXT4-fs (sdf1): recovery complete
>>>> Apr 20 07:08:02 rakete kernel: EXT4-fs (sdf1): mounted filesystem
>>>> with ordered data mode. Opts: (null)
>>>>
>>>> ####
>>>> still ro mode
>>>> /dev/sdj on /mnt/raid1 type btrfs
>>>> (ro,noatime,space_cache,autodefrag,subvolid=5,subvol=/)
>>>> ####
>>>> 14# btrfs fi show
>>>> Label: none uuid: 16d5891f-5d52-4b29-8591-588ddf11e73d
>>>> Total devices 3 FS bytes used 1.60GiB
>>>> devid 1 size 698.64GiB used 3.03GiB path /dev/sdj
>>>> devid 2 size 465.76GiB used 3.03GiB path /dev/sdk
>>>> devid 3 size 232.88GiB used 0.00B path /dev/sdl
>>>> ####
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> linux-btrfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2016-04-22 2:23 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-04-18 5:06 Question: raid1 behaviour on failure Matthias Bodenbinder
2016-04-18 7:22 ` Qu Wenruo
2016-04-20 5:17 ` Matthias Bodenbinder
2016-04-20 7:25 ` Qu Wenruo
2016-04-21 5:22 ` Matthias Bodenbinder
2016-04-21 5:43 ` Qu Wenruo
2016-04-21 6:02 ` Liu Bo
2016-04-21 6:09 ` Qu Wenruo
2016-04-21 17:40 ` Matthias Bodenbinder
2016-04-22 6:02 ` Qu Wenruo
2016-04-23 7:07 ` Matthias Bodenbinder
2016-04-23 7:17 ` Matthias Bodenbinder
2016-04-26 8:17 ` Satoru Takeuchi
2016-04-26 15:16 ` Henk Slager
2016-04-20 13:32 ` Anand Jain
2016-04-21 5:15 ` Matthias Bodenbinder
2016-04-21 7:19 ` Anand Jain
2016-04-21 6:23 ` Satoru Takeuchi
2016-04-21 11:09 ` Austin S. Hemmelgarn
2016-04-21 11:28 ` Henk Slager
2016-04-21 17:27 ` Matthias Bodenbinder
2016-04-26 16:19 ` Henk Slager
2016-04-26 16:42 ` Holger Hoffstätte
2016-04-28 5:12 ` Matthias Bodenbinder
2016-04-28 5:24 ` Gareth Pye
2016-04-28 8:08 ` Duncan
2016-04-28 5:09 ` Matthias Bodenbinder
2016-04-28 19:14 ` Henk Slager
[not found] ` <57188534.1070408@jp.fujitsu.com>
2016-04-21 11:58 ` Qu Wenruo
2016-04-22 2:21 ` Satoru Takeuchi [this message]
2016-04-22 5:32 ` Qu Wenruo
2016-04-22 6:17 ` Satoru Takeuchi
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=57198ABB.1060406@jp.fujitsu.com \
--to=takeuchi_satoru@jp.fujitsu.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=matthias@bodenbinder.de \
--cc=quwenruo.btrfs@gmx.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).