* Re: no good deed goes unpunished...
From: Phil Turmel @ 2015-03-28 1:28 UTC (permalink / raw)
To: Dave Stevens, Roger Heflin; +Cc: linux-raid
In-Reply-To: <20150327173709.19022q59t5hxh4tx@webmail.uniserve.com>
Hi Dave,
{Convention on kernel.org is reply-to-all, trim quotes, and avoid
top-posting. Please trim.}
On 03/27/2015 08:37 PM, Dave Stevens wrote:
> # cat /proc/mdstat
>
> Personalities :
>
> md12 : inactive sda2[0](S)
> 732467520 blocks
>
> md10 : inactive sdc2[2](S) sdb2[1](S)
> 1464935040 blocks
If your livecd tried to be helpful by assembling arrays, but couldn't
complete, you end up with partially-assembled inactive arrays like shown
above. The member devices shown are *busy*, under the expectation that
the remaining device(s) will show up soon. :-)
As Roger tried to suggest, you need to run:
mdadm --stop /dev/md10
mdadm --stop /dev/md12
That'll release the member devices you need. However, the device
assembly pairs above look suspicious. I suggest you show mdadm -E for
those devices again to make sure you work with the correct two disks.
(sda and sdc from the original report.)
Then you can do:
mdadm --assemble --force --verbose /dev/mdX /dev/sdY2 /dev/sdZ2
with the correct substitutions for X, Y, and Z.
If that doesn't work, show the output.
Phil
^ permalink raw reply
* Re: mdadm RAID6 "active" with spares and failed disks; need help
From: Phil Turmel @ 2015-03-28 1:59 UTC (permalink / raw)
To: Matt Callaghan, linux-raid
In-Reply-To: <BLU437-SMTP28725520E940DA3707A62F81090@phx.gbl>
Hi Matt,
On 03/27/2015 07:48 PM, Matt Callaghan wrote:
> Back at it with fresh brain and fresh hardware. (several months ago I
> got part-way through Valentine's ideas but not all the way -- decided to
> get a clean setup before progressing further)
>
> I have built a new (fresh/clean) server, and compiled+installed the
> latest mdadm v3.3.2.
> The 8x drives from this RAID6 array have also been moved to the new
> temporary server.
>
> Now of course, in the new server, the device labels are different.
> I need to map the previous "known labels" in the old server (/dev/sdX)
> to the "new labels" in order to get the drive ordering for re-assembly
> right.
> http://www.linuxquestions.org/questions/linux-server-73/mdadm-raid6-active-with-spares-and-failed-disks%3B-need-help-4175530127/
I read through this. Given all of the destructive actions you took, I
am doubtful you will ever get your data. Like mounting "readonly".
That gives you a readonly filesystem, but it writes to the device.
Possibly a great deal if there's a journal to replay. You also trimmed
much useful data with "grep" that probably would help us save you now.
However, in the hope you might have useful data that can be correlated
with current status, start with lsdrv [1]. Paste the output in your
reply with word wrap turned off. That'll at least give us a correlation
between device name and serial number.
> e.g. before I had:
> {{{
> /dev/sd[nmlpiokj]1
> }}}
FWIW, it is not safe to use square bracket notation when order matters.
> , and now I have:
> {{{
> /dev/sd[abcdefghi]1
> }}}
The linux 'sd' driver has never guaranteed consistent device names.
It's merely an artifact of boot timing that makes it look that way.
Which is why array members have superblocks that record the roles. You
absolutely *must* have accurate role numbers to get your data back.
Show complete 'mdadm -E' output for all of your member partitions as
they stand now.
> Unfortunately I don't have any smartctl output saved from the previous
> server and I can't find a way to map device drive label to serial numbers.
> Any thoughts how I could do this based on the data I have saved in that
> forum post?
Please show current 'smartctl -x' output for all of these devices, too.
Just paste it all in your reply (with word wrap turned off).
Phil
[1] https://github.com/pturmel/lsdrv
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Dave Stevens @ 2015-03-28 2:14 UTC (permalink / raw)
To: Phil Turmel; +Cc: Roger Heflin, linux-raid
In-Reply-To: <551603A2.4080808@turmel.org>
Quoting Phil Turmel <philip@turmel.org>:
> Hi Dave,
>
> {Convention on kernel.org is reply-to-all, trim quotes, and avoid
> top-posting. Please trim.}
ok, thanks
>
> On 03/27/2015 08:37 PM, Dave Stevens wrote:
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md12 : inactive sda2[0](S)
>> 732467520 blocks
>>
>> md10 : inactive sdc2[2](S) sdb2[1](S)
>> 1464935040 blocks
>
> If your livecd tried to be helpful by assembling arrays, but couldn't
> complete, you end up with partially-assembled inactive arrays like shown
> above. The member devices shown are *busy*, under the expectation that
> the remaining device(s) will show up soon. :-)
>
> As Roger tried to suggest, you need to run:
>
> mdadm --stop /dev/md10
> mdadm --stop /dev/md12
>
> That'll release the member devices you need. However, the device
> assembly pairs above look suspicious. I suggest you show mdadm -E for
> those devices again to make sure you work with the correct two disks.
> (sda and sdc from the original report.)
>
> Then you can do:
>
> mdadm --assemble --force --verbose /dev/mdX /dev/sdY2 /dev/sdZ2
>
> with the correct substitutions for X, Y, and Z.
>
> If that doesn't work, show the output.
>
> Phil
will do, thanks
D
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: mdadm RAID6 "active" with spares and failed disks; need help
From: Roman Mamedov @ 2015-03-28 10:11 UTC (permalink / raw)
To: Phil Turmel; +Cc: Matt Callaghan, linux-raid
In-Reply-To: <55160B0A.1090405@turmel.org>
[-- Attachment #1: Type: text/plain, Size: 607 bytes --]
On Fri, 27 Mar 2015 21:59:38 -0400
Phil Turmel <philip@turmel.org> wrote:
> I read through this. Given all of the destructive actions you took, I
> am doubtful you will ever get your data. Like mounting "readonly".
> That gives you a readonly filesystem, but it writes to the device.
> Possibly a great deal if there's a journal to replay.
Are you sure, which FS does that? I remember some discussion on FS lists
(Btrfs?), and IIRC the consensus and the implemented behavior was that the
device shouldn't ever be touched with writes on RO mounts, no matter what.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Read-only mounts (was mdadm RAID6 "active" with spares and failed disks; need help)
From: Phil Turmel @ 2015-03-28 15:11 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Matt Callaghan, linux-raid
In-Reply-To: <20150328151142.11db8c9a@natsu>
On 03/28/2015 06:11 AM, Roman Mamedov wrote:
> On Fri, 27 Mar 2015 21:59:38 -0400
> Phil Turmel <philip@turmel.org> wrote:
>
>> I read through this. Given all of the destructive actions you took, I
>> am doubtful you will ever get your data. Like mounting "readonly".
>> That gives you a readonly filesystem, but it writes to the device.
>> Possibly a great deal if there's a journal to replay.
>
> Are you sure, which FS does that? I remember some discussion on FS lists
> (Btrfs?), and IIRC the consensus and the implemented behavior was that the
> device shouldn't ever be touched with writes on RO mounts, no matter what.
I remember people being burned by it with ext3/4 a couple years ago.
Which is why all of the array disaster recoveries I've helped with
called for fsck -n to verify a reconstruction attempt, not a mount -o ro.
I will set up a small VM and see what recent kernels do.
Phil
^ permalink raw reply
* Test
From: Tejas Rao @ 2015-03-28 16:28 UTC (permalink / raw)
To: linux-raid
Test
^ permalink raw reply
* Re: mdadm RAID6 "active" with spares and failed disks; need help
From: Phil Turmel @ 2015-03-28 17:40 UTC (permalink / raw)
To: Matt Callaghan, linux-raid
In-Reply-To: <BLU436-SMTP135A84B36120ACD11B2783D81F70@phx.gbl>
Hi Matt,
I didn't see this make it to linux-raid, so I'll quote more than normal.
Oh, and convention on kernel.org is to reply-to-all, trim unnecessary
quotes, and avoid top-posting. Please.
On 03/27/2015 11:10 PM, Matt Callaghan wrote:
> Just noticed the lsdrv [1] link to git; got it, here's the output
> {{{
> fermulator@fermmy-mdadm:~/downloads/lsdrv/lsdrv$ ./lsdrv
> PCI [ahci] 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 40)
> ├scsi 0:x:x:x [Empty]
> └scsi 1:0:0:0 ATA Maxtor 6Y160M0
> └sda 152.67g [8:0] Empty/Unknown
> ├sda1 512.00m [8:1] Empty/Unknown
> │└Mounted as /dev/sda1 @ /boot/efi
> ├sda2 148.71g [8:2] Empty/Unknown
> │└Mounted as /dev/disk/by-uuid/5549ca2f-758a-4e04-8e36-cf4544bef4fb @ /
> └sda3 3.46g [8:3] Empty/Unknown
> PCI [mptsas] 05:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)
> ├scsi 2:0:0:0 ATA ST2000DL003-9VT1
> │└sdb 1.82t [8:16] Empty/Unknown
> │ └sdb1 1.82t [8:17] Empty/Unknown
> ├scsi 2:0:1:0 ATA ST2000DL003-9VT1
> │└sdc 1.82t [8:32] Empty/Unknown
> │ └sdc1 1.82t [8:33] Empty/Unknown
> ├scsi 2:0:2:0 ATA ST2000DL003-9VT1
> │└sdd 1.82t [8:48] Empty/Unknown
> │ └sdd1 1.82t [8:49] Empty/Unknown
> ├scsi 2:0:3:0 ATA ST2000VN000-1H31
> │└sde 1.82t [8:64] Empty/Unknown
> │ └sde1 1.82t [8:65] Empty/Unknown
> ├scsi 2:0:4:0 ATA ST2000DL003-9VT1
> │└sdf 1.82t [8:80] Empty/Unknown
> │ └sdf1 1.82t [8:81] Empty/Unknown
> ├scsi 2:0:5:0 ATA ST2000DL003-9VT1
> │└sdg 1.82t [8:96] Empty/Unknown
> │ └sdg1 1.82t [8:97] Empty/Unknown
> ├scsi 2:0:6:0 ATA ST2000DL003-9VT1
> │└sdh 1.82t [8:112] Empty/Unknown
> │ └sdh1 1.82t [8:113] Empty/Unknown
> ├scsi 2:0:7:0 ATA ST2000VN000-1H31
> │└sdi 1.82t [8:128] Empty/Unknown
> │ └sdi1 1.82t [8:129] Empty/Unknown
> └scsi 2:x:x:x [Empty]
> Other Block Devices
> ├loop0 0.00k [7:0] Empty/Unknown
> ├loop1 0.00k [7:1] Empty/Unknown
> ├loop2 0.00k [7:2] Empty/Unknown
> ├loop3 0.00k [7:3] Empty/Unknown
> ├loop4 0.00k [7:4] Empty/Unknown
> ├loop5 0.00k [7:5] Empty/Unknown
> ├loop6 0.00k [7:6] Empty/Unknown
> ├loop7 0.00k [7:7] Empty/Unknown
> ├ram0 64.00m [1:0] Empty/Unknown
> ├ram1 64.00m [1:1] Empty/Unknown
> ├ram2 64.00m [1:2] Empty/Unknown
> ├ram3 64.00m [1:3] Empty/Unknown
> ├ram4 64.00m [1:4] Empty/Unknown
> ├ram5 64.00m [1:5] Empty/Unknown
> ├ram6 64.00m [1:6] Empty/Unknown
> ├ram7 64.00m [1:7] Empty/Unknown
> ├ram8 64.00m [1:8] Empty/Unknown
> ├ram9 64.00m [1:9] Empty/Unknown
> ├ram10 64.00m [1:10] Empty/Unknown
> ├ram11 64.00m [1:11] Empty/Unknown
> ├ram12 64.00m [1:12] Empty/Unknown
> ├ram13 64.00m [1:13] Empty/Unknown
> ├ram14 64.00m [1:14] Empty/Unknown
> └ram15 64.00m [1:15] Empty/Unknown
> }}}
Ok. Not that helpful. I suspect you had error messages about missing
utilities. No serial numbers.
[trim /]
> mdadm output as of NOW. But note that the output here is likely useless
> since the last thing I was trying to was getting the array back together
> as per the forum posting... (it's definitely not in the original state
> anymore...)
Yep, useless.
[trim /]
> smartctl outputs are:
/dev/sdb:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD0XWHR
> LU WWN Device Id: 5 000c50 02f4197f5
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:05 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 113 099 006 - 51859880
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 422
> 5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
> 7 Seek_Error_Rate POSR-- 072 060 030 - 17185766
> 9 Power_On_Hours -O--CK 061 061 000 - 34871
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 71
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 099 099 000 - 1
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 094 094 000 - 6
> 190 Airflow_Temperature_Cel -O---K 059 043 045 Past 41 (5 77 42 35 0)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 420
> 193 Load_Cycle_Count -O--CK 100 100 000 - 422
> 194 Temperature_Celsius -O---K 041 057 000 - 41 (0 13 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 017 003 000 - 51859880
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 16990890657845
> 241 Total_LBAs_Written ------ 100 253 000 - 731266756
> 242 Total_LBAs_Read ------ 100 253 000 - 1129016466
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
>
> SCT Error Recovery Control command not supported
Now we know why your array fell apart. Using green and/or desktop
drives without mitigating the timeout mismatch problem.
/dev/sdc:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD1B1ZJ
> LU WWN Device Id: 5 000c50 02f361865
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:06 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 112 090 006 - 44947192
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 68
> 5 Reallocated_Sector_Ct PO--CK 078 078 036 - 14728
> 7 Seek_Error_Rate POSR-- 072 066 030 - 15873942
> 9 Power_On_Hours -O--CK 061 061 000 - 34875
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 74
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 001 001 000 - 823
> 188 Command_Timeout -O--CK 100 099 000 - 65539
> 189 High_Fly_Writes -O-RCK 093 093 000 - 7
> 190 Airflow_Temperature_Cel -O---K 058 044 045 Past 42 (2 158 44 36 0)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 65
> 193 Load_Cycle_Count -O--CK 100 100 000 - 68
> 194 Temperature_Celsius -O---K 042 056 000 - 42 (0 13 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 016 003 000 - 44947192
> 197 Current_Pending_Sector -O--C- 089 089 000 - 952
^^^^^
Wow!
> 198 Offline_Uncorrectable ----C- 089 089 000 - 952
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 141149805250605
> 241 Total_LBAs_Written ------ 100 253 000 - 3292940140
> 242 Total_LBAs_Read ------ 100 253 000 - 496297916
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
>
> SCT Error Recovery Control command not supported
And again.
/dev/sdd:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD15M4K
> LU WWN Device Id: 5 000c50 02f386588
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:07 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 117 099 006 - 153485440
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 352
> 5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
> 7 Seek_Error_Rate POSR-- 076 060 030 - 43819206
> 9 Power_On_Hours -O--CK 061 061 000 - 35013
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 74
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 097 097 000 - 3
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 099 099 000 - 1
> 190 Airflow_Temperature_Cel -O---K 057 046 045 - 43 (Min/Max 36/43)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 351
> 193 Load_Cycle_Count -O--CK 100 100 000 - 353
> 194 Temperature_Celsius -O---K 043 054 000 - 43 (0 11 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 021 003 000 - 153485440
> 197 Current_Pending_Sector -O--C- 100 100 000 - 8
More Pending sectors. These are locations where unrecoverable read
errors occurred that the firmware is waiting for an overwrite to decide
if they are fixable.
> 198 Offline_Uncorrectable ----C- 100 100 000 - 8
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 134501195876534
> 241 Total_LBAs_Written ------ 100 253 000 - 879538094
> 242 Total_LBAs_Read ------ 100 253 000 - 1783662156
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control command not supported
Sigh.
/dev/sde:
> === START OF INFORMATION SECTION ===
> Device Model: ST2000VN000-1H3164
> Serial Number: W1H25K77
> LU WWN Device Id: 5 000c50 06a40c121
> Firmware Version: SC42
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Sizes: 512 bytes logical, 4096 bytes physical
> Rotation Rate: 5900 rpm
> Device is: Not in smartctl database [for details use: -P showall]
> ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
> SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
> Local Time is: Fri Mar 27 22:57:07 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM feature is: Unavailable
> APM level is: 254 (maximum performance)
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 116 099 006 - 117001736
> 3 Spin_Up_Time PO---- 096 095 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 21
> 5 Reallocated_Sector_Ct PO--CK 100 100 010 - 0
> 7 Seek_Error_Rate POSR-- 064 060 030 - 3017660
> 9 Power_On_Hours -O--CK 085 085 000 - 13146
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 21
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 100 100 000 - 0
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 058 058 000 - 42
> 190 Airflow_Temperature_Cel -O---K 065 056 045 - 35 (Min/Max 35/37)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 21
> 193 Load_Cycle_Count -O--CK 100 100 000 - 21
> 194 Temperature_Celsius -O---K 035 044 000 - 35 (0 16 0 0 0)
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control:
> Read: 1 (0.1 seconds)
> Write: 1 (0.1 seconds)
Interesting. Is this the device default? The drives I've seen that
have a default have either 4.0s or 7.0s.
/dev/sdf:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD18S73
> LU WWN Device Id: 5 000c50 02f3fab7d
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:07 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 109 099 006 - 23951160
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 70
> 5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
> 7 Seek_Error_Rate POSR-- 075 060 030 - 39605538
> 9 Power_On_Hours -O--CK 061 061 000 - 34955
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 75
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 100 100 000 - 0
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 089 089 000 - 11
> 190 Airflow_Temperature_Cel -O---K 058 048 045 - 42 (Min/Max 34/42)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 69
> 193 Load_Cycle_Count -O--CK 100 100 000 - 70
> 194 Temperature_Celsius -O---K 042 052 000 - 42 (0 10 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 013 003 000 - 23951160
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 194931385731211
> 241 Total_LBAs_Written ------ 100 253 000 - 4208935845
> 242 Total_LBAs_Read ------ 100 253 000 - 3841138908
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control command not supported
And again.
/dev/sdg:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD1ACSD
> LU WWN Device Id: 5 000c50 02f31ac2f
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:08 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 113 099 006 - 50711848
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 70
> 5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
> 7 Seek_Error_Rate POSR-- 075 060 030 - 41597886
> 9 Power_On_Hours -O--CK 061 061 000 - 34955
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 74
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 100 100 000 - 0
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 100 100 000 - 0
> 190 Airflow_Temperature_Cel -O---K 058 048 045 - 42 (Min/Max 36/43)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 69
> 193 Load_Cycle_Count -O--CK 100 100 000 - 70
> 194 Temperature_Celsius -O---K 042 052 000 - 42 (0 10 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 017 003 000 - 50711848
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 121040768370827
> 241 Total_LBAs_Written ------ 100 253 000 - 1173584109
> 242 Total_LBAs_Read ------ 100 253 000 - 1269612579
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control command not supported
And sigh again. Broken record, I know. But this is a big deal.
/dev/sdh:
> === START OF INFORMATION SECTION ===
> Model Family: Seagate Barracuda Green (AF)
> Device Model: ST2000DL003-9VT166
> Serial Number: 5YD18S0M
> LU WWN Device Id: 5 000c50 02f3f4ec7
> Firmware Version: CC32
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Size: 512 bytes logical/physical
> Rotation Rate: 5900 rpm
> Device is: In smartctl database [for details use: -P show]
> ATA Version is: ATA8-ACS T13/1699-D revision 4
> SATA Version is: SATA 3.0, 6.0 Gb/s (current: 1.5 Gb/s)
> Local Time is: Fri Mar 27 22:57:08 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM level is: 0 (vendor specific), recommended: 254
> APM feature is: Unavailable
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 119 099 006 - 229878536
> 3 Spin_Up_Time PO---- 093 092 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 70
> 5 Reallocated_Sector_Ct PO--CK 100 100 036 - 0
> 7 Seek_Error_Rate POSR-- 075 060 030 - 38838566
> 9 Power_On_Hours -O--CK 061 061 000 - 34957
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 76
> 183 Runtime_Bad_Block -O--CK 100 100 000 - 0
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 100 100 000 - 0
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 094 094 000 - 6
> 190 Airflow_Temperature_Cel -O---K 061 051 045 - 39 (Min/Max 29/40)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 69
> 193 Load_Cycle_Count -O--CK 100 100 000 - 70
> 194 Temperature_Celsius -O---K 039 049 000 - 39 (0 11 0 0 0)
> 195 Hardware_ECC_Recovered -O-RC- 023 003 000 - 229878536
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> 240 Head_Flying_Hours ------ 100 253 000 - 30356828883085
> 241 Total_LBAs_Written ------ 100 253 000 - 16063676
> 242 Total_LBAs_Read ------ 100 253 000 - 2558000514
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control command not supported
/dev/sdi:
> === START OF INFORMATION SECTION ===
> Device Model: ST2000VN000-1H3164
> Serial Number: W1H25JXM
> LU WWN Device Id: 5 000c50 06a406dab
> Firmware Version: SC42
> User Capacity: 2,000,398,934,016 bytes [2.00 TB]
> Sector Sizes: 512 bytes logical, 4096 bytes physical
> Rotation Rate: 5900 rpm
> Device is: Not in smartctl database [for details use: -P showall]
> ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
> SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
> Local Time is: Fri Mar 27 22:57:09 2015 EDT
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
> AAM feature is: Unavailable
> APM level is: 254 (maximum performance)
> Rd look-ahead is: Enabled
> Write cache is: Enabled
> ATA Security is: Disabled, NOT FROZEN [SEC1]
> Wt Cache Reorder: Enabled
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAGS VALUE WORST THRESH FAIL RAW_VALUE
> 1 Raw_Read_Error_Rate POSR-- 119 099 006 - 218566352
> 3 Spin_Up_Time PO---- 096 096 000 - 0
> 4 Start_Stop_Count -O--CK 100 100 020 - 21
> 5 Reallocated_Sector_Ct PO--CK 100 100 010 - 0
> 7 Seek_Error_Rate POSR-- 064 060 030 - 3082219
> 9 Power_On_Hours -O--CK 085 085 000 - 13146
> 10 Spin_Retry_Count PO--C- 100 100 097 - 0
> 12 Power_Cycle_Count -O--CK 100 100 020 - 21
> 184 End-to-End_Error -O--CK 100 100 099 - 0
> 187 Reported_Uncorrect -O--CK 100 100 000 - 0
> 188 Command_Timeout -O--CK 100 100 000 - 0
> 189 High_Fly_Writes -O-RCK 050 050 000 - 50
> 190 Airflow_Temperature_Cel -O---K 064 052 045 - 36 (Min/Max 36/38)
> 191 G-Sense_Error_Rate -O--CK 100 100 000 - 0
> 192 Power-Off_Retract_Count -O--CK 100 100 000 - 21
> 193 Load_Cycle_Count -O--CK 100 100 000 - 21
> 194 Temperature_Celsius -O---K 036 048 000 - 36 (0 16 0 0 0)
> 197 Current_Pending_Sector -O--C- 100 100 000 - 0
> 198 Offline_Uncorrectable ----C- 100 100 000 - 0
> 199 UDMA_CRC_Error_Count -OSRCK 200 200 000 - 0
> ||||||_ K auto-keep
> |||||__ C event count
> ||||___ R error rate
> |||____ S speed/performance
> ||_____ O updated online
> |______ P prefailure warning
> SCT Error Recovery Control:
> Read: 1 (0.1 seconds)
> Write: 1 (0.1 seconds)
So. You have eight devices that need to make a raid6, and you have no
order information. You have two devices with pending errors that cannot
help us without role #s.
First, you need to deal with the timeout mismatch problem. Only two of
your devices support ERC, so you will need to set long driver timeouts.
Some reading:
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133665797115876&w=2
http://marc.info/?l=linux-raid&m=142504030927143&w=2
As for the latter link, I haven't tested that. When I needed such
features myself, I just put the appropriate commands into rc.local.
Since then, I've retired all of my non-raid-rated drives.
Next, you need to run numerous "mdadm --create --assume-clean" attempts
to figure out your device role order. You have 8-factorial permutations
to try (40,320). /dev/sdc and /dev/sdd have pending errors, so leave
them out (use "missing" in their places).
Your only info from the original post that shows all of the necessary
device characteristics is this:
> /dev/sdj1:
> Magic : a92b4efc
> Version : 1.1
> Feature Map : 0x2
> Array UUID : 15d2158f:5cf74d95:fd7f5607:0e447573
> Name : fermmy-server:2000 (local to host fermmy-server)
> Creation Time : Fri Apr 22 01:12:07 2011
> Raid Level : raid6
> Raid Devices : 8
>
> Avail Dev Size : 3907026816 (1863.02 GiB 2000.40 GB)
> Array Size : 11721080448 (11178.09 GiB 12002.39 GB)
> Data Offset : 304 sectors
> Super Offset : 0 sectors
> Recovery Offset : 2441891840 sectors
> State : clean
> Device UUID : eee3ae0e:f594fdba:58e19113:bc196464
>
> Update Time : Mon Jan 5 00:30:41 2015
> Checksum : 7a5a498d - correct
> Events : 42912
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Device Role : Active device 4
> Array State : A.AAAAAA ('A' == active, '.' == missing)
Note that the data offset is 304. Some of your devices reported a data
offset of 264. None of the reports were from original undisturbed
devices, so we really don't know what offset is correct. "mdadm --add"
will use that mdadm version's offset if it can.
I suggest you try to re-establish the distro you used at the time (April
2011) in a VM and create some test arrays with its version of mdadm to
get the offset to try first.
You then need to create a script that will perform the necessary "mdadm
--create --assume-clean" operations, followed by an "fsck -n" of the
device each time to see how messed up it is. Each attempt into its own
log file, so you can see (by size) which attempts were "cleanest".
Inspect the "best" log files manually to see what was found. With 40k
permutations, you may need to work out some grepping that will help
identify bad from possibly good.
If none of them come up relatively clean, try again with your next best
guess on the offset.
Good luck!
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* [PATCH 0/7] drivers: Use bool function return values true/false not 1/0
From: Joe Perches @ 2015-03-30 17:43 UTC (permalink / raw)
To: linux-kernel, dm-devel, linux-raid, platform-driver-x86, netdev,
linux-pm, linux-serial
Cc: dri-devel
Joe Perches (7):
drm: Use bool function return values of true/false not 1/0
dm_table: Use bool function return values of true/false not 1/0
genwqe: Use bool function return values of true/false not 1/0
wmi: Use bool function return values of true/false not 1/0
ssb: Use bool function return values of true/false not 1/0
thermal: Use bool function return values of true/false not 1/0
serial: kgdb_nmi: Use bool function return values of true/false not
1/0
drivers/gpu/drm/ast/ast_post.c | 2 +-
drivers/gpu/drm/vmwgfx/vmwgfx_fence.c | 2 +-
drivers/md/dm-table.c | 20 ++++++++++----------
drivers/misc/genwqe/card_base.h | 2 +-
drivers/platform/x86/wmi.c | 4 ++--
drivers/ssb/driver_gige.c | 2 +-
drivers/thermal/thermal_core.h | 2 +-
drivers/tty/serial/kgdb_nmi.c | 6 +++---
8 files changed, 20 insertions(+), 20 deletions(-)
--
2.1.2
^ permalink raw reply
* [PATCH 2/7] dm_table: Use bool function return values of true/false not 1/0
From: Joe Perches @ 2015-03-30 17:43 UTC (permalink / raw)
To: linux-kernel, Alasdair Kergon, Mike Snitzer, dm-devel, Neil Brown
Cc: linux-raid
In-Reply-To: <cover.1427736972.git.joe@perches.com>
Use the normal return values for bool functions
Signed-off-by: Joe Perches <joe@perches.com>
---
drivers/md/dm-table.c | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)
diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
index e0f618b..d9b00b8 100644
--- a/drivers/md/dm-table.c
+++ b/drivers/md/dm-table.c
@@ -1336,14 +1336,14 @@ static bool dm_table_supports_flush(struct dm_table *t, unsigned flush)
continue;
if (ti->flush_supported)
- return 1;
+ return true;
if (ti->type->iterate_devices &&
ti->type->iterate_devices(ti, device_flush_capable, &flush))
- return 1;
+ return true;
}
- return 0;
+ return false;
}
static bool dm_table_discard_zeroes_data(struct dm_table *t)
@@ -1356,10 +1356,10 @@ static bool dm_table_discard_zeroes_data(struct dm_table *t)
ti = dm_table_get_target(t, i++);
if (ti->discard_zeroes_data_unsupported)
- return 0;
+ return false;
}
- return 1;
+ return true;
}
static int device_is_nonrot(struct dm_target *ti, struct dm_dev *dev,
@@ -1405,10 +1405,10 @@ static bool dm_table_all_devices_attribute(struct dm_table *t,
if (!ti->type->iterate_devices ||
!ti->type->iterate_devices(ti, func, NULL))
- return 0;
+ return false;
}
- return 1;
+ return true;
}
static int device_not_write_same_capable(struct dm_target *ti, struct dm_dev *dev,
@@ -1465,14 +1465,14 @@ static bool dm_table_supports_discards(struct dm_table *t)
continue;
if (ti->discards_supported)
- return 1;
+ return true;
if (ti->type->iterate_devices &&
ti->type->iterate_devices(ti, device_discard_capable, NULL))
- return 1;
+ return true;
}
- return 0;
+ return false;
}
void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,
--
2.1.2
^ permalink raw reply related
* Re: [PATCH 2/7] dm_table: Use bool function return values of true/false not 1/0
From: Mike Snitzer @ 2015-03-30 17:51 UTC (permalink / raw)
To: Joe Perches
Cc: linux-kernel, Alasdair Kergon, dm-devel, Neil Brown, linux-raid
In-Reply-To: <e89210c7bf1423deef3cf16b618cd314307261db.1427736972.git.joe@perches.com>
On Mon, Mar 30 2015 at 1:43pm -0400,
Joe Perches <joe@perches.com> wrote:
> Use the normal return values for bool functions
>
> Signed-off-by: Joe Perches <joe@perches.com>
Applied, thanks.
^ permalink raw reply
* [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Shaohua Li @ 2015-03-30 22:25 UTC (permalink / raw)
To: neilb, dan.j.williams, linux-raid; +Cc: songliubraving, Kernel-team
This is my attempt to fix raid5/6 write hole issue, it's not for merge
yet, I post it out for comments. Any comments and suggestions are
welcome!
Thanks,
Shaohua
We expect a completed raid5/6 stack with reliability and high
performance. Currently raid5/6 has 2 issues:
1. read-modify-write for small size IO. To fix this issue, a cache layer
above raid5/6 can be used to aggregate write to full stripe write.
2. write hole issue. A write log below raid5/6 can fix the issue.
We plan to use a SSD to fix the two issues. Here we just fix the write
hole issue.
1. We don't try to fix the issues together. A cache layer will do write
acceleration. A log layer will fix write hole. The seperation will
simplify things a lot.
2. Current assumption is flashcache/bcache will be used as the cache
layer. If they don't work well, we can fix them or add a simple cache
layer for raid write aggregation later. We also assume cache layer will
absorb write, so log doesn't worry about write latency.
3. For log, write will hit to log disk first, then raid disks, and
finally IO completion is reported. An optimal way is to report IO
completion just after IO hits to log disk to cut write latency. But in
that way, read path need query log disk and increase complexity. And
since we don't worry about write latency, we choose a simple soltuion.
This will be revisited if there is performance issue.
This design isn't intrusive for raid5/6. Actully only very few changes
of existing code is required.
Log looks like jbd. Stripe IO to raid disks will be written to log disk
first in atomic way. Several stripe IO will consist a transaction. If
all stripes of a transaction are finished, the tranaction can be
checkpoint.
Basic logic of raid 5/6 write will be:
1. normal raid5/6 steps for a stripe (fetch data, calculate checksum,
and etc). log hooks to ops_run_io.
2. stripe is added to a transaction. Write stripe data to log disk (metadata
block, stripe data)
3. write commit block to log disk
4. flush log disk cache.
5. stripe is logged now and normal stripe handling continues
Transaction checkpoint process:
1. all stripes of a transaction are finished
2. flush disk cache of all raid disks
3. change log super to reflect new log checkpoint position
4. WRITE_FUA log super
metadata, data and commit block IO can run in the meaning time, as
checksum will be used to make sure their data is correct (like jbd2).
Log IO doesn't wait 5s to start like jbd, instead the IO will start
every time a metadata block is full. This can cut some latency.
Disk layout:
|super|metadata|data|metadata| data ... |commitdata|metadata|data| ... |commitdata|
super, metadata, commit will use one block
This is an initial version, which works but a lot of stuffes are
missing:
1. error handling
2. log recovery and impact to raid resync (don't need resync anymore)
3. utility changes
The big question is how we report log disk. In this patch, I simply use
a spare disk for testing. We need a new raid disk role for log disk.
Signed-off-by: Shaohua Li <shli@fb.com>
---
drivers/md/Makefile | 2 +-
drivers/md/raid5-log.c | 1017 ++++++++++++++++++++++++++++++++++++++++
drivers/md/raid5.c | 42 +-
drivers/md/raid5.h | 16 +
include/uapi/linux/raid/md_p.h | 64 +++
5 files changed, 1136 insertions(+), 5 deletions(-)
create mode 100644 drivers/md/raid5-log.c
diff --git a/drivers/md/Makefile b/drivers/md/Makefile
index a2da532..a0dee4c 100644
--- a/drivers/md/Makefile
+++ b/drivers/md/Makefile
@@ -16,7 +16,7 @@ dm-cache-mq-y += dm-cache-policy-mq.o
dm-cache-cleaner-y += dm-cache-policy-cleaner.o
dm-era-y += dm-era-target.o
md-mod-y += md.o bitmap.o
-raid456-y += raid5.o
+raid456-y += raid5.o raid5-log.o
# Note: link order is important. All raid personalities
# and must come before md.o, as they each initialise
diff --git a/drivers/md/raid5-log.c b/drivers/md/raid5-log.c
new file mode 100644
index 0000000..d27317f
--- /dev/null
+++ b/drivers/md/raid5-log.c
@@ -0,0 +1,1017 @@
+#include <linux/kernel.h>
+#include <linux/wait.h>
+#include <linux/blkdev.h>
+#include <linux/crc32.h>
+#include <linux/raid/md_p.h>
+#include "md.h"
+#include "raid5.h"
+
+struct r5log_journal {
+ struct mddev *mddev;
+ struct md_rdev *rdev;
+ struct page *super_page;
+
+ u64 transaction_id; /* next tid for transaction */
+ u64 checkpoint_transaction_id;
+
+ u32 block_size;
+ u32 block_sector_shift;
+ u64 total_blocks;
+ u64 first_block;
+ u64 last_block;
+
+ u64 last_checkpoint;
+ u64 log_start; /* next transaction starts here */
+
+ u8 data_checksum_type;
+ u8 meta_checksum_type;
+
+ u8 uuid[16];
+
+ struct list_head transaction_list;
+ int transaction_cnt;
+ struct r5log_transaction *current_trans;
+
+ struct list_head pending_stripes;
+ spinlock_t stripes_lock;
+
+ struct md_thread *thread;
+ unsigned long last_commit_time;
+
+ struct mutex jlock;
+
+ int aborted;
+ int do_commit;
+ int do_discard;
+};
+
+struct r5log_io_control {
+ struct r5log_transaction *trans;
+ struct page *meta_page;
+ struct page *commit_page;
+ int tag_index;
+
+ struct bio_list bios;
+ struct bio *current_bio;
+ atomic_t refcnt;
+};
+
+struct r5log_transaction {
+ struct r5log_journal *journal;
+ u64 transaction_id;
+ struct list_head stripe_list;
+ int stripe_cnt;
+ struct r5log_io_control *current_io;
+ atomic_t io_pending;
+ atomic_t stripe_pending;
+ u64 log_start; /* block */
+ u64 log_end; /* block */
+ struct list_head next_trans;
+
+ int state;
+ wait_queue_head_t wait_state;
+
+ struct mutex tlock;
+
+ struct work_struct flush_work;
+};
+
+enum {
+ /* transaction accepts new IO */
+ TRANSACTION_RUNNING = 0,
+ /* transaction is frozen. commit block IO is running */
+ TRANSACTION_COMMITED = 1,
+ /* transaction IO is finished, FLUSH request is running */
+ TRANSACTION_FLUSHING = 2,
+ /*
+ * transaction FLUSH request is finished, stripes are considered
+ * on-disk now, those stripes start hitting to raid disks
+ * */
+ TRANSACTION_STRIPE_RUNNING = 3,
+ /*
+ * Stripes of transaction hit to raid disks, transaction space can be
+ * reclaimed
+ * */
+ TRANSACTION_CHECKPOINT_READY = 4,
+};
+
+#define MAX_TAG_PER_BLOCK(journal) ((journal->block_size - \
+ sizeof(struct r5log_meta_header)) / \
+ sizeof(struct r5log_meta_block_tag))
+#define PAGE_TO_BLOCKS(journal) (PAGE_SIZE / journal->block_size)
+#define BLOCK_TO_SECTOR(b, journal) ((b) << journal->block_sector_shift)
+
+#define TRANSACTION_MAX_SIZE (2 * 1024 * 1024)
+#define TRANSACTION_MAX_STRIPES 128
+#define TRANSACTION_TIMEOUT (5 * HZ)
+
+static void r5log_journal_thread(struct md_thread *thread);
+static int r5log_do_commit(struct r5log_transaction *trans);
+static int r5log_do_checkpoint(struct r5log_journal *journal, u64 blocks);
+
+static u32 r5log_calculate_checksum(struct r5log_journal *journal, u32 crc,
+ void *buf, ssize_t size, bool data)
+{
+ if (journal->data_checksum_type != R5LOG_CHECKSUM_CRC32)
+ BUG();
+ if (journal->meta_checksum_type != R5LOG_CHECKSUM_CRC32)
+ BUG();
+ return crc32_le(crc, buf, size);
+}
+
+static int r5log_read_super(struct r5log_journal *journal)
+{
+ struct md_rdev *rdev = journal->rdev;
+ struct r5log_super_block *sb_blk;
+ struct page *page = journal->super_page;
+ u32 crc = ~0, stored_crc;
+
+ if (!sync_page_io(rdev, 0, PAGE_SIZE, page, READ, false))
+ return -EIO;
+
+ sb_blk = kmap_atomic(page);
+
+ if (le32_to_cpu(sb_blk->version) != RAID5_LOG_VERSION ||
+ le32_to_cpu(sb_blk->header.magic) != RAID5_LOG_MAGIC ||
+ le32_to_cpu(sb_blk->header.type) != R5LOG_TYPE_SUPER ||
+ le64_to_cpu(sb_blk->header.position) != 0)
+ goto error;
+
+ journal->checkpoint_transaction_id =
+ le64_to_cpu(sb_blk->header.transaction_id);
+ journal->transaction_id = journal->checkpoint_transaction_id + 1;
+
+ journal->block_size = le32_to_cpu(sb_blk->block_size);
+ journal->block_sector_shift = ilog2(journal->block_size >> 9);
+
+ /* Only support this stripe size right now */
+ if (le32_to_cpu(sb_blk->stripe_size) != PAGE_SIZE)
+ goto error;
+ if (journal->block_size > PAGE_SIZE)
+ goto error;
+
+ if (sb_blk->meta_checksum_type >= R5LOG_CHECKSUM_NR ||
+ sb_blk->data_checksum_type >= R5LOG_CHECKSUM_NR)
+ goto error;
+ journal->meta_checksum_type = sb_blk->meta_checksum_type;
+ journal->data_checksum_type = sb_blk->data_checksum_type;
+
+ stored_crc = le32_to_cpu(sb_blk->header.checksum);
+ sb_blk->header.checksum = 0;
+ crc = r5log_calculate_checksum(journal, ~0,
+ sb_blk, journal->block_size, false);
+ crc = r5log_calculate_checksum(journal, crc,
+ sb_blk->uuid, sizeof(sb_blk->uuid), false);
+ if (crc != stored_crc)
+ goto error;
+
+ if (memcmp(journal->uuid, sb_blk->uuid, sizeof(journal->uuid)))
+ goto error;
+
+ journal->first_block = le64_to_cpu(sb_blk->first_block);
+ if (journal->first_block != 1)
+ goto error;
+ journal->total_blocks = le64_to_cpu(sb_blk->total_blocks);
+ journal->last_block = journal->first_block + journal->total_blocks;
+ journal->last_checkpoint = le64_to_cpu(sb_blk->last_checkpoint);
+ kunmap_atomic(sb_blk);
+
+ return 0;
+error:
+ kunmap_atomic(sb_blk);
+ return -EINVAL;
+}
+
+static int r5log_write_super(struct r5log_journal *journal)
+{
+ struct r5log_super_block *sb_blk;
+ u32 crc;
+ struct timespec now = current_kernel_time();
+
+ sb_blk = kmap_atomic(journal->super_page);
+ sb_blk->header.checksum = 0;
+ sb_blk->header.transaction_id =
+ cpu_to_le64(journal->checkpoint_transaction_id);
+ sb_blk->last_checkpoint =
+ cpu_to_le64(journal->last_checkpoint);
+ sb_blk->update_time_sec = cpu_to_le64(now.tv_sec);
+ sb_blk->update_time_nsec = cpu_to_le64(now.tv_nsec);
+
+ crc = r5log_calculate_checksum(journal, ~0,
+ sb_blk, journal->block_size, false);
+ crc = r5log_calculate_checksum(journal, crc,
+ sb_blk->uuid, sizeof(sb_blk->uuid), false);
+ sb_blk->header.checksum = cpu_to_le32(crc);
+ kunmap_atomic(sb_blk);
+
+ if (sync_page_io(journal->rdev, 0, journal->block_size,
+ journal->super_page, WRITE_FUA, false))
+ return 0;
+ return -EIO;
+}
+
+static int r5log_recover_journal(struct r5log_journal *journal)
+{
+ return 0;
+}
+
+#define DBG 1
+#if DBG
+void r5log_fake_super(struct md_rdev *rdev)
+{
+ struct page *page = alloc_page(GFP_KERNEL|__GFP_ZERO);
+ struct r5log_super_block *sb_blk;
+ u32 crc;
+
+#define BLKSIZE 4096
+ sb_blk = kmap_atomic(page);
+ sb_blk->header.magic = cpu_to_le32(RAID5_LOG_MAGIC);
+ sb_blk->header.type = cpu_to_le32(R5LOG_TYPE_SUPER);
+ sb_blk->header.transaction_id = cpu_to_le32(0x111);
+ sb_blk->version = cpu_to_le32(RAID5_LOG_VERSION);
+ sb_blk->stripe_size = cpu_to_le32(PAGE_SIZE);
+ sb_blk->block_size = cpu_to_le32(BLKSIZE);
+ sb_blk->total_blocks = rdev->sectors * 512 / BLKSIZE;
+ sb_blk->first_block = 1;
+ sb_blk->last_checkpoint = 0;
+ sb_blk->meta_checksum_type = R5LOG_CHECKSUM_CRC32;
+ sb_blk->data_checksum_type = R5LOG_CHECKSUM_CRC32;
+ memcpy(sb_blk->uuid, rdev->mddev->uuid, sizeof(sb_blk->uuid));
+
+ crc = crc32_le(~0, sb_blk, BLKSIZE);
+ crc = crc32_le(crc, sb_blk->uuid, sizeof(sb_blk->uuid));
+ sb_blk->header.checksum = crc;
+ kunmap_atomic(sb_blk);
+
+ sync_page_io(rdev, 0, BLKSIZE, page, WRITE, false);
+ __free_page(page);
+}
+#endif
+
+struct r5log_journal *r5log_load_journal(struct md_rdev *rdev)
+{
+ struct r5log_journal *journal;
+
+#if DBG
+ r5log_fake_super(rdev);
+#endif
+
+ journal = kzalloc(sizeof(*journal), GFP_KERNEL);
+ if (!journal)
+ return NULL;
+
+ journal->super_page = alloc_page(GFP_KERNEL);
+ if (!journal->super_page)
+ goto err_page;
+
+ journal->mddev = rdev->mddev;
+ journal->rdev = rdev;
+ memcpy(journal->uuid, rdev->mddev->uuid, sizeof(journal->uuid));
+
+ INIT_LIST_HEAD(&journal->transaction_list);
+ INIT_LIST_HEAD(&journal->pending_stripes);
+ spin_lock_init(&journal->stripes_lock);
+ mutex_init(&journal->jlock);
+
+ if (r5log_read_super(journal))
+ goto err_super;
+
+ journal->do_discard = blk_queue_discard(bdev_get_queue(rdev->bdev));
+
+ if (journal->last_checkpoint != 0) {
+ if (r5log_recover_journal(journal))
+ goto err_super;
+ }
+
+ journal->log_start = 1;
+ journal->last_checkpoint = 1;
+
+ journal->last_commit_time = jiffies;
+ journal->thread = md_register_thread(r5log_journal_thread,
+ journal->mddev, "journal");
+ journal->thread->timeout = TRANSACTION_TIMEOUT;
+
+ return journal;
+err_super:
+ __free_page(journal->super_page);
+err_page:
+ kfree(journal);
+ return NULL;
+}
+
+int r5log_flush_journal(struct r5log_journal *journal)
+{
+ mutex_lock(&journal->jlock);
+ r5log_do_checkpoint(journal, -1);
+ mutex_unlock(&journal->jlock);
+ return 0;
+}
+
+void r5log_free_journal(struct r5log_journal *journal)
+{
+ r5log_flush_journal(journal);
+ md_unregister_thread(&journal->thread);
+
+ __free_page(journal->super_page);
+ kfree(journal);
+}
+
+static u64 r5log_ring_size(struct r5log_journal *journal, u64 start,
+ u64 end)
+{
+ return (journal->total_blocks + end - start) %
+ journal->total_blocks;
+}
+
+static bool r5log_has_room(struct r5log_journal *journal,
+ u64 log_end, int data_blocks)
+{
+ int tags = data_blocks * journal->block_size / PAGE_SIZE;
+ /* data + commit + meta + possible hole */
+ int blocks = data_blocks + 1 +
+ tags / MAX_TAG_PER_BLOCK(journal) + 1 +
+ PAGE_SIZE / journal->block_size - 1;
+
+ return journal->total_blocks - r5log_ring_size(journal,
+ journal->last_checkpoint, log_end)
+ >= blocks + 1;
+}
+
+static int r5log_wait_for_space(struct r5log_transaction *trans,
+ int data_blocks)
+{
+ BUG_ON(!mutex_is_locked(&trans->tlock));
+
+ if (r5log_has_room(trans->journal, trans->log_end, data_blocks))
+ return 0;
+ mutex_unlock(&trans->tlock);
+ r5log_do_checkpoint(trans->journal, data_blocks);
+ return -EAGAIN;
+}
+
+static bool r5log_check_and_freeze_current_transaction(
+ struct r5log_journal *journal)
+{
+ struct r5log_transaction *trans = journal->current_trans;
+
+ BUG_ON(!mutex_is_locked(&journal->jlock));
+
+ /*
+ * r5log_do_commit doesn't do this because journal->jlock
+ * isn't hold
+ **/
+ if (trans && trans->state >= TRANSACTION_COMMITED) {
+ journal->log_start = trans->log_end;
+ journal->current_trans = NULL;
+ return true;
+ }
+ return false;
+}
+
+static struct r5log_transaction *
+r5log_start_transaction(struct r5log_journal *journal, int data_blocks)
+{
+ struct r5log_transaction *trans;
+
+ mutex_lock(&journal->jlock);
+ /* FIXME: if transaction_cnt >= xxx, do checkpoint */
+again:
+ trans = journal->current_trans;
+ if (!trans) {
+ trans = kzalloc(sizeof(*trans), GFP_NOIO | __GFP_REPEAT);
+ trans->journal = journal;
+ trans->transaction_id = journal->transaction_id;
+ atomic_set(&trans->io_pending, 1);
+ atomic_set(&trans->stripe_pending, 1);
+ INIT_LIST_HEAD(&trans->stripe_list);
+ trans->log_start = journal->log_start;
+ trans->log_end = journal->log_start;
+ trans->state = TRANSACTION_RUNNING;
+ init_waitqueue_head(&trans->wait_state);
+ mutex_init(&trans->tlock);
+
+ list_add_tail(&trans->next_trans, &journal->transaction_list);
+ journal->transaction_id++;
+ journal->current_trans = trans;
+ journal->transaction_cnt++;
+ }
+ mutex_lock(&trans->tlock);
+
+ if (r5log_check_and_freeze_current_transaction(journal)) {
+ mutex_unlock(&trans->tlock);
+ goto again;
+ }
+
+ if (r5log_wait_for_space(trans, data_blocks))
+ goto again;
+
+ mutex_unlock(&journal->jlock);
+ return trans;
+}
+
+static u64 r5log_transaction_size(struct r5log_transaction *trans)
+{
+ struct r5log_journal *journal = trans->journal;
+
+ return r5log_ring_size(journal, trans->log_start, trans->log_end);
+}
+
+static void r5log_end_transaction(struct r5log_transaction *trans)
+{
+ struct r5log_journal *journal = trans->journal;
+
+ /* trans is big, commit it */
+ if (r5log_transaction_size(trans) * journal->block_size >=
+ TRANSACTION_MAX_SIZE || trans->stripe_cnt >=
+ TRANSACTION_MAX_STRIPES)
+ r5log_do_commit(trans);
+ mutex_unlock(&trans->tlock);
+}
+
+static void r5log_transaction_finish_stripe(struct r5log_transaction *trans)
+{
+ if (!atomic_dec_and_test(&trans->stripe_pending))
+ return;
+ trans->state = TRANSACTION_CHECKPOINT_READY;
+ wake_up(&trans->wait_state);
+}
+
+static void r5log_restore_stripe_data(struct stripe_head *sh)
+{
+ int i;
+
+ for (i = 0; i < sh->disks; i++) {
+ void *addr;
+ if (!test_and_clear_bit(R5_Escaped, &sh->dev[i].flags))
+ continue;
+ addr = kmap_atomic(sh->dev[i].page);
+ *(__le32 *)addr = cpu_to_le32(RAID5_LOG_MAGIC);
+ kunmap_atomic(addr);
+ }
+}
+
+static void r5log_trans_flush_diskcache(struct work_struct *work)
+{
+ struct r5log_transaction *trans;
+ struct stripe_head *sh;
+
+ trans = container_of(work, struct r5log_transaction, flush_work);
+ blkdev_issue_flush(trans->journal->rdev->bdev, GFP_NOIO, NULL);
+ trans->state = TRANSACTION_STRIPE_RUNNING;
+ wake_up(&trans->wait_state);
+
+ while (!list_empty(&trans->stripe_list)) {
+ sh = list_first_entry(&trans->stripe_list, struct stripe_head,
+ log_list);
+ list_del_init(&sh->log_list);
+ r5log_restore_stripe_data(sh);
+ atomic_inc(&trans->stripe_pending);
+ set_bit(STRIPE_HANDLE, &sh->state);
+ release_stripe(sh);
+ }
+ r5log_transaction_finish_stripe(trans);
+}
+
+static void r5log_transaction_finish_io(struct r5log_transaction *trans)
+{
+ if (!atomic_dec_and_test(&trans->io_pending))
+ return;
+
+ /*
+ * FIXME: we can release_stripe before flush disk cache if there is no
+ * FUA or FLUSH request
+ * */
+ trans->state = TRANSACTION_FLUSHING;
+ INIT_WORK(&trans->flush_work, r5log_trans_flush_diskcache);
+ schedule_work(&trans->flush_work);
+}
+
+static void r5log_put_io_control(struct r5log_io_control *io)
+{
+ struct r5log_transaction *trans = io->trans;
+
+ if (!atomic_dec_and_test(&io->refcnt))
+ return;
+ __free_page(io->meta_page);
+ if (io->commit_page)
+ __free_page(io->commit_page);
+ kfree(io);
+
+ r5log_transaction_finish_io(trans);
+}
+
+static void r5log_end_io(struct bio *bio, int error)
+{
+ struct r5log_io_control *io = bio->bi_private;
+
+ if (error) {
+ io->trans->journal->aborted = 1;
+ printk(KERN_ERR"r5log IO error\n");
+ }
+ r5log_put_io_control(io);
+ bio_put(bio);
+}
+
+static int r5log_submit_io(struct r5log_transaction *trans)
+{
+ struct r5log_journal *journal = trans->journal;
+ struct r5log_io_control *io = trans->current_io;
+ struct bio *bio;
+ struct r5log_meta_block *meta;
+ u32 crc;
+ u32 tag_flags;
+
+ meta = kmap_atomic(io->meta_page);
+ tag_flags = le32_to_cpu(meta->tags[io->tag_index - 1].flags);
+ tag_flags |= R5LOG_TAG_FLAG_LAST_TAG;
+ meta->tags[io->tag_index - 1].flags = cpu_to_le32(tag_flags);
+
+ crc = r5log_calculate_checksum(journal, ~0, meta,
+ journal->block_size, false);
+ crc = r5log_calculate_checksum(journal, crc, journal->uuid,
+ sizeof(journal->uuid), false);
+ meta->header.checksum = cpu_to_le32(crc);
+ kunmap_atomic(meta);
+
+ while ((bio = bio_list_pop(&io->bios)))
+ submit_bio(WRITE, bio);
+ r5log_put_io_control(io);
+ trans->current_io = NULL;
+ return 0;
+}
+
+int r5log_io_add_page(struct r5log_transaction *trans, struct page *page,
+ ssize_t size)
+{
+ struct r5log_journal *journal = trans->journal;
+ struct r5log_io_control *current_io = trans->current_io;
+ sector_t pos = BLOCK_TO_SECTOR(trans->log_end, journal);
+ struct bio *bio;
+ int blocks = size / journal->block_size;
+
+ /*
+ * if PAGE_SIZE > block size, there might be one block size hole at the
+ * tail. Recover code should be aware of this
+ * */
+ if (trans->log_end + blocks > journal->last_block) {
+ pos = BLOCK_TO_SECTOR(journal->first_block, journal);
+ goto allocate_bio;
+ }
+
+retry:
+ bio = current_io->current_bio;
+ if (!bio)
+ goto allocate_bio;
+
+ if (!bio_add_page(bio, page, size, 0))
+ goto allocate_bio;
+
+ if (trans->log_end + blocks > journal->last_block)
+ trans->log_end = journal->first_block;
+ trans->log_end += blocks;
+ return 0;
+allocate_bio:
+ current_io->current_bio = NULL;
+ bio = bio_alloc_mddev(GFP_NOIO,
+ MAX_TAG_PER_BLOCK(trans->journal), trans->journal->mddev);
+
+ bio->bi_bdev = journal->rdev->bdev;
+ bio->bi_iter.bi_sector = pos + journal->rdev->data_offset;
+ bio->bi_private = current_io;
+ bio->bi_end_io = r5log_end_io;
+
+ bio_list_add(¤t_io->bios, bio);
+ atomic_inc(¤t_io->refcnt);
+ current_io->current_bio = bio;
+ goto retry;
+}
+
+static int r5log_transaction_get_tag(struct r5log_transaction *trans)
+{
+ struct r5log_io_control *current_io;
+ struct r5log_meta_block *meta;
+
+ current_io = trans->current_io;
+ if (current_io && current_io->tag_index >=
+ MAX_TAG_PER_BLOCK(trans->journal))
+ r5log_submit_io(trans);
+
+ current_io = trans->current_io;
+ if (current_io)
+ return 0;
+
+ current_io = kmalloc(sizeof(*current_io), GFP_NOIO|__GFP_REPEAT);
+ if (!current_io)
+ return -ENOMEM;
+ current_io->meta_page = alloc_page(GFP_NOIO|__GFP_ZERO|__GFP_REPEAT);
+ if (!current_io->meta_page) {
+ kfree(current_io);
+ return -ENOMEM;
+ }
+
+ current_io->trans = trans;
+ current_io->commit_page = NULL;
+ current_io->tag_index = 0;
+ bio_list_init(¤t_io->bios);
+ current_io->current_bio = NULL;
+ atomic_set(¤t_io->refcnt, 1);
+
+ atomic_inc(&trans->io_pending);
+ trans->current_io = current_io;
+
+ r5log_io_add_page(trans, current_io->meta_page,
+ trans->journal->block_size);
+
+ meta = kmap_atomic(current_io->meta_page);
+ meta->header.magic = cpu_to_le32(RAID5_LOG_MAGIC);
+ meta->header.type = cpu_to_le32(R5LOG_TYPE_META);
+ meta->header.transaction_id =
+ cpu_to_le64(trans->transaction_id);
+ /* we never hit journal->first_block */
+ meta->header.position = cpu_to_le64(trans->log_end - 1);
+ kunmap_atomic(meta);
+ return 0;
+}
+
+int r5log_add_stripe_page(struct r5log_transaction *trans,
+ struct stripe_head *sh, int disk_index)
+{
+ struct r5log_meta_block_tag *tag;
+ struct r5log_meta_block *meta;
+ struct r5dev *rdev;
+ struct r5log_io_control *current_io;
+ u32 crc;
+
+ rdev = &sh->dev[disk_index];
+
+ if (r5log_transaction_get_tag(trans))
+ return -ENOMEM;
+ current_io = trans->current_io;
+
+ crc = r5log_calculate_checksum(trans->journal, rdev->log_checksum,
+ trans->journal->uuid, sizeof(trans->journal->uuid), true);
+ crc = r5log_calculate_checksum(trans->journal, crc,
+ &trans->transaction_id, sizeof(trans->transaction_id), true);
+
+ meta = kmap_atomic(current_io->meta_page);
+ tag = &meta->tags[current_io->tag_index];
+ if (test_bit(R5_Discard, &rdev->flags))
+ tag->flags |= R5LOG_TAG_FLAG_DISCARD;
+ if (test_bit(R5_Escaped, &rdev->flags))
+ tag->flags |= R5LOG_TAG_FLAG_ESCAPED;
+ tag->flags = cpu_to_le32(tag->flags);
+ tag->disk_index = cpu_to_le32(disk_index);
+ tag->disk_sector = cpu_to_le64(sh->sector);
+ kunmap_atomic(meta);
+
+ current_io->tag_index++;
+ r5log_io_add_page(trans, rdev->page, PAGE_SIZE);
+ return 0;
+}
+
+static int r5log_journal_one_stripe(struct r5log_journal *journal,
+ struct stripe_head *sh)
+{
+ struct r5log_transaction *trans;
+ int i;
+ int data_disks = 0;
+
+ for (i = 0; i < sh->disks; i++) {
+ if (!test_bit(R5_Wantwrite, &sh->dev[i].flags))
+ continue;
+ if (test_bit(R5_Discard, &sh->dev[i].flags))
+ continue;
+ data_disks++;
+ }
+
+ trans = r5log_start_transaction(journal,
+ data_disks * PAGE_TO_BLOCKS(journal));
+ if (!trans)
+ goto abort_trans;
+ for (i = 0; i < sh->disks; i++) {
+ if (!test_bit(R5_Wantwrite, &sh->dev[i].flags))
+ continue;
+ if (r5log_add_stripe_page(trans, sh, i))
+ goto abort_trans;
+ }
+
+ list_add_tail(&sh->log_list, &trans->stripe_list);
+ trans->stripe_cnt++;
+ sh->log_trans = trans;
+
+ r5log_end_transaction(trans);
+ return 0;
+abort_trans:
+ journal->aborted = 1;
+ printk(KERN_ERR"r5log journal failed\n");
+ r5log_restore_stripe_data(sh);
+ /* skip journal is still ok, but lose the protection */
+ set_bit(STRIPE_HANDLE, &sh->state);
+ release_stripe(sh);
+ return -ENOMEM;
+}
+
+static void r5log_journal_thread(struct md_thread *thread)
+{
+ struct mddev *mddev = thread->mddev;
+ struct r5conf *conf = mddev->private;
+ struct r5log_journal *journal = conf->journal;
+ struct r5log_transaction *trans;
+ struct stripe_head *sh;
+ LIST_HEAD(stripe_list);
+ struct blk_plug plug;
+ bool did_something;
+
+ blk_start_plug(&plug);
+again:
+ did_something = false;
+ spin_lock(&journal->stripes_lock);
+ list_splice_init(&journal->pending_stripes, &stripe_list);
+ spin_unlock(&journal->stripes_lock);
+
+ while (!list_empty(&stripe_list)) {
+ sh = list_first_entry(&stripe_list, struct stripe_head, log_list);
+ list_del_init(&sh->log_list);
+ r5log_journal_one_stripe(journal, sh);
+ did_something = true;
+ }
+
+ if (journal->do_commit || time_after(jiffies,
+ journal->last_commit_time + TRANSACTION_TIMEOUT)) {
+ mutex_lock(&journal->jlock);
+ r5log_check_and_freeze_current_transaction(journal);
+ trans = journal->current_trans;
+ if (trans)
+ mutex_lock(&trans->tlock);
+ mutex_unlock(&journal->jlock);
+ journal->do_commit = 0;
+ journal->last_commit_time = jiffies;
+
+ if (trans) {
+ r5log_do_commit(trans);
+ mutex_unlock(&trans->tlock);
+
+ did_something = true;
+ }
+ }
+
+ if (did_something)
+ goto again;
+ blk_finish_plug(&plug);
+}
+
+
+int r5log_write_stripe(struct r5log_journal *journal, struct stripe_head *sh)
+{
+ int i;
+ int write_disks = 0;
+
+ /* fulls stripe write doesn't have write hole issue */
+ if (sh->log_trans || test_bit(STRIPE_FULL_WRITE, &sh->state))
+ return -EAGAIN;
+ if (journal->aborted)
+ return -ENODEV;
+
+ for (i = 0; i < sh->disks; i++) {
+ void *addr;
+ if (!test_bit(R5_Wantwrite, &sh->dev[i].flags))
+ continue;
+ write_disks++;
+ if (test_bit(R5_Discard, &sh->dev[i].flags)) {
+ sh->dev[i].log_checksum = ~0;
+ continue;
+ }
+ addr = kmap_atomic(sh->dev[i].page);
+
+ if (*(__le32 *)addr == cpu_to_le32(RAID5_LOG_MAGIC)) {
+ *(u32 *)addr = 0;
+ set_bit(R5_Escaped, &sh->dev[i].flags);
+ }
+ sh->dev[i].log_checksum = r5log_calculate_checksum(journal,
+ ~0, addr, PAGE_SIZE, true);
+ kunmap_atomic(addr);
+ }
+ if (!write_disks)
+ return -EINVAL;
+
+ atomic_inc(&sh->count);
+
+ /*
+ * this function shouldn't wait on journal related stuff, journal
+ * checkpoint might wait for stripes to handle
+ **/
+ spin_lock(&journal->stripes_lock);
+ list_add_tail(&sh->log_list, &journal->pending_stripes);
+ spin_unlock(&journal->stripes_lock);
+ md_wakeup_thread(journal->thread);
+
+ return 0;
+}
+
+static void r5log_wait_transaction(struct r5log_transaction *trans,
+ int state)
+{
+ wait_event(trans->wait_state, trans->state >= state);
+}
+
+static int r5log_do_commit(struct r5log_transaction *trans)
+{
+ struct r5log_journal *journal = trans->journal;
+ struct r5log_io_control *current_io;
+ struct page *page;
+ struct r5log_commit_block *commit;
+ struct timespec now = current_kernel_time();
+ u32 crc;
+
+ BUG_ON(!mutex_is_locked(&trans->tlock));
+
+ if (trans->state >= TRANSACTION_COMMITED)
+ return 0;
+
+ trans->stripe_cnt = 0;
+
+ journal->last_commit_time = jiffies;
+
+ current_io = trans->current_io;
+ /* The transaction hasn't done anything yet */
+ if (!current_io)
+ BUG();
+
+ page = alloc_page(GFP_NOIO|__GFP_ZERO|__GFP_REPEAT);
+ r5log_io_add_page(trans, page, journal->block_size);
+ current_io->commit_page = page;
+
+ commit = kmap_atomic(page);
+ commit->header.magic = cpu_to_le32(RAID5_LOG_MAGIC);
+ commit->header.type = cpu_to_le32(R5LOG_TYPE_COMMIT);
+ commit->header.transaction_id =
+ cpu_to_le64(trans->transaction_id);
+ commit->commit_sec = cpu_to_le64(now.tv_sec);
+ commit->commit_nsec = cpu_to_le64(now.tv_nsec);
+ commit->header.position = cpu_to_le64(trans->log_end - 1);
+
+ crc = r5log_calculate_checksum(journal, ~0, commit,
+ journal->block_size, false);
+ crc = r5log_calculate_checksum(journal, crc, journal->uuid,
+ sizeof(journal->uuid), false);
+ commit->header.checksum = cpu_to_le32(crc);
+ kunmap_atomic(commit);
+
+ r5log_submit_io(trans);
+ trans->state = TRANSACTION_COMMITED;
+ r5log_transaction_finish_io(trans);
+
+ return 0;
+}
+
+static void r5log_start_commit(struct r5log_journal *journal)
+{
+ journal->do_commit = 1;
+ md_wakeup_thread(journal->thread);
+}
+
+static void r5log_disks_flush_end(struct bio *bio, int err)
+{
+ struct completion *io_complete = bio->bi_private;
+
+ complete(io_complete);
+ bio_put(bio);
+}
+
+static void r5log_flush_all_disks(struct r5log_journal *journal)
+{
+ struct mddev *mddev = journal->mddev;
+ struct bio *bi;
+ DECLARE_COMPLETION_ONSTACK(io_complete);
+
+ bi = bio_alloc_mddev(GFP_NOIO, 0, mddev);
+ bi->bi_end_io = r5log_disks_flush_end;
+ bi->bi_private = &io_complete;
+
+ md_flush_request(mddev, bi);
+
+ wait_for_completion_io(&io_complete);
+}
+
+static void r5log_discard_blocks(struct r5log_journal *journal,
+ u64 start, u64 end)
+{
+ if (!journal->do_discard)
+ return;
+ if (start < end) {
+ blkdev_issue_discard(journal->rdev->bdev,
+ BLOCK_TO_SECTOR(start, journal),
+ BLOCK_TO_SECTOR(end - start, journal),
+ GFP_NOIO, 0);
+ } else {
+ blkdev_issue_discard(journal->rdev->bdev,
+ BLOCK_TO_SECTOR(start, journal),
+ BLOCK_TO_SECTOR(journal->last_block - start, journal),
+ GFP_NOIO, 0);
+ blkdev_issue_discard(journal->rdev->bdev,
+ BLOCK_TO_SECTOR(journal->first_block, journal),
+ BLOCK_TO_SECTOR(end - journal->first_block, journal),
+ GFP_NOIO, 0);
+ }
+}
+
+static int r5log_do_checkpoint(struct r5log_journal *journal,
+ u64 blocks)
+{
+ u64 cp_block = journal->last_checkpoint;
+ u64 cp_tid = journal->checkpoint_transaction_id;
+ struct r5log_transaction *trans;
+ bool enough = false;
+ u64 freed = 0;
+
+ BUG_ON(!mutex_is_locked(&journal->jlock));
+
+ trans = journal->current_trans;
+ if (trans && r5log_ring_size(journal, journal->last_checkpoint,
+ trans->log_start) < blocks) {
+ mutex_lock(&trans->tlock);
+ r5log_do_commit(trans);
+ mutex_unlock(&trans->tlock);
+ }
+
+ r5log_check_and_freeze_current_transaction(journal);
+
+ while (!list_empty(&journal->transaction_list)) {
+ trans = list_first_entry(&journal->transaction_list,
+ struct r5log_transaction, next_trans);
+
+ if (cp_tid + 1 != trans->transaction_id ||
+ cp_block != trans->log_start) {
+ BUG();
+ }
+
+ if (!enough)
+ r5log_wait_transaction(trans,
+ TRANSACTION_CHECKPOINT_READY);
+ else if (trans->state < TRANSACTION_CHECKPOINT_READY)
+ break;
+ cp_block = trans->log_end;
+ cp_tid = trans->transaction_id;
+
+ freed += r5log_transaction_size(trans);
+ list_del(&trans->next_trans);
+ kfree(trans);
+
+ journal->transaction_cnt--;
+
+ if (freed > blocks)
+ enough = true;
+ }
+ /* Nothing happened */
+ if (journal->checkpoint_transaction_id == cp_tid)
+ return 0;
+
+ r5log_discard_blocks(journal, journal->last_checkpoint, cp_block);
+
+ r5log_flush_all_disks(journal);
+
+ journal->checkpoint_transaction_id = cp_tid;
+ journal->last_checkpoint = cp_block;
+ /* teardown the journal */
+ if (blocks == -1)
+ journal->last_checkpoint = 0;
+ /* FIXME: trim the range for SSD */
+ r5log_write_super(journal);
+ return 0;
+}
+
+void r5log_stripe_write_finished(struct stripe_head *sh)
+{
+ struct r5log_transaction *trans = sh->log_trans;
+
+ sh->log_trans = NULL;
+ r5log_transaction_finish_stripe(trans);
+}
+
+void r5log_flush_transaction(struct r5log_journal *journal)
+{
+ if (journal->aborted)
+ return;
+ r5log_start_commit(journal);
+}
+
+int r5log_handle_flush_request(struct mddev *mddev, struct bio *bio)
+{
+ struct r5conf *conf = mddev->private;
+ struct r5log_journal *journal = conf->journal;
+
+ if (journal->aborted)
+ return -ENODEV;
+
+ /*
+ * we flush disk cache and then release_stripe. So if a stripe is
+ * finished, the disk cache is flushed already, so we don't need flush
+ * again
+ * */
+ if (bio->bi_iter.bi_size == 0) {
+ bio_endio(bio, 0);
+ return 0;
+ }
+ bio->bi_rw &= ~REQ_FLUSH;
+ return -EAGAIN;
+}
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index cd2f96b..153cb9c 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -409,7 +409,7 @@ static int release_stripe_list(struct r5conf *conf,
return count;
}
-static void release_stripe(struct stripe_head *sh)
+void release_stripe(struct stripe_head *sh)
{
struct r5conf *conf = sh->raid_conf;
unsigned long flags;
@@ -741,6 +741,10 @@ static void ops_run_io(struct stripe_head *sh, struct stripe_head_state *s)
might_sleep();
+ if (!sh->log_trans && conf->journal) {
+ if (!r5log_write_stripe(conf->journal, sh))
+ return;
+ }
for (i = disks; i--; ) {
int rw;
int replace_only = 0;
@@ -4111,6 +4115,8 @@ static void handle_stripe(struct stripe_head *sh)
md_wakeup_thread(conf->mddev->thread);
}
+ if (s.return_bi && sh->log_trans)
+ r5log_stripe_write_finished(sh);
return_io(s.return_bi);
clear_bit_unlock(STRIPE_ACTIVE, &sh->state);
@@ -4650,8 +4656,17 @@ static void make_request(struct mddev *mddev, struct bio * bi)
bool do_prepare;
if (unlikely(bi->bi_rw & REQ_FLUSH)) {
- md_flush_request(mddev, bi);
- return;
+ int ret = -ENODEV;
+ if (conf->journal) {
+ ret = r5log_handle_flush_request(mddev, bi);
+ if (!ret)
+ return;
+ }
+ if (!conf->journal || ret == -ENODEV) {
+ md_flush_request(mddev, bi);
+ return;
+ }
+ BUG_ON(ret != -EAGAIN);
}
md_write_start(mddev, bi);
@@ -5283,6 +5298,8 @@ static void raid5_do_work(struct work_struct *work)
spin_unlock_irq(&conf->device_lock);
blk_finish_plug(&plug);
+ if (conf->journal)
+ r5log_flush_transaction(conf->journal);
pr_debug("--- raid5worker inactive\n");
}
@@ -5354,6 +5371,9 @@ static void raid5d(struct md_thread *thread)
async_tx_issue_pending_all();
blk_finish_plug(&plug);
+ if (conf->journal)
+ r5log_flush_transaction(conf->journal);
+
pr_debug("--- raid5d inactive\n");
}
@@ -5740,6 +5760,9 @@ static void raid5_free_percpu(struct r5conf *conf)
static void free_conf(struct r5conf *conf)
{
+ if (conf->journal)
+ r5log_free_journal(conf->journal);
+
free_thread_groups(conf);
shrink_stripes(conf);
raid5_free_percpu(conf);
@@ -5811,7 +5834,7 @@ static struct r5conf *setup_conf(struct mddev *mddev)
{
struct r5conf *conf;
int raid_disk, memory, max_disks;
- struct md_rdev *rdev;
+ struct md_rdev *rdev, *spare_rdev = NULL;
struct disk_info *disk;
char pers_name[6];
int i;
@@ -5914,6 +5937,12 @@ static struct r5conf *setup_conf(struct mddev *mddev)
rdev_for_each(rdev, mddev) {
raid_disk = rdev->raid_disk;
+ if (raid_disk < 0) {
+ char b[BDEVNAME_SIZE];
+ spare_rdev = rdev;
+ printk(KERN_INFO "using device %s as log\n",
+ bdevname(rdev->bdev, b));
+ }
if (raid_disk >= max_disks
|| raid_disk < 0)
continue;
@@ -5973,6 +6002,9 @@ static struct r5conf *setup_conf(struct mddev *mddev)
goto abort;
}
+ if (spare_rdev)
+ conf->journal = r5log_load_journal(spare_rdev);
+
return conf;
abort:
@@ -6873,6 +6905,8 @@ static void raid5_quiesce(struct mddev *mddev, int state)
lock_all_device_hash_locks_irq(conf));
conf->quiesce = 1;
unlock_all_device_hash_locks_irq(conf);
+ if (conf->journal)
+ r5log_flush_journal(conf->journal);
/* allow reshape to continue */
wake_up(&conf->wait_for_overlap);
break;
diff --git a/drivers/md/raid5.h b/drivers/md/raid5.h
index 983e18a..93cfb5a 100644
--- a/drivers/md/raid5.h
+++ b/drivers/md/raid5.h
@@ -198,6 +198,7 @@ struct stripe_head {
struct hlist_node hash;
struct list_head lru; /* inactive_list or handle_list */
struct llist_node release_list;
+ struct list_head log_list;
struct r5conf *raid_conf;
short generation; /* increments with every
* reshape */
@@ -215,6 +216,7 @@ struct stripe_head {
spinlock_t stripe_lock;
int cpu;
struct r5worker_group *group;
+ struct r5log_transaction *log_trans;
/**
* struct stripe_operations
* @target - STRIPE_OP_COMPUTE_BLK target
@@ -236,6 +238,7 @@ struct stripe_head {
struct bio *toread, *read, *towrite, *written;
sector_t sector; /* sector of this page */
unsigned long flags;
+ u32 log_checksum;
} dev[1]; /* allocated with extra space depending of RAID geometry */
};
@@ -300,6 +303,7 @@ enum r5dev_flags {
*/
R5_Discard, /* Discard the stripe */
R5_SkipCopy, /* Don't copy data from bio to stripe cache */
+ R5_Escaped, /* data is escaped by log */
};
/*
@@ -495,6 +499,8 @@ struct r5conf {
struct r5worker_group *worker_groups;
int group_cnt;
int worker_cnt_per_group;
+
+ struct r5log_journal *journal;
};
/*
@@ -560,4 +566,14 @@ static inline int algorithm_is_DDF(int layout)
extern void md_raid5_kick_device(struct r5conf *conf);
extern int raid5_set_cache_size(struct mddev *mddev, int size);
+
+void release_stripe(struct stripe_head *sh);
+
+struct r5log_journal *r5log_load_journal(struct md_rdev *rdev);
+int r5log_flush_journal(struct r5log_journal *journal);
+void r5log_free_journal(struct r5log_journal *journal);
+int r5log_write_stripe(struct r5log_journal *journal, struct stripe_head *sh);
+int r5log_handle_flush_request(struct mddev *mddev, struct bio *bio);
+void r5log_stripe_write_finished(struct stripe_head *sh);
+void r5log_flush_transaction(struct r5log_journal *journal);
#endif
diff --git a/include/uapi/linux/raid/md_p.h b/include/uapi/linux/raid/md_p.h
index 49f4210..84abf0c 100644
--- a/include/uapi/linux/raid/md_p.h
+++ b/include/uapi/linux/raid/md_p.h
@@ -305,4 +305,68 @@ struct mdp_superblock_1 {
|MD_FEATURE_RECOVERY_BITMAP \
)
+/* all disk position of below struct start from rdev->start_offset */
+struct r5log_meta_header {
+ __le32 magic;
+ __le32 type;
+ __le32 checksum; /* checksum(metadata block + uuid) */
+ __le32 zero_padding;
+ __le64 transaction_id;
+ __le64 position; /* block number the meta is written */
+} __attribute__ ((__packed__));
+
+#define RAID5_LOG_VERSION 0x1
+#define RAID5_LOG_MAGIC 0x28670308
+
+enum {
+ R5LOG_TYPE_META = 0,
+ R5LOG_TYPE_COMMIT = 1,
+ R5LOG_TYPE_SUPER = 2,
+};
+
+struct r5log_super_block {
+ struct r5log_meta_header header;
+ __le32 version;
+ __le32 stripe_size;
+ __le32 block_size;
+ __le64 total_blocks;
+ __le64 first_block;
+ __le64 last_checkpoint;
+ __le64 update_time_sec;
+ __le64 update_time_nsec;
+ __u8 meta_checksum_type;
+ __u8 data_checksum_type;
+ __u8 uuid[16];
+} __attribute__ ((__packed__));
+
+enum {
+ R5LOG_CHECKSUM_CRC32 = 0,
+ R5LOG_CHECKSUM_NR = 1,
+};
+
+struct r5log_meta_block_tag {
+ __le32 flags;
+ __le32 disk_index;
+ __le32 checksum; /* checksum(data + uuid + transaction_id) */
+ __le32 zero_padding;
+ __le64 disk_sector; /* raid disk sector */
+} __attribute__ ((__packed__));
+
+enum {
+ R5LOG_TAG_FLAG_ESCAPED = 1 << 0, /* data is escaped */
+ R5LOG_TAG_FLAG_LAST_TAG = 1 << 1, /* last tag in the meta block */
+ R5LOG_TAG_FLAG_DISCARD = 1 << 2, /* a discard request, no data */
+};
+
+struct r5log_meta_block {
+ struct r5log_meta_header header;
+ struct r5log_meta_block_tag tags[];
+} __attribute__ ((__packed__));
+
+struct r5log_commit_block {
+ struct r5log_meta_header header;
+ __le64 commit_sec;
+ __le64 commit_nsec;
+} __attribute__ ((__packed__));
+
#endif
--
1.8.1
^ permalink raw reply related
* Two system lockups, Magic SysKeys works, RAID6 resyncs on reboot
From: Mark Knecht @ 2015-03-31 15:00 UTC (permalink / raw)
To: Linux-RAID
Hi,
In the past week, having moved from a gentoo-sources-3.14.34 kernel
to a gentoo-sources-3.18.10 kernel, I've had two full system lockups
on what has been a very stable, day-to-day machine I use for my
everyday work at home 7 days a week. I've run this machine in this
basic configuration for about 5 years without anything like this
happening. Two in a week is alarming. The Linux part of recovering
hasn't been a big problem but making sure my Win 7 VMs are OK takes
quite awhile. I was running 3 VMs at the time of this morning's
lockup, one of them being new yesterday so it wasn't involved in the
first lockup. The other two RAID1s on the system are not mounted or
effected by this event.
I'm going to try to supply all the stuff I see asked for in most
threads like this. If I miss something please let me know. Attached
are the initramfs setup stuff, mdadm -D & mdadm -E outputs, smartctl
--all for all 5 drives, the output of lsdrv run as root and the
contents of /var/log/messages around the time of the reboot. (7AM)
By my quick read I think the disks look OK but maybe someone with a
deeper skillset can double check that and point out anything I'm
missing. smartctl --long testing is run on each drive once a week
(probably overkill) and other than a couple of system shutdowns while
in progress the smartctl data looks OK to me.
Backups are done every Sunday and the system is functional right
now (I'm writing this email) so I can do more
investigation/backups/study as needed. I don't think there's much
chance of anything really important being lost yet but I'd like to try
and understand what's causing this.
The lockups seem to effect the apps but not the mouse or keyboard.
I can switch to the console and the machine responds to the magic
syskey sequence REISUB. When the machine reboots there's no obvious
problem however the main system drive, a 5-drive RAID6, is in the
process of resyncing by the time I log in and look around. The first
time this happened the resync completed with no problems and this one
is proceeding OK so far.
c2RAID6 ~ # eix mdadm
[I] sys-fs/mdadm
Available versions: 3.2.6-r1^t 3.3.1-r2^t{tbz2} ~3.3.2^t {static}
Installed versions: 3.3.1-r2^t{tbz2}(15:11:51 11/14/14)(static)
Homepage: http://neil.brown.name/blog/mdadm
Description: A useful tool for running RAID systems - it
can be used as a replacement for the raidtools
c2RAID6 ~ #
c2RAID6 ~ # cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md10 : active raid1 sda10[0] sdb10[1]
1430135296 blocks super 1.2 [2/2] [UU]
md4 : active raid1 sdb3[1] sda3[0]
524156736 blocks super 1.2 [2/2] [UU]
md3 : active raid6 sdc3[9] sdg3[5] sdf3[6] sde3[7] sdd3[8]
1452264480 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5] [UUUUU]
[====>................] resync = 22.8% (110608304/484088160)
finish=78.5min speed=79206K/sec
unused devices: <none>
c2RAID6 ~ #
Again, nothing appears lost at this time but if there's a way to
find out what's causing this I'd very much like to learn how to do
that.
Thanks,
Mark
1) The disk is assembled in my (by hand) initramfs which is built into
the kernel.
c2RAID6 ~ # cat /usr/src/initramfs_init_new.sh
#!/bin/busybox sh
rescue_shell() {
echo "-----"
echo "!!!!!"
echo "Something went wrong."
echo "Dropping into busybox"
echo "!!!!!"
echo "-----"
busybox --install -s
exec /bin/sh
}
/bin/busybox --install -s
# Mount the /proc and /sys filesystems.
mount -t proc none /proc
mount -t sysfs none /sys
mount -t devtmpfs none /dev
# Do your stuff here.
echo " "
echo "This init script assembles the RAID6 root as /dev/md3"
echo "mounts it as rootfs and, if there are no problems,"
echo "boots it up, else calls rescue_shell/busybox"
echo " "
mdadm --assemble /dev/md3
# Mount the root filesystem.
mount -o ro /dev/md3 /realroot || rescue_shell
# Clean up.
umount /dev
umount /proc
umount /sys
# Boot the real thing.
exec switch_root /realroot /sbin/init
c2RAID6 ~ #
c2RAID6 ~ # cat /usr/src/mdadm_initramfs.conf
ARRAY /dev/md/3 metadata=1.2 UUID=de47f991:86d98467:0637635b:9c6d0591
ARRAY /dev/md/4 metadata=1.2 UUID=c3e562a4:6871679e:2d83518e:fefd237f
ARRAY /dev/md/10 metadata=1.2 UUID=d9624704:b29e1a8e:148b47be:cabad736
c2RAID6 ~ #
c2RAID6 ~ # mdadm -D /dev/md3
/dev/md3:
Version : 1.2
Creation Time : Thu Dec 30 17:40:50 2010
Raid Level : raid6
Array Size : 1452264480 (1384.99 GiB 1487.12 GB)
Used Dev Size : 484088160 (461.66 GiB 495.71 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Update Time : Tue Mar 31 07:11:01 2015
State : active, resyncing
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 16K
Resync Status : 15% complete
Name : c2stable:3
UUID : de47f991:86d98467:0637635b:9c6d0591
Events : 38369
Number Major Minor RaidDevice State
9 8 35 0 active sync /dev/sdc3
8 8 51 1 active sync /dev/sdd3
7 8 67 2 active sync /dev/sde3
6 8 83 3 active sync /dev/sdf3
5 8 99 4 active sync /dev/sdg3
c2RAID6 ~ #
c2RAID6 ~ # mdadm -E /dev/sdc3
/dev/sdc3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : de47f991:86d98467:0637635b:9c6d0591
Name : c2stable:3
Creation Time : Thu Dec 30 17:40:50 2010
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 968176345 (461.66 GiB 495.71 GB)
Array Size : 1452264480 (1384.99 GiB 1487.12 GB)
Used Dev Size : 968176320 (461.66 GiB 495.71 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=25 sectors
State : active
Device UUID : da06e9db:a8a3dffe:feec1851:d0cbdde6
Update Time : Tue Mar 31 07:26:02 2015
Checksum : 909e27f6 - correct
Events : 38372
Layout : left-symmetric
Chunk Size : 16K
Device Role : Active device 0
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
c2RAID6 ~ #
c2RAID6 ~ # mdadm -E /dev/sdd3
/dev/sdd3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : de47f991:86d98467:0637635b:9c6d0591
Name : c2stable:3
Creation Time : Thu Dec 30 17:40:50 2010
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 968176345 (461.66 GiB 495.71 GB)
Array Size : 1452264480 (1384.99 GiB 1487.12 GB)
Used Dev Size : 968176320 (461.66 GiB 495.71 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=25 sectors
State : active
Device UUID : f3dd1d1e:8fb1a087:1060acfe:b8c9a537
Update Time : Tue Mar 31 07:26:02 2015
Checksum : 59ef7dee - correct
Events : 38372
Layout : left-symmetric
Chunk Size : 16K
Device Role : Active device 1
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
c2RAID6 ~ #
c2RAID6 ~ # mdadm -E /dev/sde3
/dev/sde3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : de47f991:86d98467:0637635b:9c6d0591
Name : c2stable:3
Creation Time : Thu Dec 30 17:40:50 2010
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 968176345 (461.66 GiB 495.71 GB)
Array Size : 1452264480 (1384.99 GiB 1487.12 GB)
Used Dev Size : 968176320 (461.66 GiB 495.71 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=25 sectors
State : active
Device UUID : 6bbd236b:77c22abc:da7c8fc2:e29e601e
Update Time : Tue Mar 31 07:26:02 2015
Checksum : 861d6041 - correct
Events : 38372
Layout : left-symmetric
Chunk Size : 16K
Device Role : Active device 2
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
c2RAID6 ~ #
c2RAID6 ~ # mdadm -E /dev/sdf3
/dev/sdf3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : de47f991:86d98467:0637635b:9c6d0591
Name : c2stable:3
Creation Time : Thu Dec 30 17:40:50 2010
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 968176345 (461.66 GiB 495.71 GB)
Array Size : 1452264480 (1384.99 GiB 1487.12 GB)
Used Dev Size : 968176320 (461.66 GiB 495.71 GB)
Data Offset : 2048 sectors
Super Offset : 8 sectors
Unused Space : before=1968 sectors, after=25 sectors
State : active
Device UUID : 8a361a4a:ac34f9cc:cee70fc3:5a8bed2f
Update Time : Tue Mar 31 07:26:02 2015
Checksum : 87efa300 - correct
Events : 38372
Layout : left-symmetric
Chunk Size : 16K
Device Role : Active device 3
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
c2RAID6 ~ #
c2RAID6 ~ # mdadm -E /dev/sdg3
/dev/sdg3:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x0
Array UUID : de47f991:86d98467:0637635b:9c6d0591
Name : c2stable:3
Creation Time : Thu Dec 30 17:40:50 2010
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 968176320 (461.66 GiB 495.71 GB)
Array Size : 1452264480 (1384.99 GiB 1487.12 GB)
Data Offset : 1392 sectors
Super Offset : 8 sectors
Unused Space : before=1312 sectors, after=0 sectors
State : active
Device UUID : c56ca951:928864db:c2746525:37716872
Update Time : Tue Mar 31 07:26:02 2015
Checksum : 42ba9d48 - correct
Events : 38372
Layout : left-symmetric
Chunk Size : 16K
Device Role : Active device 4
Array State : AAAAA ('A' == active, '.' == missing, 'R' == replacing)
c2RAID6 ~ #
c2RAID6 ~ # smartctl --all /dev/sdc
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.18.10-gentoo] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital RE3 Serial ATA
Device Model: WDC WD5002ABYS-02B1B0
Serial Number: WD-WCASYA846988
LU WWN Device Id: 5 0014ee 2042c3477
Firmware Version: 02.03B03
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.5, 3.0 Gb/s
Local Time is: Tue Mar 31 07:30:21 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an
interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 9480) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 112) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 239 229 021 Pre-fail
Always - 1033
4 Start_Stop_Count 0x0032 098 098 000 Old_age
Always - 2477
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 075 075 000 Old_age
Always - 18331
10 Spin_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 098 098 000 Old_age
Always - 2474
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 48
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 2428
194 Temperature_Celsius 0x0022 092 080 000 Old_age
Always - 55
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 18262 -
# 2 Extended offline Completed without error 00% 18180 -
# 3 Extended offline Completed without error 00% 18095 -
# 4 Extended offline Completed without error 00% 18011 -
# 5 Extended offline Completed without error 00% 17930 -
# 6 Extended offline Completed without error 00% 17848 -
# 7 Extended offline Completed without error 00% 17682 -
# 8 Extended offline Completed without error 00% 17592 -
# 9 Extended offline Completed without error 00% 17505 -
#10 Extended offline Completed without error 00% 17423 -
#11 Extended offline Completed without error 00% 17340 -
#12 Extended offline Completed without error 00% 17260 -
#13 Extended offline Completed without error 00% 17173 -
#14 Extended offline Completed without error 00% 17128 -
#15 Extended offline Completed without error 00% 17032 -
#16 Extended offline Completed without error 00% 16942 -
#17 Extended offline Completed without error 00% 16865 -
#18 Extended offline Completed without error 00% 16657 -
#19 Extended offline Completed without error 00% 16581 -
#20 Extended offline Completed without error 00% 16489 -
#21 Extended offline Interrupted (host reset) 30% 16397 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
c2RAID6 ~ #
c2RAID6 ~ # smartctl --all /dev/sdd
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.18.10-gentoo] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital RE3 Serial ATA
Device Model: WDC WD5002ABYS-02B1B0
Serial Number: WD-WCASYA811977
LU WWN Device Id: 5 0014ee 2042c29e7
Firmware Version: 02.03B03
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.5, 3.0 Gb/s
Local Time is: Tue Mar 31 07:30:50 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an
interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 9480) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 112) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 239 228 021 Pre-fail
Always - 1033
4 Start_Stop_Count 0x0032 098 098 000 Old_age
Always - 2476
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 075 075 000 Old_age
Always - 18330
10 Spin_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 098 098 000 Old_age
Always - 2473
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 48
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 2427
194 Temperature_Celsius 0x0022 093 080 000 Old_age
Always - 54
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 18272 -
# 2 Extended offline Completed without error 00% 18195 -
# 3 Extended offline Completed without error 00% 18109 -
# 4 Extended offline Completed without error 00% 18023 -
# 5 Extended offline Completed without error 00% 17940 -
# 6 Extended offline Completed without error 00% 17859 -
# 7 Extended offline Completed without error 00% 17779 -
# 8 Extended offline Completed without error 00% 17692 -
# 9 Extended offline Completed without error 00% 17606 -
#10 Extended offline Interrupted (host reset) 30% 17516 -
#11 Extended offline Completed without error 00% 17437 -
#12 Extended offline Completed without error 00% 17352 -
#13 Extended offline Completed without error 00% 17183 -
#14 Extended offline Completed without error 00% 17137 -
#15 Extended offline Completed without error 00% 17045 -
#16 Extended offline Completed without error 00% 16956 -
#17 Extended offline Completed without error 00% 16876 -
#18 Extended offline Completed without error 00% 16794 -
#19 Extended offline Completed without error 00% 16668 -
#20 Extended offline Completed without error 00% 16592 -
#21 Extended offline Completed without error 00% 16501 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
c2RAID6 ~ #
c2RAID6 ~ # smartctl --all /dev/sde
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.18.10-gentoo] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital RE3 Serial ATA
Device Model: WDC WD5002ABYS-02B1B0
Serial Number: WD-WCASYA814051
LU WWN Device Id: 5 0014ee 259812251
Firmware Version: 02.03B03
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.5, 3.0 Gb/s
Local Time is: Tue Mar 31 07:31:19 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an
interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 9480) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 112) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 239 229 021 Pre-fail
Always - 1050
4 Start_Stop_Count 0x0032 098 098 000 Old_age
Always - 2476
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 075 075 000 Old_age
Always - 18329
10 Spin_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 098 098 000 Old_age
Always - 2473
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 47
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 2428
194 Temperature_Celsius 0x0022 095 084 000 Old_age
Always - 52
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 18286 -
# 2 Extended offline Completed without error 00% 18206 -
# 3 Extended offline Completed without error 00% 18119 -
# 4 Extended offline Completed without error 00% 18034 -
# 5 Extended offline Completed without error 00% 17953 -
# 6 Extended offline Completed without error 00% 17871 -
# 7 Extended offline Completed without error 00% 17790 -
# 8 Extended offline Completed without error 00% 17706 -
# 9 Extended offline Completed without error 00% 17617 -
#10 Extended offline Completed without error 00% 17446 -
#11 Extended offline Completed without error 00% 17362 -
#12 Extended offline Completed without error 00% 17279 -
#13 Extended offline Completed without error 00% 17195 -
#14 Extended offline Completed without error 00% 17061 -
#15 Extended offline Completed without error 00% 16968 -
#16 Extended offline Completed without error 00% 16885 -
#17 Extended offline Completed without error 00% 16678 -
#18 Extended offline Completed without error 00% 16512 -
#19 Extended offline Completed without error 00% 16423 -
#20 Extended offline Interrupted (host reset) 40% 16342 -
#21 Extended offline Completed without error 00% 16265 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
c2RAID6 ~ #
c2RAID6 ~ # smartctl --all /dev/sdf
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.18.10-gentoo] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital RE3 Serial ATA
Device Model: WDC WD5002ABYS-02B1B0
Serial Number: WD-WCASYA837674
LU WWN Device Id: 5 0014ee 2042bc993
Firmware Version: 02.03B03
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.5, 3.0 Gb/s
Local Time is: Tue Mar 31 07:31:53 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an
interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 9480) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 112) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 1
3 Spin_Up_Time 0x0027 239 229 021 Pre-fail
Always - 1033
4 Start_Stop_Count 0x0032 098 098 000 Old_age
Always - 2465
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 076 076 000 Old_age
Always - 18242
10 Spin_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 098 098 000 Old_age
Always - 2461
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 62
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 2402
194 Temperature_Celsius 0x0022 094 083 000 Old_age
Always - 53
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Interrupted (host reset) 10% 18135 -
# 2 Extended offline Completed without error 00% 18046 -
# 3 Extended offline Completed without error 00% 17962 -
# 4 Extended offline Completed without error 00% 17876 -
# 5 Extended offline Completed without error 00% 17719 -
# 6 Extended offline Completed without error 00% 17632 -
# 7 Extended offline Completed without error 00% 17541 -
# 8 Extended offline Completed without error 00% 17456 -
# 9 Extended offline Completed without error 00% 17372 -
#10 Extended offline Completed without error 00% 17291 -
#11 Extended offline Completed without error 00% 17204 -
#12 Extended offline Completed without error 00% 17124 -
#13 Extended offline Completed without error 00% 16991 -
#14 Extended offline Completed without error 00% 16896 -
#15 Extended offline Completed without error 00% 16808 -
#16 Extended offline Completed without error 00% 16727 -
#17 Extended offline Completed without error 00% 16646 -
#18 Extended offline Completed without error 00% 16608 -
#19 Extended offline Completed without error 00% 16526 -
#20 Extended offline Completed without error 00% 16440 -
#21 Extended offline Completed without error 00% 16350 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
c2RAID6 ~ #
c2RAID6 ~ # smartctl --all /dev/sdg
smartctl 6.3 2014-07-26 r3976 [x86_64-linux-3.18.10-gentoo] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital RE3 Serial ATA
Device Model: WDC WD5002ABYS-02B1B0
Serial Number: WD-WCASYA831311
LU WWN Device Id: 5 0014ee 259814854
Firmware Version: 02.03B03
User Capacity: 500,107,862,016 bytes [500 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.5, 3.0 Gb/s
Local Time is: Tue Mar 31 07:32:25 2015 PDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an
interrupting command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 9480) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 112) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
Always - 0
3 Spin_Up_Time 0x0027 239 229 021 Pre-fail
Always - 1050
4 Start_Stop_Count 0x0032 098 098 000 Old_age
Always - 2462
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail
Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age
Always - 0
9 Power_On_Hours 0x0032 076 076 000 Old_age
Always - 18241
10 Spin_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age
Always - 0
12 Power_Cycle_Count 0x0032 098 098 000 Old_age
Always - 2458
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age
Always - 49
193 Load_Cycle_Count 0x0032 200 200 000 Old_age
Always - 2412
194 Temperature_Celsius 0x0022 093 084 000 Old_age
Always - 54
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age
Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age
Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age
Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 18237 -
# 2 Extended offline Completed without error 00% 18160 -
# 3 Extended offline Completed without error 00% 18079 -
# 4 Extended offline Completed without error 00% 17993 -
# 5 Extended offline Completed without error 00% 17908 -
# 6 Extended offline Completed without error 00% 17828 -
# 7 Extended offline Completed without error 00% 17747 -
# 8 Extended offline Completed without error 00% 17667 -
# 9 Extended offline Completed without error 00% 17578 -
#10 Extended offline Completed without error 00% 17491 -
#11 Extended offline Completed without error 00% 17404 -
#12 Extended offline Completed without error 00% 17240 -
#13 Extended offline Completed without error 00% 17157 -
#14 Extended offline Completed without error 00% 17071 -
#15 Extended offline Completed without error 00% 17026 -
#16 Extended offline Completed without error 00% 16931 -
#17 Extended offline Completed without error 00% 16764 -
#18 Extended offline Completed without error 00% 16682 -
#19 Extended offline Completed without error 00% 16555 -
#20 Extended offline Interrupted (host reset) 40% 16475 -
#21 Extended offline Completed without error 00% 16388 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
c2RAID6 ~ #
c2RAID6 ~ # /home/mark/lsdrv
PCI [ahci] 03:00.0 SATA controller: Marvell Technology Group Ltd.
88SE9123 PCIe SATA 6.0 Gb/s controller (rev 11)
ââscsi 0:0:0:0 ATA WDC WD30EFRX-68E {WD-WCC4N0990937}
â ââsda: [8:0] Partitioned (gpt) 2.73t
â ââsda1: [8:1] (ext2) 200.00m {c0e7f8e7-0871-4b07-80da-ed0fae480491}
â ââsda2: [8:2] (swap) 6.00g {9f9d2c4a-6a47-481f-a5be-06b0bb8b0dca}
â ââsda3: [8:3] MD raid1 (0/2) 500.00g md4 clean in_sync
'c2RAID6:4' {c3e562a4-6871-679e-2d83-518efefd237f}
â â ââmd4: [9:4] (ext4) 499.87g 'RAID1root'
{08d7f067-ec60-4d80-ab28-39b3eff5896b}
â ââsda10: [8:10] MD raid1 (0/2) 1.33t md10 clean in_sync
'c2RAID6:videoBack' {d9624704-b29e-1a8e-148b-47becabad736}
â ââmd10: [9:10] (ext4) 1.33t 'VideoLib'
{3b25ecce-99f2-4e60-87fa-3c69f3a8331e}
ââscsi 1:0:0:0 ATA WDC WD30EFRX-68E {WD-WCC4N0959002}
â ââsdb: [8:16] Partitioned (gpt) 2.73t
â ââsdb1: [8:17] Empty/Unknown 200.00m
â ââsdb2: [8:18] (swap) 6.00g {307ecb02-80cf-4ff3-835b-a7b70a463282}
â ââsdb3: [8:19] MD raid1 (1/2) 500.00g md4 clean in_sync
'c2RAID6:4' {c3e562a4-6871-679e-2d83-518efefd237f}
â ââsdb10: [8:26] MD raid1 (1/2) 1.33t md10 clean in_sync
'c2RAID6:videoBack' {d9624704-b29e-1a8e-148b-47becabad736}
ââscsi 2:x:x:x [Empty]
ââscsi 3:x:x:x [Empty]
ââscsi 4:x:x:x [Empty]
ââscsi 5:x:x:x [Empty]
ââscsi 6:x:x:x [Empty]
PCI [ahci] 06:00.0 SATA controller: JMicron Technology Corp. JMB363
SATA/IDE Controller (rev 03)
ââscsi 8:x:x:x [Empty]
ââscsi 9:x:x:x [Empty]
PCI [ata_piix] 00:1f.2 IDE interface: Intel Corporation 82801JI (ICH10
Family) 4 port SATA IDE Controller #1
ââscsi 10:0:0:0 Optiarc DVD RW AD-7241S {Optiarc_DVD_RW_AD-7241S}
â ââsr0: [11:0] Empty/Unknown 1.00g
ââscsi 10:0:1:0 ATA WDC WD5002ABYS-0 {WD-WCASYA846988}
â ââsdc: [8:32] Partitioned (dos) 465.76g
â ââsdc1: [8:33] (ext2) 54.88m 'boot'
{4efd04f2-839f-4e11-a32d-8fa25129b541}
â ââsdc2: [8:34] (swap) 4.01g 'swapA'
{ed41217e-abf4-4c60-a985-c0420eb054f3}
â ââsdc3: [8:35] MD raid6 (0/5) 461.66g md3 active in_sync
'c2stable:3' {de47f991-86d9-8467-0637-635b9c6d0591}
â ââmd3: [9:3] (ext4) 1.35t 'RAID6root'
{80c3ebf0-d92f-415f-a6d7-1cdb1ceab593}
â ââMounted as /dev/md3 @ /
ââscsi 11:0:0:0 ATA WDC WD5002ABYS-0 {WD-WCASYA811977}
â ââsdd: [8:48] Partitioned (dos) 465.76g
â ââsdd1: [8:49] (ext2) 54.88m {b9dd1fb9-5c7b-459c-a7fc-ca8ebe032fcc}
â ââsdd2: [8:50] (swap) 4.01g 'swapB'
{2ea74bbb-8c1d-4a32-939d-512db4e40a54}
â ââsdd3: [8:51] MD raid6 (1/5) 461.66g md3 active in_sync
'c2stable:3' {de47f991-86d9-8467-0637-635b9c6d0591}
ââscsi 11:0:1:0 ATA WDC WD5002ABYS-0 {WD-WCASYA814051}
ââsde: [8:64] Partitioned (dos) 465.76g
ââsde1: [8:65] (ext2) 54.88m {27f276fa-6f62-4769-b9d7-deb511dfc60a}
ââsde2: [8:66] (swap) 4.01g 'swapC'
{0c6a1bd0-291e-4157-9b6a-67f7801c7a5e}
ââsde3: [8:67] MD raid6 (2/5) 461.66g md3 active in_sync
'c2stable:3' {de47f991-86d9-8467-0637-635b9c6d0591}
PCI [ata_piix] 00:1f.5 IDE interface: Intel Corporation 82801JI (ICH10
Family) 2 port SATA IDE Controller #2
ââscsi 12:0:0:0 ATA WDC WD5002ABYS-0 {WD-WCASYA837674}
â ââsdf: [8:80] Partitioned (dos) 465.76g
â ââsdf1: [8:81] Empty/Unknown 4.10g
â ââsdf3: [8:83] MD raid6 (3/5) 461.66g md3 active in_sync
'c2stable:3' {de47f991-86d9-8467-0637-635b9c6d0591}
ââscsi 13:0:0:0 ATA WDC WD5002ABYS-0 {WD-WCASYA831311}
ââsdg: [8:96] Partitioned (dos) 465.76g
ââsdg1: [8:97] (ext2) 4.10g {e7b2dbb0-df11-4bd4-902c-da2221b53e88}
ââsdg3: [8:99] MD raid6 (4/5) 461.66g md3 active in_sync
'c2stable:3' {de47f991-86d9-8467-0637-635b9c6d0591}
Other Block Devices
ââram0: [1:0] Empty/Unknown 16.00m
ââram1: [1:1] Empty/Unknown 16.00m
ââram2: [1:2] Empty/Unknown 16.00m
ââram3: [1:3] Empty/Unknown 16.00m
ââram4: [1:4] Empty/Unknown 16.00m
ââram5: [1:5] Empty/Unknown 16.00m
ââram6: [1:6] Empty/Unknown 16.00m
ââram7: [1:7] Empty/Unknown 16.00m
ââram8: [1:8] Empty/Unknown 16.00m
ââram9: [1:9] Empty/Unknown 16.00m
ââram10: [1:10] Empty/Unknown 16.00m
ââram11: [1:11] Empty/Unknown 16.00m
ââram12: [1:12] Empty/Unknown 16.00m
ââram13: [1:13] Empty/Unknown 16.00m
ââram14: [1:14] Empty/Unknown 16.00m
ââram15: [1:15] Empty/Unknown 16.00m
c2RAID6 ~ #
Mar 31 06:19:23 c2RAID6 kdm[2588]: :0[2588]:
pam_ck_connector(kde:session): nox11 mode, ignoring PAM_TTY :0
Mar 31 06:19:32 c2RAID6 dbus[2273]: [system] Activating service
name='org.freedesktop.UPower' (using servicehelper)
Mar 31 06:19:32 c2RAID6 dbus[2273]: [system] Successfully activated
service 'org.freedesktop.UPower'
Mar 31 06:19:32 c2RAID6 dbus[2273]: [system] Activating service
name='org.freedesktop.UDisks2' (using servicehelper)
Mar 31 06:19:33 c2RAID6 udisksd[3062]: udisks daemon version 2.1.3 starting
Mar 31 06:19:33 c2RAID6 dbus[2273]: [system] Successfully activated
service 'org.freedesktop.UDisks2'
Mar 31 06:19:33 c2RAID6 udisksd[3062]: Acquired the name
org.freedesktop.UDisks2 on the system message bus
Mar 31 06:19:33 c2RAID6 dbus[2273]: [system] Activating service
name='org.kde.powerdevil.backlighthelper' (using servicehelper)
Mar 31 06:19:33 c2RAID6 dbus[2273]: [system] Successfully activated
service 'org.kde.powerdevil.backlighthelper'
Mar 31 06:19:47 c2RAID6 polkitd[2954]: Registered Authentication Agent
for unix-session:/org/freedesktop/ConsoleKit/Session1 (system bus name
:1.19 [/usr/lib64/kde4/libexec/polkit-kde-authentication-agent-1],
object path /org/kde/PolicyKit1/AuthenticationAgent, locale
en_US.UTF-8)
Mar 31 06:19:59 c2RAID6 kernel: capability: warning: `VirtualBox' uses
32-bit capabilities (legacy support in use)
Mar 31 06:20:01 c2RAID6 cron[3442]: (root) CMD (test -x
/usr/sbin/run-crons && /usr/sbin/run-crons)
Mar 31 06:20:01 c2RAID6 cron[3441]: (root) CMD (updatedb)
Mar 31 06:20:08 c2RAID6 kernel: device eth0 entered promiscuous mode
Mar 31 06:30:01 c2RAID6 cron[5497]: (root) CMD (test -x
/usr/sbin/run-crons && /usr/sbin/run-crons)
Mar 31 06:40:01 c2RAID6 cron[5911]: (root) CMD (test -x
/usr/sbin/run-crons && /usr/sbin/run-crons)
Mar 31 06:50:01 c2RAID6 cron[7406]: (root) CMD (test -x
/usr/sbin/run-crons && /usr/sbin/run-crons)
Mar 31 07:01:14 c2RAID6 syslog-ng[2227]: syslog-ng starting up; version='3.6.2'
Mar 31 07:01:14 c2RAID6 acpid[2247]: starting up with netlink and the
input layer
Mar 31 07:01:14 c2RAID6 acpid[2247]: 1 rule loaded
Mar 31 07:01:14 c2RAID6 acpid[2247]: waiting for events: event logging is off
Mar 31 07:01:16 c2RAID6 kernel: sky2 0000:07:00.0 eth0: enabling interface
Mar 31 07:01:18 c2RAID6 kernel: sky2 0000:07:00.0 eth0: Link is up at
100 Mbps, full duplex, flow control both
Mar 31 07:01:19 c2RAID6 kernel: nvidia 0000:02:00.0: irq 37 for MSI/MSI-X
Mar 31 07:01:20 c2RAID6 kernel: nvidia 0000:04:00.0: irq 38 for MSI/MSI-X
Mar 31 07:01:19 c2RAID6 acpid[2247]: client connected from 2479[0:0]
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* multi-threaded raid5
From: stkoupr @ 2015-03-31 17:17 UTC (permalink / raw)
To: linux-raid
Hi,
We are testing write performance of md raid5 with SSD array and we are
hitting limit which I understand is caused by writes being handled in
single thread (md0_raid5 process is eating 100% of 1 CPU core when the
test is running). I understood that there is some work in progress which
should make write handling multi-threaded. I'd like to ask about the
status of this work, is it still in progress or planned?
Regards,
Prema
^ permalink raw reply
* Re: Two system lockups, Magic SysKeys works, RAID6 resyncs on reboot
From: Roman Mamedov @ 2015-03-31 20:30 UTC (permalink / raw)
To: Mark Knecht; +Cc: Linux-RAID
In-Reply-To: <CAK2H+efDuxdjmWZiOiJw+EK3wohY6FxRWgsm8rJc1j1+fniWig@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 490 bytes --]
On Tue, 31 Mar 2015 08:00:27 -0700
Mark Knecht <markknecht@gmail.com> wrote:
> md3 : active raid6 sdc3[9] sdg3[5] sdf3[6] sde3[7] sdd3[8]
> 1452264480 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5] [UUUUU]
> [====>................] resync = 22.8% (110608304/484088160)
> finish=78.5min speed=79206K/sec
Looks like you are running this without a write intent bitmap, why? It would
at least save you from a full resync each time.
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Dave Stevens @ 2015-03-31 21:41 UTC (permalink / raw)
To: Brad Campbell; +Cc: Roger Heflin, linux-raid
In-Reply-To: <5515FC86.7030402@fnarfbargle.com>
Quoting Brad Campbell <brad@fnarfbargle.com>:
> md12 has claimed sda2 and md10 has sdb&c. Try this
>
> mdadm --stop /dev/md12
> mdadm --stop /dev/md10
> mdadm --assemble --force /dev/md13 /dev/sd[abc]2
I owe you a beer, Brad! Now I only have to mount the LVM partition,
retrieve the Xen domains and rejig the management tools so I can do a
meaningful backup and restore but this is the bit I couldn't get my
head around.
Thanks to all who helped. Does anyone care to provide a reference to a
good intro to software RAID? This experience was educational all
right, but I would on the whole prefer to be more orderly and have a
good reference.
Dave
>
>
>
> On 28/03/15 08:37, Dave Stevens wrote:
>> Quoting Roger Heflin <rogerheflin@gmail.com>:
>>
>>> I don't know what to make of it.
>>>
>>> Everyone will ask you for this info:
>>> mdadm --examine /dev/sdc2 and the same output against
>>> /dev/sdb2
>>
>> ok, like this:
>>
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md12 : inactive sda2[0](S)
>> 732467520 blocks
>>
>> md10 : inactive sdc2[2](S) sdb2[1](S)
>> 1464935040 blocks
>>
>> unused devices: <none>
>>
>>
>> ------------------------------ reboot ----------------------
>>
>> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>>
>> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>>
>> # cat /proc/mdstat
>>
>> Personalities :
>>
>> md13 : inactive sdc2[2](S) sdb2[1](S)
>>
>> 1464935040 blocks
>>
>> unused devices : <none>
>>
>> # mkdir aa
>>
>> # mount /dev/md13 aa
>>
>> mount: /dev/md13: can't read superblock
>>
>>
>> Dave
>>
>>
>>
>>
>>>
>>> On Fri, Mar 27, 2015 at 12:33 PM, Dave Stevens <geek@uniserve.com> wrote:
>>>> From: Roger Heflin <rogerheflin@gmail.com>
>>>> To: Dave Stevens <geek@uniserve.com>
>>>> Cc: Linux RAID <linux-raid@vger.kernel.org>
>>>> Subject: Re: no good deed goes unpunished...
>>>> Date: Thu, 26 Mar 2015 19:04:45 -0500
>>>> Sender: linux-raid-owner@vger.kernel.org
>>>>
>>>> you should probably do cat /proc/mdstat
>>>>
>>>> It is likely that the livecd may have already tried to assemble it,
>>>> and that the busy is because it is already in use.
>>>>
>>>> If it is already assembled and you want to redo it first you will need
>>>> to stop the assembled array to be able to redo it.
>>>>
>>>> so this is what I did while running the live distro:
>>>>
>>>> # cat /proc/mdstat
>>>>
>>>> Personalities :
>>>>
>>>> md12 : inactive sda2[0](S)
>>>> 732467520 blocks
>>>>
>>>> md10 : inactive sdc2[2](S) sdb2[1](S)
>>>> 1464935040 blocks
>>>>
>>>> unused devices: <none>
>>>>
>>>>
>>>> It seemed to me that matters were needlessly complicated by possible
>>>> residual issues from my earlier unsuccessful attempts so I rebooted.
>>>>
>>>>
>>>> ------------------------------ reboot ----------------------
>>>>
>>>> # mdadm -A /dev/md13 /dev/sdb2 /dev/sdc2
>>>>
>>>> mdadm : /dev/md13 assembles from 1 drive - not enough to start the array
>>>>
>>>> # cat /proc/mdstat
>>>>
>>>> Personalities :
>>>>
>>>> md13 : inactive sdc2[2](S) sdb2[1](S)
>>>>
>>>> 1464935040 blocks
>>>>
>>>> unused devices : <none>
>>>>
>>>> # mkdir aa
>>>>
>>>> # mount /dev/md13 aa
>>>>
>>>> mount: /dev/md13: can't read superblock
>>>>
>>>> So I still don't see what to do. Advice welcome.
>>>>
>>>> Dave
>>>>
>>>>
>>>>
>>>> --
>>>> "As long as politics is the shadow cast on society by big business,
>>>> the attenuation of the shadow will not change the substance."
>>>>
>>>> -- John Dewey
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>>
>
>
> --
> Dolphins are so intelligent that within a few weeks they can
> train Americans to stand at the edge of the pool and throw them
> fish.
>
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Phil Turmel @ 2015-03-31 21:52 UTC (permalink / raw)
To: Dave Stevens, Brad Campbell; +Cc: Roger Heflin, linux-raid
In-Reply-To: <20150331144155.14405dy3bqxotw9v@webmail.uniserve.com>
On 03/31/2015 05:41 PM, Dave Stevens wrote:
> Quoting Brad Campbell <brad@fnarfbargle.com>:
>
>> md12 has claimed sda2 and md10 has sdb&c. Try this
>>
>> mdadm --stop /dev/md12
>> mdadm --stop /dev/md10
>> mdadm --assemble --force /dev/md13 /dev/sd[abc]2
>
> I owe you a beer, Brad! Now I only have to mount the LVM partition,
> retrieve the Xen domains and rejig the management tools so I can do a
> meaningful backup and restore but this is the bit I couldn't get my head
> around.
If you did this with those three disks, you are likely to have random
data corruption from the older mirror. If so, stop and re-assemble with
just the last good disks.
Phil
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Phil Turmel @ 2015-03-31 21:54 UTC (permalink / raw)
To: Dave Stevens, Brad Campbell; +Cc: Roger Heflin, linux-raid
In-Reply-To: <551B1717.2090809@turmel.org>
On 03/31/2015 05:52 PM, Phil Turmel wrote:
> On 03/31/2015 05:41 PM, Dave Stevens wrote:
>> Quoting Brad Campbell <brad@fnarfbargle.com>:
>>
>>> md12 has claimed sda2 and md10 has sdb&c. Try this
>>>
>>> mdadm --stop /dev/md12
>>> mdadm --stop /dev/md10
>>> mdadm --assemble --force /dev/md13 /dev/sd[abc]2
>>
>> I owe you a beer, Brad! Now I only have to mount the LVM partition,
>> retrieve the Xen domains and rejig the management tools so I can do a
>> meaningful backup and restore but this is the bit I couldn't get my head
>> around.
>
> If you did this with those three disks, you are likely to have random
> data corruption from the older mirror. If so, stop and re-assemble with
> just the last good disks.
Let me clarify: you only had *two* working disks for a month or so
before the array finally died. Only assemble those two.
Phil
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Dave Stevens @ 2015-03-31 21:55 UTC (permalink / raw)
To: Phil Turmel; +Cc: Brad Campbell, Roger Heflin, linux-raid
In-Reply-To: <551B1717.2090809@turmel.org>
Quoting Phil Turmel <philip@turmel.org>:
> On 03/31/2015 05:41 PM, Dave Stevens wrote:
>> Quoting Brad Campbell <brad@fnarfbargle.com>:
>>
>>> md12 has claimed sda2 and md10 has sdb&c. Try this
>>>
>>> mdadm --stop /dev/md12
>>> mdadm --stop /dev/md10
>>> mdadm --assemble --force /dev/md13 /dev/sd[abc]2
>>
>> I owe you a beer, Brad! Now I only have to mount the LVM partition,
>> retrieve the Xen domains and rejig the management tools so I can do a
>> meaningful backup and restore but this is the bit I couldn't get my head
>> around.
>
> If you did this with those three disks, you are likely to have random
> data corruption from the older mirror. If so, stop and re-assemble with
> just the last good disks.
>
> Phil
>
OK, thanks!
Dave
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* Re: Two system lockups, Magic SysKeys works, RAID6 resyncs on reboot
From: Mark Knecht @ 2015-03-31 22:08 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Linux-RAID
In-Reply-To: <20150401013032.5c908e85@natsu>
On Tue, Mar 31, 2015 at 1:30 PM, Roman Mamedov <rm@romanrm.net> wrote:
> On Tue, 31 Mar 2015 08:00:27 -0700
> Mark Knecht <markknecht@gmail.com> wrote:
>
>> md3 : active raid6 sdc3[9] sdg3[5] sdf3[6] sde3[7] sdd3[8]
>> 1452264480 blocks super 1.2 level 6, 16k chunk, algorithm 2 [5/5] [UUUUU]
>> [====>................] resync = 22.8% (110608304/484088160)
>> finish=78.5min speed=79206K/sec
>
> Looks like you are running this without a write intent bitmap, why? It would
> at least save you from a full resync each time.
>
> --
> With respect,
> Roman
Honestly, I don't know. The machine was the first I ever built using
RAID for the system drive and I probably didn't find out about it, or
was just trying to keep it simple? Not at all sure.
I'll check it out.
Thanks,
Mark
^ permalink raw reply
* Re: no good deed goes unpunished...
From: Dave Stevens @ 2015-03-31 23:31 UTC (permalink / raw)
To: Phil Turmel; +Cc: Brad Campbell, Roger Heflin, linux-raid
In-Reply-To: <551B179A.6020008@turmel.org>
Quoting Phil Turmel <philip@turmel.org>:
snip!
> On 03/31/2015 05:52 PM, Phil Turmel wrote:
> If you did this with those three disks, you are likely to have random
>> data corruption from the older mirror. If so, stop and re-assemble with
>> just the last good disks.
>
> Let me clarify: you only had *two* working disks for a month or so
> before the array finally died. Only assemble those two.
yes that's clear. I've done that, found the LVM volgroup and can now
see the Xen domains, working on getting data off.
many thanks!
D
>
> Phil
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
"As long as politics is the shadow cast on society by big business,
the attenuation of the shadow will not change the substance."
-- John Dewey
^ permalink raw reply
* openSUSE 13.2: md - Need advice which raid1 super format (1.0/1.2) is preferred
From: Dieter Nützel @ 2015-03-31 23:55 UTC (permalink / raw)
To: linux-raid
Hello List,
I need advice from the 'gurus' which raid1 md super format (1.0 or 1.2)
is preferred for SSD/'old' SATA disks.
Currently I have this on my gfx devel machine (Xeon X3430):
cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sdc1[0] sdd1[1]
156288896 blocks super 1.0 [2/2] [UU]
bitmap: 0/2 pages [0KB], 65536KB chunk
md1 : active raid1 sdb2[1] sda2[0]
60391744 blocks super 1.2 [2/2] [UU]
md0 : active raid1 sdb1[1] sda1[0]
2095104 blocks super 1.2 [2/2] [UU]
unused devices: <none>
md0 and md1
are on my SSD's which I've created by 'hand' (with mdadm) under openSUSE
13.2.
md2
is on my SATAs which I've created 'automatically' under yast2.
My question to you is:
Looks this sane (chunk only for super 1.0 on real SATAs and super 1.2 on
SSDs) or should I change anything, here.
I'll nuke md0 (swap) and merge it with md1 (root) 'cause I've upgraded
my RAM from 4 GB to 16 GB (soon to 24/32 GB max) and do not need swap
anylonger.
Any hints?
-Dieter
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: /dev/md0 can't be created
From: NeilBrown @ 2015-04-01 2:53 UTC (permalink / raw)
To: Xiao Ni; +Cc: linux-raid
In-Reply-To: <160793753.4322898.1427436381735.JavaMail.zimbra@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 5310 bytes --]
On Fri, 27 Mar 2015 02:06:21 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
>
> I'm lucky, I reproduced it again and I ran the command udevadm monitor > monitor before test.
> I'll put it in as an attachment.
>
> I found there is a not whole message in the last line.
>
> Xiao
The line:
UDEV [7083.531441] add /devices/virtual/block/md0 (block)
confirms that udev saw an 'add' event for /dev/md0.
That should be enough for udev to create the device node.
If it doesn't, you need to look into udev and find out why it doesn't.
I can't really help you there - I don't find udev very easy to debug.
NeilBrown
>
> ----- Original Message -----
> > From: "Xiao Ni" <xni@redhat.com>
> > To: "NeilBrown" <neilb@suse.de>
> > Cc: linux-raid@vger.kernel.org
> > Sent: Thursday, March 26, 2015 3:36:35 PM
> > Subject: Re: /dev/md0 can't be created
> >
> > Hi Neil
> >
> > I can reproduce it now. Do you want me to run udevadm monitor before the
> > test?
> > And I checked the /var/log/messages, it show the information:
> >
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-2>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-3>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-4>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-5>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-6>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-7>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-9>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: bind<dm-8>
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-7 operational as
> > raid disk 5
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-6 operational as
> > raid disk 4
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-5 operational as
> > raid disk 3
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-4 operational as
> > raid disk 2
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-3 operational as
> > raid disk 1
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: device dm-2 operational as
> > raid disk 0
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: allocated 0kB
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid:md0: raid level 5 active with 6
> > out of 7 devices, algorithm 2
> > Oct 20 19:59:59 ibm-z10-25 kernel: md/raid456: discard support disabled due
> > to uncertainty.
> > Oct 20 19:59:59 ibm-z10-25 kernel: Set
> > raid456.devices_handle_discard_safely=Y to override.
> > Oct 20 19:59:59 ibm-z10-25 kernel: md0: detected capacity change from 0 to
> > 1881145344
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: recovery of RAID array md0
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: minimum _guaranteed_ speed: 1000
> > KB/sec/disk.
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: using maximum available idle IO
> > bandwidth (but not more than 200000 KB/sec) for recovery.
> > Oct 20 19:59:59 ibm-z10-25 kernel: md: using 128k window, over a total of
> > 306176k.
> > Oct 20 19:59:59 ibm-z10-25 systemd-udevd: inotify_add_watch(7, /dev/md0, 10)
> > failed: No such file or directory
> >
> > Xiao
> >
> > ----- Original Message -----
> > > From: "NeilBrown" <neilb@suse.de>
> > > To: "Xiao Ni" <xni@redhat.com>
> > > Cc: linux-raid@vger.kernel.org
> > > Sent: Wednesday, March 25, 2015 2:35:29 PM
> > > Subject: Re: /dev/md0 can't be created
> > >
> > > On Wed, 25 Mar 2015 02:15:34 -0400 (EDT) Xiao Ni <xni@redhat.com> wrote:
> > >
> > > > Hi all
> > > >
> > > > I have encountered so many times, the raid device is created
> > > > successfully, but the directory
> > > > /dev/md0 can't be created. It can't reproduce 100%.
> > > >
> > > > [root@intel-sugarbay-do-01 create_assemble]# cat /proc/mdstat
> > > > Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
> > > > md0 : active raid10 loop7[7](S) loop6[6] loop5[5] loop4[4] loop3[3]
> > > > loop2[2] loop1[1] loop0[0]
> > > > 1788416 blocks super 1.2 512K chunks 2 near-copies [7/7] [UUUUUUU]
> > > > bitmap: 0/1 pages [0KB], 65536KB chunk
> > > >
> > > > unused devices: <none>
> > > > [root@intel-sugarbay-do-01 create_assemble]# ls /dev/md0
> > > > ls: cannot access /dev/md0: No such file or directory
> > > >
> > > > The underline devices are loop devices which are created with big
> > > > file.
> > > >
> > > > The kernel I used is RHEL7 (3.10.0-234.el7.x86_64.debug, mdadm -
> > > > v3.3.2
> > > > - 21st August 2014)
> > > > I'll try to reproduce this with upstream kernel and mdadm. But I think it
> > > > shouldn't be the problem about kernel.
> > > >
> > > > What do you think I should check for this? And which tool is
> > > > responsible for creating the directory? Maybe
> > > > I can add some log to it to find the reason.
> > > >
> > >
> > > /dev/md0 is created by udev.
> > > Run
> > > udevadm monitor
> > >
> > > to see the events that udev is processing. When and ADD event for "md0" is
> > > processed, /dev/md0 should get created.
> > >
> > > NeilBrown
> > >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Dan Williams @ 2015-04-01 3:47 UTC (permalink / raw)
To: Shaohua Li; +Cc: Neil Brown, linux-raid, Song Liu, Kernel-team
In-Reply-To: <20150330222459.GA575371@devbig257.prn2.facebook.com>
On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> This is my attempt to fix raid5/6 write hole issue, it's not for merge
> yet, I post it out for comments. Any comments and suggestions are
> welcome!
>
> Thanks,
> Shaohua
>
> We expect a completed raid5/6 stack with reliability and high
> performance. Currently raid5/6 has 2 issues:
>
> 1. read-modify-write for small size IO. To fix this issue, a cache layer
> above raid5/6 can be used to aggregate write to full stripe write.
> 2. write hole issue. A write log below raid5/6 can fix the issue.
>
> We plan to use a SSD to fix the two issues. Here we just fix the write
> hole issue.
>
> 1. We don't try to fix the issues together. A cache layer will do write
> acceleration. A log layer will fix write hole. The seperation will
> simplify things a lot.
>
> 2. Current assumption is flashcache/bcache will be used as the cache
> layer. If they don't work well, we can fix them or add a simple cache
> layer for raid write aggregation later. We also assume cache layer will
> absorb write, so log doesn't worry about write latency.
It seems neither bcache nor dm-cache are tackling the write-buffering
problem head on... they still seem to be concerned with some amount of
read caching which I can see as useful for file servers and
workstations, but not necessarily scale out storage.
I'll try to set aside time to take a look at the patch this week.
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: Shaohua Li @ 2015-04-01 5:53 UTC (permalink / raw)
To: Dan Williams; +Cc: Neil Brown, linux-raid, Song Liu, Kernel-team
In-Reply-To: <CAPcyv4jqb+OphP-1sy0dqMYG-uNK-CwtNDmOM4YeQU=MXH72Hw@mail.gmail.com>
On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
> On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> > yet, I post it out for comments. Any comments and suggestions are
> > welcome!
> >
> > Thanks,
> > Shaohua
> >
> > We expect a completed raid5/6 stack with reliability and high
> > performance. Currently raid5/6 has 2 issues:
> >
> > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> > above raid5/6 can be used to aggregate write to full stripe write.
> > 2. write hole issue. A write log below raid5/6 can fix the issue.
> >
> > We plan to use a SSD to fix the two issues. Here we just fix the write
> > hole issue.
> >
> > 1. We don't try to fix the issues together. A cache layer will do write
> > acceleration. A log layer will fix write hole. The seperation will
> > simplify things a lot.
> >
> > 2. Current assumption is flashcache/bcache will be used as the cache
> > layer. If they don't work well, we can fix them or add a simple cache
> > layer for raid write aggregation later. We also assume cache layer will
> > absorb write, so log doesn't worry about write latency.
>
> It seems neither bcache nor dm-cache are tackling the write-buffering
> problem head on... they still seem to be concerned with some amount of
> read caching which I can see as useful for file servers and
> workstations, but not necessarily scale out storage.
>
> I'll try to set aside time to take a look at the patch this week.
Thanks! The cache layer is definitely what I'll focus on next. bcache
supports writeback, I guess we can add an option to skip read data from
backing disks for read caching if it's possible. Another option is
writting a simple caching just for raid 5/6 write aggregation. We can
append all data to a log, and maintain an index in memory. At raid
shutdown, we can flush all data to raid disks, the index doesn't need
presistent in disk, which makes the caching fairly simple.
Thanks,
Shaohua
^ permalink raw reply
* Re: [RFC] raid5: add a log device to fix raid5/6 write hole issue
From: NeilBrown @ 2015-04-01 6:02 UTC (permalink / raw)
To: Shaohua Li; +Cc: Dan Williams, linux-raid, Song Liu, Kernel-team
In-Reply-To: <20150401055309.GA726662@devbig257.prn2.facebook.com>
[-- Attachment #1: Type: text/plain, Size: 2373 bytes --]
On Tue, 31 Mar 2015 22:53:21 -0700 Shaohua Li <shli@fb.com> wrote:
> On Tue, Mar 31, 2015 at 08:47:04PM -0700, Dan Williams wrote:
> > On Mon, Mar 30, 2015 at 3:25 PM, Shaohua Li <shli@fb.com> wrote:
> > > This is my attempt to fix raid5/6 write hole issue, it's not for merge
> > > yet, I post it out for comments. Any comments and suggestions are
> > > welcome!
> > >
> > > Thanks,
> > > Shaohua
> > >
> > > We expect a completed raid5/6 stack with reliability and high
> > > performance. Currently raid5/6 has 2 issues:
> > >
> > > 1. read-modify-write for small size IO. To fix this issue, a cache layer
> > > above raid5/6 can be used to aggregate write to full stripe write.
> > > 2. write hole issue. A write log below raid5/6 can fix the issue.
> > >
> > > We plan to use a SSD to fix the two issues. Here we just fix the write
> > > hole issue.
> > >
> > > 1. We don't try to fix the issues together. A cache layer will do write
> > > acceleration. A log layer will fix write hole. The seperation will
> > > simplify things a lot.
> > >
> > > 2. Current assumption is flashcache/bcache will be used as the cache
> > > layer. If they don't work well, we can fix them or add a simple cache
> > > layer for raid write aggregation later. We also assume cache layer will
> > > absorb write, so log doesn't worry about write latency.
> >
> > It seems neither bcache nor dm-cache are tackling the write-buffering
> > problem head on... they still seem to be concerned with some amount of
> > read caching which I can see as useful for file servers and
> > workstations, but not necessarily scale out storage.
> >
> > I'll try to set aside time to take a look at the patch this week.
>
> Thanks! The cache layer is definitely what I'll focus on next. bcache
> supports writeback, I guess we can add an option to skip read data from
> backing disks for read caching if it's possible. Another option is
> writting a simple caching just for raid 5/6 write aggregation. We can
> append all data to a log, and maintain an index in memory. At raid
> shutdown, we can flush all data to raid disks, the index doesn't need
> presistent in disk, which makes the caching fairly simple.
Surely if the index doesn't need to persist in disk, then the data doesn't
either, as without the index you cannot find the data...
NeilBrown
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox