It is safe to stop Raid being reshaped

It is safe to stop Raid being reshaped

[Jeremy Thompson]

Hello,

I added an extra drive to an existing Raid 5 array which consisted of
6 drives, now of course with the additional drive it will make it 7.
However, the speed of the reshape is going at 8877K/sec and sometimes
goes into 9000K/sec range however it does not go above this.

I've changed all the parameters to make it go faster for the reshaping
process which includes /proc/sys/dev/raid/speed_limit_min and
/proc/sys/dev/raid/speed_limit_max.

These were changed to 50000 each.  That did not help.  I then made
another change which was blockdev --setra 16384 /dev/md0.

Again nothing changed.  However, I do think that my drives are too
close to each other causing more heat and possibly causing issues with
the reshaping process due to some of the errors I see in the log
files.  I see some of the drives complaining about being hard reset.

I was dumb and stacked them pretty close to each other and before
since they were not under heavy use, there were no problems... however
now that the drives are under heavy use... I am seeing these errors
come up. I'd like to re-arrange the drives so they have more airflow
and also help try and prevent the errors from showing.  So with that
in mind, would it be at all possible to stop the reshaping process,
re-arrange the drives, then get the reshaping process back up and
running?

Or would it be better for me to just wait until the reshaping process
is completed?  Right now I'm look at another day to complete the
process.  If all goes well and I think it will... (first time doing
this) I would like to add another drive to the array however, I don't
want this to be my experience every time I add an additional drive to
it.  I am a bit new to Linux so if I got some of the terminology wrong
please forgive me.

Oh, and if there is a way to stop the reshape process safely and have
it restart after re-arranging of the drives, please let me know the
proper procedure to this being done.

Thanks for everyone taking their time reading this and any help anyone
can provide.

-Jeremy

Re: It is safe to stop Raid being reshaped

[Gordon Henderson]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1359 bytes --]

On Wed, 28 Dec 2011, Jeremy Thompson wrote:

> I was dumb and stacked them pretty close to each other and before
> since they were not under heavy use, there were no problems... however
> now that the drives are under heavy use... I am seeing these errors
> come up. I'd like to re-arrange the drives so they have more airflow
> and also help try and prevent the errors from showing.  So with that
> in mind, would it be at all possible to stop the reshaping process,
> re-arrange the drives, then get the reshaping process back up and
> running?

I'm not 100% sure about stopping the reshape, however I can tell you how 
to check the drive temperatures, if you've not already done this.

Use the hddtemp program.

e.g.

   # hddtemp /dev/sd{a,b}
   /dev/sda: SAMSUNG HD204UI: 25°C
   /dev/sdb: SAMSUNG HD204UI: 22°C

Then check the temperatures against the manufacturers recomendations.

IF you don't have hddtemp in your distro, try smartctl and look for 
parameter 194.

   # smartctl -a /dev/sda | fgrep 194
   194 Temperature_Celsius     0x0002   064   064   000    Old_age   Always       -       23 (Lifetime Min/Max 17/27)

However if you're seeing errors, then who knows... Cabling, motherboard, 
etc. are all possible culprits as well as physical surface defects... You
can find out more with smartctl -

   # smartctl -a /dev/sda

and so on.

Gordon

Re: It is safe to stop Raid being reshaped

[Asdo]

My opinion is: don't stop the array!
Errors are not necessarily caused by heat, probably more a controller or 
firmware issue, which will present identically if you restart the 
reshaping process.
Also, this kind of issues on my 3ware controller are harmless. You could 
better ask this to SCSI people, but there are up to 5 retries (for EACH 
scsi command) by the scsi layer after these resets happen.
Stopping the reshape is much more risky imho.

Re: It is safe to stop Raid being reshaped

[Jeremy Thompson]

The RAID array consists of drives that are on cheap SATA controllers
no RAID function on them.  That is why I chose to use mdadm instead of
a true RAID card.

So the errors would happen regardless if heat was an issue?

I'll wait until the reshape is done then, shutdown the machine and
re-arrange the drives before I add another drive to the array.

Thanks.

On Wed, Dec 28, 2011 at 11:02 AM, Asdo <asdo@shiftmail.org> wrote:
> My opinion is: don't stop the array!
> Errors are not necessarily caused by heat, probably more a controller or
> firmware issue, which will present identically if you restart the reshaping
> process.
> Also, this kind of issues on my 3ware controller are harmless. You could
> better ask this to SCSI people, but there are up to 5 retries (for EACH scsi
> command) by the scsi layer after these resets happen.
> Stopping the reshape is much more risky imho.

Re: It is safe to stop Raid being reshaped

[Asdo]

On 12/28/11 20:06, Jeremy Thompson wrote:
> The RAID array consists of drives that are on cheap SATA controllers
> no RAID function on them.  That is why I chose to use mdadm instead of
> a true RAID card.

Me too, I meant 3ware used as classic sata controller

> So the errors would happen regardless if heat was an issue?

I think that with temperature you would get a different error: disk 
going completely offline due to thermal shutdown.
Anyway Gordon showed you how to check for temperatures; try that.
Another way is via "smartctl -a /dev/sdX"

SCSI errors are usually due to cabling, not perfect firmware, not 
perfect controller, not perfect controller drivers.
If you show us the exact error we can be a bit more precise.
Note: 5 retries (for each scsi command) by the SCSI layer also applies 
to SATA disks, which is your case

> I'll wait until the reshape is done then, shutdown the machine and
> re-arrange the drives before I add another drive to the array.

Yes this would be my suggestion, but I don't know everything.

Have a look at the temperatures though, and compare to max temp by your 
HDD specs. I suggest you don't stop the array if they are lower.

Re: It is safe to stop Raid being reshaped

[Jeremy Thompson]

I checked the temperature of one of the drives, the reason I say
drives is because as soon as I wrote this email, a couple more drives
started throwing the same errors.  What boggles me is that I can't
have that many possible bad SATA cables? Can I?  The cables being used
are brand new, some off brand I know that but they are brand new.

I will certainly leave the reshape going since I'm almost 50% into it,
it would be dumb for me to stop it and then risk the whole array
losing data... not that the data is super important but I'd like to
keep it if at all possible.

Here is an excerpt from the dmesg log:

[77832.251754] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x280000 action 0x0
[77832.261281] ata3.00: BMDMA2 stat 0x696d0009
[77832.271043] ata3: SError: { 10B8B BadCRC }
[77832.280933] ata3.00: failed command: READ DMA EXT
[77832.290523] ata3.00: cmd 25/00:00:78:0f:c0/00:04:5b:00:00/e0 tag 0
dma 524288 in
[77832.290526]          res 51/04:6f:78:0f:c0/00:00:00:00:00/f0 Emask
0x1 (device error)
[77832.308903] ata3.00: status: { DRDY ERR }
[77832.318246] ata3.00: error: { ABRT }
[77832.408077] ata3.00: configured for UDMA/100
[77832.408142] ata3: EH complete

The same lines go for ata6 and ata5.  So 3 drives have bad cables?  I
have to mention before starting the reshaping process, I did not see
these errors or at least they were not as pronounced as they are
today.

Anything else you'd like me to check out?  I'd also like to know how
can I correlate between which drive is ata3, ata5, and ata6?  So ata6
could be /dev/sda for instance.

Here is what I get for the temperature from smartctl -a /dev/sdg:

190 Airflow_Temperature_Cel 0x0022   047   032   045    Old_age
Always   In_the_past 53 (77 0 55 36)
194 Temperature_Celsius     0x0022   053   068   000    Old_age
Always       -       53 (0 21 0 0)

I included both of those lines because I'm not sure which ones you
wanted to look at.

Thanks.

On Wed, Dec 28, 2011 at 6:02 PM, Asdo <asdo@shiftmail.org> wrote:
> On 12/28/11 20:06, Jeremy Thompson wrote:
>>
>> The RAID array consists of drives that are on cheap SATA controllers
>> no RAID function on them.  That is why I chose to use mdadm instead of
>> a true RAID card.
>
>
> Me too, I meant 3ware used as classic sata controller
>
>
>> So the errors would happen regardless if heat was an issue?
>
>
> I think that with temperature you would get a different error: disk going
> completely offline due to thermal shutdown.
> Anyway Gordon showed you how to check for temperatures; try that.
> Another way is via "smartctl -a /dev/sdX"
>
> SCSI errors are usually due to cabling, not perfect firmware, not perfect
> controller, not perfect controller drivers.
> If you show us the exact error we can be a bit more precise.
> Note: 5 retries (for each scsi command) by the SCSI layer also applies to
> SATA disks, which is your case
>
>
>> I'll wait until the reshape is done then, shutdown the machine and
>> re-arrange the drives before I add another drive to the array.
>
>
> Yes this would be my suggestion, but I don't know everything.
>
> Have a look at the temperatures though, and compare to max temp by your HDD
> specs. I suggest you don't stop the array if they are lower.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: It is safe to stop Raid being reshaped

[Gordon Henderson]

On Wed, 28 Dec 2011, Jeremy Thompson wrote:

> I checked the temperature of one of the drives, the reason I say
> drives is because as soon as I wrote this email, a couple more drives
> started throwing the same errors.  What boggles me is that I can't
> have that many possible bad SATA cables? Can I?  The cables being used
> are brand new, some off brand I know that but they are brand new.

They're not WDC drives are they?

> [77832.251754] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x280000 action 0x0
> [77832.261281] ata3.00: BMDMA2 stat 0x696d0009
> [77832.271043] ata3: SError: { 10B8B BadCRC }
> [77832.280933] ata3.00: failed command: READ DMA EXT
> [77832.290523] ata3.00: cmd 25/00:00:78:0f:c0/00:04:5b:00:00/e0 tag 0
> dma 524288 in
> [77832.290526]          res 51/04:6f:78:0f:c0/00:00:00:00:00/f0 Emask
> 0x1 (device error)
> [77832.308903] ata3.00: status: { DRDY ERR }
> [77832.318246] ata3.00: error: { ABRT }
> [77832.408077] ata3.00: configured for UDMA/100
> [77832.408142] ata3: EH complete

I've had very similar errors from a pair of WDC drives - bought at the 
same time, from the same batch... They didn't show any surface defects or 
sector remaps, just lots of DMA errors in both the Linux logs and the 
SMART logs in the devices.

> Anything else you'd like me to check out?  I'd also like to know how
> can I correlate between which drive is ata3, ata5, and ata6?  So ata6
> could be /dev/sda for instance.
>
> Here is what I get for the temperature from smartctl -a /dev/sdg:
>
> 190 Airflow_Temperature_Cel 0x0022   047   032   045    Old_age
> Always   In_the_past 53 (77 0 55 36)
> 194 Temperature_Celsius     0x0022   053   068   000    Old_age
> Always       -       53 (0 21 0 0)
>
> I included both of those lines because I'm not sure which ones you
> wanted to look at.

They're running at 53C. Is that good or bad? Who knows - you'll need to 
check the manufacturers specs.

Gordon

Re: It is safe to stop Raid being reshaped

[Brad Campbell]

On 29/12/11 10:23, Jeremy Thompson wrote:
> Here is an excerpt from the dmesg log:
>
> [77832.251754] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x280000 action 0x0
> [77832.261281] ata3.00: BMDMA2 stat 0x696d0009
> [77832.271043] ata3: SError: { 10B8B BadCRC }
> [77832.280933] ata3.00: failed command: READ DMA EXT
> [77832.290523] ata3.00: cmd 25/00:00:78:0f:c0/00:04:5b:00:00/e0 tag 0
> dma 524288 in
> [77832.290526]          res 51/04:6f:78:0f:c0/00:00:00:00:00/f0 Emask
> 0x1 (device error)
> [77832.308903] ata3.00: status: { DRDY ERR }
> [77832.318246] ata3.00: error: { ABRT }
> [77832.408077] ata3.00: configured for UDMA/100
> [77832.408142] ata3: EH complete
>
> The same lines go for ata6 and ata5.  So 3 drives have bad cables?  I
> have to mention before starting the reshaping process, I did not see
> these errors or at least they were not as pronounced as they are
> today.
>
I'd have a close look at your power supply rather than the cables.

I've seen odd behaviour across multiple drives when they were all under load (as they are in a 
reshape) which went away when I replaced the marginal power supply with something more substantial 
and better engineered.

Regards,
Brad

Re: It is safe to stop Raid being reshaped

[Jeremy Thompson]

I thought about this as well - I have an Enermax 450 in there right
now... I'll have to look for a higher watt power supply next I guess.

Thanks,
Jeremy

On Thu, Dec 29, 2011 at 3:35 PM, Brad Campbell <brad@fnarfbargle.com> wrote:
> On 29/12/11 10:23, Jeremy Thompson wrote:
>>
>> Here is an excerpt from the dmesg log:
>>
>> [77832.251754] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x280000 action
>> 0x0
>> [77832.261281] ata3.00: BMDMA2 stat 0x696d0009
>> [77832.271043] ata3: SError: { 10B8B BadCRC }
>> [77832.280933] ata3.00: failed command: READ DMA EXT
>> [77832.290523] ata3.00: cmd 25/00:00:78:0f:c0/00:04:5b:00:00/e0 tag 0
>> dma 524288 in
>> [77832.290526]          res 51/04:6f:78:0f:c0/00:00:00:00:00/f0 Emask
>> 0x1 (device error)
>> [77832.308903] ata3.00: status: { DRDY ERR }
>> [77832.318246] ata3.00: error: { ABRT }
>> [77832.408077] ata3.00: configured for UDMA/100
>> [77832.408142] ata3: EH complete
>>
>> The same lines go for ata6 and ata5.  So 3 drives have bad cables?  I
>> have to mention before starting the reshaping process, I did not see
>> these errors or at least they were not as pronounced as they are
>> today.
>>
> I'd have a close look at your power supply rather than the cables.
>
> I've seen odd behaviour across multiple drives when they were all under load
> (as they are in a reshape) which went away when I replaced the marginal
> power supply with something more substantial and better engineered.
>
> Regards,
> Brad
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: It is safe to stop Raid being reshaped

[Mikael Abrahamsson]

On Thu, 29 Dec 2011, Jeremy Thompson wrote:

> I thought about this as well - I have an Enermax 450 in there right 
> now... I'll have to look for a higher watt power supply next I guess.

Also look at how your feeds are balanced. Some PSUs have two rails, 
putting all drives on a single rail causes your load handling capacity to 
go down significantly. I recommend getting a single rail PSU as well, well 
above the capacity you think you need.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: It is safe to stop Raid being reshaped

[Jeremy Thompson]

OK good thing to check over, what would be a recommendation for a Power Supply?

Thanks!

On Thu, Dec 29, 2011 at 8:31 PM, Mikael Abrahamsson <swmike@swm.pp.se> wrote:
> On Thu, 29 Dec 2011, Jeremy Thompson wrote:
>
>> I thought about this as well - I have an Enermax 450 in there right now...
>> I'll have to look for a higher watt power supply next I guess.
>
>
> Also look at how your feeds are balanced. Some PSUs have two rails, putting
> all drives on a single rail causes your load handling capacity to go down
> significantly. I recommend getting a single rail PSU as well, well above the
> capacity you think you need.
>
> --
> Mikael Abrahamsson    email: swmike@swm.pp.se
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: It is safe to stop Raid being reshaped

[Brad Campbell]

On 30/12/11 12:31, Mikael Abrahamsson wrote:
> On Thu, 29 Dec 2011, Jeremy Thompson wrote:
>
>> I thought about this as well - I have an Enermax 450 in there right now... I'll have to look for 
>> a higher watt power supply next I guess.
>
> Also look at how your feeds are balanced. Some PSUs have two rails, putting all drives on a single 
> rail causes your load handling capacity to go down significantly. I recommend getting a single 
> rail PSU as well, well above the capacity you think you need.
>
I made the move to a beefy single rail supply a few years ago. Wished I'd done it sooner.

I'm using two of these :

http://www.pcpower.com/power-supply/silencer-610-eps12v.html

Both machines have 6 core Phenoms with 16GB of ram.
Machine A has 15 7200RPM drives
Machine B has 10 7200RPM and 4 15000RPM Drives

The power supplies are well within limits, run cool and I've not had an issue since I upgraded to 
them a couple of years ago.

I do staggered spin-ups on machine B, but Machine A fires all drives at boot. They've been running 
in this configuration for about 4 years on these PSU's.

Regards,
Brad

Re: It is safe to stop Raid being reshaped

[Mattias Wadenstein]

On Fri, 30 Dec 2011, Mikael Abrahamsson wrote:

> On Thu, 29 Dec 2011, Jeremy Thompson wrote:
>
>> I thought about this as well - I have an Enermax 450 in there right now... 
>> I'll have to look for a higher watt power supply next I guess.
>
> Also look at how your feeds are balanced. Some PSUs have two rails, putting 
> all drives on a single rail causes your load handling capacity to go down 
> significantly. I recommend getting a single rail PSU as well, well above the 
> capacity you think you need.

Not only that, take a good look at both the 5V and 12V specs, and compare 
them to the draw needed by the drives (and the rest of the system). I've 
found that the ammount of 5V available varies lots between PSUs, without 
much correlation with the 12V supply. Which is fine for modern CPUs/GPUs 
which mostly uses 12V, but drives seem to want quite a bit of 5V too.

One other common problem with multiple rail PSUs is that often all the 
useful power connectors (sata/molex) are only on one rail. I wouldn't mind 
multiple rails if I could actually use all of them for drives.

For the 24-drive home build I've done, I've actually ended up running two 
PSUs in parallel (joining ground), but this is hardly something to 
recommend to anyone sane. ;)

/Mattias Wadenstein

Re: It is safe to stop Raid being reshaped

[Krzysztof Adamski]

On Fri, 2011-12-30 at 13:16 +0100, Mattias Wadenstein wrote:
> On Fri, 30 Dec 2011, Mikael Abrahamsson wrote:
> 
> > On Thu, 29 Dec 2011, Jeremy Thompson wrote:
> >
> >> I thought about this as well - I have an Enermax 450 in there right now... 
> >> I'll have to look for a higher watt power supply next I guess.
> >
> > Also look at how your feeds are balanced. Some PSUs have two rails, putting 
> > all drives on a single rail causes your load handling capacity to go down 
> > significantly. I recommend getting a single rail PSU as well, well above the 
> > capacity you think you need.
> 
> Not only that, take a good look at both the 5V and 12V specs, and compare 
> them to the draw needed by the drives (and the rest of the system). I've 
> found that the ammount of 5V available varies lots between PSUs, without 
> much correlation with the 12V supply. Which is fine for modern CPUs/GPUs 
> which mostly uses 12V, but drives seem to want quite a bit of 5V too.
> 
> One other common problem with multiple rail PSUs is that often all the 
> useful power connectors (sata/molex) are only on one rail. I wouldn't mind 
> multiple rails if I could actually use all of them for drives.
> 
> For the 24-drive home build I've done, I've actually ended up running two 
> PSUs in parallel (joining ground), but this is hardly something to 
> recommend to anyone sane. ;)

I can give you my recommendation for a power supply based on my box.
The box has 22 drives (all 7200), dual core Phenom II, 8 Gig ram.
The PS is a OCZ ModXstream Pro 700. It has been running for over a year
without showing any problems. In a normal steady state the box draws 260
Watts from the wall as measures by a Kill-A-Watt device.

K