As an update,

Adam's been helping me out (and I'm not used to hitting "reply-all" for mailing lists as pretty  much all the ones I'm on set the "reply-to:")

I've turned on sct/erc for the drives... and the one that went bonkers during the rebuild (sde) still would have read issues during a rebuild.

SMART reports it's ok. but.. (shrug) I ended up running ddrescue to the new replacement drive (sdc) that kept getting put back into spare status when the rebuilds would fail.

So I just copied sde -> sdc which went pretty much flawlessly (ddrescue completed without any final complaints)

I also played with badblocks after doing my copy and could find bad blocks -- but apparnently ddrescue had no issues.

So - I went back to

*bringing up the array. No problems.
* adding ANOTHER new drive (that I ordered Sunday night) and it rebuilt fine.
* doing an FSCK -n first which reported no issues - so I did a regular fsck (without -y) and it never prompted me for anything.

My last step is to run rsync -n from my backup to see if it can find any issues between my last backup and the current data for any files with byte oddities.

All this has me wonder if those old bad sectors left some files with a sector of garbage in them or not.

Adam seems to think everything is fine -- so far, that seems to be the case.

A last few questions I have are:

The new drive I got was (supposed to be) the same model as the last Seagate I ordered, but SMART reports them differently. (see attached)

The question on the new drive is that it says it does offline collection... but with gsmartcontrol, I can't seem to turn it on.

This new drive also doesn't seem to support SCT/ERC the same way.

Again,

/dev/sdc - old new spare (bought after seagate bought Samsung and discontinued the HD103SJ model)
/dev/sdd - original RAID member
/dev/sde - brand spanking new drive purchased Sunday.
/dev/sdf - original RAID member

I realize now one says: ST1000DM005 vs ST1000DM003 - Grrr!!!

So I'd like recommendations on whether I should get better matching drives (I can use these elsewhere) or it doesn't matter.

Can I mix/match this array with WD REDs? (and eventually retire all these HD103SJ drives) Do people even like these? They seem ok?

I read a lot of conflicting info on SCT/ERC online (well, TLER anyway) -- Adam likes it enabled. What say the rest of you?

And last -- any caveats as to upgrading this array to RAID6 from RAID5? Can I even do that while in place?

Thanks all, (especially Adam!)

  -Ben

p.s. Check out some of the SMART parms on the /dev/sde. Head flying hours?? And they're not zero. Weird. :/ This drive kinda creeps me out.