3Ware on 64bit Linux.

3Ware on 64bit Linux.

[Jason Leach]

hi,

Does anyone have a 3ware 90500S card working on 64bit Linux to a point
where it is very stable and preforming as advertised?  If so, can you
send me the HW specs you are using? (mobo, cpu, chipsets).  I'm
considering putting one together but don't want an unstable system.

Jason.

Re: 3Ware on 64bit Linux.

[Mike Hardy]

I can't speak to your specific card, but I'm using another of their
cards (8-port PATA, I think that's the 7xxx?) and the driver and admin
client (3dm2) haven't had stability problems so far on a dual opteron
that previously had problems handling the interrupt traffic with
multiple promise/cmd-type PATA cards.

I won't send the details since its all very specific (hopefully someone
else has done it), but 3ware can work well on dual-opteron stuff, which
is a positive sign for you

-Mike

Jason Leach wrote:
> hi,
> 
> Does anyone have a 3ware 90500S card working on 64bit Linux to a point
> where it is very stable and preforming as advertised?  If so, can you
> send me the HW specs you are using? (mobo, cpu, chipsets).  I'm
> considering putting one together but don't want an unstable system.
> 
> Jason.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3Ware on 64bit Linux.

[Harry Mangalam]

I'm working thru this right now, so it can't be said that I HAVE a stable 
system.  What I have is a lot of experience with UNSTABLE systems, so you can 
learn from my misadventures.  

I have what is probably a relatively common setup for a medium high end, 
stand-alone, PC-based server:

- dual opteron IWILL Mobo, Gb ethernet
- Silicon image 4port SATA controller onboard (now disabled), 
- 3ware 9500S 8 port card running 8x250GB WD 2500SD disks in RAID5.  
- Disks are in 2x 4slot Chenbro hotswap RAID cages. 
- running kubuntu Linux in pure 64bit mode (altho bringing up KDE 
  currently locks the system in some configurations)
- using kernel image 2.6.11-1-amd64-k8-smp as a kubuntu debian install 
    (NOT custom-compiled)
- OS is running from a separate 200GB IDE disk 
    (which recently bonked at 3 months old)
- on an APC UPS (runnning apcupsd w/ a usb cable)

I've had nothing but trouble for a couple of months - some it my fault, some 
of it 3ware's fault, some of it WD's fault, some of it the server vendor's 
fault.

I tried SW raid 1st using the onboard controller SATA - it was a bit confusing 
to set up, but it worked well, and performed well on the bonnie++ tests and 
on real-life copy and read/write tasks.  I probably should have bought 
another similar controller (<$50) and stayed with SW RAID and mdadm.

However, because of the trickiness of it, I wanted to leave the machine 
simpler to administer, so I tried an 3ware 8506-8 controller with a full set 
of WD JD-series disks.  That was a catastrophe - all kinds of problems 
setting it up, finding out the bash script software installer installed 32b 
versions of the software and then silently died without error.  This led to a 
disk failing silently and the rebuild failed leading to the loss of the 
entire array's data.  Since we had just started using it, it was surprising, 
but not devastating.  In investigating this, I found that there ARE 64 bit 
versions of both the commandline software and the 3dm2 web-enabled software 
but they're difficult to find.  I have a much longer rant on this - I'll mail 
it to anyone interested.

The short version is that due to 'one of those things', my support call to 
3ware went awry and wasn't answered for several days - and their web site is 
truly frustrating and ill-designed (don't type anything into any of their 
support forms without having 1st typed into a local buffer - the 3ware 
support forms will allow you to type and type and type and then reject 
everything on submitting it if you've gone over their (often unannounced) 
word count.  Support URLS are hidden as plain text (their web designer is 
going to a special place in hell), click paths are taken direct from 
'adventure' (you are in a maze of dark, twisty web pages, all alike..).
However, I have to say that in the end they came thru - after the fumbled 
support call, they replaced the faulty 8506 with a 9500S, and came up with a 
lot of good, useful advice and stayed with me.  Good hardware, good tech 
people, VERY lousy web pages, lousy initial support routing, fair software, 
fairly lousy SW docs.

The 3dm2 software, once the correct version was installed, works fine (tho the 
docs are a bit fuzzy (no way to find out what the passwords were to the 3dm2 
accounts (they're buried in the installation scripts) and on Debian, it 
doesn't write a config file when you change the defaults so that it reverts 
to the defaults on reboot, leaving your spiffing new array vulnerable to 
anyone who probes port 888.  Make sure that you've 'touch'ed 
a  /etc/3dm2/3dm2.conf file before starting.

Then problems with the disks - the JD series are particularly ill suited to 
RAID5 apparently because of the BIOS algorithms and WD will replace them for 
more expensive SD series disks without complaint but the replacement web form 
is quite confusing and this delayed things even longer (they replaced only 
SOME of the disks with SD and some of them with more JDs).  This was 
admittedly my fault, but the WD support person had a similarly difficult time 
with the form.  

Dan Stromberg who has been posting here recently can speak to the 
viability/reliability of the right Maxtor disks to use, but the 3ware tech 
said that the WD SD series disks were very reliable and fast.

The Chenbro hotswap cages have also been a source of trouble - one port on one 
cage (I have 2 cages of 4 disks each) has been giving me trouble and the 
actual hardware is not well-made - the SATA connectors are just pushed onto 
very fragile pins on the circuit board, so when the cables are pulled off, 
the whole connector comes away as well and the pins can be bent out of 
position very easily when trying to put them back.
Unless you absolutely need the hotswap ability, DON'T.  It's one more piece of 
electronica that can go wrong.

The upside is that the 3ware card on a 64 bit opteron seems to be usable, tho 
I can't say that the whole system has been stable for a long time. The 3ware 
cards are supported in the kernel source so there's no need for separate 
drivers.  There's no messing with separate devices as the 3ware card presents 
as a huge SCSI disk.  The 3ware card (supposedly) takes care of mapping & 
remapping some of the low-level disk problems recently reported here.  
However, the speed of the array is really no better than SW raid5 as judged 
by bonnie++ and some large manual copies and programmatic reads/writes.  What 
I liked about the SW RAID is the degree of flexibility there is in both 
software and hardware (can mix and match much cheaper controllers), the 
price, and the much wider availability of knowlegable support (this list, 
google searches, etc).  With 3ware, you really are at the mercy of their 
support system.


 I finally have most of the above described problems addressed and as I write 
this, the last few inodes of an ext3 fs are being written to the 1.7TB array. 
(altho this was the 3rd time I tried it, using various parameters ( tho I 
suspect network problems interrupting my session rather than failure on the 
array). We'll see if it lasts).

I have been getting several emails from the 3ware card describing various 
repairs-on-the-fly that it's been making:

Wed, Jun 08, 2005 10:09.19AM - Controller 0
WARNING - (0x04:0x0023): Sector repair completed: port=6, LBA=0xDF6022E so 
that gives me a little more confidence in the system.


The UPS is yet another story for another time.

OK - Nuff said for now.  Maybe more later.

hjm



On Wednesday 08 June 2005 10:09 am, Jason Leach wrote:
> hi,
>
> Does anyone have a 3ware 90500S card working on 64bit Linux to a point
> where it is very stable and preforming as advertised?  If so, can you
> send me the HW specs you are using? (mobo, cpu, chipsets).  I'm
> considering putting one together but don't want an unstable system.
>
> Jason.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
            <<plain text preferred>>

Re: 3Ware on 64bit Linux.

[Dan Stromberg]

[-- Attachment #1: Type: text/plain, Size: 750 bytes --]

On Wed, 2005-06-08 at 11:56 -0700, Harry Mangalam wrote:

> 
> Dan Stromberg who has been posting here recently can speak to the 
> viability/reliability of the right Maxtor disks to use, but the 3ware tech 
> said that the WD SD series disks were very reliable and fast.

I have a writeup of the 3ware+maxtor SATA mess we just got a refund on
at:

http://dcs.nac.uci.edu/~strombrg/3ware-maxtor-notes.html

...but in short, we never determined definitively if the hardware
problems (there were software problems too) were due to the RAID
controllers or the disks.  The integrator we were working with was
blaming the disks, but sometimes we'd get repeated failures with
different disks in the same slot of a given RAID controller.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 3Ware on 64bit Linux.

[Konstantin Olchanski]

On Wed, Jun 08, 2005 at 11:56:30AM -0700, Harry Mangalam wrote:
> I have what is probably a relatively common setup for a medium high end, 
> stand-alone, PC-based server:
> 
> - dual opteron IWILL Mobo, Gb ethernet
> - 3ware 9500S 8 port card running 8x250GB WD 2500SD disks in RAID5.  
> - Disks are in 2x 4slot Chenbro hotswap RAID cages. 
> - running kubuntu Linux in pure 64bit mode
> - OS is running from a separate 200GB IDE disk 
> - on an APC UPS (runnning apcupsd w/ a usb cable)

I have a 3ware 8506-12 SATA controller with 12 WD 250 GByte SATA disks,
on an old dual opteron-242 Tyan mobo. The system disk is a
separate 80 GB IDE disk. The SATA disks are confugured into 2
striped (one hardware raid, one software raid) logical volumes (1.5 TB
each). The box and the hot-swap disk enclosure is el-cheapo stuff
from Hard Data in Edmonton, AB (works great!).

This system has been rock solid from day one, running RedHat 9 (32-bit),
then Fedora 1 (32-bit), than Fedora 2 and 3 (64-bit).

I use the 3ware driver that comes with the Red Hat kernels, the
additional monitoring tools from 3ware do not work. SMART monitoring
works via "smartctl -a -d 3ware,0 /dev/twe0".

-- 
Konstantin Olchanski
Data Acquisition Systems: The Bytes Must Flow!
Email: olchansk-at-triumf-dot-ca
Snail mail: 4004 Wesbrook Mall, TRIUMF, Vancouver, B.C., V6T 2A3, Canada

Re: 3Ware on 64bit Linux.

[Harry Mangalam]

On Wednesday 08 June 2005 2:36 pm, Konstantin Olchanski wrote:
> I have a 3ware 8506-12 SATA controller with 12 WD 250 GByte SATA disks,
> on an old dual opteron-242 Tyan mobo. The system disk is a
> separate 80 GB IDE disk. The SATA disks are confugured into 2
> striped (one hardware raid, one software raid) logical volumes (1.5 TB
> each). The box and the hot-swap disk enclosure is el-cheapo stuff
> from Hard Data in Edmonton, AB (works great!).
>
> This system has been rock solid from day one, running RedHat 9 (32-bit),
> then Fedora 1 (32-bit), than Fedora 2 and 3 (64-bit).

That's what you get for living the clean life in Vancouver ;)  If the 
hardware's flaky, everything else will be too.  And as KO alludes, 
reliability and cost do NOT go hand in hand.

Do you know if you're using the SD or JD versions of the disks?  

> I use the 3ware driver that comes with the Red Hat kernels, the
> additional monitoring tools from 3ware do not work. SMART monitoring
> works via "smartctl -a -d 3ware,0 /dev/twe0".

I suspect that the reason it doesn't work is that it's the 32 bit version.  
The 64b version works for me - not great, but it works.  Or are you saying 
that it IS the 64b version and it doesn't work?  The 64b version is called 
3dm2 (as opposed to the previous 3dm)

I'll try direct SMART monitoring as well - thanks for the tip.

-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
            <<plain text preferred>>

Re: 3Ware on 64bit Linux.

[Tim Moore]

Harry Mangalam wrote:

> Then problems with the disks - the JD series are particularly ill suited to 
> RAID5 apparently because of the BIOS algorithms and WD will replace them for 
> more expensive SD series disks without complaint but the replacement web form 
> is quite confusing and this delayed things even longer (they replaced only 
> SOME of the disks with SD and some of them with more JDs).  This was 
> admittedly my fault, but the WD support person had a similarly difficult time 
> with the form.  

The SD's are 100% duty cycle, 24x7, 1M hour MTBF commercial SATA II drives, 
whereas the JD's are SATA I with a "desktop" rating and a 'designed for 
Microsoft Windows' cert.  Why WD would trade a commercial drive for a 
desktop drive?

Besides that I've used 3Ware controllers since the 6400 and have had zero 
problems with anything.  The gui tool is nice but not suited to production use.

t.

Re: 3Ware on 64bit Linux.

[Harry Mangalam]

On Friday 10 June 2005 1:25 am, Tim Moore wrote:

> The SD's are 100% duty cycle, 24x7, 1M hour MTBF commercial SATA II drives,
> whereas the JD's are SATA I with a "desktop" rating and a 'designed for
> Microsoft Windows' cert.  Why WD would trade a commercial drive for a
> desktop drive?

Dunno, but they did, altho the disks back were 'recertified', not new.  As to 
the 1M MTBF rating, I also dunno.  I got 9 're-certified SD drives' back from 
WD and 2 of them (oops, make that 3 of them) have failed in the past 24 
hours, which points to some underlying problem in the system.  It's on a UPS 
which should condition the voltage. The drives that went out were on 
different ports and one of them was direct-connected, not via a hotswap cage 
(due to previous suspicions about the hotswap cage).

Any insight into this failure rate would be appreciated.  About to open 
another support call to 3ware...

> Besides that I've used 3Ware controllers since the 6400 and have had zero
> problems with anything.  

I guess I'm the reason they say YMMV. :)

> The gui tool is nice but not suited to production 
> use.

It sounded like his problem was a one-off change in structure which the gui 
was well suited for. 

Unless you're going to script it into maint mode, the gui seems to have 
benefits.
-- 
Cheers, Harry
Harry J Mangalam - 949 856 2847 (vox; email for fax) - hjm@tacgi.com 
            <<plain text preferred>>

Re: 3Ware on 64bit Linux.

[Tim Moore]

Harry Mangalam wrote:
> 
> On Friday 10 June 2005 1:25 am, Tim Moore wrote:
> 
> 
>>The SD's are 100% duty cycle, 24x7, 1M hour MTBF commercial SATA II drives,
>>whereas the JD's are SATA I with a "desktop" rating and a 'designed for
>>Microsoft Windows' cert.  Why WD would trade a commercial drive for a
>>desktop drive?
> 
> 
> Dunno, but they did, altho the disks back were 'recertified', not new.  As to 

Interesting.  Maybe there were serious mfg problems with the JDs.

> the 1M MTBF rating, I also dunno.  I got 9 're-certified SD drives' back from 
> WD and 2 of them (oops, make that 3 of them) have failed in the past 24 
> hours, which points to some underlying problem in the system.  It's on a UPS 
> which should condition the voltage. The drives that went out were on 
> different ports and one of them was direct-connected, not via a hotswap cage 
> (due to previous suspicions about the hotswap cage).
> 
> Any insight into this failure rate would be appreciated.  About to open 
> another support call to 3ware...

I've also taken "failed" WD drives on production hardware which have passed 
WD's own check program and found that badblocks -n will force remapping the 
bad sectors.

> 
> 
>>Besides that I've used 3Ware controllers since the 6400 and have had zero
>>problems with anything.  
> 
> 
> I guess I'm the reason they say YMMV. :)

Fair enough.