Tyan, RAID-6, and other recent hassles... (long, a bit OT)

Tyan, RAID-6, and other recent hassles... (long, a bit OT)

[Gordon Henderson]

This is a bit OT and long, but it might help someone in the future, you
never know!

I've been struggling recently to get a box together with some supposedly
nice hardware and it's turned out to be a bit of a nightmare.. The good
news is that it's now sorted and working well enough to go into
production.

A big thanks to everyone who's contributed both on the list and in private
email with some of the issues I've had with it.

I've been building & running servers for many years, using Linux RAID for
the past 5 or so, so thought this would be just another server (admittedly
one of the biggest in disk terms I've built) alas it was nearly my
nemesis!

It's a 3U case with 8 hot-swap SATA drives and triple redundant 600W PSU.
Nice case, 3 big fans inside, space for 3 5.25" units on one side. (I just
have a CD-ROM drive in there). I opted for a 3U case rather than 2U just
to make sure there was room inside it to take standard PCI cards without
any risers and restricted air-flows. I chose a dual Opteron mobo (clients
request) with on-board 4-port SATA controller (SII 3114) and initially got
2 more SII based 2-port PCI cards.

Mobo was a Tyan Thunder K8W. (S2885)
1GB of Crucial RAM (2x512MB PC2700)
Case: http://www.acme-technology.co.uk/acm338.htm
8 x Hitachi Deskstar 250GB SATA.
2 x Opteron 240 processors
Debian Woody with 2.6 kernel.

Then the trouble started )-:

It seems that that motherboard, or the AMD chipset just can't hack PCI-X.
(or maybe PCI cards in the PCI-X slots)  There are various jumpers and
BIOS options to fiddle with, but nothing seemed to work well. It did seem
to work better with just one PCI card though, but not perfect. I
re-flashed the BIOS to their latest (beta) version and that was better but
not 100%.

The mobo on its own, with just 4 drives on the on-board controller seemed
solid. It would boot OK, and run just fine, but as soon as I plugged
additional SII cards into the PCI slots it all went pear-shaped.

Finally, I got a 4-port Highpoint card (Rocket 1540) and that's made a lot
of difference.

I also found (along with someone else who emailled me about this), that
the SATA cables supplied with the motherboard are less than reliable.
Replacing them with nice flexable cables improved things too. I've
subsequently gone off SATA. Damnit! The cables fiddly, the connectors
fragile. Give me good old wide cables and chunky connectors anyday!
</rant> :)

(FWIW: I tested the same disks and 4 2-port SII cards in a Xeon system
with 4 PCI-X slots and it really flew, so I was confident it wasn't an OS
problem, or a problem with the cards, or disks)

The down-side is that the Highpoint driver is somewhat slower than the SII
drivers (I'm losing ~5-8Mb/sec disk performance) and it's not open source.
Another irritation is that it won't pass through the SMART commands. Yet
another irritation is that it won't compile into the kernel, and must be
loaded as a module, so I can't use auto-detection on the RAID arrays (I
don't do initrd) No real issue, as in the startup scripts, I added another
script after it checks the root filesystem, and before it checks & mounts
the others - do an explicit modload, and explicit mdadm --assemble
instructions.

I also had issues trying to boot the damn thing. It really wasn't happy
booting when the extra (SII) PCI cards installed, even when trying to just
boot off the first drive on the on-board controller. In the end, I was
booting it off an IDE flash drive and mounting / under /dev/sda1, then
subsequently /dev/md1 (raid-1 of the first 4 drives on the on-board
controller)

Now, with a different chip-set PCI card, (The Highpoint) the BIOS is happy
to boot off any one of the on-board drives, boot is /dev/md1, root is
/dev/md1 and I'm happy. (md1 is a RAID-1 comprised of /dev/sd{a,b,c,d}1
which are connected to the on-board SII 3114 controller)

I've played with 2.6.10 and 2.6.11 RC kernels. Applied patches for the
libata stuff to sort of make SMART work (the 4 drives on the 3114 need the
-w flag to hddtemp, as it thinks they are asleep all the time), the
Highpoint driver just won't pass the SMART commands.

I had concerns after I got the case about airflow and keeping the drives
cool - however, monitoring the 4 drives I can, shows them to be running at
about 30C in my non AC office. Airflow is adequate through the drives and
I'm happy. The Tyan motherboard has a plethora of sensors too - 3 on the
motherboard, as well as one in each CPU, and Tyan (to their credit!)
supply (almost) the right runes required to make lm_sensors work. The case
comes with a fan and temperature monitoring board too, with 2 temperature
probes which you can stick somewhere inside. I connected the 3 internal
case fans to the motherboard which has space for 6 fans and can read them
via lm_sensors. It's a shame the PSU doesn't provide a tacho output for
its fans. I actually ran it for a couple of hours last night with the
front and side vents blocked to see quickly it would get to an
unacceptable temperature - not a terribly scientific test, and the fans
were still running. It got to 40C then stabilised. I guess there was
enough airflow through it somehow. The place it will be installed is an AC
computer room.

After experiments with RAID-6 on a test server, I've installed this box
with RAID-6 on all partitions except the root partition which is RAID-1.
(Even swap is on 3 x 4-way RAID-6 partitions, so sue me) Performance is
adequate although not stellar - a single run of bonnie yields:

Version 1.02b       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
mayday-ext3      2G 19098  98 66984  36 33246  19 19544  96 133123  38 300.2   1
mayday-xfs       2G 20066  98 75102  27 27659  15 19826  96 126766  38 386.3   1

A bit slow on writes, but maybe thats just RAID-6, although xfs improved
writes and seeks it's slower than the other stuff. Still, benchmarks are
not much use when compared to real-life!

I've since moved the 3 data partitions on this box over to XFS. Root and
/usr are still on ext3. Under XFS, It felt more responsive to interactive
stuff when I was running 2 copies of bonnie on each of the data
partitions. ie. I was still able to compile packages, kernel, etc. under
the /usr partition in a reasonable matter. It felt clunkier under ext3,
but this is just a feeling and nothing scientific. The applications it'll
run are MySQL and CVS (data on md5, md6 being an overnight snapshot which
gets dupped to tape. md7 is just data fileserverd via nfs and samba)

If anyones interested, it looks like:

Filesystem            Size  Used Avail Use% Mounted on
/dev/md1              471M  324M  122M  73% /
/dev/md3              1.9G  1.5G  425M  78% /usr
/dev/md5               46G  5.7G   40G  13% /mounts/local0
/dev/md6               46G  4.1G   41G   9% /mounts/local0.yesterday
/dev/md7              1.3T  528k  1.2T   1% /mounts/pdrive

all 8 disks are partitioned identically:

Disk /dev/sda: 255 heads, 63 sectors, 30401 cylinders
Units = cylinders of 16065 * 512 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/sda1   *         1        62    497983+  fd  Linux raid autodetect
/dev/sda2            63       186    996030   83  Linux
/dev/sda3           187       229    345397+  83  Linux
/dev/sda4           230     30401 242356590    5  Extended
/dev/sda5           230      1225   8000338+  83  Linux
/dev/sda6          1226      2221   8000338+  83  Linux
/dev/sda7          2222     30401 226355818+  83  Linux

Swap is comprised of 3 RAID-6 units, sd{a,b,c,d}2 (md10) + sd{e,f,g,h}2
(md11) + sd{e,f,g,h}1 (md12). /proc/swaps looks like:

Filename                       Type            Size    Used    Priority
/dev/md10                     partition       1991800 0       1
/dev/md11                     partition       1991800 0       1
/dev/md12                     partition       995704  0       0

I'll be surprised if this machine ever needs any swap, but it's there just
in-case.

So there you go. I've got a 2nd one of these to build now, which I already
have the same hardware for, (it'll be acting as a backup for this one) and
possibly a few more after that, although I won't be buying a Tyan
motherboard for them (The others don't require dual CPUs, being just
filestores and not filestore + applications servers)

This box has passed an initial 3-day test and will get a full weeks
soak-testing before it finally goes live, but so-far it's looking very
good.

Cheers,

Gordon

Re: Tyan, RAID-6, and other recent hassles... (long, a bit OT)

[berk walker]

Do you want a glass or some cheese?
Actually, I am thinking that your main problem is a generic [almost] 
BIOS issue, as no one in "right mind" would expect your configuration.  
Might I suggest a somewhat more expensive, yet safer work-around?

Split your drives between more boxes and gigabite link them.  If you 
work this well, you will have increased the survival of disk/other 
failures - stick 'em in the mail room, or where-ever.

You have spent some big bux to set this up, spend a few more and harden 
it. Eh?

Just an old guy rambling-

Gordon Henderson wrote:

>This is a bit OT and long, but it might help someone in the future, you
>never know!
>
>I've been struggling recently to get a box together with some supposedly
>nice hardware and it's turned out to be a bit of a nightmare.. The good
>news is that it's now sorted and working well enough to go into
>production.
>
>A big thanks to everyone who's contributed both on the list and in private
>email with some of the issues I've had with it.
>
>I've been building & running servers for many years, using Linux RAID for
>the past 5 or so, so thought this would be just another server (admittedly
>one of the biggest in disk terms I've built) alas it was nearly my
>nemesis!
>
>It's a 3U case with 8 hot-swap SATA drives and triple redundant 600W PSU.
>Nice case, 3 big fans inside, space for 3 5.25" units on one side. (I just
>have a CD-ROM drive in there). I opted for a 3U case rather than 2U just
>to make sure there was room inside it to take standard PCI cards without
>any risers and restricted air-flows. I chose a dual Opteron mobo (clients
>request) with on-board 4-port SATA controller (SII 3114) and initially got
>2 more SII based 2-port PCI cards.
>
>Mobo was a Tyan Thunder K8W. (S2885)
>1GB of Crucial RAM (2x512MB PC2700)
>Case: http://www.acme-technology.co.uk/acm338.htm
>8 x Hitachi Deskstar 250GB SATA.
>2 x Opteron 240 processors
>Debian Woody with 2.6 kernel.
>
>Then the trouble started )-:
>
>It seems that that motherboard, or the AMD chipset just can't hack PCI-X.
>(or maybe PCI cards in the PCI-X slots)  There are various jumpers and
>BIOS options to fiddle with, but nothing seemed to work well. It did seem
>to work better with just one PCI card though, but not perfect. I
>re-flashed the BIOS to their latest (beta) version and that was better but
>not 100%.
>
>The mobo on its own, with just 4 drives on the on-board controller seemed
>solid. It would boot OK, and run just fine, but as soon as I plugged
>additional SII cards into the PCI slots it all went pear-shaped.
>
>Finally, I got a 4-port Highpoint card (Rocket 1540) and that's made a lot
>of difference.
>
>I also found (along with someone else who emailled me about this), that
>the SATA cables supplied with the motherboard are less than reliable.
>Replacing them with nice flexable cables improved things too. I've
>subsequently gone off SATA. Damnit! The cables fiddly, the connectors
>fragile. Give me good old wide cables and chunky connectors anyday!
></rant> :)
>
>(FWIW: I tested the same disks and 4 2-port SII cards in a Xeon system
>with 4 PCI-X slots and it really flew, so I was confident it wasn't an OS
>problem, or a problem with the cards, or disks)
>
>The down-side is that the Highpoint driver is somewhat slower than the SII
>drivers (I'm losing ~5-8Mb/sec disk performance) and it's not open source.
>Another irritation is that it won't pass through the SMART commands. Yet
>another irritation is that it won't compile into the kernel, and must be
>loaded as a module, so I can't use auto-detection on the RAID arrays (I
>don't do initrd) No real issue, as in the startup scripts, I added another
>script after it checks the root filesystem, and before it checks & mounts
>the others - do an explicit modload, and explicit mdadm --assemble
>instructions.
>
>I also had issues trying to boot the damn thing. It really wasn't happy
>booting when the extra (SII) PCI cards installed, even when trying to just
>boot off the first drive on the on-board controller. In the end, I was
>booting it off an IDE flash drive and mounting / under /dev/sda1, then
>subsequently /dev/md1 (raid-1 of the first 4 drives on the on-board
>controller)
>
>Now, with a different chip-set PCI card, (The Highpoint) the BIOS is happy
>to boot off any one of the on-board drives, boot is /dev/md1, root is
>/dev/md1 and I'm happy. (md1 is a RAID-1 comprised of /dev/sd{a,b,c,d}1
>which are connected to the on-board SII 3114 controller)
>
>I've played with 2.6.10 and 2.6.11 RC kernels. Applied patches for the
>libata stuff to sort of make SMART work (the 4 drives on the 3114 need the
>-w flag to hddtemp, as it thinks they are asleep all the time), the
>Highpoint driver just won't pass the SMART commands.
>
>I had concerns after I got the case about airflow and keeping the drives
>cool - however, monitoring the 4 drives I can, shows them to be running at
>about 30C in my non AC office. Airflow is adequate through the drives and
>I'm happy. The Tyan motherboard has a plethora of sensors too - 3 on the
>motherboard, as well as one in each CPU, and Tyan (to their credit!)
>supply (almost) the right runes required to make lm_sensors work. The case
>comes with a fan and temperature monitoring board too, with 2 temperature
>probes which you can stick somewhere inside. I connected the 3 internal
>case fans to the motherboard which has space for 6 fans and can read them
>via lm_sensors. It's a shame the PSU doesn't provide a tacho output for
>its fans. I actually ran it for a couple of hours last night with the
>front and side vents blocked to see quickly it would get to an
>unacceptable temperature - not a terribly scientific test, and the fans
>were still running. It got to 40C then stabilised. I guess there was
>enough airflow through it somehow. The place it will be installed is an AC
>computer room.
>
>After experiments with RAID-6 on a test server, I've installed this box
>with RAID-6 on all partitions except the root partition which is RAID-1.
>(Even swap is on 3 x 4-way RAID-6 partitions, so sue me) Performance is
>adequate although not stellar - a single run of bonnie yields:
>
>Version 1.02b       ------Sequential Output------ --Sequential Input- --Random-
>                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
>Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
>mayday-ext3      2G 19098  98 66984  36 33246  19 19544  96 133123  38 300.2   1
>mayday-xfs       2G 20066  98 75102  27 27659  15 19826  96 126766  38 386.3   1
>
>A bit slow on writes, but maybe thats just RAID-6, although xfs improved
>writes and seeks it's slower than the other stuff. Still, benchmarks are
>not much use when compared to real-life!
>
>I've since moved the 3 data partitions on this box over to XFS. Root and
>/usr are still on ext3. Under XFS, It felt more responsive to interactive
>stuff when I was running 2 copies of bonnie on each of the data
>partitions. ie. I was still able to compile packages, kernel, etc. under
>the /usr partition in a reasonable matter. It felt clunkier under ext3,
>but this is just a feeling and nothing scientific. The applications it'll
>run are MySQL and CVS (data on md5, md6 being an overnight snapshot which
>gets dupped to tape. md7 is just data fileserverd via nfs and samba)
>
>If anyones interested, it looks like:
>
>Filesystem            Size  Used Avail Use% Mounted on
>/dev/md1              471M  324M  122M  73% /
>/dev/md3              1.9G  1.5G  425M  78% /usr
>/dev/md5               46G  5.7G   40G  13% /mounts/local0
>/dev/md6               46G  4.1G   41G   9% /mounts/local0.yesterday
>/dev/md7              1.3T  528k  1.2T   1% /mounts/pdrive
>
>all 8 disks are partitioned identically:
>
>Disk /dev/sda: 255 heads, 63 sectors, 30401 cylinders
>Units = cylinders of 16065 * 512 bytes
>
>   Device Boot    Start       End    Blocks   Id  System
>/dev/sda1   *         1        62    497983+  fd  Linux raid autodetect
>/dev/sda2            63       186    996030   83  Linux
>/dev/sda3           187       229    345397+  83  Linux
>/dev/sda4           230     30401 242356590    5  Extended
>/dev/sda5           230      1225   8000338+  83  Linux
>/dev/sda6          1226      2221   8000338+  83  Linux
>/dev/sda7          2222     30401 226355818+  83  Linux
>
>Swap is comprised of 3 RAID-6 units, sd{a,b,c,d}2 (md10) + sd{e,f,g,h}2
>(md11) + sd{e,f,g,h}1 (md12). /proc/swaps looks like:
>
>Filename                       Type            Size    Used    Priority
>/dev/md10                     partition       1991800 0       1
>/dev/md11                     partition       1991800 0       1
>/dev/md12                     partition       995704  0       0
>
>I'll be surprised if this machine ever needs any swap, but it's there just
>in-case.
>
>So there you go. I've got a 2nd one of these to build now, which I already
>have the same hardware for, (it'll be acting as a backup for this one) and
>possibly a few more after that, although I won't be buying a Tyan
>motherboard for them (The others don't require dual CPUs, being just
>filestores and not filestore + applications servers)
>
>This box has passed an initial 3-day test and will get a full weeks
>soak-testing before it finally goes live, but so-far it's looking very
>good.
>
>Cheers,
>
>Gordon
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>.
>
>  
>

Re: Tyan, RAID-6, and other recent hassles... (long, a bit OT)

[Gordon Henderson]

On Sat, 19 Feb 2005, berk walker wrote:

> Do you want a glass or some cheese?

Not really... I just thought I'd pass on my experiences and thank those
who gave me support recently. By posting my configurations and thoughts
and issues I've encountered during the way, I'm essentially opening myself
up for a peer review if you like. I'm not saying my way is the best way,
but it's one way. If others can learn from it, great. If they want to
criticise it, thats also good, but it's only good if it's constructive.

But a glass of Old Peculiar would go down nicely, thanks :)

> Actually, I am thinking that your main problem is a generic [almost]
> BIOS issue, as no one in "right mind" would expect your configuration.

Expect my configuration ... what? To work? Why not? It's a motherboard
with 4 PCI-X slots and a single 32-bit PCI slot. Why shouldn't it work?

Or do you mean expect my configuration to exist at all as you think it's
utterly preposterous?

Right now it's working well, and it's about to be installed at the clients
site where it'll run and be thrashed for a week at least before we put
live data on it. Even then, we'll keep the old server (which it's
replacing) going for a month or so until we're finally happy with it.

I'm sure there is a BIOS or motherboard/chipset problem though and Tyan
have some sorting out to do.  I have emailled them with all my issues and
concerns, but not had anything back yet.

> Might I suggest a somewhat more expensive, yet safer work-around?

Feel free...

> Split your drives between more boxes and gigabite link them.  If you
> work this well, you will have increased the survival of disk/other
> failures - stick 'em in the mail room, or where-ever.

The server that this box is replacing has 12 disks. (Which I built some
years back) This has 8. It has 12x the disk capacity and cost less than
1/3 the old server did. It has redundant PSUs bought from a company that
has been supplying server cases for over 10 years. It'll be installed in
an air-conditioned machine room with dual 16KVA UPSs... Why should this
server with multiple disks pose a problem?

I've built many servers with multiple disks, and they all work well,
after-all, thats what this mailing list is about - Linux with multiple
disks!

I've had to work round buggy motherboards in the past (Dual Athlon boards)
and in that respect, this is not much different. I did at one point have 2
(server) motherboards which had the "exploding capacitor" problems, but
fortunately we were able to secure replacements before they actually
exploded.

One requirement for this server is for a very large filestore. TB or
greater. I won't get that if I split the disks between servers. (Can you
build an md device from network block devices?) I'm using RAID-6 as I've
been bitten in the past with a 2-disk failure. (and been able to recover
from it by using mdadm and advice given to others via this mailing list)

This server will have a backup server, identical in configuration
(although that's arguably not the best solution). The critical data will
be backed to tape (as it is currently on the server it's replacing) and we
have a good program of tape cycling with off-site backups being held. The
file-store part just has to be reliable - it's all re-generable (program
binaries, libraries, etc) so it doesn't have to be backed up to tape.

The client already has nearly a dozen other fileservers which I've built
for them over the years. This isn't the one server to serve them all, it's
just one small piece in their network of servers.  They are a small
silicon design co. but they have a huge data storage requirement.
(Ironically their data storage requirements increases almost in proportion
to Moores Law :)

And tomorrow I'll be installing their first Gigabit Ethernet switch. This
server box has Gb Ethernet. (although I have graphs from all their existing
switches to prove that they don't actually need Gb Ethernet, but it's the
way of the future, isn't it?)

> You have spent some big bux to set this up, spend a few more and harden
> it. Eh?

I haven't spent "big bux" at all. I've (or rather my client) has spent
less than £4K for 2 identical servers. I've spent a lot of time on it,
sure, but my time comes at a constant cost for this client, so isn't a
factor in this. I'd rather spend a lot of time on it now, than rushing it
into place and then have to spend more time on it in the future. They are
getting the server 2-3 weeks later than originally planned, but they can
live with that. (And I'm 100 miles from their site, so not having to rush
up the motorway if it does fail is a plus-point for me)

My client is a small company. 40 people, limited funds, (and no mail
room!), but great ideas. They've gone through good times and bad times
over the years I've been working with them.  Right now is a good time, but
money is still tight. I agree that you get what you pay for, but sometimes
you just have to make do. When times have been good, they've bought Dell
servers which I've installed Debian and s/w RAID on, and it's all "jsut
worked", but when times aren't so good, you have no choice but to source
from scratch and build according to budget. These servers will serve the
purpose they want - affordable storage and compute for the application
they have been built for (a combined MySQL/CVS home-grown application, as
well as a disk store).

I've been working for these guys for over 6 years now, and in that time
all the servers I've built for them have been gracefully retired rather
than gone terminally tits-up. I'm quite proud of that. It's not been easy,
but with time and perseverance, and good help from the "community"
everythings worked out just fine. The first server I built for them is now
sitting next to me at home still running. It had 4 x 18Gb drives which we
very soon upgraded to 8 18Gb drives, both drive sets running s/w RAID 5.
It had 100GB of storage on it 5 years ago, and its performance (for the
day) was stellar. Nowadays that's just peanuts, but thats progress for
you!

> Just an old guy rambling-

Gordon,
  just another old guy making a living.
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Tyan, RAID-6, and other recent hassles... (long, a bit OT)

[berk walker]

[I usually do not spend bandwidth in quoting big stuff, but your's might 
be worth it]
Properly chastised.  One CAN do net raid, 4,000 [where's my pound key?] 
is still a lot to me, [don't forget my name IS berk :)]

One doesn't always get what one pays for - but one usually pays for what 
one gets.


Too bad that you're not stateside.  I really like your attitude 
[email-wise], and would hunt you down for a job [mine sux].

b-

Gordon Henderson wrote:

>On Sat, 19 Feb 2005, berk walker wrote:
>
>  
>
>>Do you want a glass or some cheese?
>>    
>>
>
>Not really... I just thought I'd pass on my experiences and thank those
>who gave me support recently. By posting my configurations and thoughts
>and issues I've encountered during the way, I'm essentially opening myself
>up for a peer review if you like. I'm not saying my way is the best way,
>but it's one way. If others can learn from it, great. If they want to
>criticise it, thats also good, but it's only good if it's constructive.
>
>But a glass of Old Peculiar would go down nicely, thanks :)
>
>  
>
>>Actually, I am thinking that your main problem is a generic [almost]
>>BIOS issue, as no one in "right mind" would expect your configuration.
>>    
>>
>
>Expect my configuration ... what? To work? Why not? It's a motherboard
>with 4 PCI-X slots and a single 32-bit PCI slot. Why shouldn't it work?
>
>Or do you mean expect my configuration to exist at all as you think it's
>utterly preposterous?
>
>Right now it's working well, and it's about to be installed at the clients
>site where it'll run and be thrashed for a week at least before we put
>live data on it. Even then, we'll keep the old server (which it's
>replacing) going for a month or so until we're finally happy with it.
>
>I'm sure there is a BIOS or motherboard/chipset problem though and Tyan
>have some sorting out to do.  I have emailled them with all my issues and
>concerns, but not had anything back yet.
>
>  
>
>>Might I suggest a somewhat more expensive, yet safer work-around?
>>    
>>
>
>Feel free...
>
>  
>
>>Split your drives between more boxes and gigabite link them.  If you
>>work this well, you will have increased the survival of disk/other
>>failures - stick 'em in the mail room, or where-ever.
>>    
>>
>
>The server that this box is replacing has 12 disks. (Which I built some
>years back) This has 8. It has 12x the disk capacity and cost less than
>1/3 the old server did. It has redundant PSUs bought from a company that
>has been supplying server cases for over 10 years. It'll be installed in
>an air-conditioned machine room with dual 16KVA UPSs... Why should this
>server with multiple disks pose a problem?
>
>I've built many servers with multiple disks, and they all work well,
>after-all, thats what this mailing list is about - Linux with multiple
>disks!
>
>I've had to work round buggy motherboards in the past (Dual Athlon boards)
>and in that respect, this is not much different. I did at one point have 2
>(server) motherboards which had the "exploding capacitor" problems, but
>fortunately we were able to secure replacements before they actually
>exploded.
>
>One requirement for this server is for a very large filestore. TB or
>greater. I won't get that if I split the disks between servers. (Can you
>build an md device from network block devices?) I'm using RAID-6 as I've
>been bitten in the past with a 2-disk failure. (and been able to recover
>from it by using mdadm and advice given to others via this mailing list)
>
>This server will have a backup server, identical in configuration
>(although that's arguably not the best solution). The critical data will
>be backed to tape (as it is currently on the server it's replacing) and we
>have a good program of tape cycling with off-site backups being held. The
>file-store part just has to be reliable - it's all re-generable (program
>binaries, libraries, etc) so it doesn't have to be backed up to tape.
>
>The client already has nearly a dozen other fileservers which I've built
>for them over the years. This isn't the one server to serve them all, it's
>just one small piece in their network of servers.  They are a small
>silicon design co. but they have a huge data storage requirement.
>(Ironically their data storage requirements increases almost in proportion
>to Moores Law :)
>
>And tomorrow I'll be installing their first Gigabit Ethernet switch. This
>server box has Gb Ethernet. (although I have graphs from all their existing
>switches to prove that they don't actually need Gb Ethernet, but it's the
>way of the future, isn't it?)
>
>  
>
>>You have spent some big bux to set this up, spend a few more and harden
>>it. Eh?
>>    
>>
>
>I haven't spent "big bux" at all. I've (or rather my client) has spent
>less than £4K for 2 identical servers. I've spent a lot of time on it,
>sure, but my time comes at a constant cost for this client, so isn't a
>factor in this. I'd rather spend a lot of time on it now, than rushing it
>into place and then have to spend more time on it in the future. They are
>getting the server 2-3 weeks later than originally planned, but they can
>live with that. (And I'm 100 miles from their site, so not having to rush
>up the motorway if it does fail is a plus-point for me)
>
>My client is a small company. 40 people, limited funds, (and no mail
>room!), but great ideas. They've gone through good times and bad times
>over the years I've been working with them.  Right now is a good time, but
>money is still tight. I agree that you get what you pay for, but sometimes
>you just have to make do. When times have been good, they've bought Dell
>servers which I've installed Debian and s/w RAID on, and it's all "jsut
>worked", but when times aren't so good, you have no choice but to source
>from scratch and build according to budget. These servers will serve the
>purpose they want - affordable storage and compute for the application
>they have been built for (a combined MySQL/CVS home-grown application, as
>well as a disk store).
>
>I've been working for these guys for over 6 years now, and in that time
>all the servers I've built for them have been gracefully retired rather
>than gone terminally tits-up. I'm quite proud of that. It's not been easy,
>but with time and perseverance, and good help from the "community"
>everythings worked out just fine. The first server I built for them is now
>sitting next to me at home still running. It had 4 x 18Gb drives which we
>very soon upgraded to 8 18Gb drives, both drive sets running s/w RAID 5.
>It had 100GB of storage on it 5 years ago, and its performance (for the
>day) was stellar. Nowadays that's just peanuts, but thats progress for
>you!
>
>  
>
>>Just an old guy rambling-
>>    
>>
>
>Gordon,
>  just another old guy making a living.
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>.
>
>  
>

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Tyan, RAID-6, and other recent hassles... (long, a bit OT)

[Bernd Schubert]

Hi,

> Mobo was a Tyan Thunder K8W. (S2885)

personally I'm very disappointed about Tyan, well I only know their
opteron mainboards, but thats already sufficient to never buy their
mainboards again.

Server B (failover): S2880, as soon as a cable is plugged in into one of
the onboard broadcom NICs, the system crashes. Broadcom driver doesn't
need to be loaded, system may already crash during the pre-boot
initialisation or later on during the kernel boot-procedure.

Server A (main): First S2882 board crashed rather soon during pre-boot
initialisation, in the bios setup, during the boot-procedure and
sometimes it was even running stable - for a few hours. After returning
the server, we got back the system with a new mainboard (if it only
hadn't taken them two month to do this :-( ). 
With the new mainboard it was running stable for about 7 month and, we only 
experienced some rather strange ext2 errors. Interesting also to note
that those ext2 partitions were mirrored by drbd and those errors could
not be observed on the corresponding partions on Server B.
After exchanging one of the software components that required ext2, we 
could get completely rid of those ext2 partitions and everything was
fine until 1.5 weeks ago. The server suddenly crashed and crashed again
shortly after rebooting. Reiserfsck found pretty much problems on the
root partions and we decided to use our backup. For some reasons we are
pretty convinced that those ext2 and reiserfs errors are due to the
onbaord SIL3114 controller (its probably not RAM, since we monitor the
ECC memory using bluesmoke, futhermore, the much bigger data partitions
connected to the onboard SCSI controller are (so far) not affected by any
of those problems) . We now added an additional promise controller. 
Unfortunately this didn't solve our crash problem. After updating the
bios to the recent version, the promise bios complained that it can't
find any harddisk, downgrading to the version from last July solved this
problem and also almost our crash problem (bevor updating the bios was
from last March or April). It now only crashes sometimes during the bios
initialisation...

Cluster, 16 nodes: S2881 boards, master server crashed rather often
until we updated to linux-2.6.9 (old kernel was 2.4.27).


Cheers,
	Bernd


PS: I'm also mailling this to Tyan. I'm curious what they will say about
this report.

Re: Tyan, RAID-6, and other recent hassles... (long, a bit OT)

[Ricky Beam]

On Sun, 20 Feb 2005, Bernd Schubert wrote:
>> Mobo was a Tyan Thunder K8W. (S2885)
>
>personally I'm very disappointed about Tyan, well I only know their
>opteron mainboards, but thats already sufficient to never buy their
>mainboards again.

You're doing yurself a huge disservice based on very little experience.
Using one or two of a manufacturer's MBs is not sufficient grounds to
blacklist them.  Tyan sells a lot of these boards.  If there were serious
stability problems with them, tyan wouldn't be shipping them.

Tyan makes very good hardware.  I don't know what your problems are,
but I've never had any such problems.  I've used Tyan hardware for
a decade.  Yes, they've made a few mistakes over the years -- bad
SIMM slots on the earily tomcat boards, using a VIA chipset for the
tiger 133...

>Server B (failover): S2880, as soon as a cable is plugged in into one of
>the onboard broadcom NICs, the system crashes. Broadcom driver doesn't
>need to be loaded, system may already crash during the pre-boot
>initialisation or later on during the kernel boot-procedure.

I've seen a few systems do this.  It was due to bad memory and bad memory
configuration (BIOS).  Those systems (i7501's) have been running fine
(over a year now) since replacing the RAM with Tyan "ok'ed" memory and
disabling "spread spectrum" in the chipset.

--Ricky

Re: Tyan, RAID-6, and other recent hassles... (long, a bit OT)

[Alvin Oga]

hi ya

On Sat, 19 Feb 2005, Ricky Beam wrote:

> On Sun, 20 Feb 2005, Bernd Schubert wrote:
> >> Mobo was a Tyan Thunder K8W. (S2885)
> >
> >personally I'm very disappointed about Tyan, well I only know their
> >opteron mainboards, but thats already sufficient to never buy their
> >mainboards again.
> 
> You're doing yurself a huge disservice based on very little experience.
> Using one or two of a manufacturer's MBs is not sufficient grounds to
> blacklist them.  Tyan sells a lot of these boards.  If there were serious
> stability problems with them, tyan wouldn't be shipping them.
> 
> Tyan makes very good hardware.  I don't know what your problems are,
> but I've never had any such problems.

ditto ... i would buy tyan motherboards since its very cost effective
for its performance and very good reliability 
( it's my 2nd choice mb for certain apps and sometimes 1st choice )

we buy lots of um  for customers that requests certain models 

--- 

usually, if there's hardware problems, it's usually an installation
problem and/or very unlikely, 1%-5% probability of bad hardware
	- things dont always work out of the box ( cdrom installation )
	and one *-do-* need to know how to tweek it to make it work
	"right"

	- one should buy parts from reputable cpu/mb/disk distributors 
	and not any mom-n-pop website that sells "everything"

c ya
alvin

Re: Tyan, RAID-6, and other recent hassles... (long, a bit OT)

[Tim Moore]

I've been running an S2885 since Feb 1.  After flashing to 2.02b and 
replacing a faulty SATA cable, zero problems.  System runs constantly as a 
NAS load generator client except for shutdowns needed to test different 
disks or memory configs.

I'll be getting 3 more next month, racked using:
http://www.antec.com/us/productDetails.php?ProdID=94326
http://www.antec.com/us/productDetails.php?ProdID=94550

Also a good EPS12V power supply does wonders.

Cheers.

Ricky Beam wrote:
> On Sun, 20 Feb 2005, Bernd Schubert wrote:
> 
>>>Mobo was a Tyan Thunder K8W. (S2885)
>>
>>personally I'm very disappointed about Tyan, well I only know their
>>opteron mainboards, but thats already sufficient to never buy their
>>mainboards again.
> 
> 
> You're doing yurself a huge disservice based on very little experience.
> Using one or two of a manufacturer's MBs is not sufficient grounds to
> blacklist them.  Tyan sells a lot of these boards.  If there were serious
> stability problems with them, tyan wouldn't be shipping them.
> 
> Tyan makes very good hardware.  I don't know what your problems are,
> but I've never had any such problems.  I've used Tyan hardware for
> a decade.  Yes, they've made a few mistakes over the years -- bad
> SIMM slots on the earily tomcat boards, using a VIA chipset for the
> tiger 133...
> 
> 
>>Server B (failover): S2880, as soon as a cable is plugged in into one of
>>the onboard broadcom NICs, the system crashes. Broadcom driver doesn't
>>need to be loaded, system may already crash during the pre-boot
>>initialisation or later on during the kernel boot-procedure.
> 
> 
> I've seen a few systems do this.  It was due to bad memory and bad memory
> configuration (BIOS).  Those systems (i7501's) have been running fine
> (over a year now) since replacing the RAM with Tyan "ok'ed" memory and
> disabling "spread spectrum" in the chipset.
> 
> --Ricky
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 

Re: Tyan, RAID-6, and other recent hassles... (long, a bit OT)

[Gordon Henderson]

On Sat, 19 Feb 2005, berk walker wrote:

> [I usually do not spend bandwidth in quoting big stuff, but your's might
> be worth it]
> Properly chastised.  One CAN do net raid, 4,000 [where's my pound key?]
> is still a lot to me, [don't forget my name IS berk :)]

Thats for 2 servers, remember. Worry not though! I didn't post the full
background of this one, so it might have been easy to come to the wrong
conclusions.

> One doesn't always get what one pays for - but one usually pays for what
> one gets.

> Too bad that you're not stateside.  I really like your attitude
> [email-wise], and would hunt you down for a job [mine sux].

I spent 2 years working in the US once upon a time... Not sure I'd like to
go back! I've spent a lot of time working with startup companies in the
past (present and likely future!) it's a funny old business. Sometimes you
get a group that seem to do well, but fail in some odd way, sometimes
people fail, pick themselves up, fail, etc. (And some just deserve to die,
be burried and totally forgotten about!!!)

The new box was installed yesterday and has been running disk & network
tests for a while and so-far so good. I don't have Gb Ethernet in my home
office, so hadn't had a chace to test it at full line speeds, bu right now
it's going well. I typically run multiple copies of bonnie and wget
to/from another server to thrash the networking, then will do stuff via
NFS, transfer big files, checksum them, etc. as well as lots of little
files. Anyone have a good solid testsuite then like to run?

Cheers,

Gordon

Re: Tyan, RAID-6, and other recent hassles... (long, a bit OT)

[Mike Hardy]

Gordon Henderson wrote:

> to/from another server to thrash the networking, then will do stuff via
> NFS, transfer big files, checksum them, etc. as well as lots of little
> files. Anyone have a good solid testsuite then like to run?

Very off-topic at this point, but apart from transfers with checksums, I 
use a pair of ttcp processes to pack the network pipe solid, and the 
linux kernel source is a great CPU/RAM test if you unpack/untar, copy, 
diff, delete repeat

Just be sure to watch the temparature ;-)

-Mike