[parisc-linux] 'foreign' PCI cards in an rp2430

[parisc-linux] 'foreign' PCI cards in an rp2430

[Duraid Madina]

Hi all,

	I'm considering buying some rp2430s (at last, PA-RISC at the
Right Price, or so it seems), and I want to run Linux on them (they're
just a 'compute cluster')

	I'm wondering: Would I be able to use non-HP PCI cards in these
systems, if they're supported under Linux? In particular, I'm wondering
about 3ware IDE RAID cards (these work in (at least) alpha and x86
linux) and Intel gigabit ethernet NICs (based on the 82554 chip).

	And (no laughter, please) what's the maximum swap partition size
parisc linux supports? Can parisc linux swap over NFS?

	Sorry for the oddball questions,

	Duraid

Re: [parisc-linux] 'foreign' PCI cards in an rp2430

[andi]

> Can parisc linux swap over NFS?

no, swapping over nfs does not work, however, using network block device
should work, and is much faster than nfs.

do do that you need to enable network block device in the client's
kernel, and you need two special userspace tools (debian packages
nbd-server and nbd-client). the nbd-server in debian is broken you
should get the latest upstream version and compile it yourself.

good luck

andi

Re: [parisc-linux] 'foreign' PCI cards in an rp2430

[Matthew Wilcox]

On Fri, Mar 08, 2002 at 02:51:03PM +1100, Duraid Madina wrote:
> 	I'm wondering: Would I be able to use non-HP PCI cards in these
> systems, if they're supported under Linux? In particular, I'm wondering
> about 3ware IDE RAID cards (these work in (at least) alpha and x86
> linux) and Intel gigabit ethernet NICs (based on the 82554 chip).

*puts kernel hacker's hat on*
Sure!  All that stuff should work and if it doesn't it can be fixed.

*puts HP employee hat on*
Well, it might invalidate your warranty to put non-HP certified cards
in your system.

> 	And (no laughter, please) what's the maximum swap partition size
> parisc linux supports? Can parisc linux swap over NFS?

Forget about swapping over NFS.  It's prone to deadlock under
stress.  I think we support 2GB/file and up to 8 files, but I may be
underestimating... John would know.

-- 
Revolutions do not require corporate support.

Re: [parisc-linux] 'foreign' PCI cards in an rp2430

[Alan Cox]

> > 	And (no laughter, please) what's the maximum swap partition size
> > parisc linux supports? Can parisc linux swap over NFS?
> 
> Forget about swapping over NFS.  It's prone to deadlock under
> stress.  I think we support 2GB/file and up to 8 files, but I may be
> underestimating... John would know.

NFS swapping probably isnt the best of ideas - if you ensure your atomic
memory pool is big enough you can make it work though. Nbd will also work
subject to the same kind of constraints. There are some patches around to
make nfs/nbd swap reliable by putting some hacks into the network code

RE: [parisc-linux] 'foreign' PCI cards in an rp2430

[Duraid Madina]

Matthew wrote:
> *puts kernel hacker's hat on*
> Sure!  All that stuff should work and if it doesn't it can be fixed.

Sweet.
 
> *puts HP employee hat on*
> Well, it might invalidate your warranty to put non-HP 
> certified cards in your system.

Grr, someone should stop HP from calling them PCI slots since they're
only rated to 20W/slot anyway ;) (which is out of spec, iirc)
 
> > 	And (no laughter, please) what's the maximum swap 
> partition size 
> > parisc linux supports? Can parisc linux swap over NFS?
> 
> Forget about swapping over NFS.  It's prone to deadlock under 
> stress.  I think we support 2GB/file and up to 8 files, but I 
> may be underestimating... John would know.

Damn! I was hoping that I could have something like 200GB of swap or so.
Would it be difficult to add support for more than 8 swap files?

	Thanks,

	Duraid

Re: [parisc-linux] 'foreign' PCI cards in an rp2430

[Martin K. Petersen]

>>>>> "Duraid" == Duraid Madina <duraid@fl.net.au> writes:

Duraid> In particular, I'm wondering about 3ware IDE RAID cards (these
Duraid> work in (at least) alpha and x86 linux)

I don't think there are high hopes for the 3ware card.  At least not
unless you preconfigure it on a PC.

Alpha happens to work because its firmware includes a simple x86
simulator that executes the BIOS code on the card.  That way you can
configure your disk setup, etc.

We don't have similar functionality on PA-RISC.  So you are limited
what can be done from within Linux. 

Unfortunately the 3ware management utility is 386 binary-only.
I.e. you can't configure the card once you get Linux up and running
either.

-- 
Martin K. Petersen      Cereal Bowl Engineer, Linuxcare, Inc.
http://mkp.net/         SGI XFS, Linux/PA-RISC, GNOME

Re: [parisc-linux] 'foreign' PCI cards in an rp2430

[Alan Cox]

> Alpha happens to work because its firmware includes a simple x86
> simulator that executes the BIOS code on the card.  That way you can
> configure your disk setup, etc.

There is btw a passable x86 emulator for free in the XFree86 code that
is used to do things like boot video cards.

> Unfortunately the 3ware management utility is 386 binary-only.
> I.e. you can't configure the card once you get Linux up and running
> either.

Bochs 8)

Re: [parisc-linux] 'foreign' PCI cards in an rp2430

[Martin K. Petersen]

>>>>> "Alan" == Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

>> Alpha happens to work because its firmware includes a simple x86
>> simulator that executes the BIOS code on the card.  That way you
>> can configure your disk setup, etc.

Alan> There is btw a passable x86 emulator for free in the XFree86
Alan> code that is used to do things like boot video cards.

Yep.  It's going to take a fairly big hammer to beat libint10 into
submission, though.  Interactively mocking with disk configuration and
whatnot is a bit more complex than kicking off some init fluff on a
video card.


>> Unfortunately the 3ware management utility is 386 binary-only.
>> I.e. you can't configure the card once you get Linux up and running
>> either.

Alan> Bochs 8)

It would actually be useful to hack it so we had a generic way of
running x86 card ROMs on non-{PC,Alpha,Itanium} architectures.

-- 
Martin K. Petersen      Cereal Bowl Engineer, Linuxcare, Inc.
http://mkp.net/         SGI XFS, Linux/PA-RISC, GNOME

Re: [parisc-linux] 'foreign' PCI cards in an rp2430

[James S]

> > Alpha happens to work because its firmware includes a simple x86
> > simulator that executes the BIOS code on the card.  That way you can
> > configure your disk setup, etc.
> 
> There is btw a passable x86 emulator for free in the XFree86 code that
> is used to do things like boot video cards.
> 
> > Unfortunately the 3ware management utility is 386 binary-only.
> > I.e. you can't configure the card once you get Linux up and running
> > either.
> 
> Bochs 8)
> 

out of sheer curiosity, i installed bochs on my 712/60

running the linux demo that comes with it, i got a bogoMIPS rating of 0.82 

hehe :)

James S

RE: [parisc-linux] 'foreign' PCI cards in an rp2430

[Duraid Madina]

Martin wrote:
> I don't think there are high hopes for the 3ware card.  At 
> least not unless you preconfigure it on a PC.

For sure.
 
> Alpha happens to work because its firmware includes a simple 
> x86 simulator that executes the BIOS code on the card.  That 
> way you can configure your disk setup, etc.

Oops, I think I was misleading when I said 3ware 'worked' in Alpha - 3dm
doesn't (of course) and yes, you need to do all array administration
stuff out of the BIOS on a PC.

> We don't have similar functionality on PA-RISC.  So you are 
> limited what can be done from within Linux. 

Well, so long as it can find the array and read/write, I'm set.
 
> Unfortunately the 3ware management utility is 386 
> binary-only. I.e. you can't configure the card once you get 
> Linux up and running either.

I have to admit, I find hardly any need for that tool even in x86 land
;)

	Duraid

Re: [parisc-linux] 'foreign' PCI cards in an rp2430

[nick]

Erm.  Whare are you going to put the disks for the 3ware?
	Nick

On Fri, 8 Mar 2002, Duraid Madina wrote:

> Hi all,
> 
> 	I'm considering buying some rp2430s (at last, PA-RISC at the
> Right Price, or so it seems), and I want to run Linux on them (they're
> just a 'compute cluster')
> 
> 	I'm wondering: Would I be able to use non-HP PCI cards in these
> systems, if they're supported under Linux? In particular, I'm wondering
> about 3ware IDE RAID cards (these work in (at least) alpha and x86
> linux) and Intel gigabit ethernet NICs (based on the 82554 chip).
> 
> 	And (no laughter, please) what's the maximum swap partition size
> parisc linux supports? Can parisc linux swap over NFS?
> 
> 	Sorry for the oddball questions,
> 
> 	Duraid
> 
> 
> 
> 
> _______________________________________________
> parisc-linux mailing list
> parisc-linux@lists.parisc-linux.org
> http://lists.parisc-linux.org/mailman/listinfo/parisc-linux
> 

RE: [parisc-linux] 'foreign' PCI cards in an rp2430

[Duraid Madina]

> Erm.  Whare are you going to put the disks for the 3ware?

On top of the box,
unshielded disks gather dust.
I hear them rattle.

	Duraid

Re: [parisc-linux] 'foreign' PCI cards in an rp2430

[Grant Grundler]

"Duraid Madina" wrote:
> ...and Intel gigabit ethernet NICs (based on the 82554 chip).

I've tested the e1000 (4.0.24-k1, iirc) driver on HP's "2cd generation
Intel (r) Itanium (tm)" systems and it basically works. Based on
this, I would expect this e1000 driver to work on A500-like systems.
(ie rp24xx boxes). Someday, if i get the itch, I'll plug one into
my a500 and try it.

and later:
> A computational physics code. Ideally I would run it in 100s of GB of
> RAM. About two months of pain went into making it work (reasonably
> efficiently, in theory at least) in 100s of GB of swap.

The approach seems really odd - you are looking at 64-bit userspace so
you can *swap* large amounts of virtual memory in/out. If you expected
to be IO bound, tuning the app to know more about IO (instead of
swapping) makes more sense.

Have you tried begging, trading, or buying some cycles on a Superdome
with some 100GBs of RAM instead of waiting for weeks to see your jobs
run to completion?

my $0.02,
grant

RE: [parisc-linux] 'foreign' PCI cards in an rp2430

[Duraid Madina]

Grant wrote:
> and later:
> > About two months of pain went into making it work (reasonably 
> > efficiently, in theory at least) in 100s of GB of swap.
> 
> The approach seems really odd - you are looking at 64-bit 
> userspace so you can *swap* large amounts of virtual memory 
> in/out. If you expected to be IO bound, tuning the app to 
> know more about IO (instead of swapping) makes more sense.

Normally, yes. In my case, not really. To cut a long story short,
because C++ has poor intrinsic support for reflection, even making my
app *do* IO is non-trivial. E.g. my program instantiates millions of
different objects (which interact in various interesting ways). The cost
of serializing/deserializing these objects to store them in a light
database like berkeleyDB, or simply as files is great enough that the
job becomes CPU bound, and not I/O bound. Being CPU bound is not where I
want to be, when I'm seeing only 2mb/sec of I/O ;)

What's even worse is that I'm using some truly unwiedly third party
class libraries (Hi Richard! ;) which don't provide serialization, so
again, instantiating 'plain old objects' seems to be the path of least
resistance, in my case.

What I was hoping to do is take an athlon box with fast raid and gigabit
ethernet, and buy one of those nice new rp2430s, having it swap over
ethernet. It seems that I can do this with HP-UX, though perhaps not
with G++. I would prefer to run Linux on the rp2430 though, since then I
could use a cheap Intel gigabit NIC (instead of paying $$$ for an HP
one) or better yet, stick my 3ware RAID card in the rp2430, and do away
with the network altogether. That I *can't* do with HP-UX, so...

> Have you tried begging, trading, or buying some cycles on a 
> Superdome with some 100GBs of RAM instead of waiting for 
> weeks to see your jobs run to completion?

Really, my code should be pretty efficient (it takes weeks to run anyway
;) with the setup I've described above, but the killer is: I'm a
University student ("I knew it!" I hear you all say ;) and as such, my
code is perpetually in the "hmm, I wonder what'd happen if.." state. So
"some cycles" would become "some more cycles" faster than you could say
"where's all my RAM gone?!"

	Duraid

P.S. Am I really the only sicko who's wanted a 64-bit machine just so
they can abuse virtual memory? Surely not... ;)

P.P.S. It's true, I do love the smell, the feel, the sound of striped,
thrashing, consumer-level hard drives. C'mon guys, help this sicko make
his dream come true (:

Re: [parisc-linux] 'foreign' PCI cards in an rp2430

[Carlos O'Donell Jr.]

> 
> > Have you tried begging, trading, or buying some cycles on a 
> > Superdome with some 100GBs of RAM instead of waiting for 
> > weeks to see your jobs run to completion?
> 
> Really, my code should be pretty efficient (it takes weeks to run anyway
> ;) with the setup I've described above, but the killer is: I'm a
> University student ("I knew it!" I hear you all say ;) and as such, my
> code is perpetually in the "hmm, I wonder what'd happen if.." state. So
> "some cycles" would become "some more cycles" faster than you could say
> "where's all my RAM gone?!"
> 
> 	Duraid
>

This is a very similar problem that we had when calculating prime
alternating knots[1]. Essentially what was done was to use MPI[2] 
and a fast network connection, such that you could have _other_ 
computers holding data in RAM/SWAP.

Parallelize parallelize parallelize.

It's really easy to learn MPI and it's great for those big problems
that you may not be able to do with a 64-bit box :)

a. Get a 100MBit/1GBit switch and NIC's.
b. Get some more boxes (borrow/lie/cheat/steal)
c. Tie them together.

This is what I do for a hobby :) ... and yes I'm a student aswell.
It's nice to see that people like Bell and Nortel are willing to 
donate expensive gear for projects like yours or mine[3] :)

c. 

[1] http://baldric.uwo.ca/article.php3?section=baldric&article=knots
[2] http://www-unix.mcs.anl.gov/mpi/mpich/
[2] http://www.lam-mpi.org/
[3] http://www.baldric.uwo.ca/~carlos/pdisplay.html

Re: [parisc-linux] 'foreign' PCI cards in an rp2430

[Grant Grundler]

"Duraid Madina" wrote:
> Normally, yes. In my case, not really. To cut a long story short,
> because C++ has poor intrinsic support for reflection, even making my
> app *do* IO is non-trivial. E.g. my program instantiates millions of
> different objects (which interact in various interesting ways). The cost
> of serializing/deserializing these objects to store them in a light
> database like berkeleyDB, or simply as files is great enough that the
> job becomes CPU bound, and not I/O bound. Being CPU bound is not where I
> want to be, when I'm seeing only 2mb/sec of I/O ;)

FWIW, you should expect to see 30-40MB/s to single disk on rp2430.
More with more disks/channels.

> I would prefer to run Linux on the rp2430 though, since then I
> could use a cheap Intel gigabit NIC (instead of paying $$$ for an HP

Well, 2.4 has no e1000 driver (yet) much less one that is known to work
on hppa. I *think* the e1000 driver I "leveraged" from 2.5 release for
HP ia64 machine should also work for hppa. But that's just an educated
guess. FWIW, bcm5700 driver needed some patches to work on HP's
ia64 box and I'm looking forward to testing tg3 patch soon.

And Intel's change in strategy may not work so well in pratice.
So I'd stick with a tigon (acenic or bcm5700) board for the next
3 monthes.

> one) or better yet, stick my 3ware RAID card in the rp2430, and do away
> with the network altogether. That I *can't* do with HP-UX, so...

I've never liked RAID controllers. too complex and failure prone.
If you have spare CPU cycles, putting in simple SCSI controllers will
give you at least 4 x 80MB/s and it's easy to connect 40 or so 9 (or 18GB)
SCSI drives.  This is not shiny, new high tech stuff (eg u160 and 15KRPM).
but I expect it to be faster than any IDE solution (I could be wrong) and
re-usable for other computer science projects (read database or other
mutlithreaded IO) when you are done.  <RANT>And you don't need any
stinking x86 binaries-only software to manage the setup.</RANT>

But I'm not one to stop others from picking solutions that work
for them.

grant