Dual Opteron 248s w/ 8GB RAM on Tyan K8W (S2885)

Dual Opteron 248s w/ 8GB RAM on Tyan K8W (S2885)

[Bryan Stillwell]

I am having a problem with getting 8GB of RAM to work with the 2.6.5
kernel that Red Hat uses in their new Fedora Core 2 test3 release.  Yes,
not vanilla, but hopefully close enough (otherwise I can try it with a
custom kernel).  The systems (I'm testing on two mostly identical
systems) that are experiencing the problem have these specs:

Motherboard: Tyan K8W (S2885)
Processors: 2 x Opteron 248
Memory: 8 x 1GB
BIOS version: 1.02 (2-3-2004 - latest available)
SCSI card: Adaptec 29320

About 80% of the time I get this error when booting:

...
RAMDISK: Compressed image found at block 0
crc error
VFS: Cannot open root device "<NULL>" or unknown-block(3,2)
Please append a correct "root=" boot option
Kernel panic: VFS: Unable to mount root fs on unknown-block(3,2)


The other 20% of the time it boots up fine, but that's not enough
consistency for these computers final task (beowulf compute nodes).
Also I haven't really stressed the kernel on these boots.

The systems seem to work fine if I pull out half the memory so that they
only has 4GB of RAM.  I've also tested a similar setup using a Rioworks
Arima HDAMA board with 8GB of RAM and it worked.  So I'm led to believe
that this is some kind of driver issue or possibly a bios problem...

Any help/ideas are of course appreciated.

Thanks,
Bryan

-- 
Aspen Systems, Inc.    | http://www.aspsys.com/
Production Engineer    | Phone: (303)431-4606
bryans@aspsys.com      | Fax:   (303)431-7196

Re: Dual Opteron 248s w/ 8GB RAM on Tyan K8W (S2885)

[Bryan Stillwell]

On Fri, Apr 30, 2004 at 12:05:02AM +0300, Denis Vlasenko wrote:
>Latest memtest86 rumored to support testing more than 4Gb or RAM.
>You may try it and see whether it detects all the RAM.

During the 20% chance of it actually booting up, I've been able to
capture /proc/meminfo.  It reports MemTotal as 7642992 kB.  I've been
told that Tyan boards allocate almost 0.5GB for some reason for their
own use...  You can check out the logs I made from the bugzilla report I
sent to RedHat:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=121868

You'll notice that when it has 4GB in the system, the MemTotal is
3530868 kB...  The Rioworks board recognized close to the full amount
with both 4GB and 8GB installed.

I've also tested Mandrake 10.0rc1 and SuSE 9.0, and they have the same
issues that RedHat has...

Bryan

-- 
Aspen Systems, Inc.    | http://www.aspsys.com/
Production Engineer    | Phone: (303)431-4606
bryans@aspsys.com      | Fax:   (303)431-7196

Re: Dual Opteron 248s w/ 8GB RAM on Tyan K8W (S2885)

[Tom Vier]

On Thu, Apr 29, 2004 at 03:17:33PM -0600, Bryan Stillwell wrote:
> During the 20% chance of it actually booting up, I've been able to
> capture /proc/meminfo.  It reports MemTotal as 7642992 kB.  I've been
> told that Tyan boards allocate almost 0.5GB for some reason for their

interesting. where'd you hear that? mine has 2gigs and is using just under
24megs. i wonder what it's doing with all that.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: Dual Opteron 248s w/ 8GB RAM on Tyan K8W (S2885)

[Bryan Stillwell]

On Thu, Apr 29, 2004 at 10:15:16PM -0400, Tom Vier wrote:
>On Thu, Apr 29, 2004 at 03:17:33PM -0600, Bryan Stillwell wrote:
>> During the 20% chance of it actually booting up, I've been able to
>> capture /proc/meminfo.  It reports MemTotal as 7642992 kB.  I've been
>> told that Tyan boards allocate almost 0.5GB for some reason for their
>
>interesting. where'd you hear that? mine has 2gigs and is using just under
>24megs. i wonder what it's doing with all that.

I don't have any theories on what they use it for, but one of the other
production engineers at my new job says it has been that way with
multiple previous boards from Tyan we've used.

Bryan

-- 
Aspen Systems, Inc.    | http://www.aspsys.com/
Production Engineer    | Phone: (303)431-4606
bryans@aspsys.com      | Fax:   (303)431-7196

Re: Dual Opteron 248s w/ 8GB RAM on Tyan K8W (S2885)

[Richard B. Johnson]

On Thu, 29 Apr 2004, Bryan Stillwell wrote:

> On Thu, Apr 29, 2004 at 10:15:16PM -0400, Tom Vier wrote:
> >On Thu, Apr 29, 2004 at 03:17:33PM -0600, Bryan Stillwell wrote:
> >> During the 20% chance of it actually booting up, I've been able to
> >> capture /proc/meminfo.  It reports MemTotal as 7642992 kB.  I've been
> >> told that Tyan boards allocate almost 0.5GB for some reason for their
> >
> >interesting. where'd you hear that? mine has 2gigs and is using just under
> >24megs. i wonder what it's doing with all that.

It can't be. The only CPU(s) that would know anything about it and
thus be able to use that RAM are running the operating system.

It is quite obviously a BAD BIOS that doesn't follow the rules about
reporting RAM so some is wasted --- or you have 4GB on the main-board
so some gets wasted to make room for the address-space necessary for
your PCI and AGP cards.

>
> I don't have any theories on what they use it for, but one of the other
> production engineers at my new job says it has been that way with
> multiple previous boards from Tyan we've used.
>
> Bryan
>


Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5557.45 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: Dual Opteron 248s w/ 8GB RAM on Tyan K8W (S2885)

[Tom Vier]

On Fri, Apr 30, 2004 at 07:01:38AM -0400, Richard B. Johnson wrote:
> It can't be. The only CPU(s) that would know anything about it and
> thus be able to use that RAM are running the operating system.

unless it's reserving it for cross-boot data. i remember there being some
error (ecc, machine checks?) logging option in the bios setup. *shrug* i
don't know much about x86(-64) bios. openboot and srm do save state, since
you can drop back to them without rebooting.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: Dual Opteron 248s w/ 8GB RAM on Tyan K8W (S2885)

[Bryan Stillwell]

On Fri, Apr 30, 2004 at 12:02:26PM -0600, Jason Turner wrote:
>A similar (but different) problem exists with motherboards that use the
>AMD760MPX. Here is the specific problem you are seeing:
>
>>From Tyan's FAQ:(http://www.tyan.com/support/html/f_s2885.html)

I tried the beta bios mentioned in Tyan's FAQ, but it didn't help with
the booting problem I have with 8GB ram.  I did however find that I can
boot up the machine if I add "maxcpus=0" to the kernel options in the
bootloader.  Of course then only one cpu works, which is just as bad as
the other solution of having only half the memory...

Bryan

-- 
Aspen Systems, Inc.    | http://www.aspsys.com/
Production Engineer    | Phone: (303)431-4606
bryans@aspsys.com      | Fax:   (303)431-7196