Problem with Gdt driver under smp?

Problem with Gdt driver under smp?

[Pedro Pla]

Hello,

I was wondering if there is any known problem with the Gdt driver under 
an smp machine, I have a dual p3 scb2 motherboard, with 1-2GB ram, an 
scrmr intel 0 channel raid controller and 3 hotswap disks. Initially I 
thought it was a memory problem since the kernel was complaining at 
mem_alloc:89, however as long as the memory I move is not read from the 
disk (through a kernel compile, or a dd if=/dev/sda) then it seems to 
work fine. A kernel compiled with no smp works fine as well. I've tried 
kernels from the 2.2.22 to the 2.4.17-2.4.20-pre10-ac with pre11 patches 
applied on top.

The symptoms are segmentation faults, signal 11's, and occasionally 
kernel oops at mem_alloc:89, I have also seen several kernel oops in 
journal.c when I tried using ext3, I disactivated that but the faults 
remained. It has also froze once and given a kernel oops while loading 
the scsi driver at boot, and another time it simply rebooted the machine 
when loading the driver. Any ideas? For now I will work with only one 
cpu and smp disactivated since that gives no problems at all, but I 
would like to be able to unleash the full power of these machines.

Thanks

Pedro

Re: Problem with Gdt driver under smp?

[Doug Ledford]

On Thu, Oct 31, 2002 at 04:37:07PM +0800, Pedro Pla wrote:
> Hello,
> 
> I was wondering if there is any known problem with the Gdt driver under 
> an smp machine, I have a dual p3 scb2 motherboard, with 1-2GB ram, an 
> scrmr intel 0 channel raid controller and 3 hotswap disks. Initially I 
> thought it was a memory problem since the kernel was complaining at 
> mem_alloc:89, however as long as the memory I move is not read from the 
> disk (through a kernel compile, or a dd if=/dev/sda) then it seems to 
> work fine. A kernel compiled with no smp works fine as well. I've tried 
> kernels from the 2.2.22 to the 2.4.17-2.4.20-pre10-ac with pre11 patches 
> applied on top.
> 
> The symptoms are segmentation faults, signal 11's, and occasionally 
> kernel oops at mem_alloc:89, I have also seen several kernel oops in 
> journal.c when I tried using ext3, I disactivated that but the faults 
> remained. It has also froze once and given a kernel oops while loading 
> the scsi driver at boot, and another time it simply rebooted the machine 
> when loading the driver. Any ideas? For now I will work with only one 
> cpu and smp disactivated since that gives no problems at all, but I 
> would like to be able to unleash the full power of these machines.

Sounds like typical bad RAM problems.  Run a memory test on your RAM.

-- 
  Doug Ledford <dledford@redhat.com>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606
  

Re: Problem with Gdt driver under smp?

[Pedro Pla]

Sorry I accidently sent the last one in html, so it probably didn't get 
through.

Doug Ledford wrote:

>On Thu, Oct 31, 2002 at 04:37:07PM +0800, Pedro Pla wrote:
>  
>
>>Hello,
>>
>>I was wondering if there is any known problem with the Gdt driver under 
>>an smp machine, I have a dual p3 scb2 motherboard, with 1-2GB ram, an 
>>scrmr intel 0 channel raid controller and 3 hotswap disks. Initially I 
>>thought it was a memory problem since the kernel was complaining at 
>>mem_alloc:89, however as long as the memory I move is not read from the 
>>disk (through a kernel compile, or a dd if=/dev/sda) then it seems to 
>>work fine. A kernel compiled with no smp works fine as well. I've tried 
>>kernels from the 2.2.22 to the 2.4.17-2.4.20-pre10-ac with pre11 patches 
>>applied on top.
>>
>>The symptoms are segmentation faults, signal 11's, and occasionally 
>>kernel oops at mem_alloc:89, I have also seen several kernel oops in 
>>journal.c when I tried using ext3, I disactivated that but the faults 
>>remained. It has also froze once and given a kernel oops while loading 
>>the scsi driver at boot, and another time it simply rebooted the machine 
>>when loading the driver. Any ideas? For now I will work with only one 
>>cpu and smp disactivated since that gives no problems at all, but I 
>>would like to be able to unleash the full power of these machines.
>>    
>>
>
>Sounds like typical bad RAM problems.  Run a memory test on your RAM.
>
>  
>
I've run memory tests, and memtest86 v3 gives no errors at all after its 
most thorough testing, I did however find some problems when running a 
test on memtest86 v2.8, however that gave errors on 8 seperate simm 
cards running under two different dual p3 motherboards, I just find it 
strange that they would all fail, I thought it might be the motherboard 
then, but 2 motherboards failing?

Re: Problem with Gdt driver under smp?

[Doug Ledford]

On Tue, Nov 05, 2002 at 09:16:40AM +0800, Pedro Pla wrote:
> Doug Ledford wrote:
> >
> >Sounds like typical bad RAM problems.  Run a memory test on your RAM.
> >
> > 
> >
> I've run memory tests, and memtest86 v3 gives no errors at all after its 
> most thorough testing, I did however find some problems when running a 
> test on memtest86 v2.8, however that gave errors on 8 seperate simm 
> cards running under two different dual p3 motherboards, I just find it 
> strange that they would all fail, I thought it might be the motherboard 
> then, but 2 motherboards failing?
> 

I've got a memory test shell script on my web site that runs under linux.  
Do me a favor and see if it thinks your machine has bad RAM.

-- 
  Doug Ledford <dledford@redhat.com>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606
  

Re: Problem with Gdt driver under smp?

[Doug Ledford]

> Doug Ledford wrote:
> Something about his web site without citing the address, what a dork!

http://people.redhat.com/dledford 

-- 
  Doug Ledford <dledford@redhat.com>     919-754-3700 x44233
         Red Hat, Inc. 
         1801 Varsity Dr.
         Raleigh, NC 27606
  

Re: Problem with Gdt driver under smp?

[Pedro Pla]

Doug Ledford wrote:

 >On Tue, Nov 05, 2002 at 09:16:40AM +0800, Pedro Pla wrote:
 >
 >
 >>Doug Ledford wrote:
 >>
 >>
 >>>Sounds like typical bad RAM problems.  Run a memory test on your RAM.
 >>>
 >>>
 >>>
 >>>
 >>>
 >>I've run memory tests, and memtest86 v3 gives no errors at all after its
 >>most thorough testing, I did however find some problems when running a
 >>test on memtest86 v2.8, however that gave errors on 8 seperate simm
 >>cards running under two different dual p3 motherboards, I just find it
 >>strange that they would all fail, I thought it might be the motherboard
 >>then, but 2 motherboards failing?
 >>
 >>
 >>
 >
 >I've got a memory test shell script on my web site that runs under 
linux.
 >Do me a favor and see if it thinks your machine has bad RAM.
 >
 >
 >
Been running the script all day and still no errors yet, and yes, it
seems to have used up all the memory or so says a cat /proc/meminfo with
only a few megs free and no swap activated. This is under a no smp
compiled kernel, under an smp compiled kernel I think it could die while
ungzipping the first source directory.. yeah it's that bad, no "subtle"
problem, just an asap kernel oops or segmentation fault.. can't switch
to mp 1.1 from mp 1.4 on this motherboard either, or at least I haven't
found out how yet, it's an intel scb2, intel says nothing about how to
do so on their sites, and the bios has no such option... at least the
machine runs flawlessly all tests in non-smp mode which makes me think
it isn't hardware..

Re: Problem with Gdt driver under smp?

[Pedro Pla]

Doug Ledford wrote:

>On Tue, Nov 05, 2002 at 06:01:37PM +0800, Pedro Pla wrote:
>  
>
>>Been running the script all day and still no errors yet, and yes, it 
>>seems to have used up all the memory or so says a cat /proc/meminfo with 
>>only a few megs free and no swap activated. This is under a no smp 
>>compiled kernel, under an smp compiled kernel I think it could die while 
>>ungzipping the first source directory.. yeah it's that bad, no "subtle" 
>>problem, just an asap kernel oops or segmentation fault.. can't switch 
>>to mp 1.1 from mp 1.4 on this motherboard either, or at least I haven't 
>>found out how yet, it's an intel scb2, intel says nothing about how to 
>>do so on their sites, and the bios has no such option... at least the 
>>machine runs flawlessly all tests in non-smp mode which makes me think 
>>it isn't hardware..
>>    
>>
>
>Why would you think that?  You don't really think that the linux kernel 
>SMP support is so flaky that it can't survive the first round of 
>ungizzipping on an SMP machine do you?  I mean, after all, millions of SMP 
>boxes are out there running the linux kernel every day without a glitch.  
>If the kernel were that bad, they would *all* be dying.
>
Of course not :) I have been running linux smp machines for nearly 4 
years, since I had to compile the smp support in by putting the smp = 1 
manually ;) and it has been a breeze, that's why initially I did think 
it was definately a hardware issue, however these are "branded" machines 
and I have tried two different setups with the same kind of peices 
(identical cpu's, ram, motherboard, raid controller, etc) and both have 
failed. Yes it could be a fundamental flaw in the design or build of 
both of these machines, although that is a little out of the ordinary, 
that is why I suspected it might be some flaw somewhere in the kernel 
that gave it a hard time running on this particular setup. Or maybe a 
design flaw in the hardware that makes it not run correctly under linux? 
However all this hardware is certified to run under Redhat by both the 
manufacturer intel, and Redhat...

>  OTOH, several 
>things on the motherboard change when running SMP mode.  Power consumption 
>goes up (so a bad power supply can cause these problems), you start using 
>a CPU that isn't being used now (so a totally busted CPU could cause this, 
>swapping CPU sockets might make it start failing in UP mode as well), 
>amount of data your are pumping to RAM from CPU goes up (because you have 
>two CPUs pumping data instead of one, and if the memory controller or 
>cache coherency controller or RAM itself can't handle the increased speed 
>of data throughput you get all sorts of errors), etc.  There are *tons* of 
>things that change going from UP to SMP in the hardware and all sorts of 
>things that used to work fine can start breaking.  The fact that your 
>machine works OK in UP mode means absolutley nothing about how it will 
>work in SMP mode or whether or not the hardware is up to running SMP.
>
>  
>
Point taken, I will wait till I receive the other 4 machines with 
identical configurations, and if each and every one fails under smp mode 
must I still think it is hardware? that would be 6 seperate machines 
with identical configurations failing in smp? These are all pure intel 
except ram and hard disks. Although I did read on the intel site that 
the raid controller scrmr had problems with the scb2 motherboard, and I 
did apply the bios upgrade, but it still fails... sigh... maybe I should 
find a less stressful profession ;) after being on comps for 12 years 
you wonder just what more can fail when you are pressed for time and 
bosses are looking over your shoulder ;)

Re: Problem with Gdt driver under smp?

[Pedro Pla]

Just to eliminate the possibility of a faulty cpu in up mode I swapped 
cpus and it continues to run flawlessly in up, not sure how to eliminate 
the possibility of it being the cache controller or the ram since all 
simms tested so far failed in mp and ran flawlessly in up, could it be a 
bios issue? I think I'm just trying anything and everything to locate 
the exact problem now. Any test ideas that could help me pinpoint it?

Thanks,
Pedro

Pedro Pla wrote:

> Doug Ledford wrote:
>
>> On Tue, Nov 05, 2002 at 06:01:37PM +0800, Pedro Pla wrote:
>>  
>>
>>> Been running the script all day and still no errors yet, and yes, it 
>>> seems to have used up all the memory or so says a cat /proc/meminfo 
>>> with only a few megs free and no swap activated. This is under a no 
>>> smp compiled kernel, under an smp compiled kernel I think it could 
>>> die while ungzipping the first source directory.. yeah it's that 
>>> bad, no "subtle" problem, just an asap kernel oops or segmentation 
>>> fault.. can't switch to mp 1.1 from mp 1.4 on this motherboard 
>>> either, or at least I haven't found out how yet, it's an intel scb2, 
>>> intel says nothing about how to do so on their sites, and the bios 
>>> has no such option... at least the machine runs flawlessly all tests 
>>> in non-smp mode which makes me think it isn't hardware..
>>>   
>>
>>
>> Why would you think that?  You don't really think that the linux 
>> kernel SMP support is so flaky that it can't survive the first round 
>> of ungizzipping on an SMP machine do you?  I mean, after all, 
>> millions of SMP boxes are out there running the linux kernel every 
>> day without a glitch.  If the kernel were that bad, they would *all* 
>> be dying.
>>
> Of course not :) I have been running linux smp machines for nearly 4 
> years, since I had to compile the smp support in by putting the smp = 
> 1 manually ;) and it has been a breeze, that's why initially I did 
> think it was definately a hardware issue, however these are "branded" 
> machines and I have tried two different setups with the same kind of 
> peices (identical cpu's, ram, motherboard, raid controller, etc) and 
> both have failed. Yes it could be a fundamental flaw in the design or 
> build of both of these machines, although that is a little out of the 
> ordinary, that is why I suspected it might be some flaw somewhere in 
> the kernel that gave it a hard time running on this particular setup. 
> Or maybe a design flaw in the hardware that makes it not run correctly 
> under linux? However all this hardware is certified to run under 
> Redhat by both the manufacturer intel, and Redhat...
>
>>  OTOH, several things on the motherboard change when running SMP 
>> mode.  Power consumption goes up (so a bad power supply can cause 
>> these problems), you start using a CPU that isn't being used now (so 
>> a totally busted CPU could cause this, swapping CPU sockets might 
>> make it start failing in UP mode as well), amount of data your are 
>> pumping to RAM from CPU goes up (because you have two CPUs pumping 
>> data instead of one, and if the memory controller or cache coherency 
>> controller or RAM itself can't handle the increased speed of data 
>> throughput you get all sorts of errors), etc.  There are *tons* of 
>> things that change going from UP to SMP in the hardware and all sorts 
>> of things that used to work fine can start breaking.  The fact that 
>> your machine works OK in UP mode means absolutley nothing about how 
>> it will work in SMP mode or whether or not the hardware is up to 
>> running SMP.
>>
>>  
>>
> Point taken, I will wait till I receive the other 4 machines with 
> identical configurations, and if each and every one fails under smp 
> mode must I still think it is hardware? that would be 6 seperate 
> machines with identical configurations failing in smp? These are all 
> pure intel except ram and hard disks. Although I did read on the intel 
> site that the raid controller scrmr had problems with the scb2 
> motherboard, and I did apply the bios upgrade, but it still fails... 
> sigh... maybe I should find a less stressful profession ;) after being 
> on comps for 12 years you wonder just what more can fail when you are 
> pressed for time and bosses are looking over your shoulder ;)
>
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-smp" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Problem with Gdt driver under smp?

[Ishikawa]

Pedro Pla wrote:
> 
> Just to eliminate the possibility of a faulty cpu in up mode I swapped
> cpus and it continues to run flawlessly in up, not sure how to eliminate
> the possibility of it being the cache controller or the ram since all
> simms tested so far failed in mp and ran flawlessly in up, could it be a
> bios issue? I think I'm just trying anything and everything to locate
> the exact problem now. Any test ideas that could help me pinpoint it?

Hmm, the woes of sysadmins. You have my sympathy.

In the earlier post, you mentioned, 
> I've tried 
> kernels from the 2.2.22 to the 2.4.17-2.4.20-pre10-ac with pre11 patches 
> applied on top.

You also mentioned that the hardware is certified by
Intel and RedHat to be linux-compatible.

May I suggest that you try a version of RedHat linux
(and its drivers) 
instead of compiling a kernel yourself?
Maybe RedHat has applied a patch or something that
makes the board work with *THEIR* version of
linux kernel and drivers?

(I have a few DELL linux servers at the office and
they run RedHat linux pre-installed, and I have
no desire to tinker with OS upgrade
unless either DELL or RedHat assures that the
new OS release is compatible with the hardware.
They are all dual-CPU boxes. Of course, I tinker
with the kernel upgrade with my home PC all the time.
It is a single CPU athlon board and I run Debian GNU/Linux
on it.)

Also, just a hunch, you might want to try
reducing the amount of memory  (if it is easy
to do) and see if this changes the behavior.
A stock kernel might have issues with 1GB - 2GB
on the said board. Not that I am aware of
any particular bugs or anything, but you may get
a different behavior especially if you have
a bunch of inferior memory sticks and other
hardware-related problems.
(In this sense, try removing the memory stick from
the board if possible instead of just specifying
the boot parameter to use less memory.)

In the earlier post:
> The symptoms are segmentation faults, signal 11's, and occasionally 
> kernel oops at mem_alloc:89, I have also seen several kernel oops in 
> journal.c when I tried using ext3, I disactivated that but the faults 
> remained. It has also froze once and given a kernel oops while loading 
> the scsi driver at boot, and another time it simply rebooted the machine 
> when loading the driver. Any ideas? 

As someone pointed out, this certainly looks to me
a flakey hardware. What brand of memory 
boards do you use? What is the chipset of the motherboard?

It may be hard to diagnose if these behavior show up
only under SMP setup. 
Do you have a different OS that supports dual CPU operation on
this motherboard? Solaris for x86 or Windows NT or 2000?
Running them might help you in figuring out the problem is indeed
hardware-related, etc. If you don't have either of them, too bad.
Given the cost of purchase of hardware, though, 
you might slip in the purchase of Solaris for x86 (media cost only
if I am not mistaken) as a diagnostic tool :-)
If you are in USA, I think you can receive the Solaris 
media rather quickly.

Re: Problem with Gdt driver under smp?

[Pedro Pla]

Ishikawa wrote:

>In the earlier post, you mentioned, 
>  
>
>>I've tried 
>>kernels from the 2.2.22 to the 2.4.17-2.4.20-pre10-ac with pre11 patches 
>>applied on top.
>>    
>>
>
>You also mentioned that the hardware is certified by
>Intel and RedHat to be linux-compatible.
>
>May I suggest that you try a version of RedHat linux
>(and its drivers) 
>instead of compiling a kernel yourself?
>Maybe RedHat has applied a patch or something that
>makes the board work with *THEIR* version of
>linux kernel and drivers?
>
I'll try their version of the kernel, downloading it now, however the
one it is certified for (or should we say the one mentioned in the intel
docs to be more correct) is quite a bit older then the existing one, so
I wonder if it's an accumulated bug in the kernel since that version,
and if I get the latest redhat download version if it will still fail..
guess we'll have to wait till my download finishes to install and test it.

>
>(I have a few DELL linux servers at the office and
>they run RedHat linux pre-installed, and I have
>no desire to tinker with OS upgrade
>unless either DELL or RedHat assures that the
>new OS release is compatible with the hardware.
>They are all dual-CPU boxes. Of course, I tinker
>with the kernel upgrade with my home PC all the time.
>It is a single CPU athlon board and I run Debian GNU/Linux
>on it.)
>
I've always used compiled kernels on smp machines with no problems, but
hey, there's always a first time :)

>
>Also, just a hunch, you might want to try
>reducing the amount of memory  (if it is easy
>to do) and see if this changes the behavior.
>A stock kernel might have issues with 1GB - 2GB
>on the said board. Not that I am aware of
>any particular bugs or anything, but you may get
>a different behavior especially if you have
>a bunch of inferior memory sticks and other
>hardware-related problems.
>(In this sense, try removing the memory stick from
>the board if possible instead of just specifying
>the boot parameter to use less memory.)
>
I've reduced it to the lowest amount I can with the simms I have in the
office, I got it down to 1gb ram, all the simms I have are 512mb and
they have to be put in pairs, might go buy some cheap 64mb simms and see
what happens.Just spending a few hundred bucks for testing purposes
won't make the bosses happy after they spent on 6 spanking new machines...

>
>In the earlier post:
>  
>
>>The symptoms are segmentation faults, signal 11's, and occasionally 
>>kernel oops at mem_alloc:89, I have also seen several kernel oops in 
>>journal.c when I tried using ext3, I disactivated that but the faults 
>>remained. It has also froze once and given a kernel oops while loading 
>>the scsi driver at boot, and another time it simply rebooted the machine 
>>when loading the driver. Any ideas? 
>>    
>>
>
>As someone pointed out, this certainly looks to me
>a flakey hardware. What brand of memory 
>boards do you use? What is the chipset of the motherboard?
>
They are all kingston memory chips, I have tried 8 seperate simms, well
in combinations of two, so really there was only 4 different simms in,
and in any different combination. The motherboard chipset is serverworks
(intel scb2 board), I think the problem is in the dma, since it only
occurs when moving files from disk. I know the serverworks ide dma is
flakey in the kernel at the moment, but how could that affect the gdt
(scrmr) scsi? Or is it related?

>
>It may be hard to diagnose if these behavior show up
>only under SMP setup. 
>Do you have a different OS that supports dual CPU operation on
>this motherboard? Solaris for x86 or Windows NT or 2000?
>Running them might help you in figuring out the problem is indeed
>hardware-related, etc. If you don't have either of them, too bad.
>Given the cost of purchase of hardware, though, 
>you might slip in the purchase of Solaris for x86 (media cost only
>if I am not mistaken) as a diagnostic tool :-)
>If you are in USA, I think you can receive the Solaris 
>media rather quickly.
>
I'm seriously considering that, however I am not in the US, let's see if
I can download it. it is definately worth trying if everything else fails.

Thanks for all the ideas :) now to try all the ones I can out :)