high load

high load

[Phil Garcia]

Hi,

We've come across a problem that seems related to the interaction
between NFS and a 3ware 7500-8 running RAID 5 with one hot spare.  We
have a test program that opens a file and then loops, writing up to
255 bytes to the file and then sleeping for up to 10 seconds.  We run
around 200 instances of the program simultaneously for about 25
minutes.

When we run the processes on 200 separate slaves, writing to a
(reiserfs) directory under the 3ware controller, the load on the
server shoots up to over 20 and then gradually declines over the next
5 minutes or so.

When we run the processes on 200 separate slaves, writing to a
different disk/directory (again reiserfs) on the same NFS server, the
load stays fine.

When we run 200 processes locally on the server, writing to the
directory under the 3ware controller, again the load stays low.

Any insights would be much appreciated...


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: high load

[Andreas Behnert]

Phil Garcia wrote:
> Hi,
> 
> We've come across a problem that seems related to the interaction
> between NFS and a 3ware 7500-8 running RAID 5 with one hot spare.  We
> have a test program that opens a file and then loops, writing up to
> 255 bytes to the file and then sleeping for up to 10 seconds.  We run
> around 200 instances of the program simultaneously for about 25
> minutes.
> 
> When we run the processes on 200 separate slaves, writing to a
> (reiserfs) directory under the 3ware controller, the load on the
> server shoots up to over 20 and then gradually declines over the next
> 5 minutes or so.

The problem may be caused by ReiserFS:
If you are using kernel 2.2.x and ReiserFS 3.5.x - ReiserFS for
kernel 2.2.x is REALLY broken. We had similar problems here that
ended up in ReiserFS destroying itself, reproducible! Patches
from ftp.namesys.com didn't help at all. At that time ReiserFS
was nothing more than something to play with.
Can't say anything about kernel 2.4.x and ReiserFS 3.6.x - after
the 3.5.x desasters we moved back to ext2 and are now running ext3
with "data=journal" on our NFS servers without a single problem ...


Regards,
Andreas



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

High load

[Aniruddh Singh]

HI all,

I have one compaq server with 4 Intel(R) Xeon cpu's (3.1GHZ), 4GB RAM.
OS:- Fedora Core 2
Kernel:- 2.6.14

when i compile a new kernel, during th compilation process load goes
very high (10 and little above). i can not understand why does this
happen, while if i compile the same kernel on my P4 machine with 1GB ram
and 3GHZ, it remains under 3.

can somebody tell me what is wrong?
-

-- 
Regards
Aniruddh Singh
System Administrator
Monster.com India Pvt. Ltd.
FC 23, Block B, 1st Floor, Sector 16A
Film City Noida 201301 U.P.

Re: High load

[Con Kolivas]

On Thursday 05 January 2006 20:49, Aniruddh Singh wrote:
> HI all,
>
> I have one compaq server with 4 Intel(R) Xeon cpu's (3.1GHZ), 4GB RAM.
> OS:- Fedora Core 2
> Kernel:- 2.6.14
>
> when i compile a new kernel, during th compilation process load goes
> very high (10 and little above). i can not understand why does this
> happen, while if i compile the same kernel on my P4 machine with 1GB ram
> and 3GHZ, it remains under 3.
>
> can somebody tell me what is wrong?
> -

Sounds suspiciously like DMA is not working on your drives. Check your dmesg 
logs and what hdparm returns.

Con

Re: High load

[Aniruddh Singh]

Hi,

hdparm -tT /dev/cciss/c0d0p2 returns the following

/dev/cciss/c0d0p2:
 Timing buffer-cache reads:   2660 MB in  2.00 seconds = 1329.92 MB/sec
 Timing buffered disk reads:  248 MB in  3.00 seconds =  82.55 MB/sec

if i try to hdparm -I /dev/cciss/c0d0 it returns 
/dev/cciss/c0d0:
 operation not supported on SCSI disks

Do i need to installed sdparm and modify some settings?


On Thu, 2006-01-05 at 21:00 +1100, Con Kolivas wrote:
> On Thursday 05 January 2006 20:49, Aniruddh Singh wrote:
> > HI all,
> >
> > I have one compaq server with 4 Intel(R) Xeon cpu's (3.1GHZ), 4GB RAM.
> > OS:- Fedora Core 2
> > Kernel:- 2.6.14
> >
> > when i compile a new kernel, during th compilation process load goes
> > very high (10 and little above). i can not understand why does this
> > happen, while if i compile the same kernel on my P4 machine with 1GB ram
> > and 3GHZ, it remains under 3.
> >
> > can somebody tell me what is wrong?
> > -
> 
> Sounds suspiciously like DMA is not working on your drives. Check your dmesg 
> logs and what hdparm returns.
> 
> Con
-- 
Regards
Aniruddh Singh
System Administrator
Monster.com India Pvt. Ltd.
FC 23, Block B, 1st Floor, Sector 16A
Film City Noida 201301 U.P.

Re: High load

[Aniruddh Singh]

Hi,

there is a raid 0 and raid controller is Smart Array 64xx (rev 01)

hdparm -tT /dev/cciss/c0d0p2 returns the following

/dev/cciss/c0d0p2:
 Timing buffer-cache reads:   2660 MB in  2.00 seconds = 1329.92 MB/sec
 Timing buffered disk reads:  248 MB in  3.00 seconds =  82.55 MB/sec

if i try to hdparm -I /dev/cciss/c0d0 it returns 
/dev/cciss/c0d0:
 operation not supported on SCSI disks






On Thu, 2006-01-05 at 21:00 +1100, Con Kolivas wrote:
> On Thursday 05 January 2006 20:49, Aniruddh Singh wrote:
> > HI all,
> >
> > I have one compaq server with 4 Intel(R) Xeon cpu's (3.1GHZ), 4GB RAM.
> > OS:- Fedora Core 2
> > Kernel:- 2.6.14
> >
> > when i compile a new kernel, during th compilation process load goes
> > very high (10 and little above). i can not understand why does this
> > happen, while if i compile the same kernel on my P4 machine with 1GB ram
> > and 3GHZ, it remains under 3.
> >
> > can somebody tell me what is wrong?
> > -
> 
> Sounds suspiciously like DMA is not working on your drives. Check your dmesg 
> logs and what hdparm returns.
> 
> Con
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
Regards
Aniruddh Singh
System Administrator
Monster.com India Pvt. Ltd.
FC 23, Block B, 1st Floor, Sector 16A
Film City Noida 201301 U.P.

Re: High load

[Arjan van de Ven]

On Fri, 2006-01-06 at 18:06 +0530, Aniruddh Singh wrote:
> Hi,
> 
> there is a raid 0 and raid controller is Smart Array 64xx (rev 01)
> 
> hdparm -tT /dev/cciss/c0d0p2 returns the following

for measuring IO performance, I'd recommend tiobench over hdparm any day
( http://tiobench.sf.net )

Re: High load

[Aniruddh Singh]

On Fri, 2006-01-06 at 13:57 +0100, Arjan van de Ven wrote:
> On Fri, 2006-01-06 at 18:06 +0530, Aniruddh Singh wrote:
> > Hi,
> > 
> > there is a raid 0 and raid controller is Smart Array 64xx (rev 01)
> > 
> > hdparm -tT /dev/cciss/c0d0p2 returns the following
> 
> for measuring IO performance, I'd recommend tiobench over hdparm any day
> ( http://tiobench.sf.net )

i will do it, can you please tell me why load goes high when i compile
kernel (10 and above)
> 
> 
-- 
Regards
Aniruddh Singh
System Administrator
Monster.com India Pvt. Ltd.
FC 23, Block B, 1st Floor, Sector 16A
Film City Noida 201301 U.P.

Re: High load

[Arjan van de Ven]

On Fri, 2006-01-06 at 21:16 +0530, Aniruddh Singh wrote:
> On Fri, 2006-01-06 at 13:57 +0100, Arjan van de Ven wrote:
> > On Fri, 2006-01-06 at 18:06 +0530, Aniruddh Singh wrote:
> > > Hi,
> > > 
> > > there is a raid 0 and raid controller is Smart Array 64xx (rev 01)
> > > 
> > > hdparm -tT /dev/cciss/c0d0p2 returns the following
> > 
> > for measuring IO performance, I'd recommend tiobench over hdparm any day
> > ( http://tiobench.sf.net )
> 
> i will do it, can you please tell me why load goes high when i compile
> kernel (10 and above)

thats really odd, unless you do a "make -j", in which case of course
it's expected ;)

Re: High load

[Aniruddh Singh]

 
> > > for measuring IO performance, I'd recommend tiobench over hdparm any day
> > > ( http://tiobench.sf.net )
> > 
> > i will do it, can you please tell me why load goes high when i compile
> > kernel (10 and above)
> 
> thats really odd, unless you do a "make -j", in which case of course
> it's expected ;)
yes i always use make -j option ??
-- 
Regards
Aniruddh Singh
System Administrator
Monster.com India Pvt. Ltd.
FC 23, Block B, 1st Floor, Sector 16A
Film City Noida 201301 U.P.

Re: High load

[Randy.Dunlap]

On Sat, 07 Jan 2006 11:17:09 +0530 Aniruddh Singh wrote:

>  
> > > > for measuring IO performance, I'd recommend tiobench over hdparm any day
> > > > ( http://tiobench.sf.net )
> > > 
> > > i will do it, can you please tell me why load goes high when i compile
> > > kernel (10 and above)
> > 
> > thats really odd, unless you do a "make -j", in which case of course
> > it's expected ;)
> yes i always use make -j option ??

With or without a following integer?

See 'make help':
  -j [N], --jobs[=N]          Allow N jobs at once; infinite jobs with no arg.

so use make -j N to limit the number of makes/compiles etc.
or use 'make -j' to go ballistic.

---
~Randy

Re: High load

[Aniruddh Singh]

On Fri, 2006-01-06 at 21:48 -0800, Randy.Dunlap wrote:
> On Sat, 07 Jan 2006 11:17:09 +0530 Aniruddh Singh wrote:
> 
> >  
> > > > > for measuring IO performance, I'd recommend tiobench over hdparm any day
> > > > > ( http://tiobench.sf.net )
> > > > 
> > > > i will do it, can you please tell me why load goes high when i compile
> > > > kernel (10 and above)
> > > 
> > > thats really odd, unless you do a "make -j", in which case of course
> > > it's expected ;)
> > yes i always use make -j option ??
> 
> With or without a following integer?
make -j8, machine has 4 processor 
> 
> See 'make help':
>   -j [N], --jobs[=N]          Allow N jobs at once; infinite jobs with no arg.
> 
> so use make -j N to limit the number of makes/compiles etc.
> or use 'make -j' to go ballistic.
> 
> ---
> ~Randy
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
Regards
Aniruddh Singh
System Administrator
Monster.com India Pvt. Ltd.
FC 23, Block B, 1st Floor, Sector 16A
Film City Noida 201301 U.P.