I/O performance test

I/O performance test

[G.de-With]

Hello

Since a month we have a LINUX BEOWULF cluster, the clusters contains 7
P4 dual processor 2GHz computers, with 8Gb of RAM per machine. For our
network we have used Gigabit ethernet. On our cluster we are running RH
7.2, with Linux 2.4.19.

These are some of the hardware specs:

- Dual Intel Xeon 2.GHz, 512 cache 400 MHz FSB
- 8 Gb ECC DDR
- Fujitsu 72Gb 10.000 rpm Ultra 160 SCA HDD SCSI HARDDISK

The problem we have with our cluster is as follows. When running large
scientific simulations the computer performance gradually goes down as
the I/O activity is increasing. At some point the response of the
computer is so poor that we have to kill the simulation. In a worst case
when the simulation was running overnight the computer hangs and a reset
of the computer is necessary. Nevertheless, even when we manage to kill
the simulation in time the computer remains very slow and a reboot is
necessary to regain full computer power.

My first suspicion was that the computer simply started swapping, but
there is no swap space being used, instead free RAM memory is still
apparent
(between 5-10%) and only 90% of the RAM is in use whereby 50% is cached
and another 50% is in usage. In addition the cpu usage is very low as
well.

To investigate the I/O performance I installed an I/O performance
benchmark program called bonnie++. The first test I performed was a
single bonnie++ -s 16096 instance.

# bonnie++ -s 16096

Unfortunately, as a result of running bonnie++ the computer started to
slow down till it finally hang. All the symptoms I discover with
bonnie++ are identical to what I experience when running our scientific
calculations.

In order to improve the I/O performance I have add some patches to the
kernel, including the patch for 00_block-highmem-all-19-1, to avoid
bounce buffers. Unfortunately none of the patches let to any improvement
in the I/O performance.

I don't think the machine should be behaving like this. I certainly
expect some slowdowns with that much IO, but the computer should still
be reasonably responsive, particularly because no system or user files
that need to be accessed are on that channel of the SCSI controller.

As a sort of a desperate move I did two other test in addition which
could be of use to the understanding of the problem:

- I removed 6Gb from the server and run the test bonnie++ -s 16096
succesfully with 2Gb of RAM.
- I placed an IDE disk 40Gb and run the test bonnie++ -s 16096 with 8Gb
of RAM succesfully.


Any advice on approaching this problem would be appreciated.

Govert

--
 ------------------------------------------------------------
| Dr. Govert de With     Research Fellow                     |
| Fluid Mechanics Research Group                             |
| University of Hertfordshire                                |
| Tel: 01707 284124 Fax: 01707 285086                        |
 ------------------------------------------------------------
| Der Horizont vieler Menschen ist ein Kreis mit Radius Null |
| und das nennen sie ihren Standpunkt.                       |
 ------------------------------------------------------------

Re: I/O performance test

[Pasi Kärkkäinen]

On Thu, 17 Oct 2002, G.de-With wrote:

> Hello
>
> Since a month we have a LINUX BEOWULF cluster, the clusters contains 7
> P4 dual processor 2GHz computers, with 8Gb of RAM per machine. For our
> network we have used Gigabit ethernet. On our cluster we are running RH
> 7.2, with Linux 2.4.19.
>
> These are some of the hardware specs:
>
> - Dual Intel Xeon 2.GHz, 512 cache 400 MHz FSB
> - 8 Gb ECC DDR
> - Fujitsu 72Gb 10.000 rpm Ultra 160 SCA HDD SCSI HARDDISK
>
> The problem we have with our cluster is as follows. When running large
> scientific simulations the computer performance gradually goes down as
> the I/O activity is increasing. At some point the response of the
> computer is so poor that we have to kill the simulation. In a worst case
> when the simulation was running overnight the computer hangs and a reset
> of the computer is necessary. Nevertheless, even when we manage to kill
> the simulation in time the computer remains very slow and a reboot is
> necessary to regain full computer power.
>
> My first suspicion was that the computer simply started swapping, but
> there is no swap space being used, instead free RAM memory is still
> apparent
> (between 5-10%) and only 90% of the RAM is in use whereby 50% is cached
> and another 50% is in usage. In addition the cpu usage is very low as
> well.
>
> To investigate the I/O performance I installed an I/O performance
> benchmark program called bonnie++. The first test I performed was a
> single bonnie++ -s 16096 instance.
>
> # bonnie++ -s 16096
>
> Unfortunately, as a result of running bonnie++ the computer started to
> slow down till it finally hang. All the symptoms I discover with
> bonnie++ are identical to what I experience when running our scientific
> calculations.
>
> In order to improve the I/O performance I have add some patches to the
> kernel, including the patch for 00_block-highmem-all-19-1, to avoid
> bounce buffers. Unfortunately none of the patches let to any improvement
> in the I/O performance.
>
> I don't think the machine should be behaving like this. I certainly
> expect some slowdowns with that much IO, but the computer should still
> be reasonably responsive, particularly because no system or user files
> that need to be accessed are on that channel of the SCSI controller.
>
> As a sort of a desperate move I did two other test in addition which
> could be of use to the understanding of the problem:
>
> - I removed 6Gb from the server and run the test bonnie++ -s 16096
> succesfully with 2Gb of RAM.
> - I placed an IDE disk 40Gb and run the test bonnie++ -s 16096 with 8Gb
> of RAM succesfully.
>
>
> Any advice on approaching this problem would be appreciated.
>

I'm not expert on this, but I would try with -aa patches for 2.4.19
kernel. They work really well for me, and are stable, and also perform
better than vanilla 2.4.19.

2.4.19rc5aa1 applies cleanly to 2.4.19 final (because rc5 and final are
the same kernel).

-aa patches are available from kernel.org mirrors
(people/andrea/kernels/v2.4/2.4.19rc5aa1.bz2)


- Pasi Kärkkäinen

                                   ^
                                .     .
                                 Linux
                              /    -    \
                             Choice.of.the
                           .Next.Generation.

Re: I/O performance test

[Roger Larsson]

On torsdagen den 17 oktober 2002 11.58, G.de-With wrote:
> Hello
> 
> Since a month we have a LINUX BEOWULF cluster, the clusters contains 7
> P4 dual processor 2GHz computers, with 8Gb of RAM per machine. For our
> network we have used Gigabit ethernet. On our cluster we are running RH
> 7.2, with Linux 2.4.19.
> 
> These are some of the hardware specs:
> 
> - Dual Intel Xeon 2.GHz, 512 cache 400 MHz FSB
> - 8 Gb ECC DDR
> - Fujitsu 72Gb 10.000 rpm Ultra 160 SCA HDD SCSI HARDDISK

Do you have a 72 GB HD per dual processors? Or only on one?
Or do the data have to pass through the ethernet for your tests?

> 
> The problem we have with our cluster is as follows. When running large
> scientific simulations the computer performance gradually goes down as
> the I/O activity is increasing. At some point the response of the
> computer is so poor that we have to kill the simulation. In a worst case
> when the simulation was running overnight the computer hangs and a reset
> of the computer is necessary. Nevertheless, even when we manage to kill
> the simulation in time the computer remains very slow and a reboot is
> necessary to regain full computer power.
> 
> My first suspicion was that the computer simply started swapping, but
> there is no swap space being used, instead free RAM memory is still
> apparent
> (between 5-10%) and only 90% of the RAM is in use whereby 50% is cached
> and another 50% is in usage. In addition the cpu usage is very low as
> well.

Could still be that all DMA or normal zone memory is used up.
Use the magic sysrq to get zone info.
 Alt+Sys Rq+M
check in vt10 (Alt+F10 or Ctrl+Alt+F10) or with dmesg

> 
> To investigate the I/O performance I installed an I/O performance
> benchmark program called bonnie++. The first test I performed was a
> single bonnie++ -s 16096 instance.
> 
> # bonnie++ -s 16096
> 
> Unfortunately, as a result of running bonnie++ the computer started to
> slow down till it finally hang. All the symptoms I discover with
> bonnie++ are identical to what I experience when running our scientific
> calculations.

Fortunately!

You may have found a way for others to reproduce your problem :-)

> 
> In order to improve the I/O performance I have add some patches to the
> kernel, including the patch for 00_block-highmem-all-19-1, to avoid
> bounce buffers. Unfortunately none of the patches let to any improvement
> in the I/O performance.
> 
> I don't think the machine should be behaving like this. I certainly
> expect some slowdowns with that much IO, but the computer should still
> be reasonably responsive, particularly because no system or user files
> that need to be accessed are on that channel of the SCSI controller.
> 
> As a sort of a desperate move I did two other test in addition which
> could be of use to the understanding of the problem:
> 
> - I removed 6Gb from the server and run the test bonnie++ -s 16096
> succesfully with 2Gb of RAM.

This points towards a highmem issue...

> - I placed an IDE disk 40Gb and run the test bonnie++ -s 16096 with 8Gb
> of RAM succesfully.

This is why I asked the first question.

My guess: Highmem issue with Gb Ethernet?

Brand and model of the Ethernet boards?


-- 
Roger Larsson
Skellefteå
Sweden

Re: I/O performance test

[G.de-With]

Hello Roger

> > Since a month we have a LINUX BEOWULF cluster, the clusters contains 7
> > P4 dual processor 2GHz computers, with 8Gb of RAM per machine. For our
> > network we have used Gigabit ethernet. On our cluster we are running RH
> > 7.2, with Linux 2.4.19.
> >
> > These are some of the hardware specs:
> >
> > - Dual Intel Xeon 2.GHz, 512 cache 400 MHz FSB
> > - 8 Gb ECC DDR
> > - Fujitsu 72Gb 10.000 rpm Ultra 160 SCA HDD SCSI HARDDISK
>
> Do you have a 72 GB HD per dual processors? Or only on one?
> Or do the data have to pass through the ethernet for your tests?


I have 3*72Gb HD per computer. The test is carried out both on the server,
whereby data was written directly to the harddisk, as well the test was
carried out from one of the nodes whereby data had to pass through the
ethernet.
When carrying out the test of the ethernet no problems were discovered.

> > The problem we have with our cluster is as follows. When running large
> > scientific simulations the computer performance gradually goes down as
> > the I/O activity is increasing. At some point the response of the
> > computer is so poor that we have to kill the simulation. In a worst case
> > when the simulation was running overnight the computer hangs and a reset
> > of the computer is necessary. Nevertheless, even when we manage to kill
> > the simulation in time the computer remains very slow and a reboot is
> > necessary to regain full computer power.
> >
> > My first suspicion was that the computer simply started swapping, but
> > there is no swap space being used, instead free RAM memory is still
> > apparent
> > (between 5-10%) and only 90% of the RAM is in use whereby 50% is cached
> > and another 50% is in usage. In addition the cpu usage is very low as
> > well.
>
> Could still be that all DMA or normal zone memory is used up.
> Use the magic sysrq to get zone info.
>  Alt+Sys Rq+M
> check in vt10 (Alt+F10 or Ctrl+Alt+F10) or with dmesg.

OK, good idea, didn't knew this.

> > In order to improve the I/O performance I have add some patches to the
> > kernel, including the patch for 00_block-highmem-all-19-1, to avoid
> > bounce buffers. Unfortunately none of the patches let to any improvement
> > in the I/O performance.
> >
> > I don't think the machine should be behaving like this. I certainly
> > expect some slowdowns with that much IO, but the computer should still
> > be reasonably responsive, particularly because no system or user files
> > that need to be accessed are on that channel of the SCSI controller.
> >
> > As a sort of a desperate move I did two other test in addition which
> > could be of use to the understanding of the problem:
> >
> > - I removed 6Gb from the server and run the test bonnie++ -s 16096
> > succesfully with 2Gb of RAM.
>
> This points towards a highmem issue...
>
> > - I placed an IDE disk 40Gb and run the test bonnie++ -s 16096 with 8Gb
> > of RAM succesfully.
>
> This is why I asked the first question.
>
> My guess: Highmem issue with Gb Ethernet?
>
> Brand and model of the Ethernet boards?

Someone suggested me to install the -aa patch 2.4.19rc5aa1. When running the
modified kernel all test were performed well.

Hereby I send you the output from Bonnie++. In this test I run bonnie++ on
the server, whereby it was writting directly to the 72Gb HD.

[govert@galaxy govert]$ bonnie++ -s 16096
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.02c       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec
%CP
galaxy       16096M 16268  72 38745  21 17456  10 18672  82 37469  14 318.1
1
                    ------Sequential Create------ --------Random
Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec
%CP
                 16  3665  99 +++++ +++ +++++ +++  3762  99 +++++ +++  9147
99
galaxy,16096M,16268,72,38745,21,17456,10,18672,82,37469,14,318.1,1,16,3665,99,+++++,+++,+++++,+++,3762,99,+++++,+++,9147,99

[govert@galaxy govert]$ emacs bonnie++
bonnie++16096.txt  bonnie++4084.txt
[govert@galaxy govert]$


Govert


--
 ------------------------------------------------------------
| Dr. Govert de With     Research Fellow                     |
| Fluid Mechanics Research Group                             |
| University of Hertfordshire                                |
| Tel: 01707 284124 Fax: 01707 285086                        |
 ------------------------------------------------------------
| Der Horizont vieler Menschen ist ein Kreis mit Radius Null |
| und das nennen sie ihren Standpunkt.                       |
 ------------------------------------------------------------