sgp_dd uses alot of CPU time on FC3

sgp_dd uses alot of CPU time on FC3

[Martin W. Schlining III]

I posted this on the Fedora Forum. I thought this was also an 
appropriate place as well.

I am trying to use my Dell 2850 Server as a data pump using sgp_dd to 
perform large sequential read from my target through sg devices. One 
instance of sgp_dd uses about 50% of the CPU, so running a second only 
causes the CPU to become very busy and my read performance suffers as a 
result. A third and a fourth instance only make matters worse.

I tried using sgm_dd, which is not multithreaded, to run the same test. 
Though the speed is not quite as high as sgp_dd, it only uses a small 
amount of CPU resources. That lead me to believe that either sgp_dd has 
a problem in its multi-threading or maybe the POSIX threads. I'm really 
not sure at this point what to do next.

Under Windows 2003 running IOMeter, my target can be saturated at my 
expected bandwidth across 4 FC4 ports. That shows that my target and 
server are capable of delivering the speed.

I tried both the SMP and non-SMP kernels with the same results.

Here's my configuration using the non-SMP kernel:

My target w/ 4 FC4 host ports
Dell 2850 server Dual processor 1GB memory
Fedora Core 3 Distro updated to kernel 2.6.11-1.27_FC3SMP and 
2.6.11-1.27_FC3
swap size 2GB

2 Emulex LP11000 FC4 Dual HBAs, each on independant PCI busses
Emulex driver version: 2.6-8.0.16.6_x2 compiled and installed for the 
new kernels.

uname -a
Linux  2.6.11-1.27_FC3 #1 Tue May 17 20:27:37 EDT 2005 i686 i686 i386 
GNU/Linux

gcc -v
Reading specs from /usr/lib/gcc/i386-redhat-linux/3.4.3/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
--infodir=/usr/share/info --enable-shared --enable-threads=posix 
--disable-checking --with-system-zlib --enable-__cxa_atexit 
--disable-libunwind-exceptions --enable-java-awt=gtk 
--host=i386-redhat-linux
Thread model: posix
gcc version 3.4.3 20050227 (Red Hat 3.4.3-22.fc3)

lsscsi -g
[0:0:0:0] disk SEAGATE ST336607LC DS09 /dev/sda /dev/sg0
[0:0:6:0] process PE/PV 1x6 SCSI BP 1.0 - /dev/sg1
[14:0:0:0] disk E1.0 /dev/sdb /dev/sg2
[15:0:0:0] disk E1.0 /dev/sdc /dev/sg3
[16:0:0:0] disk E1.0 /dev/sdd /dev/sg4
[17:0:0:0] disk E1.0 /dev/sde /dev/sg5

cat /proc/scsi/sg/allow_dio
0

I tried using a value of 1 for allow_dio, but it had no effect.

Using sg3_utils-1.14

Running sgp_dd like this:
sgp_dd if=/dev/sg2 of=/dev/null bs=512 bpt=4096 thr=6 time=1

6 threads, 2M transfers, actually gives 2M commands sizes

[root@racerx proc]# cat cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 4
model name : Intel(R) Xeon(TM) CPU 3.00GHz
stepping : 1
cpu MHz : 2993.167
cache size : 1024 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat 
pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx lm pni monitor 
ds_cpl cid cx16 xtpr
bogomips : 5947.39

Re: sgp_dd uses alot of CPU time on FC3

[Douglas Gilbert]

Martin W. Schlining III wrote:
> I posted this on the Fedora Forum. I thought this was also an 
> appropriate place as well.
> 
> I am trying to use my Dell 2850 Server as a data pump using sgp_dd to 
> perform large sequential read from my target through sg devices. One 
> instance of sgp_dd uses about 50% of the CPU, so running a second only 
> causes the CPU to become very busy and my read performance suffers as a 
> result. A third and a fourth instance only make matters worse.
> 
> I tried using sgm_dd, which is not multithreaded, to run the same test. 
> Though the speed is not quite as high as sgp_dd, it only uses a small 
> amount of CPU resources. That lead me to believe that either sgp_dd has 
> a problem in its multi-threading or maybe the POSIX threads. I'm really 
> not sure at this point what to do next.
> 
> Under Windows 2003 running IOMeter, my target can be saturated at my 
> expected bandwidth across 4 FC4 ports. That shows that my target and 
> server are capable of delivering the speed.
> 
> I tried both the SMP and non-SMP kernels with the same results.
> 
> Here's my configuration using the non-SMP kernel:
> 
> My target w/ 4 FC4 host ports
> Dell 2850 server Dual processor 1GB memory
> Fedora Core 3 Distro updated to kernel 2.6.11-1.27_FC3SMP and 
> 2.6.11-1.27_FC3
> swap size 2GB
> 
> 2 Emulex LP11000 FC4 Dual HBAs, each on independant PCI busses
> Emulex driver version: 2.6-8.0.16.6_x2 compiled and installed for the 
> new kernels.
> 
> uname -a
> Linux  2.6.11-1.27_FC3 #1 Tue May 17 20:27:37 EDT 2005 i686 i686 i386 
> GNU/Linux
> 
> gcc -v
> Reading specs from /usr/lib/gcc/i386-redhat-linux/3.4.3/specs
> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man 
> --infodir=/usr/share/info --enable-shared --enable-threads=posix 
> --disable-checking --with-system-zlib --enable-__cxa_atexit 
> --disable-libunwind-exceptions --enable-java-awt=gtk 
> --host=i386-redhat-linux
> Thread model: posix
> gcc version 3.4.3 20050227 (Red Hat 3.4.3-22.fc3)
> 
> lsscsi -g
> [0:0:0:0] disk SEAGATE ST336607LC DS09 /dev/sda /dev/sg0
> [0:0:6:0] process PE/PV 1x6 SCSI BP 1.0 - /dev/sg1
> [14:0:0:0] disk E1.0 /dev/sdb /dev/sg2
> [15:0:0:0] disk E1.0 /dev/sdc /dev/sg3
> [16:0:0:0] disk E1.0 /dev/sdd /dev/sg4
> [17:0:0:0] disk E1.0 /dev/sde /dev/sg5
> 
> cat /proc/scsi/sg/allow_dio
> 0

Martin,
allow_dio only comes into play when the "dio=1" option
is used in sgp_dd (and sg_dd).

> I tried using a value of 1 for allow_dio, but it had no effect.
> 
> Using sg3_utils-1.14
> 
> Running sgp_dd like this:
> sgp_dd if=/dev/sg2 of=/dev/null bs=512 bpt=4096 thr=6 time=1
> 
> 6 threads, 2M transfers, actually gives 2M commands sizes

You didn't show what throughput you got.

I haven't done much timing on sgp_dd since lk 2.4
days. Here is one data point I just obtained with lk 2.6.11:

$ time sgp_dd if=/dev/sg20 of=. bs=512 bpt=4096 thr=6 time=1
time to transfer data was 558.082167 secs, 65.26 MB/sec
71132960+0 records in
71132960+0 records out
0.06user 23.95system 9:18.08elapsed 4%CPU
     (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+909minor)pagefaults 0swaps


That doesn't look too bad: 4% CPU utilization

Using a rebadged Seagate FC disk via a QLogic Corp.
QLA2312 Fibre Channel Adapter (qla2xxx LLD):

$ sdparm --page=co /dev/sg20
     /dev/sg20: HP 36.4G  ST336753FC        HP00
Control mode page:
   TST         0  [cha: n, def:  0, sav:  0]
   TMF_ONLY    0  [cha: n, def:  0, sav:  0]
   D_SENSE     0  [cha: n, def:  0, sav:  0]
   GLTSD       0  [cha: y, def:  0, sav:  0]
   RLEC        0  [cha: y, def:  0, sav:  0]
   QAM         1  [cha: y, def:  1, sav:  1]
   QERR        0  [cha: n, def:  0, sav:  0]
   RAC         0  [cha: n, def:  0, sav:  0]
   UA_INTLCK   0  [cha: n, def:  0, sav:  0]
   SWP         0  [cha: y, def:  0, sav:  0]
   ATO         0  [cha: n, def:  0, sav:  0]
   TAS         0  [cha: n, def:  0, sav:  0]
   AUTOLOAD    0  [cha: n, def:  0, sav:  0]
   BTP         0  [cha: n, def:  0, sav:  0]
   ESTCT       0  [cha: y, def:  0, sav:  0]

It is an SMP, 64 bit processor kernel:
$ cat /proc/cpuinfo
processor  : 0
vendor     : GenuineIntel
arch       : IA-64
family     : Itanium 2
model      : 1
revision   : 5
archrev    : 0
features   : branchlong
cpu number : 0
cpu regs   : 4
cpu MHz    : 1500.000000
itc MHz    : 1500.000000
BogoMIPS   : 2239.75

processor  : 1
dito


On the same disk, sgm_dd is no faster but shows
an impressive 0% CPU utilization
(0.00user 0.23system 9:18.07elapsed 0%CPU).

Doug Gilbert