Re: mlx4_en - Inconsistent TCP throughput

[Yevgeny Petrilin <yevgenyp-VPRAkNaXOzVS1MOuV/RT9w@public.gmane.org>]

Hello Brian,

I am the developer of mlx4_en driver.
I will be happy to help you in this issue, it is not rdma related, so let's take it off line.

Yevgeny

Brian Cain wrote:
> I'm trying to aggregate 10GbE ports for a realtime data acquisition
> application.  I need to be able to get consistently high throughput to
> satisfy my application's demands.
> 
> I have a dual-X58 host with a MT26448 on each IOH.  I can acquire >
> 39Gb/s in aggregate across four TCP streams (one per port).  This
> throughput remains consistent across many consecutive netperf
> TCP_STREAM trials, but trying the same experiment hours or days later,
> I'll find that the performance suffers (< 30Gb/s).
> 
> irq_balancer is disabled and all of the IRQs' affinity masks for one
> card are set to the same core, the other card's IRQs are set to a core
> on the other socket.  I'm using 9000 byte MTUs.  The majority of the
> system settings are at their defaults.  I'm keeping the task affinity
> among the netserver processes static as well.  I don't see any error
> messages from mlx* in dmesg.
> 
> Can anyone suggest some parameters that might be able to help me find
> consistently high performance?
> 
> ~~~~
> $  sudo ethtool -i eth16
> driver: mlx4_en (MT_0BD0110010)
> version: 1.4.2 (Sep 24 2009)
> firmware-version: 2.6.0
> bus-info: 0000:28:00.0
> 
> $  lsb_release -a
> LSB Version:
> core-2.0-noarch:core-3.2-noarch:core-2.0-x86_64:core-3.2-x86_64:desktop-3.2-amd64:desktop-3.2-noarch:graphics-2.0-amd64:graphics-2.0-noarch:graphics-3.2-amd64:graphics-3.2-noarch
> Distributor ID: SUSE LINUX
> Description:    openSUSE 11.1 (x86_64)
> Release:        11.1
> Codename:       n/a
> 
> $  cat /proc/version
> Linux version 2.6.29.6-9-debug (root@z800-revo-0) (gcc version 4.3.2
> [gcc-4_3-branch revision 141291] (SUSE Linux) ) #1 SMP Tue Sep 15
> 11:02:15 CDT 2009
> 
> $ sudo grep . /sys/module/mlx4_en/parameters/*
> /sys/module/mlx4_en/parameters/inline_thold:104
> /sys/module/mlx4_en/parameters/ip_reasm:1
> /sys/module/mlx4_en/parameters/num_lro:32
> /sys/module/mlx4_en/parameters/pfcrx:0
> /sys/module/mlx4_en/parameters/pfctx:0
> /sys/module/mlx4_en/parameters/rss_mask:5
> /sys/module/mlx4_en/parameters/rss_xor:0
> 
> $ sudo grep . /sys/module/mlx4_core/parameters/*
> /sys/module/mlx4_core/parameters/block_loopback:1
> /sys/module/mlx4_core/parameters/debug_level:0
> /sys/module/mlx4_core/parameters/enable_qos:N
> /sys/module/mlx4_core/parameters/internal_err_reset:1
> /sys/module/mlx4_core/parameters/log_mtts_per_seg:3
> /sys/module/mlx4_core/parameters/log_num_cq:0
> /sys/module/mlx4_core/parameters/log_num_mac:2
> /sys/module/mlx4_core/parameters/log_num_mcg:0
> /sys/module/mlx4_core/parameters/log_num_mpt:0
> /sys/module/mlx4_core/parameters/log_num_mtt:0
> /sys/module/mlx4_core/parameters/log_num_qp:0
> /sys/module/mlx4_core/parameters/log_num_srq:0
> /sys/module/mlx4_core/parameters/log_num_vlan:0
> /sys/module/mlx4_core/parameters/log_rdmarc_per_qp:0
> /sys/module/mlx4_core/parameters/msi_x:1
> /sys/module/mlx4_core/parameters/set_4k_mtu:0
> /sys/module/mlx4_core/parameters/use_prio:N
> 
> --
> -Brian
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html