Re: [PATCH 0/9 Rev3] Implement batching skb API and support in IPoIB

John Heffner
2007-08-25T00:43:11Z

Bill Fink wrote:
> Here you can see there is a major difference in the TX CPU utilization
> (99 % with TSO disabled versus only 39 % with TSO enabled), although
> the TSO disabled case was able to squeeze out a little extra performance
> from its extra CPU utilization. Interestingly, with TSO enabled, the
> receiver actually consumed more CPU than with TSO disabled, so I guess
> the receiver CPU saturation in that case (99 %) was what restricted
> its performance somewhat (this was consistent across a few test runs).

One possibility is that I think the receive-side processing tends to do
better when receiving into an empty queue. When the (non-TSO) sender is
the flow's bottleneck, this is going to be the case. But when you
switch to TSO, the receiver becomes the bottleneck and you're always
going to have to put the packets at the back of the receive queue. This
might help account for the reason why you have both lower throughput and
higher CPU utilization -- there's a point of instability right where the
receiver becomes the bottleneck and you end up pushing it over to the
bad side. :)

Just a theory. I'm honestly surprised this effect would be so
significant. What do the numbers from netstat -s look like in the two
cases?

-John