Re: Wrong network usage reported by /proc

[Eric Dumazet <dada1@cosmosbay.com>]

Willy Tarreau a écrit :
> On Mon, May 04, 2009 at 09:11:51PM +0200, Matthias Saou wrote:
>> Eric Dumazet wrote :
>>
>>> Matthias Saou a écrit :
>>>> Hi,
>>>>
>>>> I'm posting here as a last resort. I've got lots of heavily used RHEL5
>>>> servers (2.6.18 based) that are reporting all sorts of impossible
>>>> network usage values through /proc, leading to unrealistic snmp/cacti
>>>> graphs where the outgoing bandwidth used it higher than the physical
>>>> interface's maximum speed.
>>>>
>>>> For some details and a test script which compares values from /proc
>>>> with values from tcpdump :
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=489541
>>>>
>>>> The values collected using tcpdump always seem realistic and match the
>>>> values seen on the remote network equipments. So my obvious conclusion
>>>> (but possibly wrong given my limited knowledge) is that something is
>>>> wrong in the kernel, since it's the one exposing the /proc interface.
>>>>
>>>> I've reproduced what seems to be the same problem on recent kernels,
>>>> including the 2.6.27.21-170.2.56.fc10.x86_64 I'm running right now. The
>>>> simple python script available here allows to see it quite easily :
>>>> https://www.redhat.com/archives/rhelv5-list/2009-February/msg00166.html
>>>>
>>>>  * I run the script on my Workstation, I have an FTP server enabled
>>>>  * I download a DVD ISO from a remote workstation : The values match
>>>>  * I start ping floods from remote workstations : The values reported
>>>>    by /proc are much higher than the ones reported by tcpdump. I used
>>>>    "ping -s 500 -f myworkstation" from two remote workstations
>>>>
>>>> If there's anything flawed in my debugging, I'd love to have someone
>>>> point it out to me. TIA to anyone willing to have a look.
>>>>
>>>> Matthias
>>>>
>>> I could not reproduce this here... what kind of NIC are you using on
>>> affected systems ? Some ethernet drivers report stats from card itself,
>>> and I remember seeing some strange stats on some hardware, but I cannot
>>> remember which one it was (we were reading NULL values instead of
>>> real ones, once in a while, maybe it was a firmware issue...)
>> My workstation has a Broadcom BCM5752 (tg3 module). The servers which
>> are most affected have Intel 82571EB (e1000e). But the issue is that
>> with /proc, the values are a lot _higher_ than with tcpdump, and the
>> tcpdump values seem to be the correct ones.
> 
> the e1000 chip reports stats every 2 seconds. So you have to collect
> stats every 2 seconds otherwise you get "camel-looking" stats.
> 

I looked at e1000e driver, and apparently tx_packets & tx_bytes are computed
by the TX completion routine, not by the chip.

static bool e1000_clean_tx_irq(struct e1000_adapter *adapter)
{
...

                        if (cleaned) {
                                struct sk_buff *skb = buffer_info->skb;
                                unsigned int segs, bytecount;
                                segs = skb_shinfo(skb)->gso_segs ?: 1;
                                /* multiply data chunks by size of headers */
                                bytecount = ((segs - 1) * skb_headlen(skb)) +
                                            skb->len;
// maybe bytecount is wrong on some skbs ?
                                total_tx_packets += segs;
                                total_tx_bytes += bytecount;
                        }

...
    adapter->net_stats.tx_bytes += total_tx_bytes;
    adapter->net_stats.tx_packets += total_tx_packets;
...
}

and driver get_stats() does return this adapter->net_stats structure to caller

static struct net_device_stats *e1000_get_stats(struct net_device *netdev) 
{
	struct e1000_adapter *adapter = netdev_priv(netdev);

	/* only return the current stats */
	return &adapter->net_stats;
}

Could be converted to use netdev->stats... Oh well...