Re: UDP is bypassing qdisc statistics ....

Re: UDP is bypassing qdisc statistics ....

[Eric Dumazet]

Christoph Lameter a écrit :
> This is with 2.6.31-rc7. If I send icmp then its correctly registered as a
> packet by the qdisc layer:
> 
> #tc -s qdisc show
> qdisc pfifo_fast 0: dev eth0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 10491 bytes 72 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
> 
> #ping yy
> PING yy (10.2.36.120) 56(84) bytes of data.
> 64 bytes from yy (10.2.36.120): icmp_seq=1 ttl=64 time=0.933 ms
> 64 bytes from yy (10.2.36.120): icmp_seq=2 ttl=64 time=0.122 ms
> 64 bytes from yy (10.2.36.120): icmp_seq=3 ttl=64 time=0.119 ms
> ^C
> --- yy ping statistics ---
> 3 packets transmitted, 3 received, 0% packet loss, time 2003ms
> rtt min/avg/max/mdev = 0.119/0.391/0.933/0.383 ms
> 
> #tc -s qdisc show
> qdisc pfifo_fast 0: dev eth0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 10839 bytes 75 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
> 
> 
> Now I overrun the transmitter with udp multicast traffic:
> 
> #./mcast -n1 -r400000
> Receiver: Listening to control channel 239.0.192.1
> Receiver: Subscribing to 0 MC addresses 239.0.192-254.2-254 offset 0
> origin 10.2.36.120
> Sender: Sending 400000 msgs/ch/sec on 1 channels. Probe interval=0.001-1 sec.
> ^C
> Unsubscribing from multicast groups
> Done.
> 
> #tc -s qdisc show
> qdisc pfifo_fast 0: dev eth0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 13359 bytes 101 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
> 
> 
> The mcast invocation send over 3 million packets(!) but they are not
> accounted for in the qdisc. 500k of those were lost. Also not counted.
> The 26 packets receives are likely only covering the ssh session using
> TCP.
> 
> Receiver log:
> 
> yy#./mcast
> Receiver: Listening to control channel 239.0.192.1
> Receiver: Subscribing to 1 MC addresses 239.0.192-254.2-254 offset 0 origin 10.2.36.121
> 
> TotalMsg   Lost SeqErr TXDrop Msg/Sec  KB/Sec  Min/us  Avg/us  Max/us StdDv
>  3110416 531472  25555 531472  307958     0.0  151.45 1648.21 1875.68 499.92
> Unsubscribing from 1 multicast addresses origin 10.2.36.121.
> 
> What is wrong here? Is UDP bypassing the qdisc layer?
> 
> Tried UDP unicast with similar effects.


Not reproductible here, I can see bytes/pkts counts increasing while mcast -n1 -r400000 runs.

# tc -s -d qdisc show dev eth0 ; sleep 1 ; tc -s -d qdisc show dev eth0
qdisc pfifo_fast 0: root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 7694155580 bytes 36894415 pkt (dropped 0, overlimits 0 requeues 2)
 rate 0bit 0pps backlog 0b 0p requeues 2
qdisc pfifo_fast 0: root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 7778853932 bytes 37142071 pkt (dropped 0, overlimits 0 requeues 2)
 rate 0bit 0pps backlog 0b 0p requeues 2


Check :

# ip ro 
# ip ro get 10.2.36.120
# ip ro get 239.0.192.2

maybe your pings do use eth0, and your multicast doesnt.

loopback device do bypass qdisc layer for example...

Re: UDP is bypassing qdisc statistics ....

[Mark Smith]

On Mon, 31 Aug 2009 21:54:49 +0200
Eric Dumazet <eric.dumazet@gmail.com> wrote:

> Christoph Lameter a écrit :
> > This is with 2.6.31-rc7. If I send icmp then its correctly registered as a
> > packet by the qdisc layer:
> > 
<snip>
> 
> loopback device do bypass qdisc layer for example...

On occassion, I'd have found it useful if it didn't. It'd be convenient
to test out your qdisc config, or test out applications performance
behaviour over a simulated WAN via netem, without having to a
network and two hosts, and all the related miscellaneous setup work.

UDP is bypassing qdisc statistics ....

[Christoph Lameter]

This is with 2.6.31-rc7. If I send icmp then its correctly registered as a
packet by the qdisc layer:

#tc -s qdisc show
qdisc pfifo_fast 0: dev eth0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 10491 bytes 72 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0

#ping yy
PING yy (10.2.36.120) 56(84) bytes of data.
64 bytes from yy (10.2.36.120): icmp_seq=1 ttl=64 time=0.933 ms
64 bytes from yy (10.2.36.120): icmp_seq=2 ttl=64 time=0.122 ms
64 bytes from yy (10.2.36.120): icmp_seq=3 ttl=64 time=0.119 ms
^C
--- yy ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.119/0.391/0.933/0.383 ms

#tc -s qdisc show
qdisc pfifo_fast 0: dev eth0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 10839 bytes 75 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0


Now I overrun the transmitter with udp multicast traffic:

#./mcast -n1 -r400000
Receiver: Listening to control channel 239.0.192.1
Receiver: Subscribing to 0 MC addresses 239.0.192-254.2-254 offset 0
origin 10.2.36.120
Sender: Sending 400000 msgs/ch/sec on 1 channels. Probe interval=0.001-1 sec.
^C
Unsubscribing from multicast groups
Done.

#tc -s qdisc show
qdisc pfifo_fast 0: dev eth0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 13359 bytes 101 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0


The mcast invocation send over 3 million packets(!) but they are not
accounted for in the qdisc. 500k of those were lost. Also not counted.
The 26 packets receives are likely only covering the ssh session using
TCP.

Receiver log:

yy#./mcast
Receiver: Listening to control channel 239.0.192.1
Receiver: Subscribing to 1 MC addresses 239.0.192-254.2-254 offset 0 origin 10.2.36.121

TotalMsg   Lost SeqErr TXDrop Msg/Sec  KB/Sec  Min/us  Avg/us  Max/us StdDv
 3110416 531472  25555 531472  307958     0.0  151.45 1648.21 1875.68 499.92
Unsubscribing from 1 multicast addresses origin 10.2.36.121.

What is wrong here? Is UDP bypassing the qdisc layer?

Tried UDP unicast with similar effects.

Re: UDP is bypassing qdisc statistics ....

[Jarek Poplawski]

On 31-08-2009 22:58, Mark Smith wrote:
> On Mon, 31 Aug 2009 21:54:49 +0200
> Eric Dumazet <eric.dumazet@gmail.com> wrote:
> 
>> Christoph Lameter a écrit :
>>> This is with 2.6.31-rc7. If I send icmp then its correctly registered as a
>>> packet by the qdisc layer:
>>>
> <snip>
>> loopback device do bypass qdisc layer for example...
> 
> On occassion, I'd have found it useful if it didn't. It'd be convenient
> to test out your qdisc config, or test out applications performance
> behaviour over a simulated WAN via netem, without having to a
> network and two hosts, and all the related miscellaneous setup work.

Probably Eric and you mean something special, but generally a loopback
and some other virtuals bypass qdisc layer only with default qdisc.

Jarek P.

Re: UDP is bypassing qdisc statistics ....

[Eric Dumazet]

Jarek Poplawski a écrit :
> On 31-08-2009 22:58, Mark Smith wrote:
>> On Mon, 31 Aug 2009 21:54:49 +0200
>> Eric Dumazet <eric.dumazet@gmail.com> wrote:
>>
>>> Christoph Lameter a écrit :
>>>> This is with 2.6.31-rc7. If I send icmp then its correctly registered as a
>>>> packet by the qdisc layer:
>>>>
>> <snip>
>>> loopback device do bypass qdisc layer for example...
>> On occassion, I'd have found it useful if it didn't. It'd be convenient
>> to test out your qdisc config, or test out applications performance
>> behaviour over a simulated WAN via netem, without having to a
>> network and two hosts, and all the related miscellaneous setup work.
> 
> Probably Eric and you mean something special, but generally a loopback
> and some other virtuals bypass qdisc layer only with default qdisc.
> 

Yes, I was referring to Christoph use, since on its machine, only
output from "tc -s -d qdisc" is about eth0


Mark, you can certainly do something like

# tc qdisc del dev lo root
# tc qdisc add dev lo root netem delay 100ms 10ms

# ping 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
2009/08/01 08:59:22.799 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=204 ms
2009/08/01 08:59:23.804 64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=208 ms
2009/08/01 08:59:24.801 64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=204 ms
2009/08/01 08:59:25.808 64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=209 ms

# tc -s -d qdisc show dev lo
qdisc netem 8001: root limit 1000 delay 100.0ms  10.0ms
 Sent 1764 bytes 18 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0

Re: UDP is bypassing qdisc statistics ....

[Mark Smith]

Hi Eric,

On Tue, 01 Sep 2009 09:00:46 +0200
Eric Dumazet <eric.dumazet@gmail.com> wrote:

> Jarek Poplawski a écrit :
> > On 31-08-2009 22:58, Mark Smith wrote:
> >> On Mon, 31 Aug 2009 21:54:49 +0200
> >> Eric Dumazet <eric.dumazet@gmail.com> wrote:
> >>
> >>> Christoph Lameter a écrit :
> >>>> This is with 2.6.31-rc7. If I send icmp then its correctly registered as a
> >>>> packet by the qdisc layer:
> >>>>
> >> <snip>
> >>> loopback device do bypass qdisc layer for example...
> >> On occassion, I'd have found it useful if it didn't. It'd be convenient
> >> to test out your qdisc config, or test out applications performance
> >> behaviour over a simulated WAN via netem, without having to a
> >> network and two hosts, and all the related miscellaneous setup work.
> > 
> > Probably Eric and you mean something special, but generally a loopback
> > and some other virtuals bypass qdisc layer only with default qdisc.
> > 
> 
> Yes, I was referring to Christoph use, since on its machine, only
> output from "tc -s -d qdisc" is about eth0
> 
> 
> Mark, you can certainly do something like
> 
> # tc qdisc del dev lo root
> # tc qdisc add dev lo root netem delay 100ms 10ms
> 
> # ping 127.0.0.1
> PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
> 2009/08/01 08:59:22.799 64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=204 ms
> 2009/08/01 08:59:23.804 64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=208 ms
> 2009/08/01 08:59:24.801 64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=204 ms
> 2009/08/01 08:59:25.808 64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=209 ms
> 
> # tc -s -d qdisc show dev lo
> qdisc netem 8001: root limit 1000 delay 100.0ms  10.0ms
>  Sent 1764 bytes 18 pkt (dropped 0, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0

Either I tested it a couple of years back and you couldn't assign a
qdisc to loopback, and/or I read somewhere that the loopback driver was
highly optimised for looping packets back, and assumed that meant that
it bypassed the qdisc layer. 

Oh well, that's great that it already does what I was asking for.

Thanks,
Mark.

Re: UDP is bypassing qdisc statistics ....

[Patrick McHardy]

Christoph Lameter wrote:
> Eric: This occurs with a Broadcom driver (bnx2). There is only one
> network device up and the IP statistics are correctly incremented.

That explains it. The bnx2 driver uses multiple TX queues, but
tc_dump_qdisc() only dumps the statistics from queue number 0.

Not sure whether we should sum them up or dump the statistics
from each queue (or both). Summing them up avoids userspace
visible changes when drivers are converted to multiq.

Re: UDP is bypassing qdisc statistics ....

[Eric Dumazet]

Patrick McHardy a écrit :
> Christoph Lameter wrote:
>> Eric: This occurs with a Broadcom driver (bnx2). There is only one
>> network device up and the IP statistics are correctly incremented.
> 
> That explains it. The bnx2 driver uses multiple TX queues, but
> tc_dump_qdisc() only dumps the statistics from queue number 0.
> 
> Not sure whether we should sum them up or dump the statistics
> from each queue (or both). Summing them up avoids userspace
> visible changes when drivers are converted to multiq.

Ok mystery solved, partially at least :)

Hmm, my bnx2 still displays correct info here (BCM5708S) ; so maybe Christoph has a
real multiqueue NIC adapter ? I thought he mentioned a mono queue NIC earlier.

Dumping stats for each queue would be better for admin point of view IMHO
(restricted to real_num_tx_queues I guess, since bnx2 allocates 8 queues per device.

Re: UDP is bypassing qdisc statistics ....

[Eric Dumazet]

Christoph Lameter a écrit :
> On Tue, 1 Sep 2009, Eric Dumazet wrote:
> 
>> Hmm, my bnx2 still displays correct info here (BCM5708S) ; so maybe Christoph has a
>> real multiqueue NIC adapter ? I thought he mentioned a mono queue NIC earlier.
> 
> I have no clue if it is or not. I have never used those features.
> 
> dmesg says:
> 
> Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express
> 
> Is that a multiqueue adapter?

You should see that in /proc/interrupts, if I correctly understand bnx2.c

Just to be sure :
With following patch I got in my syslog :

[    8.789048] rx_rings=1 real_num_tx_queues=1
[    9.005700] rx_rings=1 real_num_tx_queues=1

So my bnx2 is mono-queue :(


diff --git a/drivers/net/bnx2.c b/drivers/net/bnx2.c
index 06b9011..573d6b3 100644
--- a/drivers/net/bnx2.c
+++ b/drivers/net/bnx2.c
@@ -6144,6 +6144,7 @@ bnx2_setup_int_mode(struct bnx2 *bp, int dis_msi)
 	bp->dev->real_num_tx_queues = bp->num_tx_rings;

 	bp->num_rx_rings = bp->irq_nvecs;
+	pr_err("rx_rings=%d real_num_tx_queues=%d\n", bp->num_rx_rings,  bp->num_tx_rings);
 }

 /* Called with rtnl_lock */

Re: UDP is bypassing qdisc statistics ....

[Eric Dumazet]

Christoph Lameter a écrit :
> On Tue, 1 Sep 2009, Patrick McHardy wrote:
> 
>> That explains it. The bnx2 driver uses multiple TX queues, but
>> tc_dump_qdisc() only dumps the statistics from queue number 0.
> 
> There are no stats in other queues either... This is the result after
> sending 10000 or so packets. Maybe I am not catching all the qdiscs?
> 
> (this is the result of a pretty raw patch to dump all qdiscs)
> 
> #cat /proc/net/qdisc_stats
> Type Device State Bytes Packets Qlen Backlog Drops Requeues Overlimits
> TX root <NULL> 0 0 0 0 0 0 0 0
> RX root <NULL> 0 0 0 0 0 0 0 0
> TX root  eth0 0 24830 155 0 0 0 0 0
> RX root <NULL> 0 0 0 0 0 0 0 0
> TX root <NULL> 0 0 0 0 0 0 0 0
> RX root <NULL> 0 0 0 0 0 0 0 0
> TX root <NULL> 0 0 0 0 0 0 0 0
> RX root <NULL> 0 0 0 0 0 0 0 0
> TX root <NULL> 0 0 0 0 0 0 0 0
> RX root <NULL> 0 0 0 0 0 0 0 0
> 
> ---
>  net/sched/sch_api.c |   64 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 64 insertions(+)
> 
> Index: linux-2.6/net/sched/sch_api.c
> ===================================================================
> --- linux-2.6.orig/net/sched/sch_api.c	2009-08-31 21:19:04.000000000 +0000
> +++ linux-2.6/net/sched/sch_api.c	2009-09-01 14:30:40.000000000 +0000
> @@ -1699,6 +1699,69 @@ static const struct file_operations psch
>  	.llseek = seq_lseek,
>  	.release = single_release,
>  };
> +
> +static void dump_qdisc(struct seq_file *seq, struct Qdisc *q, char *inout, char *text)
> +{
> +	seq_printf(seq, "%2s %2s %5s %lx %lld %d %d %d %d %d %d\n",
> +		inout, text, q->dev_queue->dev->name, q->state,
> +		q->bstats.bytes, q->bstats.packets,
> +		q->qstats.qlen, q->qstats.backlog, q->qstats.drops,
> +		q->qstats.requeues, q->qstats.overlimits);
> +}
> +
> +static void dump_qdisc_root(struct seq_file *seq, struct Qdisc *root, char *inout)
> +{
> +	struct Qdisc *q;
> +	int n = 0;
> +
> +	if (!root)
> +		return;
> +
> +	dump_qdisc(seq, root, inout, "root");
> +
> +	list_for_each_entry(q, &root->list, list) {
> +		char buffer[10];
> +
> +		sprintf(buffer,"%d", ++n);
> +		dump_qdisc(seq, q, inout, buffer);
> +	}
> +}
> +
> +
> +static int qdisc_show(struct seq_file *seq, void *v)
> +{
> +	struct net_device *dev;
> +
> +	seq_printf(seq, "Type Device State Bytes Packets "
> +			"Qlen Backlog Drops Requeues Overlimits\n");
> +
> +	read_lock(&dev_base_lock);
> +
> +	for_each_netdev(&init_net, dev) {
> +		struct netdev_queue *dev_queue;
> +
> +		dev_queue = netdev_get_tx_queue(dev, 0);
> +		dump_qdisc_root(seq, dev_queue->qdisc_sleeping, "TX");

you should iterate here 

	for (i = 0 ; i < dev->real_num_tx_queues; i++) {
		dev_queue = netdev_get_tx_queue(dev, i);
		dump_qdisc_root(seq, dev_queue->qdisc_sleeping, "TX");
	}


> +		dev_queue = &dev->rx_queue;
> +		dump_qdisc_root(seq, dev_queue->qdisc_sleeping, "RX");
> +	}
> +
> +	read_unlock(&dev_base_lock);
> +	return 0;
> +}
> +
> +static int qdisc_open(struct inode *inode, struct file *file)
> +{
> +	return single_open(file, qdisc_show, PDE(inode)->data);
> +}
> +
> +static const struct file_operations qdisc_fops = {
> +	.owner = THIS_MODULE,
> +	.open = qdisc_open,
> +	.read = seq_read,
> +	.llseek = seq_lseek,
> +	.release = single_release,
> +};
>  #endif
> 
>  static int __init pktsched_init(void)
> @@ -1706,6 +1769,7 @@ static int __init pktsched_init(void)
>  	register_qdisc(&pfifo_qdisc_ops);
>  	register_qdisc(&bfifo_qdisc_ops);
>  	proc_net_fops_create(&init_net, "psched", 0, &psched_fops);
> +	proc_net_fops_create(&init_net, "qdisc_stats", 0, &qdisc_fops);
> 
>  	rtnl_register(PF_UNSPEC, RTM_NEWQDISC, tc_modify_qdisc, NULL);
>  	rtnl_register(PF_UNSPEC, RTM_DELQDISC, tc_get_qdisc, NULL);

Re: UDP is bypassing qdisc statistics ....

[Eric Dumazet]

Christoph Lameter a écrit :
> On Tue, 1 Sep 2009, Eric Dumazet wrote:
> 
>> You should see that in /proc/interrupts, if I correctly understand bnx2.c
> 
> Hmmm I have 8 interrupts:
> 
>   62:        158          0          0          0          0          0
> 0          0          0          0          0          0          0
> 0          0          0  IR-PCI-MSI-edge      eth0-0
>   63:         84          0          0          0          0          0
> 0          0          0          0          0          0          0
> 0          0          0  IR-PCI-MSI-edge      eth0-1
>   64:        412          0          0          0          0          0
> 0          0          0          0          0          0          0
> 0          0          0  IR-PCI-MSI-edge      eth0-2
>   65:         25          0          0          0          0          0
> 0          0          0          0          0          0          0
> 0          0          0  IR-PCI-MSI-edge      eth0-3
>   66:      49718          0          0          0          0          0
> 0          0          0          0          0          0          0
> 0          0          0  IR-PCI-MSI-edge      eth0-4
>   67:         65          0          0          0          0          0
> 0          0          0          0          0          0          0
> 0          0          0  IR-PCI-MSI-edge      eth0-5
>   68:        686          0          0          0          0          0
> 0          0          0          0          0          0          0
> 0          0          0  IR-PCI-MSI-edge      eth0-6
>   69:       2582          0          0          0          0          0
> 0          0          0          0          0          0          0
> 0          0          0  IR-PCI-MSI-edge      eth0-7

Yes, this confirm you have 8 queues on this NIC

Strange thing is they seem to be all serviced by CPU-0, which is not good...

Re: UDP is bypassing qdisc statistics ....

[Eric Dumazet]

Eric Dumazet a écrit :
> Christoph Lameter a écrit :
>> On Tue, 1 Sep 2009, Eric Dumazet wrote:
>>
>>> You should see that in /proc/interrupts, if I correctly understand bnx2.c
>> Hmmm I have 8 interrupts:
>>
>>   62:        158          0          0          0          0          0
>> 0          0          0          0          0          0          0
>> 0          0          0  IR-PCI-MSI-edge      eth0-0
>>   63:         84          0          0          0          0          0
>> 0          0          0          0          0          0          0
>> 0          0          0  IR-PCI-MSI-edge      eth0-1
>>   64:        412          0          0          0          0          0
>> 0          0          0          0          0          0          0
>> 0          0          0  IR-PCI-MSI-edge      eth0-2
>>   65:         25          0          0          0          0          0
>> 0          0          0          0          0          0          0
>> 0          0          0  IR-PCI-MSI-edge      eth0-3
>>   66:      49718          0          0          0          0          0
>> 0          0          0          0          0          0          0
>> 0          0          0  IR-PCI-MSI-edge      eth0-4
>>   67:         65          0          0          0          0          0
>> 0          0          0          0          0          0          0
>> 0          0          0  IR-PCI-MSI-edge      eth0-5
>>   68:        686          0          0          0          0          0
>> 0          0          0          0          0          0          0
>> 0          0          0  IR-PCI-MSI-edge      eth0-6
>>   69:       2582          0          0          0          0          0
>> 0          0          0          0          0          0          0
>> 0          0          0  IR-PCI-MSI-edge      eth0-7
> 
> Yes, this confirm you have 8 queues on this NIC
> 
> Strange thing is they seem to be all serviced by CPU-0, which is not good...
> 
> 
> 

Given that bnx2.c uses num_online_cpus() at init time, you could
as a workaround do the insmod/modprobe bnx2 with only one online cpu,
and you'll revert to a mono-queue NIC :)

int msix_vecs = min(cpus + 1, RX_MAX_RINGS); 
...
if ((bp->flags & BNX2_FLAG_MSIX_CAP) && !dis_msi && cpus > 1)
	bnx2_enable_msix(bp, msix_vecs);


For your multicast test anyway, only one queue should be used (one flow)

Re: UDP is bypassing qdisc statistics ....

[Christoph Lameter]

Eric: This occurs with a Broadcom driver (bnx2). There is only one
network device up and the IP statistics are correctly incremented.

#ip ro
10.2.36.0/24 dev eth0  proto kernel  scope link  src 10.2.36.121
default via 10.2.36.3 dev eth0
#ip ro get 10.2.36.120
10.2.36.120 dev eth0  src 10.2.36.121
    cache  mtu 1500 advmss 1460 hoplimit 64
ip ro get 239.0.192.2
multicast 239.0.192.2 dev eth0  src 10.2.36.121
    cache <mc>  mtu 1500 advmss 1460 hoplimit 64

dmesg:

[    1.560094] Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.0.1
(May 6, 2009)
[    1.560192]   alloc irq_desc for 36 on node 0
[    1.560193]   alloc kstat_irqs on node 0
[    1.560197] alloc irq_2_iommu on node 0
[    1.560204] bnx2 0000:01:00.0: PCI INT A -> GSI 36 (level, low) -> IRQ
36
[    1.560275] bnx2 0000:01:00.0: setting latency timer to 64
[    1.560394] bnx2 0000:01:00.0: firmware: requesting
bnx2/bnx2-mips-09-4.6.17.fw


$ netstat -su
IcmpMsg:
    InType0: 4
    InType3: 1
    OutType3: 1
    OutType8: 4
Udp:
    44363 packets received
    0 packets to unknown port received.
    0 packet receive errors
    4363620 packets sent
    SndbufErrors: 738519
UdpLite:
IpExt:
    InMcastPkts: 1289
    OutMcastPkts: 5057780
    InOctets: 13956676
    OutOctets: 1672676365
    InMcastOctets: 36164
    OutMcastOctets: 1658949880

(Your patch to account for TX errors was applied)

clameter@rd-gateway3-deb64:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:21:9b:8f:9f:1b
          inet addr:10.2.36.121  Bcast:10.2.36.255  Mask:255.255.255.0
          inet6 addr: fe80::221:9bff:fe8f:9f1b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:50091 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4367178 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:14928488 (14.2 MiB)  TX bytes:1509122809 (1.4 GiB)
          Interrupt:36 Memory:d6000000-d6012800

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:89 errors:0 dropped:0 overruns:0 frame:0
          TX packets:89 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:7114 (6.9 KiB)  TX bytes:7114 (6.9 KiB)

Re: UDP is bypassing qdisc statistics ....

[Christoph Lameter]

On Tue, 1 Sep 2009, Patrick McHardy wrote:

> Christoph Lameter wrote:
> > Eric: This occurs with a Broadcom driver (bnx2). There is only one
> > network device up and the IP statistics are correctly incremented.
>
> That explains it. The bnx2 driver uses multiple TX queues, but
> tc_dump_qdisc() only dumps the statistics from queue number 0.
>
> Not sure whether we should sum them up or dump the statistics
> from each queue (or both). Summing them up avoids userspace
> visible changes when drivers are converted to multiq.

Well I have almost finished a /proc/net/qdisc_stats that shows *all*
qdisc queues. Did this primarily for debugging since I do not know too
much about the netlink stuff.

Re: UDP is bypassing qdisc statistics ....

[Christoph Lameter]

On Tue, 1 Sep 2009, Eric Dumazet wrote:

> Hmm, my bnx2 still displays correct info here (BCM5708S) ; so maybe Christoph has a
> real multiqueue NIC adapter ? I thought he mentioned a mono queue NIC earlier.

I have no clue if it is or not. I have never used those features.

dmesg says:

Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express

Is that a multiqueue adapter?

Re: UDP is bypassing qdisc statistics ....

[Christoph Lameter]

On Tue, 1 Sep 2009, Patrick McHardy wrote:

> That explains it. The bnx2 driver uses multiple TX queues, but
> tc_dump_qdisc() only dumps the statistics from queue number 0.

There are no stats in other queues either... This is the result after
sending 10000 or so packets. Maybe I am not catching all the qdiscs?

(this is the result of a pretty raw patch to dump all qdiscs)

#cat /proc/net/qdisc_stats
Type Device State Bytes Packets Qlen Backlog Drops Requeues Overlimits
TX root <NULL> 0 0 0 0 0 0 0 0
RX root <NULL> 0 0 0 0 0 0 0 0
TX root  eth0 0 24830 155 0 0 0 0 0
RX root <NULL> 0 0 0 0 0 0 0 0
TX root <NULL> 0 0 0 0 0 0 0 0
RX root <NULL> 0 0 0 0 0 0 0 0
TX root <NULL> 0 0 0 0 0 0 0 0
RX root <NULL> 0 0 0 0 0 0 0 0
TX root <NULL> 0 0 0 0 0 0 0 0
RX root <NULL> 0 0 0 0 0 0 0 0

---
 net/sched/sch_api.c |   64 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)

Index: linux-2.6/net/sched/sch_api.c
===================================================================
--- linux-2.6.orig/net/sched/sch_api.c	2009-08-31 21:19:04.000000000 +0000
+++ linux-2.6/net/sched/sch_api.c	2009-09-01 14:30:40.000000000 +0000
@@ -1699,6 +1699,69 @@ static const struct file_operations psch
 	.llseek = seq_lseek,
 	.release = single_release,
 };
+
+static void dump_qdisc(struct seq_file *seq, struct Qdisc *q, char *inout, char *text)
+{
+	seq_printf(seq, "%2s %2s %5s %lx %lld %d %d %d %d %d %d\n",
+		inout, text, q->dev_queue->dev->name, q->state,
+		q->bstats.bytes, q->bstats.packets,
+		q->qstats.qlen, q->qstats.backlog, q->qstats.drops,
+		q->qstats.requeues, q->qstats.overlimits);
+}
+
+static void dump_qdisc_root(struct seq_file *seq, struct Qdisc *root, char *inout)
+{
+	struct Qdisc *q;
+	int n = 0;
+
+	if (!root)
+		return;
+
+	dump_qdisc(seq, root, inout, "root");
+
+	list_for_each_entry(q, &root->list, list) {
+		char buffer[10];
+
+		sprintf(buffer,"%d", ++n);
+		dump_qdisc(seq, q, inout, buffer);
+	}
+}
+
+
+static int qdisc_show(struct seq_file *seq, void *v)
+{
+	struct net_device *dev;
+
+	seq_printf(seq, "Type Device State Bytes Packets "
+			"Qlen Backlog Drops Requeues Overlimits\n");
+
+	read_lock(&dev_base_lock);
+
+	for_each_netdev(&init_net, dev) {
+		struct netdev_queue *dev_queue;
+
+		dev_queue = netdev_get_tx_queue(dev, 0);
+		dump_qdisc_root(seq, dev_queue->qdisc_sleeping, "TX");
+		dev_queue = &dev->rx_queue;
+		dump_qdisc_root(seq, dev_queue->qdisc_sleeping, "RX");
+	}
+
+	read_unlock(&dev_base_lock);
+	return 0;
+}
+
+static int qdisc_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, qdisc_show, PDE(inode)->data);
+}
+
+static const struct file_operations qdisc_fops = {
+	.owner = THIS_MODULE,
+	.open = qdisc_open,
+	.read = seq_read,
+	.llseek = seq_lseek,
+	.release = single_release,
+};
 #endif

 static int __init pktsched_init(void)
@@ -1706,6 +1769,7 @@ static int __init pktsched_init(void)
 	register_qdisc(&pfifo_qdisc_ops);
 	register_qdisc(&bfifo_qdisc_ops);
 	proc_net_fops_create(&init_net, "psched", 0, &psched_fops);
+	proc_net_fops_create(&init_net, "qdisc_stats", 0, &qdisc_fops);

 	rtnl_register(PF_UNSPEC, RTM_NEWQDISC, tc_modify_qdisc, NULL);
 	rtnl_register(PF_UNSPEC, RTM_DELQDISC, tc_get_qdisc, NULL);

Re: UDP is bypassing qdisc statistics ....

[Christoph Lameter]

On Tue, 1 Sep 2009, Eric Dumazet wrote:

> You should see that in /proc/interrupts, if I correctly understand bnx2.c

Hmmm I have 8 interrupts:

  62:        158          0          0          0          0          0
0          0          0          0          0          0          0
0          0          0  IR-PCI-MSI-edge      eth0-0
  63:         84          0          0          0          0          0
0          0          0          0          0          0          0
0          0          0  IR-PCI-MSI-edge      eth0-1
  64:        412          0          0          0          0          0
0          0          0          0          0          0          0
0          0          0  IR-PCI-MSI-edge      eth0-2
  65:         25          0          0          0          0          0
0          0          0          0          0          0          0
0          0          0  IR-PCI-MSI-edge      eth0-3
  66:      49718          0          0          0          0          0
0          0          0          0          0          0          0
0          0          0  IR-PCI-MSI-edge      eth0-4
  67:         65          0          0          0          0          0
0          0          0          0          0          0          0
0          0          0  IR-PCI-MSI-edge      eth0-5
  68:        686          0          0          0          0          0
0          0          0          0          0          0          0
0          0          0  IR-PCI-MSI-edge      eth0-6
  69:       2582          0          0          0          0          0
0          0          0          0          0          0          0
0          0          0  IR-PCI-MSI-edge      eth0-7

Re: UDP is bypassing qdisc statistics ....

[Christoph Lameter]

On Tue, 1 Sep 2009, Eric Dumazet wrote:

> For your multicast test anyway, only one queue should be used (one flow)

Right.

I finally get some numbers with the qdisc_stats patch:

clameter@rd-strategy3-deb64:~$ cat /proc/net/qdisc_stats
Type     Device St  Bytes    Packts Qlen Bklg Drops  Requeu Overlimits
TXS root <NULL>   0        0      0    0    0      0      0      0
TX root <NULL>   0        0      0    0    0      0      0      0
RXS root <NULL>   0        0      0    0    0      0      0      0
RX root <NULL>   0        0      0    0    0      0      0      0
TXS root   eth0   0     2778     35    0    0      0      0      0
TX root   eth0   0     2778     35    0    0      0      0      0
TXS root   eth0   0     1622     18    0    0      0      0      0
TX root   eth0   0     1622     18    0    0      0      0      0
TXS root   eth0   0     2618     22    0    0      0      0      0
TX root   eth0   0     2618     22    0    0      0      0      0
TXS root   eth0   0     9792     65    0    0      0      0      0
TX root   eth0   0     9792     65    0    0      0      0      0
TXS root   eth0   0     9758     68    0    0      0      0      0
TX root   eth0   0     9758     68    0    0      0      0      0
TXS root   eth0   0     1413     18    0    0      0      0      0
TX root   eth0   0     1413     18    0    0      0      0      0
TXS root   eth0   0      557      8    0    0      0      0      0
TX root   eth0   0      557      8    0    0      0      0      0
TXS root   eth0   0     1220     11    0    0      0      0      0
TX root   eth0   0     1220     11    0    0      0      0      0
RXS root <NULL>   0        0      0    0    0      0      0      0

Re: UDP is bypassing qdisc statistics ....

[Jarek Poplawski]

On Tue, Sep 01, 2009 at 05:55:44PM -0400, Christoph Lameter wrote:
> 
> Ok i got some meaningful statistics now. Why is the qlen zero for all
> devices?

Looks like it's updated with q->q.qlen before dumps only. Nice work!
Btw, isn't there something with your localhost's time?

Original-Received: from ...
	...
	for <netdev@vger.kernel.org>; Tue,  1 Sep 2009 13:58:12 -0400 (EDT)
Original-Received: from cl (helo=localhost)
	...
	id 1MibKK-0007nd-Ib; Tue, 01 Sep 2009 17:55:44 -0400

Jarek P.

Re: UDP is bypassing qdisc statistics ....

[Christoph Lameter]

Ok i got some meaningful statistics now. Why is the qlen zero for all
devices?

$ cat /proc/net/qdisc_stats
Type     Device  St    Bytes Packts Qlen Bklg  Drops Requeu Overlimit
TX0 root     lo   0        0      0    0    0      0      0      0
 RX root     lo   0        0      0    0    0      0      0      0
TX0 root   eth0   0    12463     91    0    0      0      0      0
TX1 root   eth0   0     2954     39    0    0      0      0      0
TX2 root   eth0   0     2132     23    0    0      0      0      0
TX3 root   eth0   0    29293    210    0    0      0      0      0
TX4 root   eth0   0     2805     31    0    0      0      0      0
TX5 root   eth0   0 1286862317 3762765    0    0 643253  30886      0
TX6 root   eth0   0     1310     17    0    0      0      0      0
TX7 root   eth0   0     2860     31    0    0      0      0      0

---
 net/sched/sch_api.c |   72 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)

Index: linux-2.6/net/sched/sch_api.c
===================================================================
--- linux-2.6.orig/net/sched/sch_api.c	2009-09-01 12:27:24.000000000 -0500
+++ linux-2.6/net/sched/sch_api.c	2009-09-01 12:40:30.000000000 -0500
@@ -1699,6 +1699,77 @@ static const struct file_operations psch
 	.llseek = seq_lseek,
 	.release = single_release,
 };
+
+static void dump_qdisc(struct seq_file *seq, struct net_device *dev,
+				struct Qdisc *q, char *inout, char *text)
+{
+	seq_printf(seq, "%3s %2s %6s %3lx %8lld %6d %4d %4d %6d %6d %6d\n",
+		inout, text, dev->name, q->state,
+		q->bstats.bytes, q->bstats.packets,
+		q->qstats.qlen, q->qstats.backlog, q->qstats.drops,
+		q->qstats.requeues, q->qstats.overlimits);
+}
+
+static void dump_qdisc_root(struct seq_file *seq, struct net_device *dev,
+					 struct Qdisc *root, char *inout)
+{
+	struct Qdisc *q;
+	int n = 0;
+
+	if (!root)
+		return;
+
+	dump_qdisc(seq, dev, root, inout, "root");
+
+	list_for_each_entry(q, &root->list, list) {
+		char buffer[10];
+
+		sprintf(buffer,"Q%d", ++n);
+		dump_qdisc(seq, dev, q, inout, buffer);
+	}
+}
+
+
+static int qdisc_show(struct seq_file *seq, void *v)
+{
+	struct net_device *dev;
+
+	seq_printf(seq, "Type     Device  St    Bytes Packts "
+			"Qlen Bklg  Drops Requeu Overlimit\n");
+
+	read_lock(&dev_base_lock);
+
+	for_each_netdev(&init_net, dev) {
+		struct netdev_queue *dev_queue;
+		int i;
+
+		for (i = 0; i < dev->real_num_tx_queues; i++) {
+			char buffer[10];
+
+			dev_queue = netdev_get_tx_queue(dev, i);
+			sprintf(buffer, "TX%d", i);
+			dump_qdisc_root(seq, dev, dev_queue->qdisc_sleeping, buffer);
+		}
+		dev_queue = &dev->rx_queue;
+		dump_qdisc_root(seq, dev, dev_queue->qdisc_sleeping, "RX");
+	}
+
+	read_unlock(&dev_base_lock);
+	return 0;
+}
+
+static int qdisc_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, qdisc_show, PDE(inode)->data);
+}
+
+static const struct file_operations qdisc_fops = {
+	.owner = THIS_MODULE,
+	.open = qdisc_open,
+	.read = seq_read,
+	.llseek = seq_lseek,
+	.release = single_release,
+};
 #endif

 static int __init pktsched_init(void)
@@ -1706,6 +1777,7 @@ static int __init pktsched_init(void)
 	register_qdisc(&pfifo_qdisc_ops);
 	register_qdisc(&bfifo_qdisc_ops);
 	proc_net_fops_create(&init_net, "psched", 0, &psched_fops);
+	proc_net_fops_create(&init_net, "qdisc_stats", 0, &qdisc_fops);

 	rtnl_register(PF_UNSPEC, RTM_NEWQDISC, tc_modify_qdisc, NULL);
 	rtnl_register(PF_UNSPEC, RTM_DELQDISC, tc_get_qdisc, NULL);

Re: UDP is bypassing qdisc statistics ....

[Christoph Lameter]

On Tue, 1 Sep 2009, Jarek Poplawski wrote:

> Btw, isn't there something with your localhost's time?

Yup. I have virtual hosting and the provider screwed up time. I can do
nothing about it.