netlink stats and message ordering.

netlink stats and message ordering.

[Ben Greear]

I have a single netlink socket that listens for various things,
as well as requests netdev stats and such.

I am seeing a case during interface creation (mac-vlans) where I get one set of stats
that appears to show 1 packet transmitted, and then immediately after that,
a second set of stats for the same device that shows all zero counters.

Since I'm trying to handle wraps properly, my code tends to consider this
a wrap and of course the numbers go all wrong since it wasn't really a wrap.

I am wondering if it's possible that netlink messages are somehow being
re-ordered before they are sent to my application?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: netlink stats and message ordering.

[Eric Dumazet]

Le vendredi 19 novembre 2010 à 10:20 -0800, Ben Greear a écrit :
> I have a single netlink socket that listens for various things,
> as well as requests netdev stats and such.
> 
> I am seeing a case during interface creation (mac-vlans) where I get one set of stats
> that appears to show 1 packet transmitted, and then immediately after that,
> a second set of stats for the same device that shows all zero counters.
> 
> Since I'm trying to handle wraps properly, my code tends to consider this
> a wrap and of course the numbers go all wrong since it wasn't really a wrap.
> 
> I am wondering if it's possible that netlink messages are somehow being
> re-ordered before they are sent to my application?

That's strange, maybe you could add a timestamp (or a sequence number)
in the stats message to check if two messages are eventually re-ordered.

As a fast check you could try to increment an error counter like this.
If you see the rx_dropped going back, then for sure there is a
problem...

diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
index 93f0ba2..1d271ca 100644
--- a/drivers/net/macvlan.c
+++ b/drivers/net/macvlan.c
@@ -444,6 +444,7 @@ static struct rtnl_link_stats64 *macvlan_dev_get_stats64(struct net_device *dev,
 {
 	struct macvlan_dev *vlan = netdev_priv(dev);
 
+	atomic_long_inc(&dev->rx_dropped);
 	if (vlan->pcpu_stats) {
 		struct macvlan_pcpu_stats *p;
 		u64 rx_packets, rx_bytes, rx_multicast, tx_packets, tx_bytes;

Re: netlink stats and message ordering.

[Ben Greear]

On 11/19/2010 08:49 PM, Eric Dumazet wrote:
> Le vendredi 19 novembre 2010 à 10:20 -0800, Ben Greear a écrit :
>> I have a single netlink socket that listens for various things,
>> as well as requests netdev stats and such.
>>
>> I am seeing a case during interface creation (mac-vlans) where I get one set of stats
>> that appears to show 1 packet transmitted, and then immediately after that,
>> a second set of stats for the same device that shows all zero counters.
>>
>> Since I'm trying to handle wraps properly, my code tends to consider this
>> a wrap and of course the numbers go all wrong since it wasn't really a wrap.
>>
>> I am wondering if it's possible that netlink messages are somehow being
>> re-ordered before they are sent to my application?
>
> That's strange, maybe you could add a timestamp (or a sequence number)
> in the stats message to check if two messages are eventually re-ordered.

I am becoming suspicious that I may have a re-entrancy issue in my code..maybe while
handling a dev-create message I am calling into the netlink-read logic again before
handling all of the messages that were read in the initial call.

I'm going to have to think on this more, and if nothing obvious comes up,
I'll try your suggestion for incrementing one of the stats by one each time.

Thanks,
Ben

>
> diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
> index 93f0ba2..1d271ca 100644
> --- a/drivers/net/macvlan.c
> +++ b/drivers/net/macvlan.c
> @@ -444,6 +444,7 @@ static struct rtnl_link_stats64 *macvlan_dev_get_stats64(struct net_device *dev,
>   {
>   	struct macvlan_dev *vlan = netdev_priv(dev);
>
> +	atomic_long_inc(&dev->rx_dropped);
>   	if (vlan->pcpu_stats) {
>   		struct macvlan_pcpu_stats *p;
>   		u64 rx_packets, rx_bytes, rx_multicast, tx_packets, tx_bytes;
>


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com