Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Jeff Garzik]

Jeff Sipek wrote:
> +	spinlock_t	rx_packets;
> +	spinlock_t	tx_packets;
> +	spinlock_t	rx_bytes;
> +	spinlock_t	tx_bytes;
> +	spinlock_t	rx_errors;
> +	spinlock_t	tx_errors;
> +	spinlock_t	rx_dropped;
> +	spinlock_t	tx_dropped;
> +	spinlock_t	multicast;
> +	spinlock_t	collisions;
> +	spinlock_t	rx_length_errors;
> +	spinlock_t	rx_over_errors;
> +	spinlock_t	rx_crc_errors;
> +	spinlock_t	rx_frame_errors;
> +	spinlock_t	rx_fifo_errors;
> +	spinlock_t	rx_missed_errors;
> +	spinlock_t	tx_aborted_errors;
> +	spinlock_t	tx_carrier_errors;
> +	spinlock_t	tx_fifo_errors;
> +	spinlock_t	tx_heartbeat_errors;
> +	spinlock_t	tx_window_errors;
> +	spinlock_t	rx_compressed;
> +	spinlock_t	tx_compressed;

That's a fat daddy list of locks you got there.


> +	NETSTAT_TYPE	_rx_packets;		/* total packets received	*/
> +	NETSTAT_TYPE	_tx_packets;		/* total packets transmitted	*/
> +	NETSTAT_TYPE	_rx_bytes;		/* total bytes received 	*/
> +	NETSTAT_TYPE	_tx_bytes;		/* total bytes transmitted	*/
> +	NETSTAT_TYPE	_rx_errors;		/* bad packets received		*/
> +	NETSTAT_TYPE	_tx_errors;		/* packet transmit problems	*/
> +	NETSTAT_TYPE	_rx_dropped;		/* no space in linux buffers	*/
> +	NETSTAT_TYPE	_tx_dropped;		/* no space available in linux	*/
> +	NETSTAT_TYPE	_multicast;		/* multicast packets received	*/
> +	NETSTAT_TYPE	_collisions;

Increasing user-visible sizes arbitrarily breaks stuff.  Having 
config-dependent types like this increases complexity.

Short term, just sample the stats more rapidly.

Long term, I suppose with 10GbE we should start thinking about this. 
Personally, I would prefer to make the standard net device stats 
available in the format already exported by ETHTOOL_GSTATS -- which I 
note uses u64's for its counters, and it's easily extensible.  I 
received a request for this just today, even.

	Jeff


P.S.  Please cc netdev@oss.sgi.com for networking discussions.

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Jeff Sipek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 03 July 2003 23:08, Linus Torvalds wrote:
> Please do this in user space. The "overflow every 2^32 packets" thing is
> _not_ a problem, if you just gather the statistics at any kind of
> reasonable interval.

The packet counters are fine (for now, that is), but the tx_bytes and rx_bytes 
counters need those 64-bits. 4GB (= 2^32 bytes) is not enough. For example:

- - gigabit ethernet will cause 32-bit counters to overflow about every 34 
seconds (at full speed.)
- - 10Gb/s ethernet will only take about 3.4 seconds
- - a user like me, who has 5Mbit/s connection to the net can cause the counter 
to overflow in 1 hour 54 minutes

(Most of the time, the devices are not maxed out, but we have to check the 
worst case scenario.) Now, how often should the user space 
statistics-gathering program should run? Well, at least every 30 seconds, for 
now that should be good, but the rein of 10Gb/s is approaching...

> I'd hate to penalise performance for something like this. We have
> generally avoided locking _entirely_ for statistics, exactly because
> people felt that there are major performance issues wrt network packet
> handling, and that "perfect statistics" aren't important enough to
> penalize performance over.

I agree with you, that is why I made it optional so the user may choose to 
sacrifice performace for statistics when needed.

Additionally, I am sure there is a way of optimizing the patch I wrote (i.e. 
actual transmition is locked with a lock from struct net_device.) I am aware 
that this patch is a major undertaking, but it is only a matter of time 
before someone will have to do it anyway.

> Remember: "perfect is the enemy of good".

Very true.


Jeff.

- -- 
Only two things are infinite, the universe and human stupidity, and I'm 
not sure about the former.
		- Albert Einstein
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/BRBjwFP0+seVj/4RAnDSAJ90uOIpgtk0O7YLSsdj97kNbhr/jgCgrmlS
GYbA4luLnY7bli1jYVuZD3s=
=7zXz
-----END PGP SIGNATURE-----

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Jeff Sipek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 03 July 2003 22:46, Jeff Garzik wrote:
> Jeff Sipek wrote:
> > +	spinlock_t	rx_packets;
<snip>
> > +	spinlock_t	tx_compressed;
>
> That's a fat daddy list of locks you got there.

Yeah, I know, I am sure there is a way of getting rid of some of those.. (i.e.
the tx functions are inside a spinlock from struct net_device.)

> > +	NETSTAT_TYPE	_rx_packets;		/* total packets received	*/
> > +	NETSTAT_TYPE	_tx_packets;		/* total packets transmitted	*/
> > +	NETSTAT_TYPE	_rx_bytes;		/* total bytes received 	*/
> > +	NETSTAT_TYPE	_tx_bytes;		/* total bytes transmitted	*/
> > +	NETSTAT_TYPE	_rx_errors;		/* bad packets received		*/
> > +	NETSTAT_TYPE	_tx_errors;		/* packet transmit problems	*/
> > +	NETSTAT_TYPE	_rx_dropped;		/* no space in linux buffers	*/
> > +	NETSTAT_TYPE	_tx_dropped;		/* no space available in linux	*/
> > +	NETSTAT_TYPE	_multicast;		/* multicast packets received	*/
> > +	NETSTAT_TYPE	_collisions;
>
> Increasing user-visible sizes arbitrarily breaks stuff.  Having
> config-dependent types like this increases complexity.

Not really, those macros used to change the variables hide everything from the
driver programmer. Besides those changes in procfs and sysfs which always
return 64-bits, everything else is type casted (if needed) by those macros -
depending on CONFIG_NETSTATS64.

> Short term, just sample the stats more rapidly.

That's what Linus said. But it is only a temporary fix.

> Long term, I suppose with 10GbE we should start thinking about this.
> Personally, I would prefer to make the standard net device stats
> available in the format already exported by ETHTOOL_GSTATS -- which I
> note uses u64's for its counters, and it's easily extensible.  I
> received a request for this just today, even.

I was thinking about making the 64-bit stats mandatory, but then I opted to
make in an option in the config. (As Linus pointed out, some people want
performance, not statistics.)

Jeff.

- --
I'm somewhere between geek and normal.
		- Linus Torvalds
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/BRhfwFP0+seVj/4RAtBbAJ4nmbs8ZQLgFagfb4KrJGZ55AYTmwCgzkcs
1uPma124BorLUdrcsbF2Txs=
=EIag
-----END PGP SIGNATURE-----

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Jeff Sipek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Friday 04 July 2003 05:47, Jan-Benedict Glaw wrote:
<snip>
> Well... I don't really like to break userspace, but why don't we simply
> make packet/traffic counters long long / u_int64_t? This way, we'd
> simply keep almost all drivers untouched and only need to fiddle with
> some sprints()/printk() statements?

I'm no hardware expert, however, that approach contains potential race 
condition - not a system critical one, but something we should be concerned 
about. If one cpu tries to read a u_int64_t variable while another tries to 
update it, the worst case scenario is that the reader will get the high 
32-bits before the write, and low 32-bit after the write, now if the counter 
overflow, the number would be off by 4GB! (This only applies to 32-bit 
architectures.) True, there are cache coherency algorithms, etc...

> Really, how many programs use the current statistics? I'd prefer to
> modify them over adding strange patches like this one to the kernel...

I believe that on any kind of router some at some point in time would like to 
know the data transfered.

Jeff.

- -- 
Keyboard not found!
Press F1 to enter Setup
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/BcADwFP0+seVj/4RAq2TAKDS0oAnj0/PrCuPoxdQF0euBiy6LACeMHqk
gWJhwub4y0VtQmC/hcevJB4=
=RCSe
-----END PGP SIGNATURE-----

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Jeff Sipek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 03 July 2003 23:08, you wrote:
> Please do this in user space. The "overflow every 2^32 packets" thing is
> _not_ a problem, if you just gather the statistics at any kind of
> reasonable interval.
<snip>

While discussing this patch on IRC, an interesting idea came up: why not make 
the counters count something different from bytes? "Less granular stats are 
every bit (bad pun intended) as useful." This would break userspace, but 
that's what everyone has to expect during odd releases (i.e. the modules in 
2.5.)

To avoid having to change all the code, we could have (in addition to what we 
currently have) something like tx_kbytes or tx_mbytes which would be updated 
via a timer every x milliseconds (I'd say maybe 350-500). The sysfs and 
procfs interfaces would have to be modified, however those are just couple of 
lines of code.

Using KB would give us additional 10 bits (making the overflow at 4 TB.) I 
don't really like the idea of using MB, but the underlying idea is the same - 
20 more bits, making the limit 4 PB.

What is the consensus on this way of solving the problem?

Jeff.

- -- 
Failure is not an option,
It comes bundled with your Microsoft product.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD4DBQE/Bx23wFP0+seVj/4RApsVAJUaKZG6px09U87j6tCakrQQebj6AKC52f55
xSuyYxe62N8kefAoxposfg==
=As/F
-----END PGP SIGNATURE-----

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Jeff Sipek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 05 July 2003 15:58, Bernd Eckenfels wrote:
> a reader like ifconfig can easyly work around this with multiple tries, but
> incremeting those variables wont work that easy, and therefore needs a
> lock, which will be a major pita.
>
> 64bit counters should be a result of lockless per-cpu network counters
> (32bit) with some kind of async merging.

This is going to make the structure huge - not only you have the 32-bit 
variables for every CPU, but you have one global set of 64-bit variables 
(possibly you will need a lock for the 64-bit vars.)

Also another thing to consider is portability across architectures - we don't 
need all this code on 64-bit arches.

On the other hand, per-cpu stats may possibly make up for the extra code - no 
cache bouncing, etc.

> Or we wait till 64bit hardware is more common :)

Hehe, the thing is, that when 64bits beecome more common you will have this 
huge number of unused x86 computers that people will:

- - throw out
- - donate
- - convert to all sorts of "embedded" systems which need stable OS (read: 
Linux) (these include routers)

So, x86 is here to stay for some time.

Jeff.

- -- 
The Moon is Waxing Crescent (36% of Full)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/BzcbwFP0+seVj/4RAuMHAJ9sN0E4OgsPeM09D6hbgM3boECLDwCbBDTP
6u8SSobW0+Y0oWq3H4koHd0=
=Z89A
-----END PGP SIGNATURE-----

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Jeff Garzik]

The net stats are already unsigned long internally.

64-bit case is handled quite nicely today, thanks :)

I'm such a 64-bit bigot that "buy a 64-bit computer" is a solution I 
commonly suggest, and it seems to fit well here, too.

	Jeff, wondering if Intel will bother to compete w/ Athlon64

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Jeff Sipek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 05 July 2003 16:40, Jeff Garzik wrote:
> The net stats are already unsigned long internally.
>
> 64-bit case is handled quite nicely today, thanks :)

I wonder why jiffies are not unsigned long...the 64-bit case would have been 
handled quite nicely too... :-)

> I'm such a 64-bit bigot that "buy a 64-bit computer" is a solution I
> commonly suggest, and it seems to fit well here, too.

I wish I had the resources to get one... :-(

The thing is that x86 is here to stay for quite some time. Even if 64-bit 
processors take over the market, you will have so many "old" computers that 
can:

- - be thrown out
- - donated to some institution
- - converted to routers, and other "embedded" systems

Plus, they will be dirt cheap.

> 	Jeff, wondering if Intel will bother to compete w/ Athlon64

Intel screwed up with Itanium, if it had "legacy" support, they would have 
been better off...

Jeff.

- -- 
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/BzwdwFP0+seVj/4RAoUZAJ0dptf9cB60dDUQHU61zg6lO5CXSQCgqHWU
8VvyepQwEdTsNFYyozGedAY=
=OsNH
-----END PGP SIGNATURE-----

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Ben Greear]

Jeff Garzik wrote:
> The net stats are already unsigned long internally.
> 
> 64-bit case is handled quite nicely today, thanks :)
> 
> I'm such a 64-bit bigot that "buy a 64-bit computer" is a solution I 
> commonly suggest, and it seems to fit well here, too.
> 
>     Jeff, wondering if Intel will bother to compete w/ Athlon64

Untill the net-stats are 64-bit on 32-bit systems, we will need some
way to know if they have wrapped or not when reading from nettool
and getting 64-bit numbers.

I guess what I really mean to say is that, if nettool is returning 64-bit
values, we need to know which ones are obtained from 32-bit counters.
32 -> 64 bit mapping will require wrap handling on low 32-bits, but
64 -> 64 bit mapping will require wrapping about 4-billion times less often :)

Perhaps a precision field is also needed for backwards/forwards compatability,
and perhaps a nettool version field as well to also help with backwards/forwards
compat.

Ben

> 
> 
> 


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Ben Greear]

Jeff Sipek wrote:

> Using KB would give us additional 10 bits (making the overflow at 4 TB.) I 
> don't really like the idea of using MB, but the underlying idea is the same - 
> 20 more bits, making the limit 4 PB.
> 
> What is the consensus on this way of solving the problem?

I guess it could be useful for something like ifconfig, but serious
applications will need more precision and should deal with wraps anyway
(even on 64-bits, in my opinion..why have to fix bugs in 10 years because
we were too lazy to take the 10 minutes to make it right now).

Ben


-- 
Ben Greear <greearb@candelatech.com>       <Ben_Greear AT excite.com>
President of Candela Technologies Inc      http://www.candelatech.com
ScryMUD:  http://scry.wanfear.com     http://scry.wanfear.com/~greear

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Francois Romieu]

Jeff Sipek <jeffpc@optonline.net> :
[network counter overflow on 32 bits systems]
> The thing is that x86 is here to stay for quite some time. Even if 64-bit 
> processors take over the market, you will have so many "old" computers that 
> can:
> 
> - - be thrown out
> - - donated to some institution
> - - converted to routers, and other "embedded" systems
> 
> Plus, they will be dirt cheap.

- the PCI bus don't/won't/can't handle multiple 10 Gb/s adapters;
- nobody sane would recycle x86 systems as core routers after having bought
  a few Gb/s link.

--
Ueimor

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Jeff Sipek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 05 July 2003 17:51, Francois Romieu wrote:
> Jeff Sipek <jeffpc@optonline.net> :
> > The thing is that x86 is here to stay for quite some time. Even if 64-bit
> > processors take over the market, you will have so many "old" computers
> > that can:
> >
> > - - be thrown out
> > - - donated to some institution
> > - - converted to routers, and other "embedded" systems
> >
> > Plus, they will be dirt cheap.
>
> - the PCI bus don't/won't/can't handle multiple 10 Gb/s adapters;

Ok, so let's stay in the range of gigabit ethernet...

> - nobody sane would recycle x86 systems as core routers after having bought
>   a few Gb/s link.

When you have "a few Gb/s links" you would not use your beloved Pentium 100 
MHz to do the job, instead you would go for something like 1.5 GHz P4 or 
Athlon, both of which would be cheaper than the new 64-bit architecture.

Jeff.

P.S. I just looked up the cheapest gigabit copper I could find in 10 seconds, 
and I found: D-Link DGE-500T for $36.27 this is just 4 times the price of the 
cheapest fast ethernet I found on the same site (cdw.com - they are not the 
cheapest, but I like them)

- -- 
A computer without Microsoft is like chocolate cake without mustard.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/B1OxwFP0+seVj/4RAkWfAJ9lYLk9zwpR2LpVLgVIDLovQewZKwCeLivr
bRCwwzVIj29rmxiT5tpmkaM=
=HXK9
-----END PGP SIGNATURE-----

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Roland Dreier]

    Francois> - the PCI bus don't/won't/can't handle multiple 10 Gb/s adapters

In a year, there will be 32-bit x86 systems with multiple 8X PCI
Express slots (16 Gb/sec full duplex to each slot).  In five years,
those systems will sell for $10 on Ebay.

 - Roland

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Francois Romieu]

Jeff Sipek <jeffpc@optonline.net> :
[...]
> P.S. I just looked up the cheapest gigabit copper I could find in 10 seconds, 
> and I found: D-Link DGE-500T for $36.27 this is just 4 times the price of the 
> cheapest fast ethernet I found on the same site (cdw.com - they are not the 
> cheapest, but I like them)

Please google around on the topic "nanog/gigabit/routing/linux" and read
netdev archive again.

It isn't _that_ simple.

--
Ueimor

Re: [PATCH - RFC] [1/5] 64-bit network statistics - generic net

[Alan Cox]

On Sad, 2003-07-05 at 22:41, Ben Greear wrote:
> Untill the net-stats are 64-bit on 32-bit systems, we will need some
> way to know if they have wrapped or not when reading from nettool
> and getting 64-bit numbers.

iptables

Collecting the data on a need to know basis 8)