Re: [NET] 64 bit byte counter for 2.6.3

Re: [NET] 64 bit byte counter for 2.6.3

[Stephen Hemminger]

On Wed, 18 Feb 2004 18:51:19 +0200
Markus Hästbacka <midian@ihme.org> wrote:

> Ok, Here's a patch for 64 bit byte counters for 2.6.3. For any intrested
> users to try.
> 
> That means in english that the limit for RX bytes and TX bytes (in
> ifconfig for example) is much higher than the old 4GB limit on 32 bit
> systems.
> 
> Orginal patch by Miika Pekkarinen, ported forward from 2.5 by me.
> 
> Patch says 2.6.3-rc1, but patches cleanly on 2.6.3.
> 
>         Markus

Some quick comments:
 * Network changes gets discussed on netdev@oss.sgi.com 
 * 64 bit values are not atomic on 32 bit architectures 
 * wider values in /proc output risks breaking apps like ifconfig and netstat

Re: [NET] 64 bit byte counter for 2.6.3

[Markus Hästbacka]

On Wed, 2004-02-18 at 22:32, Richard B. Johnson wrote:
> Manipulation of a 'long long' is not atomic in 32 bit architectures.
> Please explain how we don't care, if we shouldn't care. Also some
> /proc entries might get read incorrectly with existing tools.
Please, tell me all the tools, I'll test them. ifconfig and netstat
works correctly atleast.

And about the caring, is rx/tx bytes so important that they can't use
long long? I would care to see more than 4GB, and maybe some error in
the counter at some point (Have you _ever_ seen that happen?) than only
4GB.

And no, I didn't post this to be merged into the mainline kernel, just
to let users know that there maybe is an option for seeing maximum 4GB.

This has been working for me since, umm.. let's say 2.4.20. All the
tools I've needed have worked.

        Markus

Re: [NET] 64 bit byte counter for 2.6.3

[Richard B. Johnson]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=X-UNKNOWN, Size: 2755 bytes --]

On Wed, 18 Feb 2004, Markus [ISO-8859-1] Hästbacka wrote:

> On Wed, 2004-02-18 at 22:32, Richard B. Johnson wrote:
> > Manipulation of a 'long long' is not atomic in 32 bit architectures.
> > Please explain how we don't care, if we shouldn't care. Also some
> > /proc entries might get read incorrectly with existing tools.
> Please, tell me all the tools, I'll test them. ifconfig and netstat
> works correctly atleast.
>

Hum, they will work when they see 2^64 written in ASCII in /proc/net/dev ?
I doubt that. Have you ever even seen 64-bit ASCII?

The largest unsigned long long will be 18446744073709551615.
If it's read with 32-bit tools (sscanf), it will read 4294967295.

> And about the caring, is rx/tx bytes so important that they can't use
> long long? I would care to see more than 4GB, and maybe some error in
> the counter at some point (Have you _ever_ seen that happen?) than only
> 4GB.
>

The 32-bit wrap is a serious problem. It can't be ignored. The writing
of any multiple-part variable in the kernel can be interrupted at
any time. Interrupting half-written variables can have dire
consequences. Let's pretend that gcc knows how to write a long-long
with minimum overhead..... high word is in edx and the low word is
in eax. The memory variable is addressed by ebx....

		addl	%eax, (%ebx)		# Sum low word
		adcl	%edx, 0x04(%ebx)	# Sum CY and high

Now, get interrupted...

		addl	%eax, (%ebx)		# Sum low word
		->>> interrupt <<<--

                Do some code that adds more stuff to
		the variable....

		Return to the interrupted code.

		adcl	%edx, 0x04(%ebx)	# Sum CY and high

The memory variable is now wrong. If it's an event counter, maybe
it doesn't make any difference. That's what needs to be addressed.
You can't just dismiss it. Any time you write code that knowingly
results in the wrong results, its impact needs to be fully understood.

Changing a bunch of variables in a 32-bit machine to 64-bit ones
is a major change, regardless of how trivial it may seem.

You need to make a spin-locked thingy for every variable you
want to manipulate if the result needs to be correct.

> And no, I didn't post this to be merged into the mainline kernel, just
> to let users know that there maybe is an option for seeing maximum 4GB.
>
> This has been working for me since, umm.. let's say 2.4.20. All the
> tools I've needed have worked.

Just wait until your packet count is 4294967296. It will likely read 0.
When 4294967297, may read 1.  That's off by quite a bit.

>
>         Markus
>

Cheers,
Dick Johnson
Penguin : Linux version 2.4.24 on an Intel Pentium III machine (797.90 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: [NET] 64 bit byte counter for 2.6.3

[Stephen Hemminger]

On Wed, 18 Feb 2004 17:45:16 -0500 (EST)
"Richard B. Johnson" <root@chaos.analogic.com> wrote:

> On Wed, 18 Feb 2004, Markus [ISO-8859-1] Hästbacka wrote:
> 
> > On Wed, 2004-02-18 at 22:32, Richard B. Johnson wrote:
> > > Manipulation of a 'long long' is not atomic in 32 bit architectures.
> > > Please explain how we don't care, if we shouldn't care. Also some
> > > /proc entries might get read incorrectly with existing tools.
> > Please, tell me all the tools, I'll test them. ifconfig and netstat
> > works correctly atleast.
> >
> 
> Hum, they will work when they see 2^64 written in ASCII in /proc/net/dev ?
> I doubt that. Have you ever even seen 64-bit ASCII?
> 
> The largest unsigned long long will be 18446744073709551615.
> If it's read with 32-bit tools (sscanf), it will read 4294967295.
> 
> > And about the caring, is rx/tx bytes so important that they can't use
> > long long? I would care to see more than 4GB, and maybe some error in
> > the counter at some point (Have you _ever_ seen that happen?) than only
> > 4GB.
> >
> 
> The 32-bit wrap is a serious problem. It can't be ignored. The writing
> of any multiple-part variable in the kernel can be interrupted at
> any time. Interrupting half-written variables can have dire
> consequences. Let's pretend that gcc knows how to write a long-long
> with minimum overhead..... high word is in edx and the low word is
> in eax. The memory variable is addressed by ebx....
> 
> 		addl	%eax, (%ebx)		# Sum low word
> 		adcl	%edx, 0x04(%ebx)	# Sum CY and high
> 
> Now, get interrupted...
> 
> 		addl	%eax, (%ebx)		# Sum low word
> 		->>> interrupt <<<--
> 
>                 Do some code that adds more stuff to
> 		the variable....
> 
> 		Return to the interrupted code.
> 
> 		adcl	%edx, 0x04(%ebx)	# Sum CY and high
> 
> The memory variable is now wrong. If it's an event counter, maybe
> it doesn't make any difference. That's what needs to be addressed.
> You can't just dismiss it. Any time you write code that knowingly
> results in the wrong results, its impact needs to be fully understood.
> 
> Changing a bunch of variables in a 32-bit machine to 64-bit ones
> is a major change, regardless of how trivial it may seem.
> 
> You need to make a spin-locked thingy for every variable you
> want to manipulate if the result needs to be correct.
> 
> > And no, I didn't post this to be merged into the mainline kernel, just
> > to let users know that there maybe is an option for seeing maximum 4GB.
> >
> > This has been working for me since, umm.. let's say 2.4.20. All the
> > tools I've needed have worked.
> 
> Just wait until your packet count is 4294967296. It will likely read 0.
> When 4294967297, may read 1.  That's off by quite a bit.
> 
> >
> >         Markus
> >
> 
> Cheers,
> Dick Johnson
> Penguin : Linux version 2.4.24 on an Intel Pentium III machine (797.90 BogoMips).
>             Note 96.31% of all statistics are fiction.
> 

Do it right:
	* use per-cpu long long for both bytes and packet counts
	  and change each driver ...
	* expose both a new 64 bit and legacy 32 bit /proc interface
	* no tools use /sys yet, so that needs to show long long
	* have both a get_stats and get_stats64 hook in netdevice so not all drivers
	  have to be converted at once.

Re: [NET] 64 bit byte counter for 2.6.3

[Jeff Garzik]

Stephen Hemminger wrote:
> Do it right:
> 	* use per-cpu long long for both bytes and packet counts
> 	  and change each driver ...
> 	* expose both a new 64 bit and legacy 32 bit /proc interface
> 	* no tools use /sys yet, so that needs to show long long
> 	* have both a get_stats and get_stats64 hook in netdevice so not all drivers
> 	  have to be converted at once.


Yep.

	Jeff

Re: [NET] 64 bit byte counter for 2.6.3

[Meelis Roos]

SH>  * Network changes gets discussed on netdev@oss.sgi.com 
SH>  * 64 bit values are not atomic on 32 bit architectures 

Agree.

SH>  * wider values in /proc output risks breaking apps like ifconfig and netstat

This is probably not a problem here, ifconfig etc have been working fine
with non-wrapping numbers on sparc64 and other 64-bit machines.

-- 
Meelis Roos

Re: [NET] 64 bit byte counter for 2.6.3

[Stephen Hemminger]

On Thu, 19 Feb 2004 10:14:55 +0200
Meelis Roos <mroos@linux.ee> wrote:

> SH>  * Network changes gets discussed on netdev@oss.sgi.com 
> SH>  * 64 bit values are not atomic on 32 bit architectures 
> 
> Agree.
> 
> SH>  * wider values in /proc output risks breaking apps like ifconfig and netstat
> 
> This is probably not a problem here, ifconfig etc have been working fine
> with non-wrapping numbers on sparc64 and other 64-bit machines.

Looking at current net-tools source, they are doing sscanf into a long-long-unsigned
value already.