fdisk units size

fdisk units size

[Matthew Eaton]

Hello, I have a quick question regarding recent fdisk.

I noticed fdisk -l now uses base 2 (GiB) instead of base 10 (GB) to
calculate device size.  This is good, but sometimes it is nice to see
the base 10 calculation, can this be toggled with a switch?  I can't
find a recent man page for fdisk.

Re: fdisk units size

[Karel Zak]

On Tue, Nov 25, 2014 at 04:19:25PM -0800, Matthew Eaton wrote:
> Hello, I have a quick question regarding recent fdisk.
> 
> I noticed fdisk -l now uses base 2 (GiB) instead of base 10 (GB) to
> calculate device size.  This is good, but sometimes it is nice to see

 when sometimes?

> the base 10 calculation, can this be toggled with a switch?  I can't
> find a recent man page for fdisk.

 Frankly, use 10 based calculation in IT is ugly thing, we are not 
 hard disk device marketing guys...

    Karel


-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: fdisk units size

[Martin Steigerwald]

Am Donnerstag, 4. Dezember 2014, 14:00:44 schrieb Karel Zak:
> On Tue, Nov 25, 2014 at 04:19:25PM -0800, Matthew Eaton wrote:
> > Hello, I have a quick question regarding recent fdisk.
> > 
> > I noticed fdisk -l now uses base 2 (GiB) instead of base 10 (GB) to
> > calculate device size.  This is good, but sometimes it is nice to see
> 
>  when sometimes?
> 
> > the base 10 calculation, can this be toggled with a switch?  I can't
> > find a recent man page for fdisk.
> 
>  Frankly, use 10 based calculation in IT is ugly thing, we are not
>  hard disk device marketing guys...

Say you want to divide a 500 GB disk in five equally sized partitions. Thats 
easier to do with base 10 calculation. 5 x 100 GB = approx. 500 GB. 5 x 100 
GiB is more.

But anyway, I use LVM these days for most stuff.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: fdisk units size

[Matthew Eaton]

>  Frankly, use 10 based calculation in IT is ugly thing, we are not
>  hard disk device marketing guys...

I work for a SSD manufacturer! :)

Actually, I agree with you.  But sometimes it's easier for me to see
240 GB in fdisk output and know that I have a 240 GB drive plugged in,
rather than see 223.5 GiB.

I like how with parted you can print whichever units you want, but
sadly doesn't work for my use case, since it just prints an error if
there is no disk label present. :(

Re: fdisk units size & disk manufacturers buying the standard

[Linda Walsh]

Matthew Eaton wrote:
>>  Frankly, use 10 based calculation in IT is ugly thing, we are not
>>  hard disk device marketing guys...
>>     
>
> I work for a SSD manufacturer! :)
>
> Actually, I agree with you.  But sometimes it's easier for me to see
> 240 GB in fdisk output and know that I have a 240 GB drive plugged in,
> rather than see 223.5 GiB.
>   
----
  The corporations told consumers how it would be so they could better sell
disks -- they bought the standard committee, and it's been screwed since.

They may a fundamental error that any engineer or mathematician could 
point out:

'B' == a prefix meaning 2^3bits.  I.e 'B' is a base-2 measurement.  In 
science
and engineering, you just don't mix units like that unless want to prove 
you don't
know what you are talking about.

NOTE:   saying 24000Gb is fine.  A bit is unit. 

But a Byte, (today), is defined as 2^3 bits.   (Unless someone want's to 
argue
that disk manufacturers really mean to use 10-12 bit bytes... *cough*).

So it becomes obvious -- in telecom, speeds are usually quoted in 
bits/time,
so decimal units make sense.  In most *physical sciences* decimal makes
sense.

Computers don't count in decimal but use binary (Show me 1 memory or
computer-cache description that tells me 16K = 16,000.

Also, disk manufacturers are lying.  Disk space is allocated in 2^9 (512 
Bytes or 2^12 bits)
or 2^12 Bytes (2^15 bits).   They ***CANNOT*** accurate quote disk space 
using
base 10.  I.e. 1MB = 2048 sectors.  But 1MBd = 1954.125 sectors -- and 
you cannot
use 1/8th of a sector.  So ANY figure they give will be a lie as 10 
doesn't divide into
a power of 2 which is how computer space is allocated and used.

Nice the way corporations can buy definitions... just like about 
anything else... ;-(

Another argument.  SI prefixes are applied to physical units (meter, 
gram, liter...etc).
A bit isn't a physical unit, so their argument that physical prefixes 
should apply to
virtual base-2 quantities becomes even more nonsensical.

But hey, what are science math and engineering to the power of ignorant 
consumer
powered corporations?

But SI overstepped their bounds, unless they want to define the 'bit' 
and the 'Byte' as
metric units and keep a representation of them in some clean room in 
Paris (or
the modern equivalent).

Re: fdisk units size & disk manufacturers buying the standard

[Felix Miata]

Linda Walsh composed on 2014-12-08 13:35 (UTC-0800):

> ...A bit isn't a physical unit,...

On a HD it has to be physical, a magnetically manipulated location on
physical media, which happens to be used in groups of 8 termed a byte,
without which the device couldn't do what it was designed to do. HD makers
logically count these groups of 8 using the same numbers most mortals use for
counting, decimals.

> ...so their argument that physical prefixes should apply to
> virtual base-2 quantities becomes even more nonsensical.

> But SI overstepped their bounds, unless they want to define the 'bit' and the 'Byte' as
> metric units and keep a representation of them in some clean room in Paris (or
> the modern equivalent).

Whether SI got the Bs & bs right I won't get into, but they did get logically
correct exposing the hijacking of the centuries old decimal concepts
represented by K, M, G, T, et al, and interjecting "i" to delineate a
considerably less ambiguous powers of two counting system.
-- 
"The wise are known for their understanding, and pleasant
words are persuasive." Proverbs 16:21 (New Living Translation)

 Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!

Felix Miata  ***  http://fm.no-ip.com/

Re: fdisk units size & disk manufacturers buying the standard

[Dale R. Worley]

Felix Miata <mrmazda@earthlink.net> writes:
> Whether SI got the Bs & bs right I won't get into, but they did get logically
> correct exposing the hijacking of the centuries old decimal concepts
> represented by K, M, G, T, et al, and interjecting "i" to delineate a
> considerably less ambiguous powers of two counting system.

Indeed, and the most important thing is to make sure that all the
utilities use GB vs. GiB correctly.  Clear communication neutralizes
many other mistakes.

Dale

Re: fdisk units size & disk manufacturers buying the standard

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12/8/2014 4:35 PM, Linda Walsh wrote:
> They may a fundamental error that any engineer or mathematician 
> could point out:
> 
> 'B' == a prefix meaning 2^3bits.  I.e 'B' is a base-2 measurement. 
> In science and engineering, you just don't mix units like that 
> unless want to prove you don't know what you are talking about.

While I do despise the HD industry for lieing about drive sizes, I
have to point out your error here.  How the unit the prefix is applied
to relates to some other unit has no bearing whatsoever on the prefix
itself.  Your argument is like saying that a kilowatt-hour should be
only 60 watt-hours instead of 1,000 because a watt-hour is 3600
joules, which is base 60 instead of base 10.

> Also, disk manufacturers are lying.  Disk space is allocated in
> 2^9 (512 Bytes or 2^12 bits) or 2^12 Bytes (2^15 bits).   They 
> ***CANNOT*** accurate quote disk space using base 10.  I.e. 1MB = 
> 2048 sectors.  But 1MBd = 1954.125 sectors -- and you cannot use 
> 1/8th of a sector.  So ANY figure they give will be a lie as 10 
> doesn't divide into a power of 2 which is how computer space is 
> allocated and used.

Rounding is not the same thing as lieing.  If they were claiming that
a disk that has 1,000,000,000 sectors was 477 GB, you couldn't really
fault them for the missing ~166 MB.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQEcBAEBAgAGBQJUqw2PAAoJENRVrw2cjl5RuaMH/A6eJT6Ww3NJR80A3s2jnieO
2W1ny8pR2J8cBRF5TfDc/LXJ3N/u9rwaVkag4DGA5LACkI6NMO9eESrbQgbCl+vs
oplEm/Ph4rvinXn78k9m0sEFS8NHtB8C023xKeqCc5exQth/lo2AA2Wkl4WylyJT
ZUtM4RlmyjEv3TuWqfyeXxHTW8HSF6Bsy5CiN2hr8Ly1VIe1uSSz+xwWz2+vjtZy
NrTod2wwwpkdXvjatzNDGkA/EDF4md/IPSuH9QNZ1TAESa/0dKa21s172Tl/y/jd
ghrqoXIQPXGXzjurWF64UG6ANX19GeG28sG4fDojbtHNAq1deIgQdwHJ5KtCYbE=
=NF/s
-----END PGP SIGNATURE-----

Re: fdisk units size & disk manufacturers buying the standard

[Linda Walsh]

Phillip Susi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 12/8/2014 4:35 PM, Linda Walsh wrote:
>> They may a fundamental error that any engineer or mathematician 
>> could point out:
>>
>> 'B' == a prefix meaning 2^3bits.  I.e 'B' is a base-2 measurement. 
>> In science and engineering, you just don't mix units like that 
>> unless want to prove you don't know what you are talking about.
>
> While I do despise the HD industry for lieing about drive sizes, I
> have to point out your error here.  How the unit the prefix is applied
> to relates to some other unit has no bearing whatsoever on the prefix
> itself.
-----
The basic SI units measure physical, real-world amounts.  Before
they were redefined in terms of atomic weights and constants, they
were based on physical objects.

Metric prefixes were designed to make human calculation easy.  Computer
software and hardware doesn't count nor is measured in powers or 10.


The metric prefixes were not designed nor intended to be used with
non-base-10 units.  If the application of prefixes had no
"proper" set of units, use of milli-miles asking for conversions
to milli-feet or microns wouldn't make your head hurt.

Converting a 'byte' to some number of bits, AT BEST, sticks out as
a non base-10 unit.  Arguably, 'Bytes' shouldn't be part of the metric
system as they don't have a fixed size based in the real world.
Even today, you can't convert bits to Bytes using a constant (ignoring
computer architectures that don't have an 8-bit byte), communications
speed in Bytes varies by protocol.  1Gb-Base-T ethernet maxes out at
a theoretical 125MB/s - divisible by 8.  But 10Gb ethernet maxes out
at 1000MB/s -- with 20% of its bandwidth going to protocol overhead.

It was inaccurate for me to call 'B' a prefix -- as it doesn't prefix
anything.  More accurately, it is variable, context-relative,
derived unit.  And is completely out of place with base-10 units.

The HD industry blew it by talking about physical memory in Bytes because
again -- what the HD provides is some number of 'bits'.  That isn't
convertible to Bytes using a fixed constant.  I'm not sure about
modern drives, but used to be you could vary the sector size on SCSI
disks and end up with a disk that had a different number of Bytes.
The physical platters still had the same number of bits, but it's
up to software to decide how many bytes are squeezed out of that space.
I.e. -- Bytes are a software-defined-unit that don't exist in the real
world -- they are logical, derived units.


It's not clear how much longer disk manufacturer's will continue
to use their 'revised' computer-units as the memory manufacturers
are slowly replacing platter-based technology.  You can't buy memory
in base-10 units.  RAM comes in sizes of base-2.  You can't buy
a 1000*1000*1000-bit RAM chip (at least not off-the-shelf).  While
flash memory chips are also sold using base-2 prefixes, its not clear
how SSD's will go.  Since they are really solid state memory chips,
it doesn't seem likely they will be measured in terms of bit density
per track.

Basically, using base 10 prefixes to describe something that only
comes in sizes of base-2 is a setup for miscommunication as well
as inherently being *unable* to accurately describe the quantities
used in the computer field.

If one wants to use base-10 prefixes they should stick with bits, but
as soon as one moves to a base-2 (usually) sized unit, one should use
base-2 prefixes.  Use of base-10 prefixes for base-2 computer Software
and Hardware creates the same inherent difficulties as trying to use
base-10 prefixes with inches, feet, and pounds and was pushed by the
HD industry for some of the exact same reasons why it is preferable
for them to stick with english units -- you generally need a
calculator to find out cost/unit.  HD manufacturers successfully
pulled a marketing scam to get those units accepted -- because
from the manufacturing standpoint, platters with 'X' bits/cm^2 can
be manufactured to decimal specs -- it's just that they can't be
*used* that way.  They only way a computer can use a hard disk is if
it if formatted into some binary size.

Anyway, sorry to confuse the issue by calling Bytes a prefix.  My bad.

Re: fdisk units size & disk manufacturers buying the standard

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 01/07/2015 07:21 PM, Linda Walsh wrote:
> The basic SI units measure physical, real-world amounts.  Before 
> they were redefined in terms of atomic weights and constants, they 
> were based on physical objects.

So what?

> Metric prefixes were designed to make human calculation easy.
> Computer software and hardware doesn't count nor is measured in
> powers or 10.

Sure, which is why I agree with the age old practice of using 1024
instead of 1000, but that has nothing to do with the fact that a byte
is 2^3 bits.

> Converting a 'byte' to some number of bits, AT BEST, sticks out as 
> a non base-10 unit.  Arguably, 'Bytes' shouldn't be part of the
> metric

And like I said, a watt-hour is in the same boat.

> system as they don't have a fixed size based in the real world. 
> Even today, you can't convert bits to Bytes using a constant
> (ignoring computer architectures that don't have an 8-bit byte),
> communications

Of course you can: that constant is x8.

> speed in Bytes varies by protocol.  1Gb-Base-T ethernet maxes out
> at a theoretical 125MB/s - divisible by 8.  But 10Gb ethernet maxes
> out at 1000MB/s -- with 20% of its bandwidth going to protocol
> overhead.

I'm not aware of any additional overhead that 10Gb ethernet has over
1Gb ethernet, which is the overhead of the packet headers, which the
125MB/s figure does not take into account ( and that is base 10 MB,
not base 2 ).

> It was inaccurate for me to call 'B' a prefix -- as it doesn't
> prefix anything.  More accurately, it is variable,
> context-relative, derived unit.  And is completely out of place
> with base-10 units.

You didn't call it a prefix, you called it a unit, and you said that
because it is 8 times another unit ( bits ), it that should inherently
alter the meaning of the prefix applied to it.

> The HD industry blew it by talking about physical memory in Bytes
> because again -- what the HD provides is some number of 'bits'.
> That isn't

No, it provides some number of sectors, which historically are each
512 bytes.

> convertible to Bytes using a fixed constant.  I'm not sure about

Again, bytes are convertible to sectors by multiplying by 8.

> modern drives, but used to be you could vary the sector size on
> SCSI disks and end up with a disk that had a different number of
> Bytes.

Perhaps on a vendor specific basis, but not as part of the scsi
standard.  People used to do the same with floppy disks since the
control logic actually was accessible to the cpu instead of being
integrated into the drive.  This really hasn't been the case since the
advent of IDE ( what?  25 years ago ) though.

> The physical platters still had the same number of bits, but it's 
> up to software to decide how many bytes are squeezed out of that
> space. I.e. -- Bytes are a software-defined-unit that don't exist
> in the real world -- they are logical, derived units.

No, the physical platters don't hold bits at all.  They record an
analog signal that the controller has to decode into bits.  The
ability to do so depends on the signal to noise ratio, which gets
worse the higher you push the buad rate.  Some controllers and media (
and different regions of the media ) could push it higher than others
before the error rate got bad.

> It's not clear how much longer disk manufacturer's will continue to
> use their 'revised' computer-units as the memory manufacturers are
> slowly replacing platter-based technology.  You can't buy memory in
> base-10 units.  RAM comes in sizes of base-2.  You can't buy a
> 1000*1000*1000-bit RAM chip (at least not off-the-shelf).  While 
> flash memory chips are also sold using base-2 prefixes, its not
> clear how SSD's will go.  Since they are really solid state memory
> chips, it doesn't seem likely they will be measured in terms of bit
> density per track.

It isn't going away; SSDs are using base 10 probably in part for the
same reason HDD makers did ( it makes them seem larger ), and in part
because they have to reserve some of the flash for over provisioning.

> Basically, using base 10 prefixes to describe something that only 
> comes in sizes of base-2 is a setup for miscommunication as well as
> inherently being *unable* to accurately describe the quantities 
> used in the computer field.

The reason base 2 is convenient for ram is because that is how it
naturally aligns due to the way it is addressed, and they don't have
physical constraints in the manufacturing process, not because there
are 8 bits in a byte.  Bytes easily could have been defined to be 10
or 12 bits and it would still make sense for ram to be built in power
of 2 units of bytes.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBCgAGBQJUrgAFAAoJENRVrw2cjl5RlXcH/2ZupptQxOjUJGS2dMyy8I8K
TltLZJYGyTWsypcgVXU9DwfwsHVNItk8ir4fYSWaJakq/FhXZjMFo2SwvFEybPwK
3Erdnr6f46lxdmU0EZ3ydwDK8X03p190gZKlqEhQgOUJbYY2IjrCIrmhgAAFD8QU
Ol+plU6hGdq5RLxnTD5hNO4rQB4KatW6TeQsY1JbIfT0X1oFJ+/jefwV1P9jpY8/
B5Zv6TtwECiga/HqaJVQ4jmxIcnHX5H56PNY1mLAporlB70m3q00iNdciAm7HKm4
uqvn4YEmnsK1eTmIN5vWFsVsvR3esOo2MtC3mUSk+6N214spi67dXLioNrKDmRw=
=y0cy
-----END PGP SIGNATURE-----

Re: fdisk units size & disk manufacturers buying the standard

[Peter Cordes]

On Wed, Jan 07, 2015 at 10:56:53PM -0500, Phillip Susi wrote:
> 
> > Basically, using base 10 prefixes to describe something that only 
> > comes in sizes of base-2 is a setup for miscommunication as well as
> > inherently being *unable* to accurately describe the quantities 
> > used in the computer field.
> 
> The reason base 2 is convenient for ram is because that is how it
> naturally aligns due to the way it is addressed, and they don't have
> physical constraints in the manufacturing process, not because there
> are 8 bits in a byte.  Bytes easily could have been defined to be 10
> or 12 bits and it would still make sense for ram to be built in power
> of 2 units of bytes.

 ECC RAM does in fact store 9 bits per byte, so there's an extra chip
on each stick.  It presents itself as an addressable container storing
64bit words, since the 72bit data + ECC is handled by the controller.
http://en.wikipedia.org/wiki/Synchronous_dynamic_random-access_memory#SDRAM_construction_and_operation
Point being, the natural word size of common (DDRwhatever) SDRAM isn't
8bits, but the convention of using bytes as the smallest logically
(not physically) addressable unit is so widespread that we measure
everything in bytes.

 And hard drives are most logically viewed as an array of sectors, of
size = 512B * 8 b/B.  Playing with the C/H/S geometry, or whatever you
did with scsi devices, just allowed access to more of fewer sectors.
You still use the device as a linear array of sectors, so it doesn't
matter how you go about translating linear sector addresses to
whatever addressing mode the hardware actually uses internally.

 There's huge inertia resisting any change to a convention with
smaller numbers, for marketting reasons.  This is probably also why
wire and wireless transmission protocols measure bits / sec.  (baud is
symbols / sec, BTW, which isn't bits or bytes per sec, in case anyone
needed reminding about that.)

 What are we arguing about here, anyway?  MB vs MiB?  Yeah it's
annoying that HD manufacturers used MB even before MiB was a thing,
and everyone else meant what we now call MiB.  Water under the bridge,
it's just how things are done, and I'd assume we're stuck with it.  It
only bites you when you forget to do the conversion when thinking
about if a 1TB hard drive will hold a directory with 950 GiB of data,
according to du output.  Filesystem overhead eats up space, too,
depending on the FS.

 But it's usually clear what's what.  Printing out storage device
sizes in TB and TiB together seems like a good idea, to make sure
people remember that the conversion is needed, until our wishes come
true and storage hardware is sold with sizes measured with power-of-2
prefixes, maybe by the year 2100.  Or maybe not until an alien
invasion forces the US to drop the ridiculous non-metric system, too.

 These days, everything is logically an array of bits, (almost always
a whole number of bytes), and any funky addressing is hidden behind a
hardware or at least software translation layer.

 PS, are we supposed to reply-all instead of just the list, on
util-linux?

-- 
#define X(x,y) x##y
Peter Cordes ;  e-mail: X(peter@cor , des.ca)

"The gods confound the man who first found out how to distinguish the hours!
 Confound him, too, who in this place set up a sundial, to cut and hack
 my day so wretchedly into small pieces!" -- Plautus, 200 BC

Re: fdisk units size & disk manufacturers buying the standard

[Linda Walsh]

Phillip Susi wrote:
>> Converting a 'byte' to some number of bits, AT BEST, sticks out as 
>> a non base-10 unit.  Arguably, 'Bytes' shouldn't be part of the
>> metric
>
> And like I said, a watt-hour is in the same boat.
Not at all.  An hour is NOT a metric unit and is not, officially,
used with SI prefixes.  Clearly, if it is in the same boat,
you would be claiming that the Byte is not a metric unit -- and you
would be right.

SI-base units include: meter, kilogram, second ampere, kelvin, candela,
mole with 'liter' being an acceptable unit but not formally part of the
system.

No where does it talk about things like hours and Bytes being metric
units (http://en.wikipedia.org/wiki/SI_base_unit).


>> speed in Bytes varies by protocol.  1Gb-Base-T ethernet maxes out
>> at a theoretical 125MB/s - divisible by 8.  But 10Gb ethernet maxes
>> out at 1000MB/s -- with 20% of its bandwidth going to protocol
>> overhead.
>
> I'm not aware of any additional overhead that 10Gb ethernet has over
> 1Gb ethernet, 
----
See kernel messages for a 10b-T ethernet.

[   21.224641] ixgbe 0000:05:00.0: PCI Express bandwidth of 32GT/s available
[   21.224644] ixgbe 0000:05:00.0: (Speed:5.0GT/s, Width: x8, Encoding 
Loss:20%)

I don't recall a 20% encoding loss in 1Gb or 100Mb ethernet and the 
kernel displays
no such messages for the slower speed cards.


> which is the overhead of the packet headers, which the
> 125MB/s figure does not take into account ( and that is base 10 MB,
> not base 2 ).
----
    I said theoretical speed -- which excludes packet headers.  
Theoretical speed
excludes optional protocols.  The 125MB/s is a max theoretical rate and 
is in base 2.
In practice, 125 millionBytes for writes and 119 millionBytes are a 
benchmark
maximum that includes headers (SMB/CIFS for the test I most frequently run).

    I.e. 125millionBytes/s -- the base10 number IS a benchmark maximum 
that includes
protocol headers (specifically for SMB).
>
>> It was inaccurate for me to call 'B' a prefix -- as it doesn't
>> prefix anything.  More accurately, it is variable,
>> context-relative, derived unit.  And is completely out of place
>> with base-10 units.
>
> You didn't call it a prefix, you called it a unit, and you said that
> because it is 8 times another unit ( bits ), it that should inherently
> alter the meaning of the prefix applied to it.
----
    I'm sorry, I thought I wrote this:
-------- Original Message --------
Subject: 	Re: fdisk units size & disk manufacturers buying the standard
Date: 	Mon, 08 Dec 2014 13:35:43 -0800
From: 	Linda Walsh
To: 	util-linux@

'B' == a prefix meaning 2^3bits.
-----------

Glad to know that wasn't me....
>
>> The HD industry blew it by talking about physical memory in Bytes
>> because again -- what the HD provides is some number of 'bits'.
>> That isn't
>
> No, it provides some number of sectors, which historically are each
> 512 bytes.
----
I would assert you are contradicting yourself.  You said the platters 
don't contain any
logical computer storage unit:

    No, the physical platters don't hold bits at all.  They record an
    analog signal that the controller has to decode into bits.  The
    ability to do so depends on the signal to noise ratio, which gets
    worse the higher you push the buad rate.  ...


>   This really hasn't been the case since the
> advent of IDE ( what?  25 years ago ) though.
----
SCSI didn't go away with the advent of IDE.  Though it is true IDE drivers
were dumbed down for cost
>
>> The physical platters still had the same number of bits, but it's 
>> up to software to decide how many bytes are squeezed out of that
>> space. I.e. -- Bytes are a software-defined-unit that don't exist
>> in the real world -- they are logical, derived units.
>
> No, the physical platters don't hold bits at all.  
But you said above they hold sectors @ 512 bytes each,
which you have defined as being 8 bits each.  That would
imply 4kbit - data / physical sector.

Bytes are a unit specific to the computer field. 
They are not metric any more than hours.

But their core unit 'bit' like the 'second',
is based on a minimal distinct magnetic flux variation
on the media.  It is directly usable with SI prefixes as
there is a 1-1 mapping of the minimum sized, stable flux changes
per unit area and the devices maximum bit storage. 

You can't get the correct number of bytes transfered over
1 300bps modem by dividing by 8.  (Note, encoding overhead is not
the same type of overhead as protocol overhead, as the encoding overhead
is media specific, while protocol overhead is not).

Re: fdisk units size & disk manufacturers buying the standard

[Matthias Schniedermeyer]

On 08.01.2015 18:37, Linda Walsh wrote:
> Phillip Susi wrote:
> >>speed in Bytes varies by protocol.  1Gb-Base-T ethernet maxes out
> >>at a theoretical 125MB/s - divisible by 8.  But 10Gb ethernet maxes
> >>out at 1000MB/s -- with 20% of its bandwidth going to protocol
> >>overhead.
> >
> >I'm not aware of any additional overhead that 10Gb ethernet has over
> >1Gb ethernet,
> ----
> See kernel messages for a 10b-T ethernet.
> 
> [   21.224641] ixgbe 0000:05:00.0: PCI Express bandwidth of 32GT/s available
> [   21.224644] ixgbe 0000:05:00.0: (Speed:5.0GT/s, Width: x8, Encoding
> Loss:20%)
> 
> I don't recall a 20% encoding loss in 1Gb or 100Mb ethernet and the kernel
> displays
> no such messages for the slower speed cards.

The message speaks about PCIe.

So the 40GBit/s (a.k.a. 40GT/s) is in effect 32GBit/s on the PCIe side.
5.0 GT/s = PCIe Gen. 2. PCIe Gen. 1 & 2 use 8b/10b encoding.
IOW for every 8 bits of payload 10 bits go ever the wire. This is 20% 
the enconding loss the message speaks about.

PCIe Gen 3 (and 4) use an enhanced encoding called 128b/130b. IOW for
every 128 Bit of data 130Bits are send over the wire, so only 8GT/s
(instead of 10GT/s) were needed to (nearly) double the effective
datarate in Gen 3.

See:
http://en.wikipedia.org/wiki/PCI_Express

Which still leaves well enough headroom to get the 10Git/s for the 
ethernet-connection across.




-- 

Matthias

Re: fdisk units size & disk manufacturers buying the standard

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 1/8/2015 9:37 PM, Linda Walsh wrote:
> Phillip Susi wrote:
>> And like I said, a watt-hour is in the same boat.
> Not at all.  An hour is NOT a metric unit and is not, officially, 
> used with SI prefixes.  Clearly, if it is in the same boat, you
> would be claiming that the Byte is not a metric unit -- and you 
> would be right.

Umm... yea.. you said bytes are not SI, I'm saying watt-hours aren't
either.  You said that means the kilo prefix should be base 2 since a
byte is 2^3 bits.  I'm saying by the same silly logic, a
killowatt-hour should be 3600 watt-hours since a watt-hour is 60^2
joules.  Are you getting it now?

> See kernel messages for a 10b-T ethernet.
> 
> [   21.224641] ixgbe 0000:05:00.0: PCI Express bandwidth of 32GT/s 
> available [   21.224644] ixgbe 0000:05:00.0: (Speed:5.0GT/s, Width:
> x8, Encoding Loss:20%)
> 
> I don't recall a 20% encoding loss in 1Gb or 100Mb ethernet and
> the kernel displays no such messages for the slower speed cards.

That message is referring to the encoding loss on the pci express gen
2 bus, not ethernet.

> excludes optional protocols.  The 125MB/s is a max theoretical rate
> and is in base 2.

No, as I said, that is base 10, not base 2.  1,000,000,000 bps / 8
bits per byte = 125,000,000 bytes / second, not 131,072,000 bytes.

> In practice, 125 millionBytes for writes and 119 millionBytes are
> a benchmark maximum that includes headers (SMB/CIFS for the test I
> most frequently run).

Nope, not possible.

> I.e. 125millionBytes/s -- the base10 number IS a benchmark maximum 
> that includes protocol headers (specifically for SMB).

Nope.

> I would assert you are contradicting yourself.  You said the
> platters don't contain any logical computer storage unit:
> 
> No, the physical platters don't hold bits at all.  They record an 
> analog signal that the controller has to decode into bits.  The 
> ability to do so depends on the signal to noise ratio, which gets 
> worse the higher you push the buad rate.  ...
> 
> 
>> This really hasn't been the case since the advent of IDE ( what?
>> 25 years ago ) though.
> ---- SCSI didn't go away with the advent of IDE.  Though it is true
> IDE drivers were dumbed down for cost

I didn't say it did.  I said the SCSI standards never have specified a
way to ask the drive to change its low level format.  IIRC, you could
do this with MFM/RLL hard drives, but when IDE came along, it too
decided not to give a way to do a low level format.  Thus the nature
of how the drive actually stores data on the platter is specifically
hidden from the computer and the drive deals only with whole sectors,
as far as the computer is concerned.

>>> The physical platters still had the same number of bits, but
>>> it's up to software to decide how many bytes are squeezed out
>>> of that space. I.e. -- Bytes are a software-defined-unit that
>>> don't exist in the real world -- they are logical, derived
>>> units.
>> 
>> No, the physical platters don't hold bits at all.
> But you said above they hold sectors @ 512 bytes each, which you
> have defined as being 8 bits each.  That would imply 4kbit - data /
> physical sector.

There is no such thing as a "physical sector".  On the platter, we are
in the analog domain now rather than the digital, so here everything
is a continuous function.  You can not point to an exact spot and say
right there is the first bit in the sector, right there is the second,
and so on.  You get a continuous signal that you have to decide to
recognize as a series of symbols that can be mapped to some number of
bits.  Exactly where they start and end is a bit fuzzy, which is why
decoding it sometimes gets it wrong.

> But their core unit 'bit' like the 'second', is based on a minimal
> distinct magnetic flux variation on the media.  It is directly
> usable with SI prefixes as

No, it isn't.  The bit is based purely on a mathematical notion of a
base 2 numbering system because digital logic is very good at
representing two distinct states.  The fact that we came up with ways
of recording bits in signals of magnetic flux on spinning rust has
nothing to do with the definition of a bit.  Even such recordings
typically do not record a single bit per baud, but use multi bit
symbols, such as 8PSK or 16QAM.

> You can't get the correct number of bytes transfered over 1 300bps
> modem by dividing by 8.  (Note, encoding overhead is not the same
> type of overhead as protocol overhead, as the encoding overhead is
> media specific, while protocol overhead is not).

Umm, yes, that is exactly what you do.  It kind of follows from the
definition of bits and bytes.  Of course, there weren't 1,300 bps
modems, but still.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQEcBAEBAgAGBQJUr+ziAAoJENRVrw2cjl5RS0kIAIqE61iOR3mFfqBz0gt9eP7B
PylDcPBdJw1kg7BNSTpMvLBRqutePvIDGIDr9+b9xKsx24g6+Dl42AdZj3dNdfak
7yih6goDCgAjARpSoSsYNGS7IiTTDtIEpc+bXIyQJzKzBl5n8MQV2VDyYEUXjZvN
sN0vZHnM2g3BoXUK0nNJbnvgiykyS964QSv0UgmdqYfqg6KtAxV3tFZtnZt/EYYU
XGxJaK8wvoIOL+lb+qxOpr5Dsa9XEFvHNFD6yM+W2/9bIe0TE9wb5IQJGF+b/IiM
b8EsVdKHem2Inorp6Yfuxklt0Dy5TWMhME7p0k2afo9gE/v4MmwE9s1Rbde5+OM=
=mvaH
-----END PGP SIGNATURE-----

Re: fdisk units size & disk manufacturers buying the standard

[Linda Walsh]

[-- Attachment #1: Type: text/plain, Size: 6248 bytes --]

Phillip Susi wrote:
> Umm... yea.. you said bytes are not SI, I'm saying watt-hours aren't
> either.  You said that means the kilo prefix should be base 2 since a
> byte is 2^3 bits.  I'm saying by the same silly logic, a
> killowatt-hour should be 3600 watt-hours since a watt-hour is 60^2
> joules.  Are you getting it now?
>   
====
    I was hoping you'd ask that.
    The 'kilo' applies to the 'watt', not the hour.  If you had
1000 kWHr, you wouldn't see normal usage calling it 1 MWHr.  It's ok
to use Metric prefixes with metric units. Multiplying a metric
quantity by a non-metric value doesn't change that -- but you
won't see power plants rated that way, nor will you see
that on your power bill.

    You will see articles talking about hard disk technologies speaking
of density in 'bits'/area (where either bit OR area can have an SI
quantity), since a bit requires no conversion constant to go from
flux-changes/area. HD-densities (in the US) are expressed in
<metric-prefix>-bits/in².  Again, with the metric prefix applying to
the unit that requires no conversion constant (the bit) to be expressed
as some combination of SI units.

    This is a great point here -- concerning the HD industry.
If they were so concerned about using the metric system, why wouldn't
they specify density in <prfx>-bits/cm².  Snort.
 

>   
>> See kernel messages for a 10b-T ethernet.
>>
>> [   21.224641] ixgbe 0000:05:00.0: PCI Express bandwidth of 32GT/s 
>> available [   21.224644] ixgbe 0000:05:00.0: (Speed:5.0GT/s, Width:
>> x8, Encoding Loss:20%)
>>     
>
> That message is referring to the encoding loss on the pci express gen
> 2 bus, not ethernet.
>   
---
    Thank you.  I didn't know that.  Now I know I need a HW upgrade... ;-)


>   
>> excludes optional protocols.  The 125MB/s is a max theoretical rate
>> and is in base 2.
>>     
>
> No, as I said, that is base 10, not base 2.  1,000,000,000 bps / 8
> bits per byte = 125,000,000 bytes / second, not 131,072,000 bytes.
>   
---
    My bad... you're right.  I *thought* it was the other way
around at one point, but later realized that the output of 'dd'
was in decimal prefixes (regardless of user input).  My test
script even tries to convert units back to base2... ;-)
>   
>> In practice, 125 millionBytes for writes and 119 millionBytes are
>> a benchmark maximum that includes headers (SMB/CIFS for the test I
>> most frequently run).
>>     
>
> Nope, not possible.
>   
===
    Maybe not, but it is repeatable by myself and other people.
Do you have samba and a 1Gb connection?  It's fairly easy to setup.
I'll even attach the primitive test script (bash) -- easy to use
once you get the files setup in a server-dir. 

I'm on an internal network (so no encryption on the link).  Client
is Win7SP1, linux flavor = opensuse, but kernel is vanilla,
currently at 3.17.3. 
Measuring line speeds only, I use the following files in my home
directory on the server:

>  cd
>  ll zero null
crwxrw-rw- 1 1, 3 Jan  9 13:55 null
crwxrw-rw- 1 1, 5 May  7  2013 zero
---
Using cygwin 'dd', for client writes I read from /dev/zero and write
to 'null' oflag=direct conv=notrunc,nocreat, BS=16M, total size=4GB
for client reads: if=zero of=/dev/null.

Note -- if you make them regular files, at least defrag them.
Using 'xfs', you can use "xfs_fsr" to make a file contiguous.

Other people on the samba list report similar results, though
it is definitely a minority who have tuned their systems for that.
Besides what I can do on windows, I have also used a fair number
of large or insane parameters on the linux networking stack.

Maybe it's your hardware?  For my fastest results, I've been using
Intel dual-port cards:

Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
Intel Corporation Ethernet Controller 10 Gigabit X540-AT2 (rev 01)

Identical cards on server and client -- though my client only uses the
10Gb card now...(300-650MB/s transfer tops... varies widely).

I also use 9kB (9000) byte packets and whatever offload features I
find yield a positive effect.  You aren't going to get those speeds
out of the box... it does take a bit of performance tuning.

>   
>> I.e. 125millionBytes/s -- the base10 number IS a benchmark maximum 
>> that includes protocol headers (specifically for SMB).
>>     
>
> Nope.
>   
---
I've seen it multiple times -- others have reproduced it.
Vs. the max ranges for 10Gb (8Gb on my setup), have been in the
600's (usually lower) (smb protocol is cpu bound and usually w/1 user
connection -- though newer smb (win8+) seems to have some beginnings
for support for using parallelism in the processors.
>   
>> But their core unit 'bit' like the 'second', is based on a minimal
>> distinct magnetic flux variation on the media.  It is directly
>> usable with SI prefixes.
>>     
>
> No, it isn't.
Reality disagrees: See:

   http://en.wikipedia.org/wiki/Bit_cell
   http://www.softpres.org/?id=glossary:bit_cell

  Specifically: (quote)

This is essentially the distance along the track allocated for the 
recording of an area of Flux. That is, in each “cell” there are magnetic 
particles that together indicate a detectable magnetic polarity of set 
length. Note that bit cells are not read as bits of data directly, they 
are used to form the flux transitions used to make up the Encoding. 
Every bit cell contains either a change of polarity (1) or not (0).

--endquote--

Perhaps you have some references that would convince me that the
above terminology isn't both common and standard in this field?


Only when you get into marketing speak (getting away from tech speak),
will you see things like $$$/[MGT]B, where the prefix applies
to 'Byte'.  But again, that's marketing speak.

Disk manufacturers describe and talk about "bits"/area when describing
disk technologies.


>   
>> You can't get the correct number of bytes transfered over 1 300bps
>> modem by dividing by 8.
>>     
>
> Umm, yes, that is exactly what you do.  It kind of follows from the
> definition of bits and bytes.  Of course, there weren't 1,300 bps
> modems, but still.
>   
---
    That's "one" 300bps modem.  And no, you don't divide by 8
since 300bps modems only had a 30cps throughput -- not 37.5 as you
would seem to indicate.


>   


[-- Attachment #2: iotest --]
[-- Type: text/plain, Size: 3062 bytes --]

#!/bin/bash -u
# iotest v0.1 - lawalsh: open usage allowed

_prgpth="${0:?}"; _prg="${_prgpth##*/}"; _prgdr="${_prgpth%/$_prg}"
[[ -z $_prgdr || $_prg == $_prgdr ]] && $_prgdr="$PWD"
export PATH="$_prgdr:$_prgdr/lib:$PATH"
shopt -s expand_aliases extglob sourcepath ; set -o pipefail

#include stdalias

Dd=$(type -P dd)

[[ $Dd ]] || { echo "Cannot find dd.  Cannot proceed.";  exit 1; }

alias intConst=declare\ -ix int=declare\ -i my=declare
alias string=declare sub=function array=declare\ -a
# 1 num = block size
# num-num = range of block sizes to test; w/increment = "2x", so
# 4M-16M tests 4M, 8M, 16M
# 4M-12M test 4M, 8M, 12M
# count adjusted to xfer 4G, rounding up
#----

#all xfers are using 'devices' (/dev/zero for source, /dev/null for target)
# remote filenames "zero" and "null" should be setup to be remote devices

intConst K=1024
intConst M=$[K*K]
intConst G=$[M*K]
intConst T=$[G*K]

int BS=$[16*M]
int count=256
int IOSIZE=${IOSIZE:-4*G}

#		desuffix 	1st arg = num+suffix -> convert to int
#							2nd arg = optional buff name (else print to stdout)
#							return 0 if no error

sub desuffix {						#convert num+Suff => int store in optional Buff
	string str="${1:?}" ; shift
	string bufnam=""; (($#)) && bufnam=$1
	if [[ $p =~ ^([0-9]+)([KMGT])$ ]]; then 
		int num=${BASH_REMATCH[1]}*${BASH_REMATCH[2]}
		((num)) || return 1
		if [[ $bufnam ]] ; then printf -v $bufnam "%d" "$num"
		else printf "%d" "$num" ; fi
	else 
		return 1
	fi
}




sub hdisp {
	int num=${1:?}; shift
	string bufnam=""; (($#)) && bufnam=$1
	string suf=""
	int ans=num
	array pows=('K' 'M' 'G' 'T')
	for s in ${pows[@]};do
		int si=$s
		if (((num/si)*si==num)); then
			ans=num/si;suf="$s"
		fi
	done
}

sub check_params {
	int num=0
	if (($#)) ; then 
		string p="$1"; shift;
		if [[ $p =~ ([0-9]+)([KMGT]) ]]; then 
			num=${BASH_REMATCH[1]}*${BASH_REMATCH[2]}
		fi
	fi
	((num)) && {
		BS=num
		count=IOSIZE/BS
	}
}

(($#)) && check_params "$@"

array reada=(/h/zero /dev/null)
array writea=(/dev/zero /h/null conv=nocreat,notrunc)

sub dd_need_io  {
	local if="$1" of="$2"; shift 2
	nice --19 $Dd if="$if" of="$of" bs="$BS" count="$count"\
	 		oflag=direct iflag=direct conv=nocreat "$@"
}

sub dd {
	local if="$1" of="$2" ;shift 2
	#echo $dd if="$if" of="$of" bs="$BS" count="$count" "$@" >&2
	array out err
	readarray err < <( \
		readarray out < <(dd_need_io "$if" "$of" "$@";int s=$?; 
											((s)) && echo "stat:$s">&2  ) 2>&1 )
	#
	if ((${#err[@]})) ;then echo "${err[@]}"; exit 1; fi
	return 0
}
	
function dd_format { my ln
	while read ln;do
		echo $ln | while read bytes btxt pnum suffp \
											copt time unitc rate ra_unit; do
			[[ $bytes == records ]] && continue
			[[ ${pnum:0:1} != \( || ${suffp:0-1:1} != \) ]] && continue
			num="${pnum:1}" 
			suff="${suffp%?}" 
			unit="${unitc%?}"
			printf "%s:%s:%s:%s:%s:%s:\n"	"$num" "$suff" "$time" "$unit" "$rate" "$ra_unit"
		done
	done
}

sub onecycle {
	echo -n "R:"; { dd "${reada[@]}"|| exit $?; } | dd_format 
	echo -n "W:";	{ dd "${writea[@]}" || exit $?; } | dd_format 
}

onecycle

Re: fdisk units size & disk manufacturers buying the standard

[Phillip Susi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 1/9/2015 7:29 PM, Linda Walsh wrote:
> You will see articles talking about hard disk technologies
> speaking of density in 'bits'/area (where either bit OR area can
> have an SI quantity), since a bit requires no conversion constant
> to go from flux-changes/area. HD-densities (in the US) are
> expressed in <metric-prefix>-bits/in².  Again, with the metric
> prefix applying to the unit that requires no conversion constant
> (the bit) to be expressed as some combination of SI units.

And this is a purely synthetic value obtained by dividing the capacity
of the disk by its size.  If you look at the distance between any two
symbols on the disk, it is not precisely constant but varies
throughout the disk, and there are gaps between sectors and between
the sector header and the payload where there is no data at all (
because it takes the logic some time to switch from reading to writing
).  The size of these gaps is not infinitely accurate; they vary
within some tolerance.

> Maybe not, but it is repeatable by myself and other people. Do you
> have samba and a 1Gb connection?  It's fairly easy to setup.

Then your measurements are not accurate enough to detect the overhead.
 If you are using 9k jumbo frames that overhead would be rather small,
lowering throughput to something like 124,305,555 bytes per second (
assuming 50 bytes out of every 9000 ).

> Maybe it's your hardware?  For my fastest results, I've been using 
> Intel dual-port cards:

No; it has nothing to do with hardware, but rather the specs, which
state that the speed of 1000Base-T is 1,000,000,000 bps with a pretty
narrow tolerance, and you just aren't going to get that unless you
have zero overhead.

>> No, it isn't.
> Reality disagrees: See:
> 
> http://en.wikipedia.org/wiki/Bit_cell 
> http://www.softpres.org/?id=glossary:bit_cell
> 
> Specifically: (quote)
> 
> This is essentially the distance along the track allocated for the 
> recording of an area of Flux. That is, in each “cell” there are
> magnetic particles that together indicate a detectable magnetic
> polarity of set length. Note that bit cells are not read as bits of
> data directly, they are used to form the flux transitions used to
> make up the Encoding. Every bit cell contains either a change of
> polarity (1) or not (0).

Reality != a description on wikipedia.  This is an abstraction used to
conceptualize how the device works; it is not reality.  Until you get
down to the quantum scale, the real world does not operate in discrete
quantities.  When you pass the read head over the medium, you do not
get a 1 or a 0; you get an analog signal whose voltage varies
continuously between, for example, 0 and 1.0 volts.  Manchester
encoding uses an edge detector to detect a sharp change between
somewhere less than 0.5 volts and somewhere over 0.5 volts and
combined with a clock source whose timing is kept synchronized to the
detected edges and whose duration is set correctly can manage to
decode bits from that continuous signal.  Note that manchester
encoding is very simplistic and inefficient, which is why 10Base-T
ethernet required 20 MHz of bandwidth but 100Base-T delivers 10 times
the data rate but only increased the required bandwidth to 25 MHz.
Modern communications systems use better modulation techniques that
pack multiple bits per baud, so the description of a "bit cell" either
being a 1 or a 0 is entirely non applicable.  For example, 1000Base-T
uses 5 different voltage levels to encode 3 bits per baud.

> Disk manufacturers describe and talk about "bits"/area when
> describing disk technologies.

Yes, and this is merely an approximation, not a hard constant across
the entire disk surface.  Much like how you can divide the number of
gallons of gas in your tank into the number of miles you drive before
having to refill to estimate the fuel economy of your car.  That
doesn't mean you can put one gallon of gas in and drive exactly that
far under any conditions before running out of gas; the economy varies
quite a bit.

>>> You can't get the correct number of bytes transfered over 1
>>> 300bps modem by dividing by 8.
>>> 
>> 
>> Umm, yes, that is exactly what you do.  It kind of follows from
>> the definition of bits and bytes.  Of course, there weren't 1,300
>> bps modems, but still.
>> 
> --- That's "one" 300bps modem.  And no, you don't divide by 8 since
> 300bps modems only had a 30cps throughput -- not 37.5 as you would
> seem to indicate.

Ahh, yes... the original 300 baud modem was external and connected
with an RS-232 cable, which bracketed every byte in a start and stop
bit, effectively using 10 baud to transmit each byte.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQEcBAEBAgAGBQJUtBdZAAoJENRVrw2cjl5RECcH/0gWjtJyN7gnAG9htHcIrpn2
QB02igaPQgn7Gzs0BIlI9fp965jWDuYGmSomg4XVVAIIsWFpJzB+ks7WnYiOS+wc
tZprznTJCngSqjGQ/pcfDjO6M7mDmAeF8lSJFUMukVtiKr6SsAy1Qwe9b7H4b/Wm
lQQJPiECjbC4mbn3HURl8H1/NIwqqkLjcdOEmM0uMF7nYG3BBKrcTJg+D6GxmaYa
JrpMIfYleS8eZJfZfvnwaOIaAadjIiTPGPXrv5mkmorio6sZBDzK66w4nxITRyj3
ALRSa9hMBs2/e6cRR3AkMtKE5HMkc0Gdi4wo5WSFva0EP8Q1iqY/38IOkEl5K9s=
=L5il
-----END PGP SIGNATURE-----