Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andre Hedrick]

Rik and Richard,

As you see, I in good faith prior to this holy war, had initiated a formal
request include a new protocol into the Linux kernel prior to the freeze.
The extention was requested to insure the product was of the highest
quality and not limited with excessive erratium as the ratification of the
IETF modified, postponed, and delayed ; regardless of reason.

Obviously, PyX had (has) on its schedule to product a high quality target
which is transport independent on each side of the protocol.  We are not
sure of this position because of the uncertain nature of the basic usages
of headers and export_symbols.

Regards,

Andre Hedrick, CTO & Founder 
iSCSI Software Solutions Provider
http://www.PyXTechnologies.com/

---------- Forwarded message ----------
Date: 23 Oct 2002 09:57:43 +0100
From: Alan Cox <alan@lxorguk.ukuu.org.uk>
To: Andre Hedrick <andre@pyxtechnologies.com>
Cc: Linus Torvalds <torvalds@transmeta.com>
Subject: Re: Linux iSCSI Initiator, OpenSource

On Wed, 2002-10-23 at 09:30, Andre Hedrick wrote:
> 
> Greetings Linus and Alan,
> 
> PyX Technologies is asking for a formal extention beyond the Oct 31st
> Feature Freeze.  The exception to the rules is generally granted to no
> one, and is well understood by PyX.  Only because PyX is under review for
> full funding on "Oct 31st" in the morning, would I even consider this
> motion.  Our position is to formally open source a full feature with all
> corner cases resolved under "error recovery level zero".

I can't see an iscsi initiator needing to touch core code anyway. Its
just another (slightly demented) scsi adapter

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andrew Morton]

Andre Hedrick wrote:
> 
> Rik and Richard,
> 
> As you see, I in good faith prior to this holy war, had initiated a formal
> request include a new protocol into the Linux kernel prior to the freeze.
> The extention was requested to insure the product was of the highest
> quality and not limited with excessive erratium as the ratification of the
> IETF modified, postponed, and delayed ; regardless of reason.
> 
> Obviously, PyX had (has) on its schedule to product a high quality target
> which is transport independent on each side of the protocol.  We are not
> sure of this position because of the uncertain nature of the basic usages
> of headers and export_symbols.
> 

I suggest that if a function happens to be implemented as an inline
in a header then it should be treated (for licensing purposes) as
an exported-to-all-modules symbol.  So in Linux, that would be LGPL-ish.

The fact that a piece of kernel functionality happens to be inlined
is a pure technical detail of linkage.

If there really is inlined functionality which we do not wish made
available to non-GPL modules then it should be either uninlined and
not exported or it should be wrapped in #ifdef GPL.

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[NEURONET]

> The fact that a piece of kernel functionality happens to be inlined
> is a pure technical detail of linkage.

Absolutely.

> If there really is inlined functionality which we do not wish made
> available to non-GPL modules then it should be either uninlined and
> not exported or it should be wrapped in #ifdef GPL.

I don't even think there should be any such case, 
as the real point in having LGPL is, supposedly, to 
support *interfacing*, so using headers (source-code 
*interfaces*) and linking to stuff (binding binary 
*interfaces*) should be just fine under LGPL (or a 
better LGPL? IGPL? dunno the details that deep, sorry).

Sab

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[NEURONET]

> > If there really is inlined functionality which we do not wish made
> > available to non-GPL modules then it should be either uninlined and
> > not exported or it should be wrapped in #ifdef GPL.
> 
> I don't even think there should be any such case, 
> as the real point in having LGPL is, supposedly, to 
> support *interfacing*, so using headers (source-code 
> *interfaces*) and linking to stuff (binding binary 
> *interfaces*) should be just fine under LGPL (or a 
> better LGPL? IGPL? dunno the details that deep, sorry).

(Umm, sorry, I guess I should've been sleeping 
fast already, instead of confusing things publicly 
here... :-/ )

Sab

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andre Hedrick]

Andrew et al.,

Now this is a point of responsible policy making, and your previous
position to characterize me as somebody who solely wants to rip off and
use the work of our peers was unwarrented!  Have a glass of milk to help
the crow, trust me it works.

Now the question before developers of "Linux", and not the FSF, GNU
Project, so Richard it has been fun but we shall let you go do what you do
best for now.

Granting special rights to developers is not acceptable, nor would I even
assume or expect to be above the rules.

Grandfathering our design mistakes, and giving notice now that 2.6/3.0
shall have explicit rules and provisions for binary only seems
reasonable.  It would allow time for everyone to get their house in order.
Additionally the developers should not impose restrictions or willfully
change the the current exported_symbols of today for 2.6/3.0 to provide
backwards compatability for another development cycle.

If the kernel developers force out all current binary only vendors of
today now, the ramp up and recovery time to support our current shortages
is incredible and steep.

I will have to take a calculated risk of exposure and trust our peers are
reasonable and will not seek to destroy one another.

Just to let you all in on the scope of the project, I am hoping to empower
all the folks who have shipped and promoted Linux on the hardware side.
Yeah, I mean all the white box builders who are having it as rough as
everyone else.  The big storage vendors are users-keepers of all of our
hard work.  iSCSI is the first time Enterprise storage is not controlled
by the few and the powerful who are basically untouchable.

I have a goal and a vision to enable Linux to dominate the Enterprise
Storage arena and do it in a cost effective manner.  The time is now as
the hardware vendors who will only ship binary modules and look to other
architects to embed the protocol.

If you are against this idea of bringing enterprise storage to the masses,
under Linux, I will be forced to migrate the most valuable piece of IP to
another platform.  To give each of you an idea how much work is involved
to achieve interoperability ...

http://www.PyXTechnologies.com/target.html

This is a snapshot of MicroSoft NT4+SP6 running on a Dell Beetle Box.
The initiator is the IBM v8 1.2.2 windows initiator.
The target is PyX-Target using 3ware 8500-8 SATA with 6 Maxtor 160/5400.
The benchmark is Intel's IOmeter.
I have no control of the initiator environment :-/

The image break down goes as the following.

                        Random Access | Sequential Access
                        ---------------------------------
 67% read,  33% write  | 112 MB/sec   | 112 MB/sec
  0% read, 100% write  |  69 MB/sec   |  69 MB/sec
100% read,   0% write  | 180 MB/sec   | 171 MB/sec

I can make this an absolute win for Linux from my side.

The strenght of Dave Miller and Jeff Garzik on the netork stack for TOE's.
The vision of Jens Axboe to deploy BIO successfully.
The stomachs of James Bottomley and Douglas Gilbert to fix SCSI.
The insanity of Rik van Riel, Andrea Arcangeli, yourself for MM.
The new RAID 6 by H. Peter Anvin.
Plus the MD/LVM gang.
With grit of Alex Viro to keep us all straight!

It takes all of us, all of you, and more to make Linux the the absolute
winner take all in the future of enterprise storage.

---------

I am either in or out, but I will not risk loosing the ability to recover
all of my development costs and fill the bank account first before I give
it up and grab the next generation of storage technology roller coaster in
another three years.  There are three levels of Error Recovery, and one
released a year is reasonable to me.  That implies, I could publish ERL=0
today before I have recovered a DIME, with 18 months in the hole now.

Again all I want to know is where the threshold of fair usage lays.

This posting made by Linus to the gnu.misc.discuss newsgroup (Message-ID
"4b0rbb$5iu@klaava.helsinki.fi") on December 17, 1995 where he
basically gave his permission for the EXPORT_SYMBOL
vs. EXPORT_SYMBOL_GPL system hereby proprietary modules that call only
EXPORT_SYMBOL symbols are allowed:

http://groups.google.com/groups?as_umsgid=4b0rbb%245iu%40klaava.helsinki.fi

Until there is some type of agreement ratified by all of us, this is a
safe position for setting and holding a precedence.  If any one of the
copyright holders in the kernel wishes to formally object, please do so
now.  If you are not one of these people I would ask you to hold your
comments, because FSF has "released" responsiblity of enforcement to them.
Please respect my request.

Regards,

Andre Hedrick, CTO & Founder
iSCSI Software Solutions Provider
http://www.PyXTechnologies.com/


On Sat, 4 Jan 2003, Andrew Morton wrote:

> Andre Hedrick wrote:
> > 
> > Rik and Richard,
> > 
> > As you see, I in good faith prior to this holy war, had initiated a formal
> > request include a new protocol into the Linux kernel prior to the freeze.
> > The extention was requested to insure the product was of the highest
> > quality and not limited with excessive erratium as the ratification of the
> > IETF modified, postponed, and delayed ; regardless of reason.
> > 
> > Obviously, PyX had (has) on its schedule to product a high quality target
> > which is transport independent on each side of the protocol.  We are not
> > sure of this position because of the uncertain nature of the basic usages
> > of headers and export_symbols.
> > 
> 
> I suggest that if a function happens to be implemented as an inline
> in a header then it should be treated (for licensing purposes) as
> an exported-to-all-modules symbol.  So in Linux, that would be LGPL-ish.
> 
> The fact that a piece of kernel functionality happens to be inlined
> is a pure technical detail of linkage.
> 
> If there really is inlined functionality which we do not wish made
> available to non-GPL modules then it should be either uninlined and
> not exported or it should be wrapped in #ifdef GPL.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andrew Morton]

Andre Hedrick wrote:
> 
> ..
> Again all I want to know is where the threshold of fair usage lays.

Yes, it needs to be clarified.
 
> This posting made by Linus to the gnu.misc.discuss newsgroup (Message-ID
> "4b0rbb$5iu@klaava.helsinki.fi") on December 17, 1995 where he
> basically gave his permission for the EXPORT_SYMBOL
> vs. EXPORT_SYMBOL_GPL system hereby proprietary modules that call only
> EXPORT_SYMBOL symbols are allowed:
> 
> http://groups.google.com/groups?as_umsgid=4b0rbb%245iu%40klaava.helsinki.fi

I wasn't aware of that posting until Adam pointed it out.  It seems
to be a sensible and easily understandable position.
 
> Until there is some type of agreement ratified by all of us, this is a
> safe position for setting and holding a precedence.  If any one of the
> copyright holders in the kernel wishes to formally object, please do so
> now.

Yup.  Now is their chance.  Is everyone OK with treating the contents
of header files in the same was as EXPORT_SYMBOL()?  ie: LGPL?

Really, I don't see any alternative.  Even things like the semaphore
down() function are inlined.  Linus's 1995 intentions are infeasible
unless we also treat that part of the kernel API which is implemented
in headers as being exported.

It might make sense to be more selective, by putting #ifdef GPL around
some portions.  If anyone cares, and can be bothered.  If any inlined
function is complex enough to justify that then it's too damn big and
should be zoomed into a .c file anyway.

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Adam J. Richter]

Andrew Morton writes:
>Is everyone OK with treating the contents
>of header files in the same was as EXPORT_SYMBOL()? ie: LGPL? 

	I don't believe that anyone other than Linus has explicity
given their permission for the "EXPORT_SYMBOL_GPL / EXPORT_SYMBOL"
system.  However, I'm not trying to obstruct other people giving
what permissions they want with the code that they wrote, so I'd
like to point out a problem and suggest a fix to your proposal.

	Your proposal would prevent "EXPORT_SYMBOL_GPL" inline
functions.

	I can think of a solution which would
		(1) allow both kinds of functions and
		(2) which I think would be more legally defensible than
		    posting a message to a mailing list and asking "is
		    everyone OK with it]".

	Have the appropriate copyright owners move their "proprietary
callers OK" inline functions to files with the appropriate copyright
statements (such as LGPL or something similar), or submit changes to
the copyright statements of entire .h files in cases where you've got
the explicit agreement of all the copyright owners for that file.

	Along the same lines, those copyright owners (individuals,
companies, etc.) who seek to create a kernel that contains only
content that allows proprietary modules could start by explicitly
stating that they now irrevocably give the same permission that Linus
gave in his gnu.misc.discuss posting for their past contributions and
any future contributions that they make before they say otherwise.  At
least socially, it would come with better graces for people to grant
permissions on their copyrights before trying to interpret away other
contributors' copyrights.

	However, before you grant such permission, I recommend that
you consider whether permission that allows proprietary linking
against only EXPORT_SYMBOL functions really means allowing proprietary
modules to call *all* functions by just including a GPL-compatible
library module like so:

void loophole_ide_setup_pci_device (struct pci_dev *dev, ide_pci_device_t *d)
{
	return ide_setup_pci_device(dev, d);
	/* ide_setup_pci_device is an EXPORT_SYMBOL_GPL function in 2.5.54. */
}
EXPORT_SYMBOL(loophole_pci_hp_register);


	One could presumably do the same with an function that is not
even exported, and perhaps even avoid the subroutine jump by using
static inline declarations.  As far as I can tell, it would be
essentially equivalent to switching to the LGPL.  Come to think of it,
if this _is_ what you want, then maybe you find it simpler to just
grant permission to copy your code under the LGPL.

	I'm not a lawyer, so don't take this is as legal advice.

Adam J. Richter     __     ______________   575 Oroville Road
adam@yggdrasil.com     \ /                  Milpitas, California 95035
+1 408 309-6081         | g g d r a s i l   United States of America
                         "Free Software For The Rest Of Us."

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Oliver Xymoron]

On Sat, Jan 04, 2003 at 06:47:43PM -0800, Andrew Morton wrote:
> Andre Hedrick wrote:
> > 
> > Rik and Richard,
> > 
> > As you see, I in good faith prior to this holy war, had initiated a formal
> > request include a new protocol into the Linux kernel prior to the freeze.
> > The extention was requested to insure the product was of the highest
> > quality and not limited with excessive erratium as the ratification of the
> > IETF modified, postponed, and delayed ; regardless of reason.
> > 
> > Obviously, PyX had (has) on its schedule to product a high quality target
> > which is transport independent on each side of the protocol.  We are not
> > sure of this position because of the uncertain nature of the basic usages
> > of headers and export_symbols.
> > 
> 
> I suggest that if a function happens to be implemented as an inline
> in a header then it should be treated (for licensing purposes) as
> an exported-to-all-modules symbol.  So in Linux, that would be LGPL-ish.
> 
> The fact that a piece of kernel functionality happens to be inlined
> is a pure technical detail of linkage.
> 
> If there really is inlined functionality which we do not wish made
> available to non-GPL modules then it should be either uninlined and
> not exported or it should be wrapped in #ifdef GPL.

More pragmatically, who cares? There's already at least one vendor
(Cisco) who ships a perfectly good fully GPLed iSCSI initiator module
that doesn't need to touch any core code. It's already the benchmark
for compatibility at interoperability tests. And it's following the
IETF drafts closely too. Once we actually have an iSCSI RFC, it might
be worth pulling it into the kernel tree. I believe Red Hat is
shipping it some form already.

That leaves the question of using Linux as an iSCSI target, and I've
yet to see any reason why this couldn't be done in userspace. In fact,
in a lot of ways that's the right thing to do as it lets you take
proper advantage of MD/LVM/EVMS/crypto, etc..

There are a few other free implementations out there too.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.." 

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Richard Stallman]

    I suggest that if a function happens to be implemented as an inline
    in a header then it should be treated (for licensing purposes) as
    an exported-to-all-modules symbol.  So in Linux, that would be LGPL-ish.

The Linux developers can certainly do this, if the copyright holders
of the substantial functions in question go along with it.  Even if
they already went along with linking to their functions from non-free
modules, this is still somewhat different.

The question only arises for the specific non-small functions that are
to be inlined in headers in this way.  (Inlining a very small function
from a header is probably not significant for copyright.)  Perhaps the
copyright holders of these functions are few and easy to ask.

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andre Hedrick]

On Sun, 5 Jan 2003, Oliver Xymoron wrote:

> On Sat, Jan 04, 2003 at 06:47:43PM -0800, Andrew Morton wrote:
> > Andre Hedrick wrote:
> > > 
> > > Rik and Richard,
> > > 
> > > As you see, I in good faith prior to this holy war, had initiated a formal
> > > request include a new protocol into the Linux kernel prior to the freeze.
> > > The extention was requested to insure the product was of the highest
> > > quality and not limited with excessive erratium as the ratification of the
> > > IETF modified, postponed, and delayed ; regardless of reason.
> > > 
> > > Obviously, PyX had (has) on its schedule to product a high quality target
> > > which is transport independent on each side of the protocol.  We are not
> > > sure of this position because of the uncertain nature of the basic usages
> > > of headers and export_symbols.
> > > 
> > 
> > I suggest that if a function happens to be implemented as an inline
> > in a header then it should be treated (for licensing purposes) as
> > an exported-to-all-modules symbol.  So in Linux, that would be LGPL-ish.
> > 
> > The fact that a piece of kernel functionality happens to be inlined
> > is a pure technical detail of linkage.
> > 
> > If there really is inlined functionality which we do not wish made
> > available to non-GPL modules then it should be either uninlined and
> > not exported or it should be wrapped in #ifdef GPL.
> 
> More pragmatically, who cares? There's already at least one vendor
> (Cisco) who ships a perfectly good fully GPLed iSCSI initiator module
> that doesn't need to touch any core code. It's already the benchmark
> for compatibility at interoperability tests. And it's following the
> IETF drafts closely too. Once we actually have an iSCSI RFC, it might
> be worth pulling it into the kernel tree. I believe Red Hat is
> shipping it some form already.

If you know anything about iSCSI RFC draft and how storage truly works.
Cisco gets it wrong, they do not believe in supporting the full RFC.
So you get ERL=0, and now they turned of the "Header and Data Digests",
this is equal to turning off the iCRC in ATA, or CRC in SCSI between the
controller and the device.  For those people who think removing the
checksum test for the integrity of the data and command operations, you
get what you deserve.

> That leaves the question of using Linux as an iSCSI target, and I've
> yet to see any reason why this couldn't be done in userspace. In fact,
> in a lot of ways that's the right thing to do as it lets you take
> proper advantage of MD/LVM/EVMS/crypto, etc..

You go right ahead, and see if you can move at near wire speed.
Next try to support any filesystem regardless of platform.
Specifically anything Microsoft does to thwart Linux, I have already
covered.

> There are a few other free implementations out there too.

Please go use them and in two seconds my product can bring them all to the
ground with the full error injection tool kit from both sides.  My team
has gone through supporting every optional feature in the RFC draft as
manditory to remove any possible vendor unique opportunities.

There are grey areas in the RFC draft to support every corner case of
ERL=1 and ERL=2.

You figure out how to support the marker stream to perform a
Sync-and-Steering layer.

PyX is the second in the world to support Sync-and-Steering, and the first
to do it software only.

Please go for it, and you will spend at least 18-24 months to develop.

The target(erl=0) is what would be the second phase to open source, but I
see you and other want to do the hard way and that is fine.

In two week I will have NetBSD certified, and 4 weeks later should have
Solaris certifed.

Cheers,

Andre Hedrick, CTO & Founder 
iSCSI Software Solutions Provider
http://www.PyXTechnologies.com/

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andre Hedrick]

Richard,

Can you admit the follow, that GPL has everything to control
redistribution, and has ZERO context for copyright.  The holders of the
copyright control the issues.

See your whole hook is "Derivative Works" well, I implimented a protocol.
It works regardless of platform or OS.  All it uses are simple and
standard kernel services.

Since I am willing to list every kernel function I call, and see who is
going to object to me doing what everyone else is doing and is clearly
positioned as accepted as of 1995.  If you were not aware of this
position, and I was not either, tough.  Linus sets the rules and he
controls the key interfaces.

Next you have NO ownership in the kernel, so unless you try to collect a
group of copyright holders and advocate on their behalf, I think you are
way out of bounds to even be here, period.  As you have stated already,
this is an issue exclusive to the copyright holders, and you are not one
of us.  So please live by your own words, or state you legal position to
be here in the affairs of the Copyright holders.

Regards,

Andre Hedrick, CTO & Founder
iSCSI Software Solutions Provider
http://www.PyXTechnologies.com/

On Sun, 5 Jan 2003, Richard Stallman wrote:

>     I suggest that if a function happens to be implemented as an inline
>     in a header then it should be treated (for licensing purposes) as
>     an exported-to-all-modules symbol.  So in Linux, that would be LGPL-ish.
> 
> The Linux developers can certainly do this, if the copyright holders
> of the substantial functions in question go along with it.  Even if
> they already went along with linking to their functions from non-free
> modules, this is still somewhat different.
> 
> The question only arises for the specific non-small functions that are
> to be inlined in headers in this way.  (Inlining a very small function
> from a header is probably not significant for copyright.)  Perhaps the
> copyright holders of these functions are few and easy to ask.

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Oliver Xymoron]

On Sun, Jan 05, 2003 at 07:38:15PM -0800, Andre Hedrick wrote:
> On Sun, 5 Jan 2003, Oliver Xymoron wrote:
> 
> > On Sat, Jan 04, 2003 at 06:47:43PM -0800, Andrew Morton wrote:
> > > Andre Hedrick wrote:
> > > > 
> > > > Rik and Richard,
> > > > 
> > > > As you see, I in good faith prior to this holy war, had initiated a formal
> > > > request include a new protocol into the Linux kernel prior to the freeze.
> > > > The extention was requested to insure the product was of the highest
> > > > quality and not limited with excessive erratium as the ratification of the
> > > > IETF modified, postponed, and delayed ; regardless of reason.
> > > > 
> > > > Obviously, PyX had (has) on its schedule to product a high quality target
> > > > which is transport independent on each side of the protocol.  We are not
> > > > sure of this position because of the uncertain nature of the basic usages
> > > > of headers and export_symbols.
> > > > 
> > > 
> > > I suggest that if a function happens to be implemented as an inline
> > > in a header then it should be treated (for licensing purposes) as
> > > an exported-to-all-modules symbol.  So in Linux, that would be LGPL-ish.
> > > 
> > > The fact that a piece of kernel functionality happens to be inlined
> > > is a pure technical detail of linkage.
> > > 
> > > If there really is inlined functionality which we do not wish made
> > > available to non-GPL modules then it should be either uninlined and
> > > not exported or it should be wrapped in #ifdef GPL.
> > 
> > More pragmatically, who cares? There's already at least one vendor
> > (Cisco) who ships a perfectly good fully GPLed iSCSI initiator module
> > that doesn't need to touch any core code. It's already the benchmark
> > for compatibility at interoperability tests. And it's following the
> > IETF drafts closely too. Once we actually have an iSCSI RFC, it might
> > be worth pulling it into the kernel tree. I believe Red Hat is
> > shipping it some form already.
> 
> If you know anything about iSCSI RFC draft and how storage truly works.
> Cisco gets it wrong, they do not believe in supporting the full RFC.
> So you get ERL=0, and now they turned of the "Header and Data Digests",
> this is equal to turning off the iCRC in ATA, or CRC in SCSI between the
> controller and the device.  For those people who think removing the
> checksum test for the integrity of the data and command operations, you
> get what you deserve.

CRC code seems to be functional, at least in their most recent drop.
As for ERL, the state of error handling in the rest of the Linux IO
layer suggests that's a lower triage priority..

If and when it becomes a high priority, well, the community is free to do what
it likes with the source.

> Next try to support any filesystem regardless of platform.
> Specifically anything Microsoft does to thwart Linux, I have already
> covered.

I'm having a very hard time making any sense of that statement.
There's no reason an iSCSI initiator on the MS side should care what
OS is serving it iSCSI. And the target shouldn't give a damn whether
it's serving up a filesystem.

> The target(erl=0) is what would be the second phase to open source, but I
> see you and other want to do the hard way and that is fine.
>
> In two week I will have NetBSD certified, and 4 weeks later should have
> Solaris certifed.

Frankly, I think you're the one choosing the hard path. Proprietary
code is the domain of corporate giants and you're likely to get
squished - marketing matters more than quality. If you choose to go
that road, I wish you the best of luck.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.." 

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Lincoln Dale]

Andre,

At 07:38 PM 5/01/2003 -0800, Andre Hedrick wrote:
>If you know anything about iSCSI RFC draft and how storage truly works.
>Cisco gets it wrong, they do not believe in supporting the full RFC.

i can tell you that you're mistaken in your belief.

[..]
>Next try to support any filesystem regardless of platform.
>Specifically anything Microsoft does to thwart Linux, I have already
>covered.

you seem to miss the basic fact that iSCSI is a block-layer 
transport.  file system != block layer.
supporting any filesystem with iSCSI is trivial - its just the same as 
supporting any filesystem on any other block device.

[..]
>In two week I will have NetBSD certified, and 4 weeks later should have
>Solaris certifed.

we certainly don't care about any "certifications" you have for NetBSD or 
solaris.

if you wish to discuss the various merits of parts of the iSCSI protocol, 
there are forums for that kind of thing.
linux-kernel is not one of them.


cheers,

lincoln.

Re: Linux iSCSI Initiator

[Andre Hedrick]

On Mon, 6 Jan 2003, Lincoln Dale wrote:

> Andre,
> 
> At 07:38 PM 5/01/2003 -0800, Andre Hedrick wrote:
> >If you know anything about iSCSI RFC draft and how storage truly works.
> >Cisco gets it wrong, they do not believe in supporting the full RFC.
> 
> i can tell you that you're mistaken in your belief.

So Cisco now will turn on and leave the Header and Data Digests on, this
differs from your last initiator test.

> [..]
> >Next try to support any filesystem regardless of platform.
> >Specifically anything Microsoft does to thwart Linux, I have already
> >covered.
> 
> you seem to miss the basic fact that iSCSI is a block-layer 
> transport.  file system != block layer.
> supporting any filesystem with iSCSI is trivial - its just the same as 
> supporting any filesystem on any other block device.

No, you just lost the context of Oliver's question about doing it all from
user space.  Whereas it could be suggested "a target" drops in to the
respective builting FS support.  Why, because attempting to bypass the
double memcpy to/from user/kernel space.

Since it is a pure "block-transport", and execution the raw data transport
to the spindle, the stack does not care about anything about.  Block is a
bit-bucket as what defines SAN.

> [..]
> >In two week I will have NetBSD certified, and 4 weeks later should have
> >Solaris certifed.
> 
> we certainly don't care about any "certifications" you have for NetBSD or 
> solaris.

As you should not, these are data transport QAQC for the respective
"targets".  Initiators make no money, but "targets" do.

> if you wish to discuss the various merits of parts of the iSCSI protocol, 
> there are forums for that kind of thing.
> linux-kernel is not one of them.

Yep, will see you that refector to discuss a hole in the ERL=1 with a
header digest failure w/ the S_BIT set.  It get tossed out and now you end
up with a status sequuence number order problem.

Who owns the mess, Target or Initiator ?

I do not care, because I have both sides covered to support all 11:11
regardless of the support level when talking to your switch with my
initiator, or your initiator talking to my target.

Cheers,

Andre Hedrick
LAD Storage Consulting Group

Re: Linux iSCSI Initiator

[Lincoln Dale]

At 11:53 PM 5/01/2003 -0800, Andre Hedrick wrote:
[..]

as a discussion about various merits of parts of the iSCSI protocol are not 
relevant to linux-kernel, i've taken the discussion(sic) off-list.

see my private reply.


cheers,

lincoln.

Re: Linux iSCSI Initiator

[Andre Hedrick]

On Mon, 6 Jan 2003, Lincoln Dale wrote:

> At 11:53 PM 5/01/2003 -0800, Andre Hedrick wrote:
> [..]
> 
> as a discussion about various merits of parts of the iSCSI protocol are not 
> relevant to linux-kernel, i've taken the discussion(sic) off-list.
> 
> see my private reply.

it is now way past time and over due to kill the battle.
It is a little less gray, and a little more clear.

I guess I will hang on a little longer with the old hat, while figuring
out what to do with a new one.

Oh and you guys who emailed, got me with the guilt trip to go back to my
old directory.  I do not know how, or I am starting to age into a softy
but I am back.  Where was Bart?  I would have dumped on his plate at the
time.

Regards,

Andre Hedrick
LAD Storage Consulting Group

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andrew McGregor]

It's a common large-vendor tactic to create bizzarre specs, and the IETF 
gets sidetracked that way very easily (I'm not blaming Cisco or anyone in 
particular, I am too busy at IETFs to track the politics in iSCSI). 
Andre's showing some of the difficulty of implementing any IETF spec; they 
are very incomplete.  You can't write an IP stack purely from the spec and 
get anywhere.

Also, in the case of something like iSCSI, you get two kinds of early 
implementations; experimental (in this case Cisco) and production (Andre, 
and presumably Microsoft).  Often the quality of the experimental 
implementations is awful, but nevertheless they become the reference. The 
community has recently been able to produce production quality code faster 
than the experimenters (anyone remember the first Halloween memo talking 
about how long it would take to implement WebDAV?  There was a complete, 
production opensource implementation available three days after the release 
of the memo, about a year ahead of MS).  Looks like Andre's done that with 
iSCSI.

Andrew

--On Sunday, January 05, 2003 19:38:15 -0800 Andre Hedrick 
<andre@pyxtechnologies.com> wrote:

> On Sun, 5 Jan 2003, Oliver Xymoron wrote:
>
>> On Sat, Jan 04, 2003 at 06:47:43PM -0800, Andrew Morton wrote:
>> > Andre Hedrick wrote:
>> > >
>> > > Rik and Richard,
>> > >
>> > > As you see, I in good faith prior to this holy war, had initiated a
>> > > formal request include a new protocol into the Linux kernel prior to
>> > > the freeze. The extention was requested to insure the product was of
>> > > the highest quality and not limited with excessive erratium as the
>> > > ratification of the IETF modified, postponed, and delayed ;
>> > > regardless of reason.
>> > >
>> > > Obviously, PyX had (has) on its schedule to product a high quality
>> > > target which is transport independent on each side of the protocol.
>> > > We are not sure of this position because of the uncertain nature of
>> > > the basic usages of headers and export_symbols.
>> > >
>> >
>> > I suggest that if a function happens to be implemented as an inline
>> > in a header then it should be treated (for licensing purposes) as
>> > an exported-to-all-modules symbol.  So in Linux, that would be
>> > LGPL-ish.
>> >
>> > The fact that a piece of kernel functionality happens to be inlined
>> > is a pure technical detail of linkage.
>> >
>> > If there really is inlined functionality which we do not wish made
>> > available to non-GPL modules then it should be either uninlined and
>> > not exported or it should be wrapped in #ifdef GPL.
>>
>> More pragmatically, who cares? There's already at least one vendor
>> (Cisco) who ships a perfectly good fully GPLed iSCSI initiator module
>> that doesn't need to touch any core code. It's already the benchmark
>> for compatibility at interoperability tests. And it's following the
>> IETF drafts closely too. Once we actually have an iSCSI RFC, it might
>> be worth pulling it into the kernel tree. I believe Red Hat is
>> shipping it some form already.
>
> If you know anything about iSCSI RFC draft and how storage truly works.
> Cisco gets it wrong, they do not believe in supporting the full RFC.
> So you get ERL=0, and now they turned of the "Header and Data Digests",
> this is equal to turning off the iCRC in ATA, or CRC in SCSI between the
> controller and the device.  For those people who think removing the
> checksum test for the integrity of the data and command operations, you
> get what you deserve.
>
>> That leaves the question of using Linux as an iSCSI target, and I've
>> yet to see any reason why this couldn't be done in userspace. In fact,
>> in a lot of ways that's the right thing to do as it lets you take
>> proper advantage of MD/LVM/EVMS/crypto, etc..
>
> You go right ahead, and see if you can move at near wire speed.
> Next try to support any filesystem regardless of platform.
> Specifically anything Microsoft does to thwart Linux, I have already
> covered.
>
>> There are a few other free implementations out there too.
>
> Please go use them and in two seconds my product can bring them all to the
> ground with the full error injection tool kit from both sides.  My team
> has gone through supporting every optional feature in the RFC draft as
> manditory to remove any possible vendor unique opportunities.
>
> There are grey areas in the RFC draft to support every corner case of
> ERL=1 and ERL=2.
>
> You figure out how to support the marker stream to perform a
> Sync-and-Steering layer.
>
> PyX is the second in the world to support Sync-and-Steering, and the first
> to do it software only.
>
> Please go for it, and you will spend at least 18-24 months to develop.
>
> The target(erl=0) is what would be the second phase to open source, but I
> see you and other want to do the hard way and that is fine.
>
> In two week I will have NetBSD certified, and 4 weeks later should have
> Solaris certifed.
>
> Cheers,
>
> Andre Hedrick, CTO & Founder
> iSCSI Software Solutions Provider
> http://www.PyXTechnologies.com/
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
>

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Roman Zippel]

Hi,

> If you know anything about iSCSI RFC draft and how storage truly works.
> Cisco gets it wrong, they do not believe in supporting the full RFC.
> So you get ERL=0, and now they turned of the "Header and Data Digests",
> this is equal to turning off the iCRC in ATA, or CRC in SCSI between the
> controller and the device.  For those people who think removing the
> checksum test for the integrity of the data and command operations, you
> get what you deserve.

Ever heard of TCP checksums? Ever heard of ethernet checksums? Which
transport doesn't use checksums nowadays? The digest makes only sense if
you can generate it for free in hardware or for debugging, otherwise
it's only a waste of cpu time. This makes the complete ERL 1 irrelevant
for a software implementation. With block devices you can even get away
with just ERL 0 to implement transparent recovery.

bye, Roman

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Richard Stallman]

    Can you admit the follow, that GPL has everything to control
    redistribution, and has ZERO context for copyright.

It is not clear what those words mean, so I won't agree or disagree.

      The holders of the
    copyright control the issues.

That is certainly true.  The copyright holders of the code can permit
whatever they wish to permit, for that code.  The copyright holders of
Linux can permit whatever they wish to permit, for Linux.  I've said
this many times.  The last occasion was in the message that you just
responded to:

> The Linux developers can certainly do this, if the copyright holders
> of the substantial functions in question go along with it.

When I say X and you respond by demanding angrily that I agree to X, I
have to think we're failing to communicate.

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andre Hedrick]

On Mon, 6 Jan 2003, Roman Zippel wrote:

> Hi,
> 
> > If you know anything about iSCSI RFC draft and how storage truly works.
> > Cisco gets it wrong, they do not believe in supporting the full RFC.
> > So you get ERL=0, and now they turned of the "Header and Data Digests",
> > this is equal to turning off the iCRC in ATA, or CRC in SCSI between the
> > controller and the device.  For those people who think removing the
> > checksum test for the integrity of the data and command operations, you
> > get what you deserve.
> 
> Ever heard of TCP checksums? Ever heard of ethernet checksums? Which
> transport doesn't use checksums nowadays? The digest makes only sense if
> you can generate it for free in hardware or for debugging, otherwise
> it's only a waste of cpu time. This makes the complete ERL 1 irrelevant
> for a software implementation. With block devices you can even get away
> with just ERL 0 to implement transparent recovery.

Please continue to think of TCP checksums as valid for a data transport,
you data will be gone soon enough.

Initiator == Controller
Target == Disk
iSCSI == cable or ribbon

Please turn off the CRC on your disk drive and see if you still have data.

Cheers,

Andre Hedrick, CTO & Founder 
iSCSI Software Solutions Provider
http://www.PyXTechnologies.com/

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andrew McGregor]

Hmm.  The problem here is that there is a nontrivial probability that a 
packet can pass both ethernet and TCP checksums and still not be right, 
given the gigantic volumes of data that iSCSI is intended to be used with. 
Back up a 100 terabyte array and it's more than 1%, back of the envelope.

Ethernet and TCP were both designed to be cheap to evaluate, not the 
absolute last word in integrity.  There is a move underway to provide an 
optional stronger TCP digest for IPv6, and if used with that then there is 
no need for the iSCSI digest.  Otherwise, well, play dice with the data. 
Loaded in your favour, but still dice.

Andrew

--On Monday, January 06, 2003 17:51:13 +0100 Roman Zippel 
<zippel@linux-m68k.org> wrote:

> Hi,
>
>> If you know anything about iSCSI RFC draft and how storage truly works.
>> Cisco gets it wrong, they do not believe in supporting the full RFC.
>> So you get ERL=0, and now they turned of the "Header and Data Digests",
>> this is equal to turning off the iCRC in ATA, or CRC in SCSI between the
>> controller and the device.  For those people who think removing the
>> checksum test for the integrity of the data and command operations, you
>> get what you deserve.
>
> Ever heard of TCP checksums? Ever heard of ethernet checksums? Which
> transport doesn't use checksums nowadays? The digest makes only sense if
> you can generate it for free in hardware or for debugging, otherwise
> it's only a waste of cpu time. This makes the complete ERL 1 irrelevant
> for a software implementation. With block devices you can even get away
> with just ERL 0 to implement transparent recovery.
>
> bye, Roman
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
>

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Oliver Xymoron]

On Tue, Jan 07, 2003 at 01:39:38PM +1300, Andrew McGregor wrote:
> Hmm.  The problem here is that there is a nontrivial probability that a 
> packet can pass both ethernet and TCP checksums and still not be right, 
> given the gigantic volumes of data that iSCSI is intended to be used with. 
> Back up a 100 terabyte array and it's more than 1%, back of the envelope.

What was the underlying error rate and distribution you assumed? I
figure if it were high enough to get to your 1%, you'd have such high
retry rates (and resulting throughput loss) that the operator would
notice his LAN was broken weeks before said transfer completed.
 
-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.." 

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]

On Mon, 06 Jan 2003 22:20:46 CST, Oliver Xymoron said:

> What was the underlying error rate and distribution you assumed? I
> figure if it were high enough to get to your 1%, you'd have such high
> retry rates (and resulting throughput loss) that the operator would
> notice his LAN was broken weeks before said transfer completed.

The average ISP wouldn't notice things were broken unless enough magic
smoke escaped to cause a Halon dump.

Consider as evidence the following NANOG presentation:

http://www.nanog.org/mtg-0210/wessels.html

Some *98* percent of all queries at one of the root nameservers over a 24-hour
period were broken in some way.  And there wasn't even a DDoS in progress
at the time...

Also, I think Andrew was computing the chances that *SOME* packet in the
100T would be mangled in an undetected fashion, so 99% of the time all 100T
would be OK, but 1% of the time there was some subtle block mangling some
dozens of terabytes into the transfer.  Given that the TCP slow-start code
is currently busticated for gigabit and higher (it takes *hours* without a
packet drop to get the window open *all* the way - there's IETF drafts
in process about this), it's quite possible that you'd not notice packet
drops due to error among all the congestion drops kicking the window size
down.....
-- 
				Valdis Kletnieks
				Computer Systems Senior Engineer
				Virginia Tech


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Werner Almesberger]

Valdis.Kletnieks@vt.edu wrote:
> is currently busticated for gigabit and higher (it takes *hours* without a
> packet drop to get the window open *all* the way

Don't use 2.0.21 for gigabit traffic :-)

(2.0.21 and earlier initialized sstresh to zero, which would indeed
cause the behaviour you're describing.)

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 2637 bytes --]

On Tue, 07 Jan 2003 03:16:38 -0300, Werner Almesberger said:
> Valdis.Kletnieks@vt.edu wrote:
> > is currently busticated for gigabit and higher (it takes *hours* without a
> > packet drop to get the window open *all* the way
> 
> Don't use 2.0.21 for gigabit traffic :-)
> 
> (2.0.21 and earlier initialized sstresh to zero, which would indeed
> cause the behaviour you're describing.)

That's not the problem. The problem is that TCP slow-start itself (and some of
the related congestion control stuff) has some issues scaling to the very high
end.  Floyd (of RFC3168 fame) has done some work in the area:

from http://www.ietf.org/internet-drafts/draft-floyd-tcp-highspeed-01.txt

Abstract

   This document proposes HighSpeed TCP, a modification to TCP's
   congestion control mechanism for use with TCP connections with large
   congestion windows.  The congestion control mechanisms of the current
   Standard TCP constrains the congestion windows that can be achieved
   by TCP in realistic environments.  For example, for a Standard TCP
   connection with 1500-byte packets and a 100 ms round-trip time,
   achieving a steady-state throughput of 10 Gbps would require an
   average congestion window of 83,333 segments, and a packet drop rate
   of at most one congestion event every 5,000,000,000 packet (or
   equivalently, at most one congestion event every 1 2/3 hours).  This
   is widely acknowledged as an unrealistic constraint.  To address this
   limitation of TCP, this document proposes HighSpeed TCP, and solicits
   experimentation and feedback from the wider community.

Also http://www.ietf.org/internet-drafts/draft-floyd-tcp-slowstart-01.txt

Abstract

   This note proposes a modification for TCP's slow-start for use with
   TCP connections with large congestion windows.  For TCP connections
   that are able to use congestion windows of thousands (or tens of
   thousands) of MSS-sized segments (for MSS the sender's MAXIMUM
   SEGMENT SIZE), the current slow-start procedure can result in
   increasing the congestion window by thousands of segments in a single
   round-trip time.  Such an increase can easily result in thousands of
   packets being dropped in one round-trip time.  This is often counter-
   productive for the TCP flow itself, and is also hard on the rest of
   the traffic sharing the congested link.  This note proposes Limited
   Slow-Start as an optional mechanism for limiting the number of
   segments by which the congestion window is increased for one window
   of data during slow-start, in order to improve performance for TCP
   connections with large congestion windows.





[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Lincoln Dale]

At 12:38 AM 7/01/2003 -0500, Valdis.Kletnieks@vt.edu wrote:
> > What was the underlying error rate and distribution you assumed? I
> > figure if it were high enough to get to your 1%, you'd have such high
> > retry rates (and resulting throughput loss) that the operator would
> > notice his LAN was broken weeks before said transfer completed.
>
>The average ISP wouldn't notice things were broken unless enough magic
>smoke escaped to cause a Halon dump.
>
>Consider as evidence the following NANOG presentation:
>http://www.nanog.org/mtg-0210/wessels.html
>
>Some *98* percent of all queries at one of the root nameservers over a 24-hour
>period were broken in some way.

please don't confuse issues.
i think you just epitomized the quote: "there are lies, damn lies, and 
statistics".

you're trying to say that because there is some broken/buggy nameserver 
code out there, it means that the error-rate for TCP is correct?


cheers,

lincoln.

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1122 bytes --]

On Tue, 07 Jan 2003 17:45:03 +1100, Lincoln Dale said:
> At 12:38 AM 7/01/2003 -0500, Valdis.Kletnieks@vt.edu wrote:
> > > What was the underlying error rate and distribution you assumed? I
> > > figure if it were high enough to get to your 1%, you'd have such high
> > > retry rates (and resulting throughput loss) that the operator would
> > > notice his LAN was broken weeks before said transfer completed.
> >
> >The average ISP wouldn't notice things were broken unless enough magic
> >smoke escaped to cause a Halon dump.
> >
> >Consider as evidence the following NANOG presentation:
> >http://www.nanog.org/mtg-0210/wessels.html
> >
> >Some *98* percent of all queries at one of the root nameservers over a 24-ho
ur
> >period were broken in some way.
> 
> please don't confuse issues.
> i think you just epitomized the quote: "there are lies, damn lies, and 
> statistics".
> 
> you're trying to say that because there is some broken/buggy nameserver 
> code out there, it means that the error-rate for TCP is correct?

No, I'm saying the assertion that "the operator would notice his LAN was broken"
is incorrect.

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Werner Almesberger]

Valdis.Kletnieks@vt.edu wrote:
> That's not the problem. The problem is that TCP slow-start itself (and some of
> the related congestion control stuff) has some issues scaling to the very high
> end.

I'm very well aware of that ;-) But what you wrote was:

> it takes *hours* without a
> packet drop to get the window open *all* the way

Or did you mean "after" instead of "without" ? Or maybe "into
equilibrium" instead of "the window open ..." ? (After all, the
window isn't only open, but it's been blown off its hinges.)

In any case, your statement accurately describes a somewhat
surprising quirk in Linux TCP performance as of only a bit more
than six years ago :)

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1389 bytes --]

On Tue, 07 Jan 2003 04:08:29 -0300, Werner Almesberger said:
> Valdis.Kletnieks@vt.edu wrote:

> > it takes *hours* without a
> > packet drop to get the window open *all* the way
> 
> Or did you mean "after" instead of "without" ? Or maybe "into
> equilibrium" instead of "the window open ..." ? (After all, the
> window isn't only open, but it's been blown off its hinges.)

"without".  Let's say it takes 4 hours to recover from a drop, and
you have another one 3 hours into recovery - it will now take more than
one more hour to recover.

"into equilibrium fully open".  It's easy enough to see it in equilibrium
(more or less) not fully open.. ;)

> In any case, your statement accurately describes a somewhat
> surprising quirk in Linux TCP performance as of only a bit more
> than six years ago :)

OK, I tuned in late - are you saying that the 6-year-old Linux quirk
happened to have the same symptoms as Floyd's current work, or that
the slow-start tweaks were designed in 6 years ago, or that a fix for
the quirk accidentally did the same thing as Floyd's stuff?

The whole slow-start/ack/retransmit has been chewed over so many times in the
last 20 years that it's hard to keep track of which vendors picked up which
tweaks when, and which vendors accidentally invented them again, and which
vendors invented the tweaks independently and didn't publicize them more....

/Valdis


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Werner Almesberger]

Valdis.Kletnieks@vt.edu wrote:
> "without".  Let's say it takes 4 hours to recover from a drop, and
> you have another one 3 hours into recovery - it will now take more than
> one more hour to recover.

Ah, now I understand what you meant. I read that as "even if there is
no drop at all, it can take hours".

> The whole slow-start/ack/retransmit has been chewed over so many times in the
> last 20 years that it's hard to keep track of which vendors picked up which
> tweaks when, and which vendors accidentally invented them again, and which
> vendors invented the tweaks independently and didn't publicize them more....

... or figure out which combination of RFCs, I-Ds, and ad hoc genius
makes up Linux TCP, yes ;-)

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 813 bytes --]

On Tue, 07 Jan 2003 05:14:41 -0300, Werner Almesberger said:

> ... or figure out which combination of RFCs, I-Ds, and ad hoc genius
> makes up Linux TCP, yes ;-)

That's the easy part.  :)  You then get to take a cross-product of that and
its interactions with whatever combination of RFC/I-D/ad-crockery are in
the 4 different Solaris releases, the 2 or 3 AIX releases, the Tru64 releases,
the myriad different Microsoft-based servers - and that's just the 10K square
feet in our machine room.  A lot of our gear keeps trying to talk to the
outside world, where ALL bets are off. ;)

Does anybody think it would be worthwhile to collate a document of which
RFCs/I-Ds are supported, and to what extent (the MUST/SHOULD/MAY stuff)?
Or is there one already and my 3:30AM search for same is missing it? ;)

/Valdis

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Olivier Galibert]

On Tue, Jan 07, 2003 at 01:39:38PM +1300, Andrew McGregor wrote:
> Ethernet and TCP were both designed to be cheap to evaluate, not the 
> absolute last word in integrity.  There is a move underway to provide an 
> optional stronger TCP digest for IPv6, and if used with that then there is 
> no need for the iSCSI digest.  Otherwise, well, play dice with the data. 
> Loaded in your favour, but still dice.

Ethernet's checksum is a standard crc32, with all the usual good
properties and, at least on FE and lower, 1500bytes max of payload.
So it's quite reasonable.  TCP's checksum, though, is crap.

I'm not entirely sure how crc32 would behave on jumbo frames.

  OG.

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andrew McGregor]

No no, that's a 1% chance that one packet in the terabyte is broken.

But actually, it's not that hard to construct a peturbation to the packet 
that will beat both the ethernet and TCP checksums (I gave an example that 
beats TCP before).  That kind of change is not likely for random bit 
errors, but is quite likely to occur in just slightly marginal hardware. 
Partial packet duplication or byte reordering on the highly ordered data 
patterns you find in filesystem metadata could be really bad.

Like I say, debugging one crypto protocol I've seen this happen for real. 
Twice in about 10000 packets, on an otherwise apparently perfectly fine 
LAN.  I suspect bad cabling, and changed it, but it's hard to tell that 
anything has changed.  That shows that my 1% is probably quite conservative 
for that particular link.

Internet protocols (changing to IETF hat now) are supposed to work on the 
global internet, and that means iSCSI has to be engineered to work on the 
worst links imaginable, because sometime, somewhere, someone's data is 
going to cross a really broken backup link that they have no way of knowing 
has just come on.  Possibly it's wireless, where packet corruption due to 
undetected collisions happens quite frequently.

Andre routinely tests it with the IBM team in Israel, with his end in 
California.

Andrew

--On Monday, January 06, 2003 22:20:46 -0600 Oliver Xymoron 
<oxymoron@waste.org> wrote:

> On Tue, Jan 07, 2003 at 01:39:38PM +1300, Andrew McGregor wrote:
>> Hmm.  The problem here is that there is a nontrivial probability that a
>> packet can pass both ethernet and TCP checksums and still not be right,
>> given the gigantic volumes of data that iSCSI is intended to be used
>> with.  Back up a 100 terabyte array and it's more than 1%, back of the
>> envelope.
>
> What was the underlying error rate and distribution you assumed? I
> figure if it were high enough to get to your 1%, you'd have such high
> retry rates (and resulting throughput loss) that the operator would
> notice his LAN was broken weeks before said transfer completed.
>
> --
>  "Love the dolphins," she advised him. "Write by W.A.S.T.E.."
>
>

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andrew McGregor]

No, he really meant without.  I don't know if Valdis saw the presentation 
that went with that draft, but I did (last IETF in Yokohama).  The example 
was a 10Gbps link with a 250ms RTT (plausibly, a trans-pacific cable 
lambda).  There are tens of thousands of 9k packets in the window (yep, 100 
*megabytes* in flight in the cable!), and it does take several hours with 
exactly zero drops to get the connection to fill the 10Gbps.  After one 
dropped packet, it can take an hour to get back to full utilisation.  The 
graphs are really startling to look at :-)

That quirk just meant the numbers were off by a few orders of magnitude.

If anyone wants to look at that further, I think there are URLs in the 
draft.  If not, I can dig them out of the proceedings.

Andrew

--On Tuesday, January 07, 2003 04:08:29 -0300 Werner Almesberger 
<wa@almesberger.net> wrote:

> Valdis.Kletnieks@vt.edu wrote:
>> That's not the problem. The problem is that TCP slow-start itself (and
>> some of the related congestion control stuff) has some issues scaling to
>> the very high end.
>
> I'm very well aware of that ;-) But what you wrote was:
>
>> it takes *hours* without a
>> packet drop to get the window open *all* the way
>
> Or did you mean "after" instead of "without" ? Or maybe "into
> equilibrium" instead of "the window open ..." ? (After all, the
> window isn't only open, but it's been blown off its hinges.)
>
> In any case, your statement accurately describes a somewhat
> surprising quirk in Linux TCP performance as of only a bit more
> than six years ago :)
>
> - Werner
>
> --
>
> _________________________________________________________________________
>  / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net
> /
> /_http://www.almesberger.net/____________________________________________/
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>
>

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Alan Cox]

On Tue, 2003-01-07 at 00:39, Andrew McGregor wrote:
> optional stronger TCP digest for IPv6, and if used with that then there is 
> no need for the iSCSI digest.  Otherwise, well, play dice with the data. 
> Loaded in your favour, but still dice.

There is no need for the iSCSI digest anyway. You can use IP-AH to achieve 
precisely the same thing, and strong guarantees already in a standards 
compliant manner.

Alan

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andrew McGregor]

Or ESP, with or without encryption as well.

But that does not acheive quite the same thing, because the iSCSI digest is 
another lightweight checksum, albeit stronger than most, and does not 
provide authentication.  So AH or ESP is stronger, but slower.

Maybe Cisco are assuming another layer deals with the errors.  However, to 
get an interoperable and efficient implementation requires the capability 
to do whatever combination is required, along with sensible defaults.

Andrew

--On Tuesday, January 07, 2003 12:31:18 +0000 Alan Cox 
<alan@lxorguk.ukuu.org.uk> wrote:

> On Tue, 2003-01-07 at 00:39, Andrew McGregor wrote:
>> optional stronger TCP digest for IPv6, and if used with that then there
>> is  no need for the iSCSI digest.  Otherwise, well, play dice with the
>> data.  Loaded in your favour, but still dice.
>
> There is no need for the iSCSI digest anyway. You can use IP-AH to
> achieve  precisely the same thing, and strong guarantees already in a
> standards  compliant manner.
>
> Alan
>
>

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Lionel Bouton]

On lun, jan 06, 2003 at 05:51:13 +0100, Roman Zippel wrote:
> Hi,
> 
> > If you know anything about iSCSI RFC draft and how storage truly works.
> > Cisco gets it wrong, they do not believe in supporting the full RFC.
> > So you get ERL=0, and now they turned of the "Header and Data Digests",
> > this is equal to turning off the iCRC in ATA, or CRC in SCSI between the
> > controller and the device.  For those people who think removing the
> > checksum test for the integrity of the data and command operations, you
> > get what you deserve.
> 
> Ever heard of TCP checksums? Ever heard of ethernet checksums? Which
> transport doesn't use checksums nowadays? The digest makes only sense if
> you can generate it for free in hardware or for debugging, otherwise
> it's only a waste of cpu time. This makes the complete ERL 1 irrelevant
> for a software implementation. With block devices you can even get away
> with just ERL 0 to implement transparent recovery.
>

Some others already stated that TCP checksums aren't reliable enough.
I'll add to these remarks the following :

2 years ago we installed a Linux based VPN for one of our customers. We got
anomaly reports where the symptoms were "SMB transfers died unexpectedly".
We suspected a bug in the Windows 2000 SMB client code that talked to the
Samba server accross the VPN link (very large files over slow and long path,
this was not exactly the usual environment for an SMB client code thought
for the LAN) until I tested scp transfers between the two routers.

Scheduling scp transfers/md5 checks during a whole day revealed that some
files were corrupted.
We guessed that one router along the path recomputed TCP checksums unconditionnaly
(even if the packets arrived corrupted) and this was confirmed by other
customers of the same ISP.

Next time we will use a more robust VPN than vtun :)

If you want to use iSCSI in an uncontrolled environment you can't trust the
data transport. This is sad, but many vendors violates RFCs on a regular
basis.

I've even learned recently that one vendor developped quite some time ago a
TCP stack that pre-ACKed packets in order to speed up transfers !
As you can guess they didn't have much success out of their labs...

LB

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Alan Cox]

On Tue, 2003-01-07 at 12:31, Andrew McGregor wrote:
> Or ESP, with or without encryption as well.
> 
> But that does not acheive quite the same thing, because the iSCSI digest is 
> another lightweight checksum, albeit stronger than most, and does not 
> provide authentication.  So AH or ESP is stronger, but slower.

AH permits multiple digests, they also happen to correspond to the hardware
accelerated ones on things like the 3c990...

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Oliver Xymoron]

On Wed, Jan 08, 2003 at 01:31:36AM +1300, Andrew McGregor wrote:
> Or ESP, with or without encryption as well.
> 
> But that does not acheive quite the same thing, because the iSCSI digest is 
> another lightweight checksum, albeit stronger than most, and does not 
> provide authentication.  So AH or ESP is stronger, but slower.
> 
> Maybe Cisco are assuming another layer deals with the errors.  However, to 
> get an interoperable and efficient implementation requires the capability 
> to do whatever combination is required, along with sensible defaults.

Actually, as I pointed out before, the current Cisco iSCSI driver does
support CRC (32-bit), though it's presumably off by default.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.." 

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Werner Almesberger]

Valdis.Kletnieks@vt.edu wrote:
> Does anybody think it would be worthwhile to collate a document of which
> RFCs/I-Ds are supported, and to what extent (the MUST/SHOULD/MAY stuff)?
> Or is there one already and my 3:30AM search for same is missing it? ;)

Well, every once in a while, that accumulated wisdom gets
summarized and written down in some RFC. Of course, it's been a
while since e.g. RFC2581 ...

What might be useful at some point, is to document Linux TCP, and
how it is linked to all the RFCs, I-Ds, research papers, etc.

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Roman Zippel]

Hi,

Andre Hedrick wrote:

> Please continue to think of TCP checksums as valid for a data transport,
> you data will be gone soon enough.
> 
> Initiator == Controller
> Target == Disk
> iSCSI == cable or ribbon
> 
> Please turn off the CRC on your disk drive and see if you still have data.

This maybe works as PR, but otherwise it's crap.
With a network protocol you have multiple possibilities to increase the
reliability. The lower you do it in the network layer the easier is it
to put it into hardware and to optimize it and the more generically it's
usable. Doing it in the protocol is only the last resort. The iSCSI
protocol is a nice protocol - if you ignore all the crap the hardware
vendors put in (that stuff only makes sense if you want to produce ultra
cheap hardware).

bye, Roman

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andre Hedrick]

On Tue, 7 Jan 2003, Roman Zippel wrote:

> Hi,
> 
> Andre Hedrick wrote:
> 
> > Please continue to think of TCP checksums as valid for a data transport,
> > you data will be gone soon enough.
> > 
> > Initiator == Controller
> > Target == Disk
> > iSCSI == cable or ribbon
> > 
> > Please turn off the CRC on your disk drive and see if you still have data.
> 
> This maybe works as PR, but otherwise it's crap.

So, please turn off the CRC's in your onboard storage today and see how
long it lasts.

> With a network protocol you have multiple possibilities to increase the
> reliability. The lower you do it in the network layer the easier is it
> to put it into hardware and to optimize it and the more generically it's
> usable. Doing it in the protocol is only the last resort. The iSCSI
> protocol is a nice protocol - if you ignore all the crap the hardware
> vendors put in (that stuff only makes sense if you want to produce ultra
> cheap hardware).

I will be happy to see everyone turn off the CRC's on the data and headers
on their products or the open sources ones which fail to follow the rules.
I am well away of everyones contempt for standards.

Cheers,

Andre Hedrick, CTO & Founder 
iSCSI Software Solutions Provider
http://www.PyXTechnologies.com/

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andrew McGregor]

Hardware acceleration is the right way to do any of this, agreed :-)

--On Tuesday, January 07, 2003 13:58:50 +0000 Alan Cox 
<alan@lxorguk.ukuu.org.uk> wrote:

> On Tue, 2003-01-07 at 12:31, Andrew McGregor wrote:
>> Or ESP, with or without encryption as well.
>>
>> But that does not acheive quite the same thing, because the iSCSI digest
>> is  another lightweight checksum, albeit stronger than most, and does
>> not  provide authentication.  So AH or ESP is stronger, but slower.
>
> AH permits multiple digests, they also happen to correspond to the
> hardware accelerated ones on things like the 3c990...
>
>

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Roman Zippel]

Hi,

Andre Hedrick wrote:

> > > Please continue to think of TCP checksums as valid for a data transport,
> > > you data will be gone soon enough.
> > >
> > > Initiator == Controller
> > > Target == Disk
> > > iSCSI == cable or ribbon
> > >
> > > Please turn off the CRC on your disk drive and see if you still have data.
> >
> > This maybe works as PR, but otherwise it's crap.
> 
> So, please turn off the CRC's in your onboard storage today and see how
> long it lasts.

If you want to compare apples with apples, you should rather tell me how
I turn off the checksumming of my nic.

> > With a network protocol you have multiple possibilities to increase the
> > reliability. The lower you do it in the network layer the easier is it
> > to put it into hardware and to optimize it and the more generically it's
> > usable. Doing it in the protocol is only the last resort. The iSCSI
> > protocol is a nice protocol - if you ignore all the crap the hardware
> > vendors put in (that stuff only makes sense if you want to produce ultra
> > cheap hardware).
> 
> I will be happy to see everyone turn off the CRC's on the data and headers
> on their products or the open sources ones which fail to follow the rules.
> I am well away of everyones contempt for standards.

You know RFC2119?

bye, Roman

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[walt]

Andre Hedrick wrote:

> Richard,
>
> Can you admit the follow, that GPL has everything to control
> redistribution, and has ZERO context for copyright.  The holders of the
> copyright control the issues.

RMS didn't understand this paragraph, and neither did I.  Would you
please clarify?

> See your whole hook is "Derivative Works" well, I implimented a protocol.
> It works regardless of platform or OS.  All it uses are simple and
> standard kernel services.


If you did this because of RMS or his GPL then I think all of us owe him
a big 'thank you'.

Andre, I've been following this list for at least two years.  Although I
contribute nothing except an occasional bug report I care about what
happens here and I and I care about the people who do contribute.  I know
that includes you -- in a big way.

Please let me make some observations:

1) I admire anyone who speaks more than one language.  I suspect that
    includes you.

2) Much of the time I don't know what the hell you're talking about.
    (I'll bet you a Porsche that I'm not the only one ;-)

3) You'll make back your expenses a lot faster if you'll get rid of
    the Porsche.

4) I thank you for all your contributions to Open Source.  I benefit
    from your work every day and I want you to know that I appreciate it.

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Larry McVoy]

On Wed, Jan 08, 2003 at 01:43:01AM +0000, Alan Cox wrote:
> On Wed, 2003-01-08 at 00:04, Roman Zippel wrote:
> > If you want to compare apples with apples, you should rather tell me how
> > I turn off the checksumming of my nic.
> 
> For some cards you can do this. For instructive information on the effects 
> look at the saga of sunos 3.x and NFS over wans. Old SunOS turned off UDP
> checksums for NFS. It provided an adequate demonstration that UDP checksums
> for NFS are needed. Sun of course addressed this design error long long 
> ago.

BK has a pretty lame checksum but good enough to catch a lot of errors
and we still catch 'em.  Software, hardware, network, whatever, they
happen all the time.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Alan Cox]

On Wed, 2003-01-08 at 00:04, Roman Zippel wrote:
> If you want to compare apples with apples, you should rather tell me how
> I turn off the checksumming of my nic.

For some cards you can do this. For instructive information on the effects 
look at the saga of sunos 3.x and NFS over wans. Old SunOS turned off UDP
checksums for NFS. It provided an adequate demonstration that UDP checksums
for NFS are needed. Sun of course addressed this design error long long 
ago.

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Vojtech Pavlik]

On Tue, Jan 07, 2003 at 02:45:51PM -0800, Andre Hedrick wrote:

> > Andre Hedrick wrote:
> > 
> > > Please continue to think of TCP checksums as valid for a data transport,
> > > you data will be gone soon enough.
> > > 
> > > Initiator == Controller
> > > Target == Disk
> > > iSCSI == cable or ribbon
> > > 
> > > Please turn off the CRC on your disk drive and see if you still have data.
> > 
> > This maybe works as PR, but otherwise it's crap.
> 
> So, please turn off the CRC's in your onboard storage today and see how
> long it lasts.

1) Bad comparison.

2) It'd last very very long, because I never get a CRC error anyway.

-- 
Vojtech Pavlik
SuSE Labs

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[H. Peter Anvin]

Followup to:  <20030107053146.A16578@kerberos.ncsl.nist.gov>
By author:    Olivier Galibert <galibert@pobox.com>
In newsgroup: linux.dev.kernel
>
> On Tue, Jan 07, 2003 at 01:39:38PM +1300, Andrew McGregor wrote:
> > Ethernet and TCP were both designed to be cheap to evaluate, not the 
> > absolute last word in integrity.  There is a move underway to provide an 
> > optional stronger TCP digest for IPv6, and if used with that then there is 
> > no need for the iSCSI digest.  Otherwise, well, play dice with the data. 
> > Loaded in your favour, but still dice.
> 
> Ethernet's checksum is a standard crc32, with all the usual good
> properties and, at least on FE and lower, 1500bytes max of payload.
> So it's quite reasonable.  TCP's checksum, though, is crap.
> 
> I'm not entirely sure how crc32 would behave on jumbo frames.
> 

AUTODIN-II CRC32 (the one used by Ethernet) is stable up to 11454
bytes.  The jumbo frame size was chosen as the largest multiple of the
standard IP payload size to fit within this number.

	-hpa







-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andre Hedrick]

On Wed, 8 Jan 2003, Vojtech Pavlik wrote:

> On Tue, Jan 07, 2003 at 02:45:51PM -0800, Andre Hedrick wrote:
> 
> > > Andre Hedrick wrote:
> > > 
> > > > Please continue to think of TCP checksums as valid for a data transport,
> > > > you data will be gone soon enough.
> > > > 
> > > > Initiator == Controller
> > > > Target == Disk
> > > > iSCSI == cable or ribbon
> > > > 
> > > > Please turn off the CRC on your disk drive and see if you still have data.
> > > 
> > > This maybe works as PR, but otherwise it's crap.
> > 
> > So, please turn off the CRC's in your onboard storage today and see how
> > long it lasts.
> 
> 1) Bad comparison.
> 
> 2) It'd last very very long, because I never get a CRC error anyway.

So turn them off so it never checks, nevermind :-)

--ah

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Andrew McGregor]

Actually, talking to some people off-list, I realised that what happened 
with the instances I saw was probably that the packets were corrupted 
inside the host, somewhere after the ethernet checksum had done its job. 
DMA problems or a slow address line on some RAM somewhere could easily beat 
the TCP checksum, but as many folks have pointed out, ethernet CRC is much 
stronger.

Having seen something odd like that in practice, I overestimated the 
probability of these problems.

It was also pointed out that iSCSI also makes it's CRC optional only if 
there is some other mechanism (ESP, AH or some other high-integrity 
transport) providing the data integrity.  Partly this is because that 
checksum is the same as used by Fiber Channel, and is therefore available 
'for free' in some, but not all, hardware, so there needs to be another way 
to integrity protect the data.

Andrew

--On Wednesday, January 08, 2003 11:10:44 -0800 "H. Peter Anvin" 
<hpa@zytor.com> wrote:

> Followup to:  <20030107053146.A16578@kerberos.ncsl.nist.gov>
> By author:    Olivier Galibert <galibert@pobox.com>
> In newsgroup: linux.dev.kernel
>>
>> On Tue, Jan 07, 2003 at 01:39:38PM +1300, Andrew McGregor wrote:
>> > Ethernet and TCP were both designed to be cheap to evaluate, not the
>> > absolute last word in integrity.  There is a move underway to provide
>> > an  optional stronger TCP digest for IPv6, and if used with that then
>> > there is  no need for the iSCSI digest.  Otherwise, well, play dice
>> > with the data.  Loaded in your favour, but still dice.
>>
>> Ethernet's checksum is a standard crc32, with all the usual good
>> properties and, at least on FE and lower, 1500bytes max of payload.
>> So it's quite reasonable.  TCP's checksum, though, is crap.
>>
>> I'm not entirely sure how crc32 would behave on jumbo frames.
>>
>
> AUTODIN-II CRC32 (the one used by Ethernet) is stable up to 11454
> bytes.  The jumbo frame size was chosen as the largest multiple of the
> standard IP payload size to fit within this number.
>
> 	-hpa

Re: Linux iSCSI Initiator, OpenSource (fwd) (Re: Gauntlet Set NOW!)

[Richard B. Johnson]

On Thu, 9 Jan 2003, Andrew McGregor wrote:

> Actually, talking to some people off-list, I realised that what happened 
> with the instances I saw was probably that the packets were corrupted 
> inside the host, somewhere after the ethernet checksum had done its job. 
> DMA problems or a slow address line on some RAM somewhere could easily beat 
> the TCP checksum, but as many folks have pointed out, ethernet CRC is much 
> stronger.
> 
> Having seen something odd like that in practice, I overestimated the 
> probability of these problems.
> 
> It was also pointed out that iSCSI also makes it's CRC optional only if 
> there is some other mechanism (ESP, AH or some other high-integrity 
> transport) providing the data integrity.  Partly this is because that 
> checksum is the same as used by Fiber Channel, and is therefore available 
> 'for free' in some, but not all, hardware, so there needs to be another way 
> to integrity protect the data.
> 
> Andrew
> 
> --On Wednesday, January 08, 2003 11:10:44 -0800 "H. Peter Anvin" 
> <hpa@zytor.com> wrote:
> 
> > Followup to:  <20030107053146.A16578@kerberos.ncsl.nist.gov>
> > By author:    Olivier Galibert <galibert@pobox.com>
> > In newsgroup: linux.dev.kernel
> >>
> >> On Tue, Jan 07, 2003 at 01:39:38PM +1300, Andrew McGregor wrote:
> >> > Ethernet and TCP were both designed to be cheap to evaluate, not the
> >> > absolute last word in integrity.  There is a move underway to provide
> >> > an  optional stronger TCP digest for IPv6, and if used with that then
> >> > there is  no need for the iSCSI digest.  Otherwise, well, play dice
> >> > with the data.  Loaded in your favour, but still dice.
> >>
> >> Ethernet's checksum is a standard crc32, with all the usual good
> >> properties and, at least on FE and lower, 1500bytes max of payload.
> >> So it's quite reasonable.  TCP's checksum, though, is crap.
> >>
> >> I'm not entirely sure how crc32 would behave on jumbo frames.
> >>
> >
> > AUTODIN-II CRC32 (the one used by Ethernet) is stable up to 11454
> > bytes.  The jumbo frame size was chosen as the largest multiple of the
> > standard IP payload size to fit within this number.
> >
> > 	-hpa
> 
> -

The TCP/IP checksum is 'strange' in that if all 0x00 are changed to 0xff,
it will not detect the error. But... in the 'real world', the TCP/IP
checksum does quite well detecting the kinds of errors likely in
serial links. Many years ago, in one of our products, some 'junior'
designer once decided that the way to control some equipment would be to
use RS-232C. Nobody at the design reviews caught this. RS-232C is about
99.999 percent reliable. That means that one byte in 10,000 may be
damaged. Normally humans correct RS-232C errors by looking at echo and
fixing the 'typo'. The product ran off a VAXen serial link at 19,200 baud,
the highest speed to which the VAX could be set.

The machine would not work. I ended up having to 'fix' the problem
in software. I used a TCP/IP checksum. The communications then never
failed (after the necessary retries). Since this was a 'success' I
decided to keep a record of the number of bad blocks retransmitted.

The customer got interested and emphatically stated; "The TCP/IP
checksum is wortless because.....". The result being pages of the
known anomolies of the checksum. The customer then required a
32-bit CRC they they specified. It was a standard polynominal,
but it was initialized with 0xaa55aa55 rather than 0xffffffff
(that's what then wanted). So, I used both and I logged the
operation of both.

There were never any errors, detected by the CRC, that were not
also detected by the checksum. This interface was for a spectrometer
that ran continuous serial commands (and data) for months at a time.
I recall that there were typically 40 to 50 recovered errors per
hour of operation. We delivered thousands of these.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.18 on an i686 machine (797.90 BogoMips).
Why is the government concerned about the lunatic fringe? Think about it.