[GIT PATCH] scsi bug fixes for 2.6.23-rc2

[GIT PATCH] scsi bug fixes for 2.6.23-rc2

[James Bottomley]

This is mainly bug fixes ... there's one or two features completions
that have been delayed pending ack and review to do with bsg (headers
and passthrough) but these are really required to complete already
upstream code.

The patch is available here:

master.kernel.org:/pub/scm/linux/kernel/git/jejb/scsi-rc-fixes-2.6.git

The short changelog is:

Boaz Harrosh (6):
      aha152x: use data accessors and !use_sg cleanup
      aha152x: Fix check_condition code-path
      aha152x: Clean Reset path
      aha152x: preliminary fixes and some comments
      aha152x: use bounce buffer
      aha152x: fix debug mode symbol conflict

David Miller (1):
      ESP: Revert ESP_BUS_TIMEOUT back down to 250

FUJITA Tomonori (6):
      initialize shost_data to zero
      mptsas: add SMP passthrough support via bsg
      bsg: update sg_io_v4 structure
      ibmvscsi: use shost_priv
      ibmvscsi: remove unnecessary map_sg check
      zfcp: convert to use the data buffer accessors

James Bottomley (4):
      sd: disentangle barriers in SCSI
      aic7xxx: cap maxsync according to correct card limits
      mpt fusion: make logging a global sysfs parameter
      libsas: fix build dependencies on libata

James Smart (9):
      lpfc : scsi command accessor fix for 8.2.2
      lpfc 8.2.2 : Change version number to 8.2.2
      lpfc 8.2.2 : Style cleanups
      lpfc 8.2.2 : Miscellaneous Bug Fixes
      lpfc 8.2.2 : Miscellaneous management and logging mods
      lpfc 8.2.2 : Rework the lpfc_printf_log() macro
      lpfc 8.2.2 : Attribute and Parameter splits for vport and physical port
      lpfc 8.2.2 : Fix locking around HBA's port_list
      lpfc 8.2.2 : Error messages and debugfs updates

Jeff Garzik (1):
      gdth: remove redundant PCI stuff

Mark Fortescue (1):
      qlogicpti: Some cosmetic changes

Matthew Wilcox (1):
      dpt_i2o: convert to SCSI hotplug model

Matthias Kaehlcke (1):
      st: Use mutex instead of semaphore

Salyzyn, Mark (1):
      aacraid: prevent panic on adapter resource failure

Seokmann Ju (1):
      qla2xxx: fix panic caused by previous patch


and the diffstat:

 block/bsg.c                         |   10 
 drivers/message/fusion/mptbase.c    |   17 
 drivers/message/fusion/mptsas.c     |  126 ++++++
 drivers/s390/scsi/zfcp_fsf.c        |    5 
 drivers/s390/scsi/zfcp_qdio.c       |   41 --
 drivers/scsi/aacraid/linit.c        |    4 
 drivers/scsi/aha152x.c              |  169 ++++----
 drivers/scsi/aha152x.h              |    2 
 drivers/scsi/aic7xxx/aic7xxx_core.c |   22 +
 drivers/scsi/dpt_i2o.c              |  132 +++---
 drivers/scsi/dpti.h                 |    9 
 drivers/scsi/esp_scsi.h             |    2 
 drivers/scsi/gdth.c                 |   48 +-
 drivers/scsi/gdth.h                 |    6 
 drivers/scsi/hosts.c                |    2 
 drivers/scsi/ibmvscsi/ibmvscsi.c    |   39 --
 drivers/scsi/libsas/Kconfig         |    3 
 drivers/scsi/lpfc/lpfc.h            |   72 ++-
 drivers/scsi/lpfc/lpfc_attr.c       |  423 +++++++++++++++-------
 drivers/scsi/lpfc/lpfc_crtn.h       |   28 -
 drivers/scsi/lpfc/lpfc_ct.c         |  243 ++++++------
 drivers/scsi/lpfc/lpfc_debugfs.c    |  595 ++++++++++++++++++++++++++++---
 drivers/scsi/lpfc/lpfc_debugfs.h    |    2 
 drivers/scsi/lpfc/lpfc_els.c        |  679 ++++++++++++++++--------------------
 drivers/scsi/lpfc/lpfc_hbadisc.c    |  539 ++++++++++++----------------
 drivers/scsi/lpfc/lpfc_hw.h         |   14 
 drivers/scsi/lpfc/lpfc_init.c       |  284 +++++++--------
 drivers/scsi/lpfc/lpfc_logmsg.h     |   10 
 drivers/scsi/lpfc/lpfc_mbox.c       |   20 -
 drivers/scsi/lpfc/lpfc_mem.c        |   32 +
 drivers/scsi/lpfc/lpfc_nportdisc.c  |  162 +++-----
 drivers/scsi/lpfc/lpfc_scsi.c       |  413 ++++++++++-----------
 drivers/scsi/lpfc/lpfc_sli.c        |  423 +++++++++++-----------
 drivers/scsi/lpfc/lpfc_sli.h        |   10 
 drivers/scsi/lpfc/lpfc_version.h    |    4 
 drivers/scsi/lpfc/lpfc_vport.c      |  164 +++++---
 drivers/scsi/lpfc/lpfc_vport.h      |    2 
 drivers/scsi/qla2xxx/qla_os.c       |   14 
 drivers/scsi/qlogicpti.c            |   50 +-
 drivers/scsi/scsi_lib.c             |   17 
 drivers/scsi/sd.c                   |   14 
 drivers/scsi/st.c                   |   16 
 drivers/scsi/st.h                   |    3 
 include/linux/bsg.h                 |   13 
 include/scsi/scsi_driver.h          |    2 
 include/scsi/sd.h                   |    2 
 46 files changed, 2837 insertions(+), 2050 deletions(-)

James

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Linus Torvalds]

On Sat, 4 Aug 2007, James Bottomley wrote:
>
> This is mainly bug fixes ... there's one or two features completions
> that have been delayed pending ack and review to do with bsg (headers
> and passthrough) but these are really required to complete already
> upstream code.

James, this is the last time *ever* I apply patches from you after -rc1.

You used to have serious problems with the merge window, but for a few 
releases you then seemed to "get it" and got on with the program.

But now it's back to "anythign goes", apparently. And I'm going to take a 
hard-line approach with you now. 

For SCSI merges, if I don't get the first pull request in the FIRST week 
of the merge window, don't bother sending one later, unless it's pure 
fixes and regressions.

And after -rc1, I don't want to see crap like this:

	 46 files changed, 2837 insertions(+), 2050 deletions(-)

because that simply is *not* appropriate after -rc1, much less -rc2.

So I pulled, but I wanted to make it very clear that I'm very unhappy with 
you right now, and you're on my shit-list for the next few releases. Get 
the changes in before -rc1, or just *wait*. If they aren't ready before 
the merge window opens, they simply shouldn't be merged at all.

		Linus

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[James Bottomley]

On Mon, 2007-08-06 at 17:51 -0700, Linus Torvalds wrote:
> 
> On Sat, 4 Aug 2007, James Bottomley wrote:
> >
> > This is mainly bug fixes ... there's one or two features completions
> > that have been delayed pending ack and review to do with bsg (headers
> > and passthrough) but these are really required to complete already
> > upstream code.
> 
> James, this is the last time *ever* I apply patches from you after -rc1.
> 
> You used to have serious problems with the merge window, but for a few 
> releases you then seemed to "get it" and got on with the program.
> 
> But now it's back to "anythign goes", apparently. And I'm going to take a 
> hard-line approach with you now. 
> 
> For SCSI merges, if I don't get the first pull request in the FIRST week 
> of the merge window, don't bother sending one later, unless it's pure 
> fixes and regressions.

Confused ... you did get the first pull request in the first week.  That
was this:

                           Subject: 
[GIT PATCH] first SCSI merge for
2.6.22
                              Date: 
        Sun, 15 Jul 2007 10:24:17
-0500
190 files changed, 21725 insertions(+), 26337 deletions(-)

Then there was the last piece before the merge window closed:

                           Subject: 
[GIT PATCH] final piece of the SCSI
merge for 2.6.22
                              Date: 
        Sun, 22 Jul 2007 13:28:53
-0500
74 files changed, 3649 insertions(+), 1295 deletions(-)



> And after -rc1, I don't want to see crap like this:
> 
> 	 46 files changed, 2837 insertions(+), 2050 deletions(-)
> 
> because that simply is *not* appropriate after -rc1, much less -rc2.
> 
> So I pulled, but I wanted to make it very clear that I'm very unhappy with 
> you right now, and you're on my shit-list for the next few releases. Get 
> the changes in before -rc1, or just *wait*. If they aren't ready before 
> the merge window opens, they simply shouldn't be merged at all.

OK ... that's arguable.  This one is larger than I like because of the
lpfc bug fix patch ... I accept I need to do a better job getting these
into the merge window via the scsi-misc tree.  So I will accept the "too
big" criticism and try to manage the driver maintainers better.

However, I won't accept the "not bug fixes only" criticism at -rc1.  The
problem is that we're trying to stabilise a new feature: bsg.
Unfortunately, the closure of the merge window was really the first time
anyone got to play with all of these features together.  The non-bug fix
changes around bsg have been trying to achieve stability.  The problem
is that there were a few fairly problematic pieces: dependence on
non-modular SCSI; SG header layout and driver implementation.  What we
really don't want is to have a problematic API baked in stone because we
can't do anything other than bug fix updates once the merge window
closes.

The real root cause of all of this is that there's no tree I can
persuade all the interested parties to test that includes all of these
features.  In spite of the fact they've all been incubating in -mm for
at least 3 months, no-one apparently tested all the features together
until 2.6.23-rc1 was released, so then we're scrambling to address the
issues as they arise.

I really, *really* think we need a pre-release tree that consists of all
the upstream targetted features (i.e. all of the for the next merge
window git trees) and nothing else.  -mm doesn't really satisfy this,
because it has so much other stuff that the people I need to get testing
this don't trust it.  The lack of a tree like this that we could have
persuaded people to test for the last month is what's causing us to
scramble like this at the closure of the merge window.

James

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Linus Torvalds]

On Mon, 6 Aug 2007, James Bottomley wrote:
>
> Confused ... you did get the first pull request in the first week.

Here's the problem. Let me repeat it again:

> > And after -rc1, I don't want to see crap like this:
> > 
> > 	 46 files changed, 2837 insertions(+), 2050 deletions(-)

It DOES NOT MATTER if I get a first pull request in the first week, if 
that pull request is purely cosmetic, and is followed by stuff that 
*should* have been in the merge window four weeks afterwards.

> OK ... that's arguable.

There's nothing arguable at all about it.

If you have 5000 lines of changes, that's not a "bugfix" any more. That's 
a big damn change, and it should have happened in the merge window. Or if 
it doesn't make it in time, in the *next* merge window.

			Linus

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[James Smart]

In defense of my maintainer, who was working on my behalf! ...

The lpfc mods were the bulk of the +/- counts.  We batch our bug fixes
together and then push to James as a large lump. Unfortunately, we had
a change that changed logging from a base object to a subobject. Although
not risky, it did account for a lot of +/- changes.  The way we pushed
to James, did not allow for him to easily segment one set of changes
from the other. Emulex will change this behavior, hopefully making this
easier on James to keep you happy.

However, I take issue with looking at line counts as the sole basis
for what's appropriate or not. It can be argued that some bug fixes may be
larger in scope than others, or patch batching so that the bug fix count is
higher will skew this perception. I also believe that more "lesser" bugfixes
should be allowed in an earlier -rc? than later, so a hard-and-fast rule for
line counts seem odd.  Also - what's a bug fix ?  There are many things
which are not "features" but are necessities for diagnosis or support of the
larger change. Some of these you simply don't find in time to make sure they
are in place for the -rc1 merge. Do you hold off on them, or do you make a
choice based risk/reward based on where the -rc is ? I vote for the latter.
I realize that the Linux kernel is such a beast overall that you must have
some simple guidelines, but basing it solely on numbers is a very bad pitfall.

-- james s


Linus Torvalds wrote:
> 
> On Mon, 6 Aug 2007, James Bottomley wrote:
>> Confused ... you did get the first pull request in the first week.
> 
> Here's the problem. Let me repeat it again:
> 
>>> And after -rc1, I don't want to see crap like this:
>>>
>>> 	 46 files changed, 2837 insertions(+), 2050 deletions(-)
> 
> It DOES NOT MATTER if I get a first pull request in the first week, if 
> that pull request is purely cosmetic, and is followed by stuff that 
> *should* have been in the merge window four weeks afterwards.
> 
>> OK ... that's arguable.
> 
> There's nothing arguable at all about it.
> 
> If you have 5000 lines of changes, that's not a "bugfix" any more. That's 
> a big damn change, and it should have happened in the merge window. Or if 
> it doesn't make it in time, in the *next* merge window.
> 
> 			Linus
> -
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jeff Garzik]

James Smart wrote:
> However, I take issue with looking at line counts as the sole basis
> for what's appropriate or not. It can be argued that some bug fixes may be
> larger in scope than others, or patch batching so that the bug fix count is
> higher will skew this perception. I also believe that more "lesser" 
> bugfixes
> should be allowed in an earlier -rc? than later, so a hard-and-fast rule 
> for
> line counts seem odd.  Also - what's a bug fix ?  There are many things
> which are not "features" but are necessities for diagnosis or support of 
> the
> larger change. Some of these you simply don't find in time to make sure 
> they
> are in place for the -rc1 merge. Do you hold off on them, or do you make a
> choice based risk/reward based on where the -rc is ? I vote for the latter.
> I realize that the Linux kernel is such a beast overall that you must have
> some simple guidelines, but basing it solely on numbers is a very bad 
> pitfall.


It's straightforward engineering math:  the more LOC that changed, the 
more important it is to /not/ stuff it into a stabilization release, 
because of the greater potential for breaking stuff and negating all the 
existing testing so far.

Once -rc1 is out there, that means the focus should be on stabilizing 
the existing codebase.  Pushing a big driver update means that effort 
must restart from scratch.  We just don't want to go down that road, 
which a big reason for the merge window in general.

If you miss the merge window, tough cookies :)  You gotta deal with it 
just like I do, and everyone else does.

Remember -- the more disciplined we all are with the merge window, the 
more likely it is that a release can be stabilized quickly, and thus, 
the more quickly we will reach the next merge window.

In contrast, increasing violations of the merge window mean increasing 
time between releases.

	Jeff

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[James Bottomley]

On Mon, 2007-08-06 at 21:01 -0700, Linus Torvalds wrote:
> 
> On Mon, 6 Aug 2007, James Bottomley wrote:
> >
> > Confused ... you did get the first pull request in the first week.
> 
> Here's the problem. Let me repeat it again:
> 
> > > And after -rc1, I don't want to see crap like this:
> > > 
> > > 	 46 files changed, 2837 insertions(+), 2050 deletions(-)
> 
> It DOES NOT MATTER if I get a first pull request in the first week, if 
> that pull request is purely cosmetic, and is followed by stuff that 
> *should* have been in the merge window four weeks afterwards.
> 
> > OK ... that's arguable.
> 
> There's nothing arguable at all about it.
> 
> If you have 5000 lines of changes, that's not a "bugfix" any more. That's 
> a big damn change, and it should have happened in the merge window. Or if 
> it doesn't make it in time, in the *next* merge window.

I'm not arguing that the bug fix piece wasn't too big (although
realistically, line counts are only a guide not a rule.  If we discover
something like a calling convention bug in SCSI [reversed kmalloc
arguments, say], I could see a huge patch to fix all of the call
sites) ... I've said I'll take responsibility for that and fix it.

I'm arguing that a too strict an interpretation of bugfix only post -rc1
will damage feature stabilisation.  Please think carefully about this.
If we go out in a released kernel with a problematic user space ABI, we
end up being committed to it forever.

James

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jeff Garzik]

James Bottomley wrote:
> I'm arguing that a too strict an interpretation of bugfix only post -rc1
> will damage feature stabilisation.  Please think carefully about this.
> If we go out in a released kernel with a problematic user space ABI, we
> end up being committed to it forever.


IMO you're going off on your own tangent.  Linus never singled out bsg 
(far from it, in fact, since bsg was not a major LOC contributor) or 
declared ABI-related fixes verboten.

I don't think anyone wants to release a userspace ABI with problems, 
since we all know that's basically locked in stone once its in a 
mainline release.

AFAICS his main complaint was he felt your push was a big honking huge 
change, late in the game, that included obvious non-fixes.  And it was. 
  lpfc was probably the biggest part of that, not bsg, and it's pretty 
clear such a big lpfc update should have gone in when the merge window 
was open.  The [non-lpfc] cleanups were also not -rc2 material.

	Jeff

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[James Bottomley]

On Tue, 2007-08-07 at 12:20 -0400, Jeff Garzik wrote:
> James Bottomley wrote:
> > I'm arguing that a too strict an interpretation of bugfix only post -rc1
> > will damage feature stabilisation.  Please think carefully about this.
> > If we go out in a released kernel with a problematic user space ABI, we
> > end up being committed to it forever.
> 
> 
> IMO you're going off on your own tangent.  Linus never singled out bsg 
> (far from it, in fact, since bsg was not a major LOC contributor) or 
> declared ABI-related fixes verboten.
> 
> I don't think anyone wants to release a userspace ABI with problems, 
> since we all know that's basically locked in stone once its in a 
> mainline release.
> 
> AFAICS his main complaint was he felt your push was a big honking huge 
> change, late in the game, that included obvious non-fixes.  And it was. 
>   lpfc was probably the biggest part of that, not bsg, and it's pretty 
> clear such a big lpfc update should have gone in when the merge window 
> was open.  The [non-lpfc] cleanups were also not -rc2 material.

I think you'll find that part of the complaint was addressed in the
first part of the email which you didn't quote.

James

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Andrew Morton]

On Mon, 06 Aug 2007 22:55:41 -0500 James Bottomley <James.Bottomley@SteelEye.com> wrote:

> The real root cause of all of this is that there's no tree I can
> persuade all the interested parties to test that includes all of these
> features.  In spite of the fact they've all been incubating in -mm for
> at least 3 months, no-one apparently tested all the features together
> until 2.6.23-rc1 was released, so then we're scrambling to address the
> issues as they arise.

I pulled git-scsi-misc on July 19 and there was no bsg code in there at
all.  I pulled again on July 20 and all the bsg code was in mainline.  So
it appears that the bsg code went mailing-list -> mainline in less than 24
hours, so there wasn't a lot of opportunity for -mm testing there.

A lot of the stupid it-doesn't-compile stuff would have been fixed in -mm,
but more substantial problems might not have been picked up.  But one can
say that about anything.

> I really, *really* think we need a pre-release tree that consists of all
> the upstream targetted features (i.e. all of the for the next merge
> window git trees) and nothing else.

That *is* -mm.  The vast majority of -mm is the 75-odd subsystem trees. 
What you're suggesting amounts to omitting some of those trees for test
purposes (I think).  If so, which ones?

Now it coud be argued that subsystem maintainers should run two trees in
the last 2.6.x-rcN phase: one tree for 2.6.x+1 and one tree for 2.6.x+2. 
Then someone could pull all that together as the "Linus tree in a month,
minus insufficiently baked stuff" tree.  But frankly, I don't expect that
people will want to do that, nor will they be able to do it reliably.

Plus, an *amazing* amount of stuff turns up in the git trees which was
committed just a few days prior to the merge window opening, or even after
it opening.  eg, bsg which was, afaict, first committed to the scsi tree
eleven days after the 2.6.22 release.

>  -mm doesn't really satisfy this,
> because it has so much other stuff that the people I need to get testing
> this don't trust it.

Right.  75-odd developers need to stop committing bugs to their devel
trees.  Interesting project ;)

>  The lack of a tree like this that we could have
> persuaded people to test for the last month is what's causing us to
> scramble like this at the closure of the merge window.

Nope.  The scramble is caused by subsystem maintainers jamming stuff into
mainline at the last minute so they don't have to sit on it for the next
two months.

Look.  If we're serious about this then the rule needs to be something like

  If it wasn't committed to your tree *at least* two weeks prior to the
  2.6.x merge window opening, it shouldn't go into 2.6.x.

People are not presently observing this sort of discipline by a metric
mile.  And I'm not sure that we should, really.


I don't think it's terribly bad to whack half-baked things (bsg ;)) into
mainline during the merge window, as long as a) we're sure that we want the
feature in Linux and b) we're confident that we can get it fixed up within
a couple of months.  Two months is a long time.

But that's just me, and it is not the approach which Linus wants taken.

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[FUJITA Tomonori]

On Tue, 7 Aug 2007 00:14:29 -0700
Andrew Morton <akpm@linux-foundation.org> wrote:

> On Mon, 06 Aug 2007 22:55:41 -0500 James Bottomley <James.Bottomley@SteelEye.com> wrote:
> 
> > The real root cause of all of this is that there's no tree I can
> > persuade all the interested parties to test that includes all of these
> > features.  In spite of the fact they've all been incubating in -mm for
> > at least 3 months, no-one apparently tested all the features together
> > until 2.6.23-rc1 was released, so then we're scrambling to address the
> > issues as they arise.
> 
> I pulled git-scsi-misc on July 19 and there was no bsg code in there at
> all.  I pulled again on July 20 and all the bsg code was in mainline.  So
> it appears that the bsg code went mailing-list -> mainline in less than 24
> hours, so there wasn't a lot of opportunity for -mm testing there.

bsg was merged via Jens' branch. After that, I asked James to send
some fixes via the scsi-rc-fixes.


> A lot of the stupid it-doesn't-compile stuff would have been fixed in -mm,
> but more substantial problems might not have been picked up.  But one can
> say that about anything.

My mistake. I should have sent bsg to -mm. Sorry about that.

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jeff Garzik]

FUJITA Tomonori wrote:
> On Tue, 7 Aug 2007 00:14:29 -0700
> Andrew Morton <akpm@linux-foundation.org> wrote:
> 
>> On Mon, 06 Aug 2007 22:55:41 -0500 James Bottomley <James.Bottomley@SteelEye.com> wrote:
>>
>>> The real root cause of all of this is that there's no tree I can
>>> persuade all the interested parties to test that includes all of these
>>> features.  In spite of the fact they've all been incubating in -mm for
>>> at least 3 months, no-one apparently tested all the features together
>>> until 2.6.23-rc1 was released, so then we're scrambling to address the
>>> issues as they arise.
>> I pulled git-scsi-misc on July 19 and there was no bsg code in there at
>> all.  I pulled again on July 20 and all the bsg code was in mainline.  So
>> it appears that the bsg code went mailing-list -> mainline in less than 24
>> hours, so there wasn't a lot of opportunity for -mm testing there.
> 
> bsg was merged via Jens' branch. After that, I asked James to send
> some fixes via the scsi-rc-fixes.

ISTR that Jens doesn't regularly push / get picked up by -mm?  That 
seems like an easy problem to solve.

	Jeff

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Andrew Morton]

On Tue, 07 Aug 2007 10:21:18 -0400 Jeff Garzik <jeff@garzik.org> wrote:

> FUJITA Tomonori wrote:
> > On Tue, 7 Aug 2007 00:14:29 -0700
> > Andrew Morton <akpm@linux-foundation.org> wrote:
> > 
> >> On Mon, 06 Aug 2007 22:55:41 -0500 James Bottomley <James.Bottomley@SteelEye.com> wrote:
> >>
> >>> The real root cause of all of this is that there's no tree I can
> >>> persuade all the interested parties to test that includes all of these
> >>> features.  In spite of the fact they've all been incubating in -mm for
> >>> at least 3 months, no-one apparently tested all the features together
> >>> until 2.6.23-rc1 was released, so then we're scrambling to address the
> >>> issues as they arise.
> >> I pulled git-scsi-misc on July 19 and there was no bsg code in there at
> >> all.  I pulled again on July 20 and all the bsg code was in mainline.  So
> >> it appears that the bsg code went mailing-list -> mainline in less than 24
> >> hours, so there wasn't a lot of opportunity for -mm testing there.
> > 
> > bsg was merged via Jens' branch. After that, I asked James to send
> > some fixes via the scsi-rc-fixes.
> 
> ISTR that Jens doesn't regularly push / get picked up by -mm?  That 
> seems like an easy problem to solve.
> 

-mm includes
git+ssh://master.kernel.org/pub/scm/linux/kernel/git/axboe/linux-2.6-block.git#for-akpm,
but it's up to Jens to choose what goes in there.

git-block has been dropped from -mm since 2.6.23-rc1-mm1 (I think) because
it has something in there which breaks sata on the Vaio.  Prior to that (in
2.6.22-rc-late) there was nothing in that tree.

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[James Bottomley]

On Tue, 2007-08-07 at 00:14 -0700, Andrew Morton wrote:
> On Mon, 06 Aug 2007 22:55:41 -0500 James Bottomley <James.Bottomley@SteelEye.com> wrote:
> 
> > The real root cause of all of this is that there's no tree I can
> > persuade all the interested parties to test that includes all of these
> > features.  In spite of the fact they've all been incubating in -mm for
> > at least 3 months, no-one apparently tested all the features together
> > until 2.6.23-rc1 was released, so then we're scrambling to address the
> > issues as they arise.
> 
> I pulled git-scsi-misc on July 19 and there was no bsg code in there at
> all.  I pulled again on July 20 and all the bsg code was in mainline.  So
> it appears that the bsg code went mailing-list -> mainline in less than 24
> hours, so there wasn't a lot of opportunity for -mm testing there.

The initial bsg submit went via the block git tree ... which I believe
you have in -mm.  We only started taking the updates via the scsi tree
when it became evident that they were entangling both scsi and bsg too
deeply to be split between trees.

> A lot of the stupid it-doesn't-compile stuff would have been fixed in -mm,
> but more substantial problems might not have been picked up.  But one can
> say that about anything.

Actually, it was fixed ... just in a fashion I found to be unacceptable:
making SCSI built in if bsg was selected.

> > I really, *really* think we need a pre-release tree that consists of all
> > the upstream targetted features (i.e. all of the for the next merge
> > window git trees) and nothing else.
> 
> That *is* -mm.  The vast majority of -mm is the 75-odd subsystem trees. 
> What you're suggesting amounts to omitting some of those trees for test
> purposes (I think).  If so, which ones?

The problem is that there's too much other stuff in -mm.  Whenever
anyone asks where they can get scsi-misc (which is my tree for the next
merge window) from without constructing it themselves in git, I always
say use -mm.  Unfortunately, the attrition rate after telling them this
seems to be really high.

> Now it coud be argued that subsystem maintainers should run two trees in
> the last 2.6.x-rcN phase: one tree for 2.6.x+1 and one tree for 2.6.x+2. 
> Then someone could pull all that together as the "Linus tree in a month,
> minus insufficiently baked stuff" tree.  But frankly, I don't expect that
> people will want to do that, nor will they be able to do it reliably.

A sort of pre merge window freeze point?

> Plus, an *amazing* amount of stuff turns up in the git trees which was
> committed just a few days prior to the merge window opening, or even after
> it opening.  eg, bsg which was, afaict, first committed to the scsi tree
> eleven days after the 2.6.22 release.

Yes ... particularly in large trees like SCSI, there's the maintainer
"bugger if I don't mail it out now I don't get it in for another three
months" factor.

bsg had actually been sitting in the block tree since 2.6.21, so it had
followed the delayed merge rule ... it just seems that it didn't get
enough integration testing in that six months.  This is what I consider
the real problem to be.

> >  -mm doesn't really satisfy this,
> > because it has so much other stuff that the people I need to get testing
> > this don't trust it.
> 
> Right.  75-odd developers need to stop committing bugs to their devel
> trees.  Interesting project ;)
> 
> >  The lack of a tree like this that we could have
> > persuaded people to test for the last month is what's causing us to
> > scramble like this at the closure of the merge window.
> 
> Nope.  The scramble is caused by subsystem maintainers jamming stuff into
> mainline at the last minute so they don't have to sit on it for the next
> two months.
> 
> Look.  If we're serious about this then the rule needs to be something like
> 
>   If it wasn't committed to your tree *at least* two weeks prior to the
>   2.6.x merge window opening, it shouldn't go into 2.6.x.
> 
> People are not presently observing this sort of discipline by a metric
> mile.  And I'm not sure that we should, really.

I don't disagree; my point is that bsg did follow this rule (in fact it
tried to stabilise itself outside of mainline for far longer than this
rule implies)  the problem is that it didn't get sufficient integration
testing.

> I don't think it's terribly bad to whack half-baked things (bsg ;)) into

I wouldn't call bsg half baked ... it was very carefully matured.  There
were just a few integration issues.

> mainline during the merge window, as long as a) we're sure that we want the
> feature in Linux and b) we're confident that we can get it fixed up within
> a couple of months.  Two months is a long time.
> 
> But that's just me, and it is not the approach which Linus wants taken.

I agree with this approach too ... that's what I've been doing.  It
means that feature stabilisation must finish at around -rc3.  The
relative quiet from SCSI over the last two releases was because I didn't
have any new features.

I fully agree, and firmly believe that the current stabilisation works
incredibly well for shaking out bugs.  My problem is that it doesn't
work for stabilising features.  Either we have to get far more people
doing feature integration testing before the merge window, or we have to
accept feature updates after the merge window (for existing features
that are having stability issues).

James

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Alan Cox]

> I fully agree, and firmly believe that the current stabilisation works
> incredibly well for shaking out bugs.  My problem is that it doesn't
> work for stabilising features.  Either we have to get far more people
> doing feature integration testing before the merge window, or we have to
> accept feature updates after the merge window (for existing features
> that are having stability issues).

The other alternative is that if Linus won't take updates you ask him to
revert bsg so that you don't get a half baked merge as a result of this.
I'm not sure that is a good path to follow either however.

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jeff Garzik]

Alan Cox wrote:
>> I fully agree, and firmly believe that the current stabilisation works
>> incredibly well for shaking out bugs.  My problem is that it doesn't
>> work for stabilising features.  Either we have to get far more people
>> doing feature integration testing before the merge window, or we have to
>> accept feature updates after the merge window (for existing features
>> that are having stability issues).
> 
> The other alternative is that if Linus won't take updates you ask him to
> revert bsg so that you don't get a half baked merge as a result of this.
> I'm not sure that is a good path to follow either however.

Like everything else in life, it's a balance.  If something is clearly 
half-baked and requires a bunch of post-rc1 changes just to be usable, a 
revert might make a lot of sense.

It's questions of: how much further change is required, how invasive are 
those changes, how half-baked and incomplete is the feature really, what 
is the downside of a revert, ...

	Jeff

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jeff Garzik]

James Bottomley wrote:
> The initial bsg submit went via the block git tree ... which I believe
> you have in -mm.  We only started taking the updates via the scsi tree

Seven hours before you posted this, in 
<20070807001429.f8cb3b22.akpm@linux-foundation.org>, Andrew already 
noted it was not in -mm.

A trivial examination of the broken-out mm patches backs up the absence 
  of Jens' block tree, too.

So let's put this myth / bad assumption to rest, shall we?


> Yes ... particularly in large trees like SCSI, there's the maintainer
> "bugger if I don't mail it out now I don't get it in for another three
> months" factor.

That factor always exists.  It's not confined to SCSI or large trees. 
It's basic the nature of the merge window.  Nothing new or shocking here.


> bsg had actually been sitting in the block tree since 2.6.21, so it had
> followed the delayed merge rule ... it just seems that it didn't get
> enough integration testing in that six months.  This is what I consider

It didn't get integration testing, at least in part, because it did not 
hit our official pre-release tree.  Quoth Andrew:
> I pulled git-scsi-misc on July 19 and there was no bsg code in there at
> all.  I pulled again on July 20 and all the bsg code was in mainline.



> I don't disagree; my point is that bsg did follow this rule (in fact it

Evidence says otherwise.


> I wouldn't call bsg half baked ... it was very carefully matured.  There
> were just a few integration issues.

I wouldn't call bsg carefully matured, if in addition to not really 
gracing -mm with its presence, the userland API structure is still 
getting changes on July 29, 2007 (0c6a89ba640d28e1dcd7fd1a217d2cfb92ae4953).

	Jeff

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[James Bottomley]

On Tue, 2007-08-07 at 11:11 -0400, Jeff Garzik wrote:
> James Bottomley wrote:
> > The initial bsg submit went via the block git tree ... which I believe
> > you have in -mm.  We only started taking the updates via the scsi tree
> 
> Seven hours before you posted this, in 
> <20070807001429.f8cb3b22.akpm@linux-foundation.org>, Andrew already 
> noted it was not in -mm.
> 
> A trivial examination of the broken-out mm patches backs up the absence 
>   of Jens' block tree, too.
> 
> So let's put this myth / bad assumption to rest, shall we?

Sorry ... I just assumed from the fact that it had been in the block git
tree for six months that it was also in -mm.

> > Yes ... particularly in large trees like SCSI, there's the maintainer
> > "bugger if I don't mail it out now I don't get it in for another three
> > months" factor.
> 
> That factor always exists.  It's not confined to SCSI or large trees. 
> It's basic the nature of the merge window.  Nothing new or shocking here.
> 
> 
> > bsg had actually been sitting in the block tree since 2.6.21, so it had
> > followed the delayed merge rule ... it just seems that it didn't get
> > enough integration testing in that six months.  This is what I consider
> 
> It didn't get integration testing, at least in part, because it did not 
> hit our official pre-release tree.  Quoth Andrew:
> > I pulled git-scsi-misc on July 19 and there was no bsg code in there at
> > all.  I pulled again on July 20 and all the bsg code was in mainline.
> 
> 
> 
> > I don't disagree; my point is that bsg did follow this rule (in fact it
> 
> Evidence says otherwise.

It followed the rule of trying to stabilise outside mainline ... it just
didn't get sufficient integration testing.

> > I wouldn't call bsg half baked ... it was very carefully matured.  There
> > were just a few integration issues.
> 
> I wouldn't call bsg carefully matured, if in addition to not really 
> gracing -mm with its presence, the userland API structure is still 
> getting changes on July 29, 2007 (0c6a89ba640d28e1dcd7fd1a217d2cfb92ae4953).

This would be the ABI change I talked about in the previous emails.

So would this problem have been fixed simply by adding the missing block
tree to -mm?

James

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jeff Garzik]

James Bottomley wrote:
> It followed the rule of trying to stabilise outside mainline ... it just
> didn't get sufficient integration testing.

IMO it's self-evident that pushing to a git tree few ever see or test is 
not following the spirit of the rule.

In practice, stabilize outside mainline implies -mm integration, in 
addition to whatever else a maintainer does specific to their subsystem.


>>> I wouldn't call bsg half baked ... it was very carefully matured.  There
>>> were just a few integration issues.
>> I wouldn't call bsg carefully matured, if in addition to not really 
>> gracing -mm with its presence, the userland API structure is still 
>> getting changes on July 29, 2007 (0c6a89ba640d28e1dcd7fd1a217d2cfb92ae4953).
> 
> This would be the ABI change I talked about in the previous emails.
> 
> So would this problem have been fixed simply by adding the missing block
> tree to -mm?

IMO 70% of the problem would be solved by that.

It wouldn't have solved the late ABI change though (which was first 
posted in earlier form on April 4th, and so, had time for integration).

	Jeff

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Andrew Morton]

On Tue, 07 Aug 2007 10:38:44 -0500 James Bottomley <James.Bottomley@SteelEye.com> wrote:

> On Tue, 2007-08-07 at 11:11 -0400, Jeff Garzik wrote:
> > James Bottomley wrote:
> > > The initial bsg submit went via the block git tree ... which I believe
> > > you have in -mm.  We only started taking the updates via the scsi tree
> > 
> > Seven hours before you posted this, in 
> > <20070807001429.f8cb3b22.akpm@linux-foundation.org>, Andrew already 
> > noted it was not in -mm.
> > 
> > A trivial examination of the broken-out mm patches backs up the absence 
> >   of Jens' block tree, too.
> > 
> > So let's put this myth / bad assumption to rest, shall we?
> 
> Sorry ... I just assumed from the fact that it had been in the block git
> tree for six months that it was also in -mm.

bsg was never in the #for-akpm branch of git-block.  So I assume that
Jens had it in some other branch and for some reason never pulled it
across into #for-akpm.

It was most reasonable of you to expect that bsg had received a decent
run in -mm.

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jens Axboe]

On Tue, Aug 07 2007, Andrew Morton wrote:
> On Tue, 07 Aug 2007 10:38:44 -0500 James Bottomley <James.Bottomley@SteelEye.com> wrote:
> 
> > On Tue, 2007-08-07 at 11:11 -0400, Jeff Garzik wrote:
> > > James Bottomley wrote:
> > > > The initial bsg submit went via the block git tree ... which I believe
> > > > you have in -mm.  We only started taking the updates via the scsi tree
> > > 
> > > Seven hours before you posted this, in 
> > > <20070807001429.f8cb3b22.akpm@linux-foundation.org>, Andrew already 
> > > noted it was not in -mm.
> > > 
> > > A trivial examination of the broken-out mm patches backs up the absence 
> > >   of Jens' block tree, too.
> > > 
> > > So let's put this myth / bad assumption to rest, shall we?
> > 
> > Sorry ... I just assumed from the fact that it had been in the block git
> > tree for six months that it was also in -mm.
> 
> bsg was never in the #for-akpm branch of git-block.  So I assume that
> Jens had it in some other branch and for some reason never pulled it
> across into #for-akpm.

#for-akpm is usually only in very few -mm release anyway, so it's not
like it would have made much difference. We/you/I need to improve that,
certainly.

Honestly, for bsg, it wasn't much of an issue. We had build problems
when bsg was merged which was unfortunate but got fixed quickly. Having
bsg in -mm would not have caused any testing of the driver in question
outside of what it already received, given the nature of it.

-- 
Jens Axboe

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jeff Garzik]

Jens Axboe wrote:
> #for-akpm is usually only in very few -mm release anyway, so it's not
> like it would have made much difference. We/you/I need to improve that,
> certainly.
> 
> Honestly, for bsg, it wasn't much of an issue. We had build problems
> when bsg was merged which was unfortunate but got fixed quickly. Having
> bsg in -mm would not have caused any testing of the driver in question
> outside of what it already received, given the nature of it.


That's just an excuse for what happened -- you made an end run around 
our test tree.  Pretty please with sugar on it -- make sure changes show 
up in -mm.

	Jeff

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jens Axboe]

On Mon, Aug 13 2007, Jeff Garzik wrote:
> Jens Axboe wrote:
> >#for-akpm is usually only in very few -mm release anyway, so it's not
> >like it would have made much difference. We/you/I need to improve that,
> >certainly.
> >
> >Honestly, for bsg, it wasn't much of an issue. We had build problems
> >when bsg was merged which was unfortunate but got fixed quickly. Having
> >bsg in -mm would not have caused any testing of the driver in question
> >outside of what it already received, given the nature of it.
> 
> 
> That's just an excuse for what happened -- you made an end run around 
> our test tree.  Pretty please with sugar on it -- make sure changes show 
> up in -mm.

It's not just an excuse, there are repeatably problems with getting the
block stuff pulled into -mm. Sometimes it's my problem, but not always.
Then I get a note saying that pulling was disabled due to merge
problems, usually right before -mmX goes out. So it misses that release.

With bsg it wasn't a huge issue imho, since it was in fairly good shape
and it the issue were mainly with integration. I do try to make sure
stuff gets tested, but I also know which stuff is more important to get
exposure and multi-user testing on. bsg wasn't one of those, it's just a
lowly driver.

-- 
Jens Axboe

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jens Axboe]

On Mon, Aug 13 2007, Jens Axboe wrote:
> On Mon, Aug 13 2007, Jeff Garzik wrote:
> > Jens Axboe wrote:
> > >#for-akpm is usually only in very few -mm release anyway, so it's not
> > >like it would have made much difference. We/you/I need to improve that,
> > >certainly.
> > >
> > >Honestly, for bsg, it wasn't much of an issue. We had build problems
> > >when bsg was merged which was unfortunate but got fixed quickly. Having
> > >bsg in -mm would not have caused any testing of the driver in question
> > >outside of what it already received, given the nature of it.
> > 
> > 
> > That's just an excuse for what happened -- you made an end run around 
> > our test tree.  Pretty please with sugar on it -- make sure changes show 
> > up in -mm.
> 
> It's not just an excuse, there are repeatably problems with getting the
> block stuff pulled into -mm. Sometimes it's my problem, but not always.
> Then I get a note saying that pulling was disabled due to merge
> problems, usually right before -mmX goes out. So it misses that release.

This may sound like I'm blaming Andrew, but that is not my intention. He
does a lot of work and he can't (and should not) attempt to fix every
integration and merge issue. Perhaps if I/we were more strict in pushing
every block bit through the block git tree, it wouldn't be so much of an
integration pain. Then the work would be in me to sort out, which I'm
happy to do.

Then there are things like the sg chaining bits, which touch a lot of
arch code. Those are just painful to deal with, for obvious reasons.
Right now there are not in -mm, but that's because of a bug that
prevents Andrews laptop from booting properly.

-- 
Jens Axboe

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jeff Garzik]

Andrew Morton wrote:
> On Mon, 06 Aug 2007 22:55:41 -0500 James Bottomley <James.Bottomley@SteelEye.com> wrote:
>> I really, *really* think we need a pre-release tree that consists of all
>> the upstream targetted features (i.e. all of the for the next merge
>> window git trees) and nothing else.
> 
> That *is* -mm.  The vast majority of -mm is the 75-odd subsystem trees. 

Not quite.

-mm is git trees plus an amazing amount of random patches that turned up 
on LKML, a lot of which is not destined for kernel release 2.6.(X+1) or 
2.6.(X+2),


> Plus, an *amazing* amount of stuff turns up in the git trees which was
> committed just a few days prior to the merge window opening, or even after
> it opening.

Yes :(  That's a tough problem to solve, too.

Deadlines always motivate people, and so -- as in almost every other 
software project I've worked with -- everybody seems to submit their 
work on the day of the deadline.

Realistically, for the merge window to work perfectly, each step down 
the maintainership ladder needs to have time to review and integrate the 
changes destined for that merge window.  Ideally, people would do all 
this work beforehand, so that each step up the ladder has time prior to 
merge window for review and testing.

But that's just not software engineers as we know them ;-)


>>  The lack of a tree like this that we could have
>> persuaded people to test for the last month is what's causing us to
>> scramble like this at the closure of the merge window.
> 
> Nope.  The scramble is caused by subsystem maintainers jamming stuff into
> mainline at the last minute so they don't have to sit on it for the next
> two months.

Indeed.  Particularly in this case, where bsg didn't really grace -mm at 
all.


> Look.  If we're serious about this then the rule needs to be something like
> 
>   If it wasn't committed to your tree *at least* two weeks prior to the
>   2.6.x merge window opening, it shouldn't go into 2.6.x.
> 
> People are not presently observing this sort of discipline by a metric
> mile.  And I'm not sure that we should, really.

My goal AT A MINIMUM with netdev and libata is to get stuff in at least 
one -mm release prior to merge window opening (though preferably a 
longer lead time than that).  Of course, reality intrudes, but that's my 
goal.

And I think it's a reasonable goal to push upon others (but I'm biased:))

	Jeff

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Rene Herman]

On 08/07/2007 05:55 AM, James Bottomley wrote:

> I really, *really* think we need a pre-release tree that consists of all 
> the upstream targetted features (i.e. all of the for the next merge 
> window git trees) and nothing else.  -mm doesn't really satisfy this, 
> because it has so much other stuff that the people I need to get testing 
> this don't trust it.

I much agree with this. I used to run -mm at least somewhat frequently if it 
included stuff I was interested in (random features, or for example latest 
ALSA) but gave that up after I had it break on something unrelated and (to 
me) uninteresting a few times too many. There's so much in there that before 
you know it you end up spending time not on that which you wanted to spend 
time on but on something completely unrelated which given a fixed amount of 
available total time (and level/spread of knowledge/interest) gets to be a 
problem.

For latest ALSA, I now sometimes look at the ALSA repositories directly but 
otherwise it seems I'm not testing much of anything before late -rc's.

Not everything going to Linus gets there by way of -mm, but perhaps it could 
help if Andrew split off -merge from -mm, where -merge contained only stuff 
(from -mm) expected to go upstream in the next merge window.

Not sure if this is proposing that other people do more work -- wouldn't 
want do to anything of the sort...

Rene.

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jeff Garzik]

James Bottomley wrote:
> OK ... that's arguable.  This one is larger than I like because of the
> lpfc bug fix patch ... I accept I need to do a better job getting these
> into the merge window via the scsi-misc tree.  So I will accept the "too
> big" criticism and try to manage the driver maintainers better.
> 
> However, I won't accept the "not bug fixes only" criticism at -rc1.  The
> problem is that we're trying to stabilise a new feature: bsg.

Just so we don't lose the forest for the trees...

Not trying to put words in Linus's mouth, but it seems to me he wasn't 
complaining specifically about bsg.  "style cleanups", "cosmetic 
cleanups", ancient ISA driver polishing (1542, my gdth patch) are 
definitely not "bug fix only" material.

The lpfc update was probably the biggest thing, LOC-wise.  And even 
though that was mostly bug fixes -- and notably NOT 100% fixes -- it is 
big enough to warrant integration testing and exposure prior to 
mainline.  Definitely merge-window-open material AFAICS.

	Jeff

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[James Smart]

Jeff Garzik wrote:
> The lpfc update was probably the biggest thing, LOC-wise.  And even 
> though that was mostly bug fixes -- and notably NOT 100% fixes -- it is 
> big enough to warrant integration testing and exposure prior to 
> mainline.  Definitely merge-window-open material AFAICS.

FYI - it is integrated and tested prior to mainline, by Emulex (and who
else *really* tests it close to the degree we do ?). We do so, as a whole,
weeks ahead of the submit to the maintainer. Usually, there's only a couple
of small api changes that are picked up when we merge into the maintainers
pool.  And most of these are caught by us prior anyway as we package the
patchsets and ensure the integration into the maintainers pool is smooth.

-- james s

Re: [GIT PATCH] scsi bug fixes for 2.6.23-rc2

[Jeff Garzik]

James Smart wrote:
> Jeff Garzik wrote:
>> The lpfc update was probably the biggest thing, LOC-wise.  And even 
>> though that was mostly bug fixes -- and notably NOT 100% fixes -- it 
>> is big enough to warrant integration testing and exposure prior to 
>> mainline.  Definitely merge-window-open material AFAICS.
> 
> FYI - it is integrated and tested prior to mainline, by Emulex (and who
> else *really* tests it close to the degree we do ?). We do so, as a whole,
> weeks ahead of the submit to the maintainer. Usually, there's only a couple
> of small api changes that are picked up when we merge into the maintainers
> pool.  And most of these are caught by us prior anyway as we package the
> patchsets and ensure the integration into the maintainers pool is smooth.

This is a highly common pattern, and unfortunately you get the highly 
common Linux response:

In Linux we never ever assume a driver is working simply because the 
hardware vendor tested it.  A decade of real world experience PROVES 
precisely the opposite -- getting code out into the world early and 
often repeatedly turned up problems not seen in hardware vendor's testing.

Take a lesson from when I was on Linus's shit-list... twice:  Twice, 
Intel submitted an e1000 update after the merge window closed.  Twice, 
they claimed the driver passed their quite-exhaustive internal testing. 
  And twice, the most popular network driver broke for large masses of 
users because I took a hardware vendor's word on testing rather than 
rely on the testing PROVEN to flush out problems:  public linux kernel 
testing.

I'm not singling out Intel, there are plenty of other hardware vendors 
that repeat the exact same pattern.

It's quite simply impossible for a hardware vendor to test all the weird 
combinations in the field.  Our test lab -- the Internet -- is the one 
we trust.

	Jeff