Error report: perfbook Section 11.6.3.2

Error report: perfbook Section 11.6.3.2

[Isaac To]

Hi,

Thanks a lot for the free book, it is very useful!

I'd like to point out an error I see in Section 11.6.3.2, after Quick Quiz
11.10.  It says the following:

  Suppose that a given test fails about once every hour, but after a bug fix,
  a 24-hour test run fails only twice. What is the probability of this being
  due to random chance, in other words, what is the probability that the fix
  had no statistical effect?

In other parts of the book, care has been taken to say something like
"confidence level" to make the probability statements correct.  Not here.

The only thing that we know about the probability of failing after the
experiment is that "the probability of the test failing now is not zero".
To know the "probability of this being due to random chance", or
"probability that the fix had no effect", requires the knowledge of the
behaviour of the bug itself (or, at least, the prior probability that the
perhaps buggy system has a certain behaviour), and it is not specified in
the question.  E.g., in an extreme world that the test can only fail at a
rate of once every hour or never at all, we must admit that we are very
very lucky during this particular test.

The probability being calculated needs to be specified like "the
probability of this happening *in case* the fix has no effect".  That "the
fix has no effect" needs to be a *prerequisite*, and cannot itself be the
probability to be computed.  The complement, i.e., 1 - 1.2e-8, is "the
confidence level that the probability of failing is less than the
original".  I'd suggest using the "confidence level" wording here, but
explain what it is earlier in the book to tell the less mathematically apt
readers understand the wordings.

Regards,
Isaac

Re: Error report: perfbook Section 11.6.3.2

[Paul E. McKenney]

On Mon, Jul 21, 2014 at 04:24:26PM +0800, Isaac To wrote:
> Hi,
> 
> Thanks a lot for the free book, it is very useful!

Glad you like it!

> I'd like to point out an error I see in Section 11.6.3.2, after Quick Quiz
> 11.10.  It says the following:
> 
>   Suppose that a given test fails about once every hour, but after a bug fix,
>   a 24-hour test run fails only twice. What is the probability of this being
>   due to random chance, in other words, what is the probability that the fix
>   had no statistical effect?
> 
> In other parts of the book, care has been taken to say something like
> "confidence level" to make the probability statements correct.  Not here.
> 
> The only thing that we know about the probability of failing after the
> experiment is that "the probability of the test failing now is not zero".
> To know the "probability of this being due to random chance", or
> "probability that the fix had no effect", requires the knowledge of the
> behaviour of the bug itself (or, at least, the prior probability that the
> perhaps buggy system has a certain behaviour), and it is not specified in
> the question.  E.g., in an extreme world that the test can only fail at a
> rate of once every hour or never at all, we must admit that we are very
> very lucky during this particular test.
> 
> The probability being calculated needs to be specified like "the
> probability of this happening *in case* the fix has no effect".  That "the
> fix has no effect" needs to be a *prerequisite*, and cannot itself be the
> probability to be computed.  The complement, i.e., 1 - 1.2e-8, is "the
> confidence level that the probability of failing is less than the
> original".  I'd suggest using the "confidence level" wording here, but
> explain what it is earlier in the book to tell the less mathematically apt
> readers understand the wordings.

Good catch!

How about if I reworded that paragraph as follows?

	Suppose that a given test fails about once every hour, but after a
	bug fix, a 24-hour test run fails only twice.  Assuming that the
	failure leading to the bug is a random occurrence, and further
	assuming that the alleged fix actually had no effect on this
	particular bug, what is the probability that the small number
	of failures in the second run was due to random chance? This
	probability may be calculated by summing Equation 11.26 as
	follows:

I am shying away from explaining "confidence level" because I haven't
yet come up with a compact and accurate way of doing so.  However, I am
taking this email as encouragement to keep trying.  ;-)

							Thanx, Paul

PS.  Congratulations!  You are the first to use the new mailing list
     to report a bug in this book.  ;-)

Re: Error report: perfbook Section 11.6.3.2

[Isaac To]

On Tue, Jul 22, 2014 at 11:57 PM, Paul E. McKenney
<paulmck@linux.vnet.ibm.com> wrote:
>> The probability being calculated needs to be specified like "the
>> probability of this happening *in case* the fix has no effect".  That "the
>> fix has no effect" needs to be a *prerequisite*, and cannot itself be the
>> probability to be computed.  The complement, i.e., 1 - 1.2e-8, is "the
>> confidence level that the probability of failing is less than the
>> original".  I'd suggest using the "confidence level" wording here, but
>> explain what it is earlier in the book to tell the less mathematically apt
>> readers understand the wordings.
>
> Good catch!
>
> How about if I reworded that paragraph as follows?
>
>         Suppose that a given test fails about once every hour, but after a
>         bug fix, a 24-hour test run fails only twice.  Assuming that the
>         failure leading to the bug is a random occurrence, and further
>         assuming that the alleged fix actually had no effect on this
>         particular bug, what is the probability that the small number
>         of failures in the second run was due to random chance? This
>         probability may be calculated by summing Equation 11.26 as
>         follows:

I think if you already say "assuming that the failure ... is a random
occurrence", then "was due to random chance" is redundant.  Perhaps
"What is the probability of the small number of failures, assuming
that the alleged fix actually has no effect on the test failure?"
suffices.  On the other hand, that would not be very consistent with
other parts of the book that uses the "confidence level" terminology.
And the term is not defined elsewhere either.

> I am shying away from explaining "confidence level" because I haven't
> yet come up with a compact and accurate way of doing so.  However, I am
> taking this email as encouragement to keep trying.  ;-)

Yes that is hard.  Actually I like the confidence level wording more,
not only because it is more compact, but also because the wording
clearly explain why the probability calculated is noteworthy.  On the
other hand, I think the book is more on programming rather than
mathematics, so I won't complain if the book does not try to teach
fancy statistical ideas if that is not needed in the main theme.

Regards,
Isaac

Re: Error report: perfbook Section 11.6.3.2

[Paul E. McKenney]

On Sat, Jul 26, 2014 at 04:47:51PM +0800, Isaac To wrote:
> On Tue, Jul 22, 2014 at 11:57 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> >> The probability being calculated needs to be specified like "the
> >> probability of this happening *in case* the fix has no effect".  That "the
> >> fix has no effect" needs to be a *prerequisite*, and cannot itself be the
> >> probability to be computed.  The complement, i.e., 1 - 1.2e-8, is "the
> >> confidence level that the probability of failing is less than the
> >> original".  I'd suggest using the "confidence level" wording here, but
> >> explain what it is earlier in the book to tell the less mathematically apt
> >> readers understand the wordings.
> >
> > Good catch!
> >
> > How about if I reworded that paragraph as follows?
> >
> >         Suppose that a given test fails about once every hour, but after a
> >         bug fix, a 24-hour test run fails only twice.  Assuming that the
> >         failure leading to the bug is a random occurrence, and further
> >         assuming that the alleged fix actually had no effect on this
> >         particular bug, what is the probability that the small number
> >         of failures in the second run was due to random chance? This
> >         probability may be calculated by summing Equation 11.26 as
> >         follows:
> 
> I think if you already say "assuming that the failure ... is a random
> occurrence", then "was due to random chance" is redundant.  Perhaps
> "What is the probability of the small number of failures, assuming
> that the alleged fix actually has no effect on the test failure?"
> suffices.  On the other hand, that would not be very consistent with
> other parts of the book that uses the "confidence level" terminology.
> And the term is not defined elsewhere either.

OK, I now say the following, which removes the redundancy but links to
the concept of confidence:

	Suppose that a given test fails about once every hour, but after
	a bug fix, a 24-hour test run fails only twice.  Assuming that
	the failure leading to the bug is a random occurrence, what
	is the probability that the small number of failures in the
	second run was due to random chance?  In other words, how
	confident should we be that the fix actually had some effect
	on the bug?  This probability may be calculated by summing
	Equation 11.26 as follows:

> > I am shying away from explaining "confidence level" because I haven't
> > yet come up with a compact and accurate way of doing so.  However, I am
> > taking this email as encouragement to keep trying.  ;-)
> 
> Yes that is hard.  Actually I like the confidence level wording more,
> not only because it is more compact, but also because the wording
> clearly explain why the probability calculated is noteworthy.  On the
> other hand, I think the book is more on programming rather than
> mathematics, so I won't complain if the book does not try to teach
> fancy statistical ideas if that is not needed in the main theme.

Well, I do say this earlier on:

	These tools are extremely helpful, but please note that reading
	this section not a substitute for taking a good set of statistics
	classes.

And in case you were wondering about the derivation of the Poisson CDF,
that will be moving to an appendix shortly, and will probably be dropped
entirely to make room for more relevant things.  What can I say?  I was
too proud of suddenly realizing how to derive it from scratch.  ;-)

							Thanx, Paul