Article in Phoronix about loss of performance in 2.6.35 release candidates

Article in Phoronix about loss of performance in 2.6.35 release candidates

[Alex Buell]

http://www.phoronix.com/vr.php?view=14976

Question: Why?
-- 
http://www.munted.org.uk

One very high maintenance cat living here.

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Robert Hancock]

On 05/31/2010 05:19 PM, Alex Buell wrote:
> http://www.phoronix.com/vr.php?view=14976
>
> Question: Why?

Good question.. I guess it would too much to ask of them to try to 
figure out what area the problem lies in (even to the point of figuring 
out if it's a CPU or IO-bound problem), or try to bisect, or at least 
report it to LKML before going to the trouble of creating 5 pages of 
graphs.. Given the 20x slowdown in some of the benchmarks you'd think it 
wouldn't be too hard to narrow down.

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Alex Buell]

On Mon, 2010-05-31 at 17:39 -0600, Robert Hancock wrote:
> On 05/31/2010 05:19 PM, Alex Buell wrote:
> > http://www.phoronix.com/vr.php?view=14976
> >
> > Question: Why?
> 
> Good question.. I guess it would too much to ask of them to try to 
> figure out what area the problem lies in (even to the point of figuring 
> out if it's a CPU or IO-bound problem), or try to bisect, or at least 
> report it to LKML before going to the trouble of creating 5 pages of 
> graphs.. Given the 20x slowdown in some of the benchmarks you'd think it 
> wouldn't be too hard to narrow down.

What I really don't get is why they didn't talk to people on
linux-kernel before posting those claims on phoronix. :S 
-- 
http://www.munted.org.uk

One very high maintenance cat living here.

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Dave Airlie]

On Tue, Jun 1, 2010 at 10:46 AM, Alex Buell <alex.buell@munted.org.uk> wrote:
> On Mon, 2010-05-31 at 17:39 -0600, Robert Hancock wrote:
>> On 05/31/2010 05:19 PM, Alex Buell wrote:
>> > http://www.phoronix.com/vr.php?view=14976
>> >
>> > Question: Why?
>>
>> Good question.. I guess it would too much to ask of them to try to
>> figure out what area the problem lies in (even to the point of figuring
>> out if it's a CPU or IO-bound problem), or try to bisect, or at least
>> report it to LKML before going to the trouble of creating 5 pages of
>> graphs.. Given the 20x slowdown in some of the benchmarks you'd think it
>> wouldn't be too hard to narrow down.
>
> What I really don't get is why they didn't talk to people on
> linux-kernel before posting those claims on phoronix. :S

Well thats kinda obvious, its "journalism" in the days of google
adwords. They make revenue by making people click on their website,
they don't make money being useful or interacting with others.
Phoronix in all the years I've been dealing with them as the only
place doing any reporting on graphics, have never once confirmed a
source, asked for information directly or anything you'd expect from
real journalists, again because that doesn't drive page hits, whereas
sensationalist useless headlines are the main point of the site.

Dave.

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Bill Davidsen]

Robert Hancock wrote:
> On 05/31/2010 05:19 PM, Alex Buell wrote:
>> http://www.phoronix.com/vr.php?view=14976
>>
>> Question: Why?
> 
> Good question.. I guess it would too much to ask of them to try to 
> figure out what area the problem lies in (even to the point of figuring 
> out if it's a CPU or IO-bound problem), or try to bisect, or at least 
> report it to LKML before going to the trouble of creating 5 pages of 
> graphs.. Given the 20x slowdown in some of the benchmarks you'd think it 
> wouldn't be too hard to narrow down.

That's true, but 20x should be too hard for people to detect when they do QA 
after creating a patch, before sending it to LKML in the first place, either. If 
such a regression made it to an -rc1 then it really is kind of a big deal. Of 
course Phoronics running the tests on netbook processors is probably a good 
thing, I doubt many developers and testers are compiling kernels on a rig like 
that, or doing much of anything else demanding.

I guess I would expect people to react with dismay to the fact that such a 
problem made it undetected to rc stage, but perhaps I have too much respect for 
developers. This looks more like "how dare they not keep it quiet and just tell 
us" indignation. In the end I doubt it makes a lot of difference, if someone 
posted to LKML and Slashdot picked it up, be sure it would have hit the media 
anyway.

Should any media keep a defect quiet when they make their living informing the 
readers? I see a lot of glee among Linux users every few days when a new Windows 
bug becomes public. Phoronics tested and reported, why is that less honorable 
than Tom's Hardware telling us a new CPU sucks?


-- 
Bill Davidsen <davidsen@tmr.com>
   "We have more to fear from the bungling of the incompetent than from
the machinations of the wicked."  - from Slashdot

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Robert Hancock]

On Mon, May 31, 2010 at 9:15 PM, Bill Davidsen <davidsen@tmr.com> wrote:
> Robert Hancock wrote:
>>
>> On 05/31/2010 05:19 PM, Alex Buell wrote:
>>>
>>> http://www.phoronix.com/vr.php?view=14976
>>>
>>> Question: Why?
>>
>> Good question.. I guess it would too much to ask of them to try to figure
>> out what area the problem lies in (even to the point of figuring out if it's
>> a CPU or IO-bound problem), or try to bisect, or at least report it to LKML
>> before going to the trouble of creating 5 pages of graphs.. Given the 20x
>> slowdown in some of the benchmarks you'd think it wouldn't be too hard to
>> narrow down.
>
> That's true, but 20x should be too hard for people to detect when they do QA
> after creating a patch, before sending it to LKML in the first place,
> either. If such a regression made it to an -rc1 then it really is kind of a
> big deal. Of course Phoronics running the tests on netbook processors is
> probably a good thing, I doubt many developers and testers are compiling
> kernels on a rig like that, or doing much of anything else demanding.
>
> I guess I would expect people to react with dismay to the fact that such a
> problem made it undetected to rc stage, but perhaps I have too much respect
> for developers. This looks more like "how dare they not keep it quiet and
> just tell us" indignation. In the end I doubt it makes a lot of difference,
> if someone posted to LKML and Slashdot picked it up, be sure it would have
> hit the media anyway.
>
> Should any media keep a defect quiet when they make their living informing
> the readers? I see a lot of glee among Linux users every few days when a new
> Windows bug becomes public. Phoronics tested and reported, why is that less
> honorable than Tom's Hardware telling us a new CPU sucks?

Of course they shouldn't keep it quiet. The problem is they went and
wrote an article that was basically "OMG HUGE PERFORMANCE LOSS!!1!!"
without reporting the problem to people that can actually do something
about it, and also didn't provide any very useful details like dmesg,
config, etc. that might let someone figure out what's going on.

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Mike Galbraith]

On Mon, 2010-05-31 at 21:22 -0600, Robert Hancock wrote:
> On Mon, May 31, 2010 at 9:15 PM, Bill Davidsen <davidsen@tmr.com> wrote:
> > Robert Hancock wrote:
> >>
> >> On 05/31/2010 05:19 PM, Alex Buell wrote:
> >>>
> >>> http://www.phoronix.com/vr.php?view=14976
> >>>
> >>> Question: Why?
> >>
> >> Good question.. I guess it would too much to ask of them to try to figure
> >> out what area the problem lies in (even to the point of figuring out if it's
> >> a CPU or IO-bound problem), or try to bisect, or at least report it to LKML
> >> before going to the trouble of creating 5 pages of graphs.. Given the 20x
> >> slowdown in some of the benchmarks you'd think it wouldn't be too hard to
> >> narrow down.
> >
> > That's true, but 20x should be too hard for people to detect when they do QA
> > after creating a patch, before sending it to LKML in the first place,
> > either. If such a regression made it to an -rc1 then it really is kind of a
> > big deal. Of course Phoronics running the tests on netbook processors is
> > probably a good thing, I doubt many developers and testers are compiling
> > kernels on a rig like that, or doing much of anything else demanding.
> >
> > I guess I would expect people to react with dismay to the fact that such a
> > problem made it undetected to rc stage, but perhaps I have too much respect
> > for developers. This looks more like "how dare they not keep it quiet and
> > just tell us" indignation. In the end I doubt it makes a lot of difference,
> > if someone posted to LKML and Slashdot picked it up, be sure it would have
> > hit the media anyway.
> >
> > Should any media keep a defect quiet when they make their living informing
> > the readers? I see a lot of glee among Linux users every few days when a new
> > Windows bug becomes public. Phoronics tested and reported, why is that less
> > honorable than Tom's Hardware telling us a new CPU sucks?
> 
> Of course they shouldn't keep it quiet. The problem is they went and
> wrote an article that was basically "OMG HUGE PERFORMANCE LOSS!!1!!"
> without reporting the problem to people that can actually do something
> about it, and also didn't provide any very useful details like dmesg,
> config, etc. that might let someone figure out what's going on.

Shrug.  If eggshells land in our omelet, they can make a buck telling
people about it.  Who cares?  If tasty bacon bits land, they'll make a
buck on that event.  Either way, we get some test coverage.

	-Mike

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[tytso]

On Tue, Jun 01, 2010 at 12:19:29AM +0100, Alex Buell wrote:
> http://www.phoronix.com/vr.php?view=14976
> 
> Question: Why?

One of the theories that has been advanced is that it's simply this
problem:

	http://lkml.org/lkml/2010/5/22/154

If so, it points out how idiotic Phoronix is about not being able to
notice udev pegging the CPU at 100% being someone bad for its
benchmark runs.  :-)

OTOH, this bug has been known for over a week, and it is sort sad that
we haven't reverted this patch.  It looks like the conversation has
died, but without a fix?

						- Ted

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Pekka Enberg]

On Tue, Jun 1, 2010 at 7:10 AM,  <tytso@mit.edu> wrote:
> On Tue, Jun 01, 2010 at 12:19:29AM +0100, Alex Buell wrote:
>> http://www.phoronix.com/vr.php?view=14976
>>
>> Question: Why?
>
> One of the theories that has been advanced is that it's simply this
> problem:
>
>        http://lkml.org/lkml/2010/5/22/154
>
> If so, it points out how idiotic Phoronix is about not being able to
> notice udev pegging the CPU at 100% being someone bad for its
> benchmark runs.  :-)
>
> OTOH, this bug has been known for over a week, and it is sort sad that
> we haven't reverted this patch.  It looks like the conversation has
> died, but without a fix?

It's fixed by 1eb2cbb6d5efe129 so the problem doesn't exist for 2.6.35-r1.

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Ingo Molnar]

* Pekka Enberg <penberg@cs.helsinki.fi> wrote:

> On Tue, Jun 1, 2010 at 7:10 AM,  <tytso@mit.edu> wrote:
> > On Tue, Jun 01, 2010 at 12:19:29AM +0100, Alex Buell wrote:
> >> http://www.phoronix.com/vr.php?view=14976
> >>
> >> Question: Why?
> >
> > One of the theories that has been advanced is that it's simply this
> > problem:
> >
> > ? ? ? ?http://lkml.org/lkml/2010/5/22/154
> >
> > If so, it points out how idiotic Phoronix is about not being able to
> > notice udev pegging the CPU at 100% being someone bad for its
> > benchmark runs. ?:-)
> >
> > OTOH, this bug has been known for over a week, and it is sort sad that
> > we haven't reverted this patch. ?It looks like the conversation has
> > died, but without a fix?
> 
> It's fixed by 1eb2cbb6d5efe129 so the problem doesn't exist for 2.6.35-r1.

Btw., the changelog in 1eb2cbb6d5efe129:

 | From 1eb2cbb6d5efe129cd006691267ce513c0aa59da Mon Sep 17 00:00:00 2001
 | From: Al Viro <viro@zeniv.linux.org.uk>
 | Date: Thu, 27 May 2010 11:11:06 -0400
 | Subject: [PATCH] Revert "anon_inode: set S_IFREG on the anon_inode"
 | 
 | This reverts commit a7cf4145bb86aaf85d4d4d29a69b50b688e2e49d.
 | ---
 |  fs/anon_inodes.c |    2 +-
 |  1 files changed, 1 insertions(+), 1 deletions(-)

Is lacking pretty much all essential pieces of information which are required 
for reverts:

 - it has no description about who reported the bug.

 - it has no description about what the problem was and what the effects were. 
   People have little chance to link up the 'udev is spinning' bug to
   this fix.

 - it has no link back to '3836a03: anon_inodes: mark the anon inode private' 
   which introduced it and hence it has no -stable tag either.

 - has no signoff line

	Ingo

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> Is lacking pretty much all essential pieces of information which are 
> required for reverts:
> 
>  - it has no description about who reported the bug.

In fact it was _bisected_ by Eric Paris, so a Bisected-by tag is a must.

Eric is a nice guy and i'm sure wont complain about this, but i can tell it 
with a 100% certainty that to most testers and developers bisecting bugs is a 
tiresome and error-prone process that involves quite a bit of work.

Bisecting a bug is often more work to do than the work which went into the 
original commit and the revert, combined.

So if we want more people to test our commits then the least we can do is to 
credit testers by making proper use of the Reported-by and Bisected-by tags.

	Ingo

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

>  - it has no link back to '3836a03: anon_inodes: mark the anon inode private' 
>    which introduced it and hence it has no -stable tag either.

Ah, it got introduced by a7cf4145bb86 so no -stable relevancy.

	Ingo

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Chris Mason]

On Tue, Jun 01, 2010 at 08:00:39AM +0300, Pekka Enberg wrote:
> On Tue, Jun 1, 2010 at 7:10 AM,  <tytso@mit.edu> wrote:
> > On Tue, Jun 01, 2010 at 12:19:29AM +0100, Alex Buell wrote:
> >> http://www.phoronix.com/vr.php?view=14976
> >>
> >> Question: Why?
> >
> > One of the theories that has been advanced is that it's simply this
> > problem:
> >
> >        http://lkml.org/lkml/2010/5/22/154
> >
> > If so, it points out how idiotic Phoronix is about not being able to
> > notice udev pegging the CPU at 100% being someone bad for its
> > benchmark runs.  :-)
> >
> > OTOH, this bug has been known for over a week, and it is sort sad that
> > we haven't reverted this patch.  It looks like the conversation has
> > died, but without a fix?
> 
> It's fixed by 1eb2cbb6d5efe129 so the problem doesn't exist for 2.6.35-r1.

When I first read this email, I thought it meant the test was done
on rc1, but reading the article:

	To cut to the chase, between the 22nd and 24th of May there
	looks to be at least one commit (though perhaps multiple based
	upon the different data) within the Linus Torvalds 2.6 Git tree
	that are negatively affecting many different server/desktop
	benchmarks. We waited nearly a week to see if these regressions
	would be organically caught and addressed, but they have not
	been at least of the Linux 2.6 Git state as of 2010-05-26.

I'll give the udev fix a try w/the btrfs tests.

-chris

Re: Article in Phoronix about loss of performance in 2.6.35 release candidates

[Ingo Molnar]

* tytso@mit.edu <tytso@mit.edu> wrote:

> On Tue, Jun 01, 2010 at 12:19:29AM +0100, Alex Buell wrote:
> > http://www.phoronix.com/vr.php?view=14976
> > 
> > Question: Why?
> 
> One of the theories that has been advanced is that it's simply this problem:
> 
> 	http://lkml.org/lkml/2010/5/22/154
> 
> If so, it points out how idiotic Phoronix is about not being able to notice 
> udev pegging the CPU at 100% being someone bad for its benchmark runs.  :-)

They might be 'minimally helpful' and they might want to maximize clicks, but 
IMO that doesnt equate to 'idiotic', at all. The slowdown was real and they 
have no obligation to figure out what caused it.

Think what an ordinary tester would do: install -rc1, see a slowdown, skip 
back to v2.6.34. Reporting the 'macro performance' metrics is useful.

As Linux gets more and more mainstream we should get used to 
selfish/hostile/unhelpful journalism, just like we had to get used to 
selfish/hostile/unhelpful users, developers, etc. This isnt directed at Linux, 
this is simply an unavoidable side effect of becoming more mainstream. And 
there will be more of this, hopefully ;-)

Thanks,

	Ingo