Re: LSF Papers online?

Re: LSF Papers online?

[Jeff Garzik]

Bartlomiej Zolnierkiewicz wrote:
> On Tuesday 14 April 2009 00:24:57 Jeff Garzik wrote:
>> Bartlomiej Zolnierkiewicz wrote:
>>> I've started reading it and immediately noticed a thing which made by day. :-)
>>>
>>> Sorry if it will sound off-topic or undiplomatic but it is the best occasion
>>> to straighten up some facts:
>>>
>>>  "Discussion then moved on to the current status of getting libata out of
>>>   SCSI: we have had several successes, notably timer handling and pieces of
>>>   error handling have moved up to block. Unfortunately, the current progress
>>>   has reached the point where it's being impeded by the legacy IDE subsystem
>>>
>>> Heh, you can also blame the lack of world peace on the legacy IDE subsystem.
>>>
>>> I wonder who came up with this ridiculous excuse (I'm sure it wasn't James!).
> 
> It was you!? :)
> 
>>> The thing is that during last _five_ years almost nothing was done in this
>>> direction.  Despite the fact that it was #1 condition under which the whole
>>> code has been merged.  Sorry to say it but it seems like the whole merge
>>> strategy was to over-promise things now and worry about delivery later.
>> Yet, shockingly, users have been happily using libata despite all these 
>> horrors.
> 
> That was not the issue raised:
> 
> If you think that you can take a "I will deliver later" credit from the
> developers community and later cover it up by "this is still my goal, I
> just need to find some suckers to do it for me" and think that you won by
> fooling people you're sadly mistaken and will most likely have a reality
> check one day (not from me, I really don't care that much to waste my
> precious time on proving you wrong).

The project you refer to -- move libata out of SCSI -- is far less 
important than another project:  keep libata going amidst new SAS and 
SATA hardware.

Choosing to use the SCSI driver infrastructure was a solid technical 
decision in the beginning, and time has proven that true:  since we were 
inside SCSI, ATAPI and SAT support came naturally.  Support for SAS+SATA 
controllers came naturally.

So it was absolutely the right thing to do for Linux users, to 
de-prioritize the libata-out-of-SCSI project.

The users were not, and are not, asking for it.  It will even introduce 
some breakage if you're not careful.

The only people who even mention it are a few key IDE and block layer 
developers - me, you, Tejun, Jens, sometimes James B.  Linus has 
probably forgotten, but for I occasionally mention it at kernel summits.

I think Linux users are happy that they were delivered a working ATA 
driver of a much more clean design.  That is the delivery that matters.

Even with the benefit of hindsight, I don't see that libata development 
should have happened any other way.

Moving libata out of SCSI is now a long term, far off goal.  A goal that 
implies many intermediate steps, cleanups to block, libata, IDE, SCSI 
and other block drivers.

I am highly confident we will reach this goal eventually, but there is 
no rush.  If it takes ten years, fine.  THIS IS THE PROCESS.  The end 
result will be that all storage drivers in the kernel are improved.

We steer this ship by having a general idea of where we want to go, not 
a specific roadmap.  Interesting, unexpected things happen during the 
journey, perhaps taking you down a different course.  Open source... 
it's all very zen.  :)

	Jeff

Re: LSF Papers online?

[Bartlomiej Zolnierkiewicz]

On Tuesday 14 April 2009 12:14:00 Jeff Garzik wrote:
> Bartlomiej Zolnierkiewicz wrote:
> > On Tuesday 14 April 2009 00:24:57 Jeff Garzik wrote:
> >> Bartlomiej Zolnierkiewicz wrote:
> >>> I've started reading it and immediately noticed a thing which made by day. :-)
> >>>
> >>> Sorry if it will sound off-topic or undiplomatic but it is the best occasion
> >>> to straighten up some facts:
> >>>
> >>>  "Discussion then moved on to the current status of getting libata out of
> >>>   SCSI: we have had several successes, notably timer handling and pieces of
> >>>   error handling have moved up to block. Unfortunately, the current progress
> >>>   has reached the point where it's being impeded by the legacy IDE subsystem
> >>>
> >>> Heh, you can also blame the lack of world peace on the legacy IDE subsystem.
> >>>
> >>> I wonder who came up with this ridiculous excuse (I'm sure it wasn't James!).
> > 
> > It was you!? :)
> > 
> >>> The thing is that during last _five_ years almost nothing was done in this
> >>> direction.  Despite the fact that it was #1 condition under which the whole
> >>> code has been merged.  Sorry to say it but it seems like the whole merge
> >>> strategy was to over-promise things now and worry about delivery later.
> >> Yet, shockingly, users have been happily using libata despite all these 
> >> horrors.
> > 
> > That was not the issue raised:
> > 
> > If you think that you can take a "I will deliver later" credit from the
> > developers community and later cover it up by "this is still my goal, I
> > just need to find some suckers to do it for me" and think that you won by
> > fooling people you're sadly mistaken and will most likely have a reality
> > check one day (not from me, I really don't care that much to waste my
> > precious time on proving you wrong).
> 
> The project you refer to -- move libata out of SCSI -- is far less 
> important than another project:  keep libata going amidst new SAS and 
> SATA hardware.
> 
> Choosing to use the SCSI driver infrastructure was a solid technical 
> decision in the beginning, and time has proven that true:  since we were 
> inside SCSI, ATAPI and SAT support came naturally.  Support for SAS+SATA 
> controllers came naturally.
> 
> So it was absolutely the right thing to do for Linux users, to 
> de-prioritize the libata-out-of-SCSI project.

You made this decision yourself without consulting it with anybody.

> The users were not, and are not, asking for it.  It will even introduce 
> some breakage if you're not careful.

The users were not asking for many other things, i.e. libata PATA.

> The only people who even mention it are a few key IDE and block layer 
> developers - me, you, Tejun, Jens, sometimes James B.  Linus has 
> probably forgotten, but for I occasionally mention it at kernel summits.

Did you you forget about Intel guys and other people working on SSDs?

> I think Linux users are happy that they were delivered a working ATA 
> driver of a much more clean design.  That is the delivery that matters.

It was clean design in 2003.

Now it is yet another fairly complex piece of code.

i.e. please go into libata-eh.c, try to make sense out of it and track all
subtle libata-SCSI-block interactions (yes, the code is clean by mingo's
cleanness standard -- this is not the point here)

To keep the design clean and modern the constant _significant_ maintenance
work is needed.

I also don't buy this "That is the delivery that matters" mantra repeated
by some people.  Agreed delivery is the most important thing but how you get
there is also very important in the longer-term.

> Even with the benefit of hindsight, I don't see that libata development 
> should have happened any other way.

How's about starting with working on the existing ATA/IDE subsystem?

You lacked _any_ hands-on experience with it.

You were given green light on libata simply because we had no other choice:
you came up with the full solution without any previous discussions and used
the same "think about users" excuse.

The SATA features that needed SCSI infrastructure came 2 years later.

> Moving libata out of SCSI is now a long term, far off goal.  A goal that 
> implies many intermediate steps, cleanups to block, libata, IDE, SCSI 
> and other block drivers.

"far off"?

The fact that it is much harder to do nowadays than in 2004-2005 (without
ATAPI support, PATA support and heavy dependence on SCSI infrastructure)
is only _your_ fault.

> I am highly confident we will reach this goal eventually, but there is 
> no rush.  If it takes ten years, fine.  THIS IS THE PROCESS.  The end 
> result will be that all storage drivers in the kernel are improved.

"eventually"?

"no rush"?

"ten years"?

I don't have that much patience left.

> We steer this ship by having a general idea of where we want to go, not 
> a specific roadmap.  Interesting, unexpected things happen during the 
> journey, perhaps taking you down a different course.  Open source... 
> it's all very zen.  :)

Please save us the management fairy-tales and just show us the code.

Thanks,
Bart

Re: LSF Papers online?

[Jeff Garzik]

Bartlomiej Zolnierkiewicz wrote:
> On Tuesday 14 April 2009 12:14:00 Jeff Garzik wrote:


> The SATA features that needed SCSI infrastructure came 2 years later.
> 
>> Moving libata out of SCSI is now a long term, far off goal.  A goal that 
>> implies many intermediate steps, cleanups to block, libata, IDE, SCSI 
>> and other block drivers.
> 
> "far off"?
> 
> The fact that it is much harder to do nowadays than in 2004-2005 (without
> ATAPI support, PATA support and heavy dependence on SCSI infrastructure)
> is only _your_ fault.

Of course it is.  Use of SCSI driver infrastructure was a sound 
technical decision, I'll happily defend.  Key reasons SCSI core was used:

* ATA-SCSI convergence was clear when libata began.  Time has proven 
this true:

ATAPI was always SCSI-like.  SAS is plug-compatible with SATA [for some 
SAS plugs], and SAS transmits SATA frames from SAS expanders and SATA 
port multipliers.  T10 and T13 standards committees actively 
collaborate.  SCSI even has a specification, SAT, that describes how to 
best co-mingle ATA with SCSI.

* SCSI driver infrastructure was the only one advanced enough to support 
controller hotplug, device hotplug, and all sorts of queueing contortions.

* SCSI was the only infrastructure that _guaranteed_ it would work with 
existing installers and distros.  For users, there is a clear level of 
difference in support between /dev/hdXX, /dev/sdXX, and every other 
block device in the kernel.

SCSI had a higher Just Works(tm) value at the time.

	Jeff

Re: LSF Papers online?

[Alan Cox]

> The users were not asking for many other things, i.e. libata PATA.

Yes they were, and at the time libata PATA got done the old IDE code had
been rotting for a long time. It was only after that you decided to
revamp it all, duplicating work when you could have helped work on libata
PATA. Maybe then there would have been more time to continue working on
the split.

> > probably forgotten, but for I occasionally mention it at kernel summits.
> Did you you forget about Intel guys and other people working on SSDs?

I seem to be an Intel guy now 8)

> i.e. please go into libata-eh.c, try to make sense out of it and track all
> subtle libata-SCSI-block interactions (yes, the code is clean by mingo's
> cleanness standard -- this is not the point here)

libata-eh is pretty clear to follow and because it uses the SCSI approach
of quiescing the queue doesn't have the creepy horrors of the old IDE
layer which was trying to do resets and polled speed changes in IRQ
context racing against timers and completion events.

It does that despite handling NCQ, barriers and hotplug. It does this
despite being vastly smarter than the old IDE code in its error
processing heuristics.

> How's about starting with working on the existing ATA/IDE subsystem?
> 
> You lacked _any_ hands-on experience with it.

But those helping him had extensive, painful, experience with it having
maintained the relic for years. The decision to go with libata PATA was
well informed and made from a deep understanding of just how big a pile of
turd there was lurking in the old drivers, coupled with a view that it
would be better to create the new drivers in parallel rather than
destabilize the old code while progressing it.

> The fact that it is much harder to do nowadays than in 2004-2005 (without
> ATAPI support, PATA support and heavy dependence on SCSI infrastructure)
> is only _your_ fault.

Really - you could have contributed too. Anyway I wrote most of the PATA
bits, Tejun wrote a ton of stuff including EH, and Albert Lee wrote
chunks of it too.

So that would be Red Hat, SuSE and IBM who are to blame right, not just
Jeff ? Sounds like a conspiracy to me ;)

> Please save us the management fairy-tales and just show us the code.

You've misunderstood free software Bartlomiej. If you want something that
badly *you* should us the code or work with other like minded people.

Alan

Re: LSF Papers online?

[Bartlomiej Zolnierkiewicz]

On Tuesday 14 April 2009 18:54:41 Alan Cox wrote:
> > The users were not asking for many other things, i.e. libata PATA.
> 
> Yes they were, and at the time libata PATA got done the old IDE code had
> been rotting for a long time. It was only after that you decided to

Since you seem to love going over this over and over and over again... ;)

I would like you to remind you that I wasn't saying NAK to libata PATA
despite the fact that some people spend quite a big time on trying to
blame every single IDE problem on me (while I took over only in 2.5.7x,
received next to none help and wasn't even paid for this work)...

I just wanted to have clear vision of the process instead of "lets merge
it now and worry about real users later".

I couldn't get this vision neither from you nor Jeff.

There was no technical discussion with Red Hat people at all.

Every uncomfortable technical issue raised was addressed with direct
"IDE is teh crap" shouting or indirect "you're being difficult" clues.

And don't try to tell me that I was difficult.  Sure I was. :)

Like the vase majority of other kernel people.

In the end we should handle things at the technical level.

This was clearly not the case back then.

I could have also continued to work on IDE after libata PATA.

Even if I would eventually out do it, what would it change?

Fedora will still start shipping with CONFIG_IDE=n and Red Hat will
still bless libata PATA.

SuSE and Canonical will jump on on ship just to not divert too much.

In the best case I would get a lovely "Bart was right!" comment
somewhere in libata-core.c. ;)

I was able to see that much and my motivation was completely gone,
IDE blame shifting was also really very successful in this regard.

So I took a break limiting my involvement to patch reviewing and waiting
to see how would the situation develop.

It had developed more-or-less like it was predicted.

The "lets build a great new shiny drivers" utopia ended and the real-world
"oh we have dozens of regressions and still lack hardware coverage" phase
started.

> revamp it all, duplicating work when you could have helped work on libata
> PATA. Maybe then there would have been more time to continue working on
> the split.

Well, it is not like I was working day and night on fixing old IDE,
I just dusted off old patches and then the ball started to roll.

I did it purely because it was fun to work with many great people (like
Sergei or Borislav), there were still a lot of IDE users left behind by
libata PATA and I also didn't want to let my old patches go to waste.

Have you ever wondered why people are still using IDE or sending me
IDE patches?  No, it is not because I'm such a great guy to work with. :)

Hint: the most reasons are purely technical.

There are still dozens of IDE -> libata regressions.  The fact that such
fixes are being mislabeled as an "improvements" is hardly fooling anybody.

There are also uses (embedded world) when IDE is better cause it is
smaller and _much_ simpler.

Maybe you really shouldn't have brought this topic up. ;)

However since you did lets get to some productive conclusions this time.

I'm ready to discuss anything as long as we stick to technical facts.

libata PATA is still not a replacement for IDE and cannot be removed
anytime soon.

This is the technical fact.

How we should start addressing it and who is going to do work?

>From my side I can help with setting up a another quilt tree and starting
addressing host driver regressions (there still quite a few of them left),
and maybe porting a driver or two but this is it.

I'm quite time constrained and I'm also not leaving IDE behind (we still
have some pending changes there) so I need a lot of help on cleaning up,
porting and testing of some quite peculiar host drivers.

The whole process will go slowly and will take up long time.

No problem with this as long as we make progress.

This is of course given that you want to finish the work that you started
with libata PATA and not leave it half-way done.

Comments?  Thoughts?

> > > probably forgotten, but for I occasionally mention it at kernel summits.
> > Did you you forget about Intel guys and other people working on SSDs?
> 
> It seem to be an Intel guy now 8)
> 
> > i.e. please go into libata-eh.c, try to make sense out of it and track all
> > subtle libata-SCSI-block interactions (yes, the code is clean by mingo's
> > cleanness standard -- this is not the point here)
> 
> libata-eh is pretty clear to follow and because it uses the SCSI approach
> of quiescing the queue doesn't have the creepy horrors of the old IDE
> layer which was trying to do resets and polled speed changes in IRQ
> context racing against timers and completion events.
> 
> It does that despite handling NCQ, barriers and hotplug. It does this
> despite being vastly smarter than the old IDE code in its error
> processing heuristics.

Well, old IDE is a pretty weak reference point.

> > How's about starting with working on the existing ATA/IDE subsystem?
> > 
> > You lacked _any_ hands-on experience with it.
> 
> But those helping him had extensive, painful, experience with it having
> maintained the relic for years. The decision to go with libata PATA was
> well informed and made from a deep understanding of just how big a pile of
> turd there was lurking in the old drivers, coupled with a view that it
> would be better to create the new drivers in parallel rather than
> destabilize the old code while progressing it.

Well, you are not the only person that had such bad experiences with it.

Plus I also put a significant effort into gaining insight into the
subsystem during 2001-2004 days -- this wasn't clearly visible cause
my patch game sucked big time compared to my review game (yes, it
still sucks now but it is getting bit better).  Moreover I learned
a lot do-nots from the results of 2.5.x IDE rewrite attempt.

I saw that it could have been done in a evolutionary way (which worked
miracles with SCSI subsystem) without destabilizing it too much and in
a timely manner.  Got the agreement about this with libata people and
I started the work in 2003-2004.

Lets skip the historical part here.

The rework was later finished in 2007-2008.

Despite facing heavy competition and lacking any corporate support.

With a few voluntary IDE developers and a bit of help from community.

We actually did a whole lot more changes that were originally intended.

Hey, I've never imagined that we will rewrite almost whole subsystem.

It was possible.  See?

We actually used quite a lot of libata PATA changes on the host driver front
(though because of the lack of incremental libata PATA changes and the need
to fix some IDE specific issues first this was not that easy) so maybe if we
could agree on some mid-point way back in 2005 we would be in a completely
different place today.

Lets not repeat the history again.

> > The fact that it is much harder to do nowadays than in 2004-2005 (without
> > ATAPI support, PATA support and heavy dependence on SCSI infrastructure)
> > is only _your_ fault.
> 
> Really - you could have contributed too. Anyway I wrote most of the PATA
> bits, Tejun wrote a ton of stuff including EH, and Albert Lee wrote
> chunks of it too.

I did a bit work.  Check your host drivers. :)

> So that would be Red Hat, SuSE and IBM who are to blame right, not just
> Jeff ? Sounds like a conspiracy to me ;)

There is no conspiracy there.

Just the good old game known from elsewhere.

Top layer modern day kernel hacking is based on business principles.

However, like they say "Hate the Game, Not the Player". :)

There is really nothing wrong with it as long as we don't state
otherwise to new people so they are fully aware of the situation.

> > Please save us the management fairy-tales and just show us the code.
> 
> You've misunderstood free software Bartlomiej. If you want something that

Indeed, I misunderstood it, not this point though.

Anyway, you guys know what I'm getting at.

The opportunity window that was there in 2004-2006 to push things
forward in a *really* major way and we failed to use.

As I see it now it was in large part my fault of seeing the chance
and not being able to convince people about it so I'm not pointing
finger at anybody else.

> badly *you* should us the code or work with other like minded people.

Well, it wasn't me who promised the delivery within the year and later
avoided the topic for half of the decade.

If Jeff changed his mind or simply didn't want to do this work it was as
simple as saying it, in PM or otherwise.  Seriously.

Avoiding discussion about uncomfortable topics prevents a progress.

Thanks,
Bart

"I feel so right, yet I'm so wrong"

Re: LSF Papers online?

[James Bottomley]

On Wed, 2009-04-15 at 00:09 +0200, Bartlomiej Zolnierkiewicz wrote:

OK, look, guys, could we stop this argument?  It's becoming a bit old.

For the record, I was unhappy to have libata pata in SCSI, but it got
moved into drivers/ata shortly after, limiting my influence.  I thought
having a slow wind down of legacy pata in drivers/ide but moving to
libata for sata was the correct split.  I agree with Jeff that the SCSI
layer did provide unique features that SATA/NCQ needed at the time ...
but I also think we need to move them into block sooner rather than
later.

I'm the last person ever to proscribe a kernel subsystem that has
willing volunteers, so drivers/ide is yours as long as you want to
maintain it.  I can certainly see the merits of a thinner stack to the
embedded world, and not a few embedded developers seem to agree.

As far as moving it out of SCSI goes, I've always been a supporter of
this.  Tejun looks like he's willing to execute, so I feel much more
sanguine that it will happen.  The point of the notes was really to draw
attention to something I hadn't realised:  we can't refactor block to
get libata out of SCSI without also changing drivers/ide, which does
make the problem harder.

On a final note about the urgency of getting libata out of SCSI: Intel
has been worrying for a while about the fatness of the SCSI/libata
stack, and its effects on performance, especially command transmission
via SAT, so I'm hoping they'll be supporting the effort.

Jmaes

Re: LSF Papers online?

[Robert Hancock]

James Bottomley wrote:
> 
> On a final note about the urgency of getting libata out of SCSI: Intel
> has been worrying for a while about the fatness of the SCSI/libata
> stack, and its effects on performance, especially command transmission
> via SAT, so I'm hoping they'll be supporting the effort.

I really don't see this as being a big driver for this move. If you look 
at the code that does the translation of SCSI commands to ATA commands, 
there really is not much there at all of any consequence to CPU usage. 
Compared to any kind of hardware/controller interactions I wouldn't say 
it's likely to be a significant bottleneck at all. In oprofile runs I've 
done with heavy ATA activity, the top time consumers are the interrupt 
handlers, command issue paths, code that actually is poking IO 
registers. The libata-scsi code hasn't even shown up on the radar in my 
experience.

Re: LSF Papers online?

[James Bottomley]

On Tue, 2009-04-14 at 19:39 -0600, Robert Hancock wrote:
> James Bottomley wrote:
> > 
> > On a final note about the urgency of getting libata out of SCSI: Intel
> > has been worrying for a while about the fatness of the SCSI/libata
> > stack, and its effects on performance, especially command transmission
> > via SAT, so I'm hoping they'll be supporting the effort.
> 
> I really don't see this as being a big driver for this move. If you look 
> at the code that does the translation of SCSI commands to ATA commands, 
> there really is not much there at all of any consequence to CPU usage. 
> Compared to any kind of hardware/controller interactions I wouldn't say 
> it's likely to be a significant bottleneck at all. In oprofile runs I've 
> done with heavy ATA activity, the top time consumers are the interrupt 
> handlers, command issue paths, code that actually is poking IO 
> registers. The libata-scsi code hasn't even shown up on the radar in my 
> experience.

Been there, said that and got the nicely embroidered polo shirt to prove
it.

The point is, it doesn't really matter what I say or believe, it matters
what they do ... and they believe fat stacks impede the performance of
their SSDs.

James

Re: LSF Papers online?

[Alan Cox]

> The point is, it doesn't really matter what I say or believe, it matters
> what they do ... and they believe fat stacks impede the performance of
> their SSDs.

The 'fat' is higher up, as is being clearly shown now people are finally
looking at aspects of the performance problems introduced around 2.6.19
in the fs layers.

Re: LSF Papers online?

[Grant Grundler]

On Tue, Apr 14, 2009 at 6:39 PM, Robert Hancock <hancockrwd@gmail.com> wrote:
> James Bottomley wrote:
>>
>> On a final note about the urgency of getting libata out of SCSI: Intel
>> has been worrying for a while about the fatness of the SCSI/libata
>> stack, and its effects on performance, especially command transmission
>> via SAT, so I'm hoping they'll be supporting the effort.
>
> I really don't see this as being a big driver for this move. If you look at
> the code that does the translation of SCSI commands to ATA commands, there
> really is not much there at all of any consequence to CPU usage.

It's measurable. Just need to get sufficient IO rates going. Normal JBOD/RAID
controllers don't care.

See Matthew "the Intel guy" :) Wilcox and Kristen Accardi's paper at LSF2008:
    http://iou.parisc-linux.org/lsf2008/IO-latency-Kristen-Carlson-Accardi.pdf

or just google for "Matthew Wilcox SSD perf" - first four hits I
looked at are what I expected:
    hardware.slashdot.org/article.pl?sid=09/02/13/2337258&from=rss
    markmail.org/message/w22r6y4gik7bjf2w
    https://kerneltrap.org/mailarchive/linux-scsi/2008/12/10/4388804/thread
    www.usenix.org/events/lsf08/tech/IO_Carlson_Accardi_SATA.pdf

(last one is the official location of the same paper)

> Compared to
> any kind of hardware/controller interactions I wouldn't say it's likely to
> be a significant bottleneck at all. In oprofile runs I've done with heavy
> ATA activity, the top time consumers are the interrupt handlers,

interrupts just introduce completion reporting latency. Interrupt mitigation
techinques/smarter controller can reduce this.

> command issue paths,

Hrm? Is this in the device driver?

> code that actually is poking IO registers.

Stupid controller design. NICs have been able to run without
MMIO *Reads* in the performance for more than 5 years now.
New "Enterprise" SAS/SATA controllers are better but I'm not
at liberty to discuss those. (sorry)

> The libata-scsi code
> hasn't even shown up on the radar in my experience.

It won't for normal disk IOPS rates (<1000 IOPS per disk).
Run it at 20K or 50K IOPS and see again.
NICs are pushing alot more than that.

hth,
grant

Re: LSF Papers online?

[James Bottomley]

On Wed, 2009-04-15 at 23:31 -0700, Grant Grundler wrote:
> On Tue, Apr 14, 2009 at 6:39 PM, Robert Hancock <hancockrwd@gmail.com> wrote:
> > James Bottomley wrote:

[...]
> > Compared to
> > any kind of hardware/controller interactions I wouldn't say it's likely to
> > be a significant bottleneck at all. In oprofile runs I've done with heavy
> > ATA activity, the top time consumers are the interrupt handlers,
> 
> interrupts just introduce completion reporting latency. Interrupt mitigation
> techinques/smarter controller can reduce this.

For sure smarter controllers; I'm less convinced on interrupt mitigation
techniques, primarily because smart controllers tend to do batch
completion anyway.

The reason storage shouldn't really need interrupt mitigation (like NAPI
for networks) is that we shouldn't ever really be surprised by an
interrupt, unlike networks.  We should always know what payloads will be
coming at us from the device (and have buffers ready and waiting).  Most
SCSI controllers (including the SAS ones) do all of this today: You can
send out I/O and on some of them get under two interrupts per gigabyte
of data.   I fully agree that some of the less smart SATA controllers
have a lot of catching up to do in this space, but that isn't
necessarily a driver issue; you can't polish a turd as the saying
goes ...

> > command issue paths,
> 
> Hrm? Is this in the device driver?
> 
> > code that actually is poking IO registers.
> 
> Stupid controller design. NICs have been able to run without
> MMIO *Reads* in the performance for more than 5 years now.
> New "Enterprise" SAS/SATA controllers are better but I'm not
> at liberty to discuss those. (sorry)

That's both a protocol and a controller issue for SATA: some of the ATA
transfer modes (like pio) have a lot higher overhead (and,
unfortunately, less offload in the controller) in the protocol stack and
can be unavoidable on certain transactions.

> > The libata-scsi code
> > hasn't even shown up on the radar in my experience.
> 
> It won't for normal disk IOPS rates (<1000 IOPS per disk).
> Run it at 20K or 50K IOPS and see again.
> NICs are pushing alot more than that.

So again, we get to this terminology problem:  NICs tend to have a fixed
packet size (the network MTU) so IOPS/s makes a lot of sense for them.
In most storage transactions we don't really have MTU limitations, so we
try to right size the outgoing transactions to maximize for bandwidth,
so IOPS don't tell the full story.  (after all, if I see all my packets
are 128k, I can artificially reduce the merge limit to 64k and double my
IOPS).

IOPS are starting to come up because SSDs are saying they prefer many
smaller transactions to an accumulated larger one.  I'm still not
entirely convinced that trying to rightsize is wrong here:  most of the
FS data is getting more contiguous, so even for SSDs we can merge
without a lot of work.  A simple back of the envelope calculation can
give the rightizing:  If you want a SSD to max out at its 31 allowed
tags saturating a 3G sata link, then you're talking 10M per tag per
second.  If we assume a 4k sector size, that's 2500 IOPS per tag
(there's no real point doing less than 4k, because that has us splitting
the page cache). Or, to put it another way, over 75k IOPS for a single
SSD doesn't make sense ... the interesting question is whether it would
make more sense to align on, say 16k io and so expect to max out at 20k
IOPS.

James

Re: LSF Papers online?

[Matthew Wilcox]

On Thu, Apr 16, 2009 at 11:37:02AM -0500, James Bottomley wrote:
> of data.   I fully agree that some of the less smart SATA controllers
> have a lot of catching up to do in this space, but that isn't
> necessarily a driver issue; you can't polish a turd as the saying
> goes ...

I guess you haven't seen the episode of Mythbusters where they manage
to do exactly that?  ;-)

> IOPS are starting to come up because SSDs are saying they prefer many
> smaller transactions to an accumulated larger one.  I'm still not

I don't think that's what SSDs are saying.  The protocol (and controllers)
still work better if you send down one 128k IO than 32 4k IOs.  But with
the low latency of doing accesses, it's better to send down a 16k IO
now than it is to wait around a bit and see if another 16k IO comes along.

> entirely convinced that trying to rightsize is wrong here:  most of the
> FS data is getting more contiguous, so even for SSDs we can merge
> without a lot of work.  A simple back of the envelope calculation can
> give the rightizing:  If you want a SSD to max out at its 31 allowed
> tags saturating a 3G sata link, then you're talking 10M per tag per

Better than that, only 8MB of data per tag per second.  SATA effectively
limits you to 250MB/s.  That's 2016 IOPS per tag.  Of course, this
assumes you're only doing the NCQ commands and not, say, issuing TRIM
or something.

> second.  If we assume a 4k sector size, that's 2500 IOPS per tag
> (there's no real point doing less than 4k, because that has us splitting
> the page cache). Or, to put it another way, over 75k IOPS for a single
> SSD doesn't make sense ... the interesting question is whether it would
> make more sense to align on, say 16k io and so expect to max out at 20k
> IOPS.

If we're serious about getting 2000 IOPS per tag, then the round-trip
inside the kernel to recycle a tag has to be less than 500 microseconds.
Do you have a good idea about how to measure what that is today?
Here's the call path taken by the AHCI driver:

ahci_interrupt()
ahci_port_intr()
ata_qc_complete_multiple()
ata_qc_complete()
__ata_qc_complete()
ata_scsi_qc_complete() [qc->complete_fn]
scsi_done() [qc->scsidone]
blk_complete_request()
__blk_complete_request()
raise_softirq_irqoff()
...
blk_done_softirq()
scsi_softirq_done() [rq->q->softirq_done_fn]
scsi_finish_command()
scsi_io_completion()
scsi_end_request()
scsi_next_command()
scsi_run_queue()
__blk_run_queue()
blk_invoke_request_fn()
scsi_request_fn() [q->request_fn]
scsi_dispatch_cmd()
ata_scsi_translate() [host->hostt->queuecommand]
ata_qc_issue()
ahci_qc_issue() [ap->ops->qc_issue]

I can see a few ways to cut down the latency between knowing a tag is
no longer used and starting the next command.

We could pretend the AHCI driver has a queue depth of 64, queue up
commands in the driver, swap the tags over, and send out the next command
before we process this command.

This is similar to a technique that's used in some old SCSI drivers that
didn't support tagged commands at all -- a second command was queued
inside the driver while the first was executing on the device.

But then, we had that big movement towards elimintaing queues from inside
drivers ... maybe we need another way.

-- 
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: LSF Papers online?

[Jeff Garzik]

Bartlomiej Zolnierkiewicz wrote:
> We actually used quite a lot of libata PATA changes on the host driver front
> (though because of the lack of incremental libata PATA changes and the need
> to fix some IDE specific issues first this was not that easy) so maybe if we
> could agree on some mid-point way back in 2005 we would be in a completely
> different place today.
> 
> Lets not repeat the history again.
> 
>>> The fact that it is much harder to do nowadays than in 2004-2005 (without
>>> ATAPI support, PATA support and heavy dependence on SCSI infrastructure)
>>> is only _your_ fault.
>> Really - you could have contributed too. Anyway I wrote most of the PATA
>> bits, Tejun wrote a ton of stuff including EH, and Albert Lee wrote
>> chunks of it too.
> 
> I did a bit work.  Check your host drivers. :)
> 
>> So that would be Red Hat, SuSE and IBM who are to blame right, not just
>> Jeff ? Sounds like a conspiracy to me ;)
> 
> There is no conspiracy there.
> 
> Just the good old game known from elsewhere.
> 
> Top layer modern day kernel hacking is based on business principles.
> 
> However, like they say "Hate the Game, Not the Player". :)
> 
> There is really nothing wrong with it as long as we don't state
> otherwise to new people so they are fully aware of the situation.

Actually, libata got started even before I was at Red Hat.  As I alluded 
to at the bottom of the libata announcement[1], rewriting the IDE driver 
into something clean and modern was one of my Projects To Do Before I 
Die -- that is, projects not sponsored by any business or group or 
conspiracy, but rather things I feel personally are very important to 
the advancement of Linux.

And I have been very fortunate that other talented hackers (including 
yourself) have been willing to work on one of my dream projects.

	Jeff



[1] http://www.kernel-traffic.org/kernel-traffic/kt20030616_219.html#7

Re: LSF Papers online?

[Alan Cox]

> I just wanted to have clear vision of the process instead of "lets merge
> it now and worry about real users later".

There was a simple vision which was to provide full coverage for all the
relevant ATA controllers using the libata core ASAP and leave the old IDE
code "as is". That was what was done.

> Every uncomfortable technical issue raised was addressed with direct
> "IDE is teh crap" shouting 

Sorry but you are re-inventing history here. The old IDE layer was crap,
and for sound technical reasons including the error handling and the
horrible locking problems from the design level up caused by the
reset/error/timeout handling paths and the polled speed change on CRC
error change down.

I spent years working on that stuff. Throwing it all away was not my idea
of fun, but it was clearly the right technical decision. I'm very happy
with the state of libata PATA.

Alan

Re: LSF Papers online?

[Bartlomiej Zolnierkiewicz]

On Wednesday 15 April 2009 11:28:29 Alan Cox wrote:
> > I just wanted to have clear vision of the process instead of "lets merge
> > it now and worry about real users later".
> 
> There was a simple vision which was to provide full coverage for all the
> relevant ATA controllers using the libata core ASAP and leave the old IDE
> code "as is". That was what was done.

You took the "shortcut" instead of fixing the issue the "proper" way.

Namely, you defined "relevant" and "full coverage" as you wanted.

Depending on the level of difficulty of the task.

Wasn't IDE PMAC relevant back in 2005?

It was much more relevant than CMD640 or legacy VLB drivers that you ported.

It is 2009 now and IDE PMAC is still much more relevant than some other
stuff that you're working on.

Wasn't 32-bit PIO important for embedded systems back then?
[it happened only recently]

You also threw in a bunch of regressions because you didn't start from
fixing IDE host drivers in incremental evolutionary way.

You didn't even care to backport some obvious/critical fixes to IDE,
the same IDE that all distributions (including _your_ past employer)
were still shipping.

Yes, doing it your way was faster for _you_ and gave _you_ huge competition
advantage but prevented any reasonable review/help from _other_ people.

Reinventing the whole subsystem is not a single person project.

It is just physically impossible to do it properly.

The complexity and the scale is too large and there will be always things
that you miss or that other people are better at.

> > Every uncomfortable technical issue raised was addressed with direct
> > "IDE is teh crap" shouting 
> 
> Sorry but you are re-inventing history here. The old IDE layer was crap,
> and for sound technical reasons including the error handling and the
> horrible locking problems from the design level up caused by the
> reset/error/timeout handling paths and the polled speed change on CRC
> error change down.

That was not the issue raised.

> I spent years working on that stuff. Throwing it all away was not my idea

So what?

This doesn't make you special in any way.

Few other people had similar experiences.

> of fun, but it was clearly the right technical decision. I'm very happy
> with the state of libata PATA.

Maybe it is the time to retire from kernel hacking then? ;)

You see, I'm not happy with the state of IDE, libata, SCSI, block
and many other things.  I know that I'll _never_ be.  I also bet that
most of people working with me feel in the exactly same way.

More seriously, it is 2009 now.

Four years after initial libata PATA introduction and I could still
spend few next weeks sending one IDE -> libata regression fix per day,
ranging from possible data corruption through lack of hardware support
to duplicated code [most of issues can be tracked back to 2005].

However I simply do not care.

I don't need libata PATA.

I don't use it, I'm not paid to work on it and it is not my project.

It seems that it I'm not the only one who misunderstood the free software.

If you are not interested in people wanting to improve _your_ project
then we can end up the discussion here.

You may not like me but this is pretty lame excuse for denying me
merit and in turn damaging _your_ own project.

Re-read my previous mail and ping me when you are back to the rational
thinking.

Till then I'm done with this topic.

Re: LSF Papers online?

[Alan Cox]

> You may not like me but this is pretty lame excuse for denying me
> merit and in turn damaging _your_ own project.

Oh finally we get the truth. You are peeved someone went off and did the
job that needed doing and it wasn't you.

Firstly it isn't *my* project. I just contribute chunks of code to it. I
didn't even start the shift to put PATA drivers into libata, I just
picked up a job that needed doing, did it and for the most part have now
moved on to trying to tackle the tty layer.

I really don't care about the whole merit thing. You'll note when Linus
asked me to maintain 2.4 I said no, when some young punk called DaveM
showed up and proved way better than me at networking code I simply
handed over maintainership.

There are plenty of projects where you can go play full contact ego wars
all day but Linux isn't one of them. I fix stuff that needs fixing, I
write stuff that needs writing and above all I try and get things into a
state where other people take them over because they feel confident they
can. If someone comes along and does the job better I'm not peeved I'm
happy because it means Linux is better too.

Alan

Re: LSF Papers online?

[Bartlomiej Zolnierkiewicz]

On Wednesday 15 April 2009 16:56:14 Alan Cox wrote:
> > You may not like me but this is pretty lame excuse for denying me
> > merit and in turn damaging _your_ own project.
> 
> Oh finally we get the truth. You are peeved someone went off and did the
> job that needed doing and it wasn't you.

Peeved about that?

Only a bit. :)  This was not the real reason.

> Firstly it isn't *my* project. I just contribute chunks of code to it. I
> didn't even start the shift to put PATA drivers into libata, I just
> picked up a job that needed doing, did it and for the most part have now
> moved on to trying to tackle the tty layer.

However, it is all good now.  I look at tty commits and I'm happy.

Thanks,
Bart