PATA IDE is slower in newer versions of kernel

PATA IDE is slower in newer versions of kernel

[Joshua Hintze]

Hello, 

I've been digging into this for a couple weeks now. I have an embedded board
that runs Linux Kernel 2.6.10 that is connected to a SSD over a PATA
connection. Data is transferred using UDMA4 speeds and I get a nice
sustained write speed of about 20 MBps which is adequate for us. 

I have recently taken the plunge to update to a newer version of the kernel
starting at 2.6.32 (also tried 2.6.33) and what I am seeing writes speeds
drop to about 18 MBps peak with large jumps going from 8 MBps->16 MBs for
sustained throughput.

I decided to read up on the ATA specification and I connected a logic
analyzer to the PATA bus and here is what is happening...after a large chunk
of data is written to the device, instead of the device pausing the transfer
by asserting DDMARDY it actual initiates a device data-out termination by
pulling DMARQ low.

The old kernel 2.6.10 responds to this by strobing the STOP line 8 times
within 800uS and the device releases DMARQ shortly afterwards. On the newer
kernels the strobing of the STOP line takes near 8mS of time. Since the hard
drive only release DMARQ after this 8 pulse strobe I believe this is the
cause of the slower write speeds on the newer kernels. This whole process
happens thousands of times when writing megs of data. So those extra 7mS
begin to add up fast.

My problem is I've been digging through the ide.c/ide-dma.c and other code
but I'm not exactly sure where the code would jump to upon a device
initiated termination on a data-out dma transfer.

My guess is ide_dma_intr(...). Is this correct?

Any other areas I could look at?

Thanks in advance.

Re: PATA IDE is slower in newer versions of kernel

[Alejandro Riveira Fernández]

El Thu, 16 Sep 2010 17:30:51 -0600
"Joshua Hintze" <joshh@imsar.com> escribió:

 [ Just CC linux-ide ]

> Hello, 
> 
> I've been digging into this for a couple weeks now. I have an embedded board
> that runs Linux Kernel 2.6.10 that is connected to a SSD over a PATA
> connection. Data is transferred using UDMA4 speeds and I get a nice
> sustained write speed of about 20 MBps which is adequate for us. 
> 
> I have recently taken the plunge to update to a newer version of the kernel
> starting at 2.6.32 (also tried 2.6.33) and what I am seeing writes speeds
> drop to about 18 MBps peak with large jumps going from 8 MBps->16 MBs for
> sustained throughput.
> 
> I decided to read up on the ATA specification and I connected a logic
> analyzer to the PATA bus and here is what is happening...after a large chunk
> of data is written to the device, instead of the device pausing the transfer
> by asserting DDMARDY it actual initiates a device data-out termination by
> pulling DMARQ low.
> 
> The old kernel 2.6.10 responds to this by strobing the STOP line 8 times
> within 800uS and the device releases DMARQ shortly afterwards. On the newer
> kernels the strobing of the STOP line takes near 8mS of time. Since the hard
> drive only release DMARQ after this 8 pulse strobe I believe this is the
> cause of the slower write speeds on the newer kernels. This whole process
> happens thousands of times when writing megs of data. So those extra 7mS
> begin to add up fast.
> 
> My problem is I've been digging through the ide.c/ide-dma.c and other code
> but I'm not exactly sure where the code would jump to upon a device
> initiated termination on a data-out dma transfer.
> 
> My guess is ide_dma_intr(...). Is this correct?
> 
> Any other areas I could look at?
> 
> Thanks in advance.
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

RE: PATA IDE is slower in newer versions of kernel

[Joshua Hintze]

Some updated information. I created a screen capture of what I am seeing. First is the logic analyzer when the newer linux kernel is writing slow to the PATA port.

http://i56.tinypic.com/oqhrug.png

Notice in the above link how there is a lot of activating and then large gaps of no activity. After about 9 mS the STOP line is strobed a few times and then DMARQ finally returns to normal operating. The link below shows the picture of an old 2.6.10 kernel where write speeds are much faster.

http://i51.tinypic.com/a9wiur.png

In the above image notice how the wait time is considerable less 100's of microseconds before the stop lines get strobed.

I'm dedicated to continue working on the problem, I just need to advice from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.

Thanks!



-----Original Message-----
From: linux-ide-owner@vger.kernel.org [mailto:linux-ide-owner@vger.kernel.org] On Behalf Of Alejandro Riveira Fernández
Sent: Friday, September 17, 2010 7:48 AM
To: Joshua Hintze
Cc: linux-kernel@vger.kernel.org; linux-ide@vger.kernel.org
Subject: Re: PATA IDE is slower in newer versions of kernel

El Thu, 16 Sep 2010 17:30:51 -0600
"Joshua Hintze" <joshh@imsar.com> escribió:

 [ Just CC linux-ide ]

> Hello, 
> 
> I've been digging into this for a couple weeks now. I have an embedded board
> that runs Linux Kernel 2.6.10 that is connected to a SSD over a PATA
> connection. Data is transferred using UDMA4 speeds and I get a nice
> sustained write speed of about 20 MBps which is adequate for us. 
> 
> I have recently taken the plunge to update to a newer version of the kernel
> starting at 2.6.32 (also tried 2.6.33) and what I am seeing writes speeds
> drop to about 18 MBps peak with large jumps going from 8 MBps->16 MBs for
> sustained throughput.
> 
> I decided to read up on the ATA specification and I connected a logic
> analyzer to the PATA bus and here is what is happening...after a large chunk
> of data is written to the device, instead of the device pausing the transfer
> by asserting DDMARDY it actual initiates a device data-out termination by
> pulling DMARQ low.
> 
> The old kernel 2.6.10 responds to this by strobing the STOP line 8 times
> within 800uS and the device releases DMARQ shortly afterwards. On the newer
> kernels the strobing of the STOP line takes near 8mS of time. Since the hard
> drive only release DMARQ after this 8 pulse strobe I believe this is the
> cause of the slower write speeds on the newer kernels. This whole process
> happens thousands of times when writing megs of data. So those extra 7mS
> begin to add up fast.
> 
> My problem is I've been digging through the ide.c/ide-dma.c and other code
> but I'm not exactly sure where the code would jump to upon a device
> initiated termination on a data-out dma transfer.
> 
> My guess is ide_dma_intr(...). Is this correct?
> 
> Any other areas I could look at?
> 
> Thanks in advance.
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

No virus found in this incoming message.
Checked by AVG - www.avg.com 
Version: 9.0.851 / Virus Database: 271.1.1/3132 - Release Date: 09/16/10 12:34:00

RE: PATA IDE is slower in newer versions of kernel

[Maxim Levitsky]

On Sat, 2010-09-18 at 11:12 -0600, Joshua Hintze wrote:
> Some updated information. I created a screen capture of what I am seeing. First is the logic analyzer when the newer linux kernel is writing slow to the PATA port.
> 
> http://i56.tinypic.com/oqhrug.png
> 
> Notice in the above link how there is a lot of activating and then large gaps of no activity. After about 9 mS the STOP line is strobed a few times and then DMARQ finally returns to normal operating. The link below shows the picture of an old 2.6.10 kernel where write speeds are much faster.
> 
> http://i51.tinypic.com/a9wiur.png
> 
> In the above image notice how the wait time is considerable less 100's of microseconds before the stop lines get strobed.
> 
> I'm dedicated to continue working on the problem, I just need to advice from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.
> 
> Thanks!
> 
Maybe you can just bisect this?
Just a general advice, I don't know ide subsystem at all.

Best regards,
	Maxim Levitsky

RE: PATA IDE is slower in newer versions of kernel

[Joshua Hintze]

Thanks,

I'm not sure what you mean by bisect, other than just the definition of the word. I am breaking this problem down adding printks etc.  I think I'm finally beginng to understand how this works :)

Josh


-----Original Message-----
From: linux-ide-owner@vger.kernel.org [mailto:linux-ide-owner@vger.kernel.org] On Behalf Of Maxim Levitsky
Sent: Saturday, September 18, 2010 1:23 PM
To: Joshua Hintze
Cc: linux-kernel@vger.kernel.org; linux-ide@vger.kernel.org
Subject: RE: PATA IDE is slower in newer versions of kernel

On Sat, 2010-09-18 at 11:12 -0600, Joshua Hintze wrote:
> Some updated information. I created a screen capture of what I am seeing. First is the logic analyzer when the newer linux kernel is writing slow to the PATA port.
> 
> http://i56.tinypic.com/oqhrug.png
> 
> Notice in the above link how there is a lot of activating and then large gaps of no activity. After about 9 mS the STOP line is strobed a few times and then DMARQ finally returns to normal operating. The link below shows the picture of an old 2.6.10 kernel where write speeds are much faster.
> 
> http://i51.tinypic.com/a9wiur.png
> 
> In the above image notice how the wait time is considerable less 100's of microseconds before the stop lines get strobed.
> 
> I'm dedicated to continue working on the problem, I just need to advice from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.
> 
> Thanks!
> 
Maybe you can just bisect this?
Just a general advice, I don't know ide subsystem at all.

Best regards,
	Maxim Levitsky

--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

No virus found in this incoming message.
Checked by AVG - www.avg.com 
Version: 9.0.851 / Virus Database: 271.1.1/3132 - Release Date: 09/18/10 00:34:00

Re: PATA IDE is slower in newer versions of kernel

[Randy Dunlap]

On Sat, 18 Sep 2010 13:43:40 -0600 Joshua Hintze wrote:

> Thanks,
> 
> I'm not sure what you mean by bisect, other than just the definition of the word. I am breaking this problem down adding printks etc.  I think I'm finally beginng to understand how this works :)
> 

http://www.kernel.org/pub/software/scm/git/docs/git-bisect.html
http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#using-bisect


> 
> 
> -----Original Message-----
> From: linux-ide-owner@vger.kernel.org [mailto:linux-ide-owner@vger.kernel.org] On Behalf Of Maxim Levitsky
> Sent: Saturday, September 18, 2010 1:23 PM
> To: Joshua Hintze
> Cc: linux-kernel@vger.kernel.org; linux-ide@vger.kernel.org
> Subject: RE: PATA IDE is slower in newer versions of kernel
> 
> On Sat, 2010-09-18 at 11:12 -0600, Joshua Hintze wrote:
> > Some updated information. I created a screen capture of what I am seeing. First is the logic analyzer when the newer linux kernel is writing slow to the PATA port.
> > 
> > http://i56.tinypic.com/oqhrug.png
> > 
> > Notice in the above link how there is a lot of activating and then large gaps of no activity. After about 9 mS the STOP line is strobed a few times and then DMARQ finally returns to normal operating. The link below shows the picture of an old 2.6.10 kernel where write speeds are much faster.
> > 
> > http://i51.tinypic.com/a9wiur.png
> > 
> > In the above image notice how the wait time is considerable less 100's of microseconds before the stop lines get strobed.
> > 
> > I'm dedicated to continue working on the problem, I just need to advice from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.
> > 
> > Thanks!
> > 
> Maybe you can just bisect this?
> Just a general advice, I don't know ide subsystem at all.
> 
> Best regards,
> 	Maxim Levitsky


---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

RE: PATA IDE is slower in newer versions of kernel

[Joshua Hintze]

I got you. 

Thanks.


-----Original Message-----
From: Randy Dunlap [mailto:rdunlap@xenotime.net] 
Sent: Saturday, September 18, 2010 1:48 PM
To: Joshua Hintze
Cc: 'Maxim Levitsky'; linux-kernel@vger.kernel.org;
linux-ide@vger.kernel.org
Subject: Re: PATA IDE is slower in newer versions of kernel

On Sat, 18 Sep 2010 13:43:40 -0600 Joshua Hintze wrote:

> Thanks,
> 
> I'm not sure what you mean by bisect, other than just the definition of
the word. I am breaking this problem down adding printks etc.  I think I'm
finally beginng to understand how this works :)
> 

http://www.kernel.org/pub/software/scm/git/docs/git-bisect.html
http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#using-bisec
t


> 
> 
> -----Original Message-----
> From: linux-ide-owner@vger.kernel.org
[mailto:linux-ide-owner@vger.kernel.org] On Behalf Of Maxim Levitsky
> Sent: Saturday, September 18, 2010 1:23 PM
> To: Joshua Hintze
> Cc: linux-kernel@vger.kernel.org; linux-ide@vger.kernel.org
> Subject: RE: PATA IDE is slower in newer versions of kernel
> 
> On Sat, 2010-09-18 at 11:12 -0600, Joshua Hintze wrote:
> > Some updated information. I created a screen capture of what I am
seeing. First is the logic analyzer when the newer linux kernel is writing
slow to the PATA port.
> > 
> > http://i56.tinypic.com/oqhrug.png
> > 
> > Notice in the above link how there is a lot of activating and then large
gaps of no activity. After about 9 mS the STOP line is strobed a few times
and then DMARQ finally returns to normal operating. The link below shows the
picture of an old 2.6.10 kernel where write speeds are much faster.
> > 
> > http://i51.tinypic.com/a9wiur.png
> > 
> > In the above image notice how the wait time is considerable less 100's
of microseconds before the stop lines get strobed.
> > 
> > I'm dedicated to continue working on the problem, I just need to advice
from the linux community. I've tried different I/O schedulers such as noop
and anticipatory and I'm studying the ide-dma code but still lost to where
this is happening.
> > 
> > Thanks!
> > 
> Maybe you can just bisect this?
> Just a general advice, I don't know ide subsystem at all.
> 
> Best regards,
> 	Maxim Levitsky


---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

No virus found in this incoming message.
Checked by AVG - www.avg.com 
Version: 9.0.851 / Virus Database: 271.1.1/3132 - Release Date: 09/18/10
00:34:00

Re: PATA IDE is slower in newer versions of kernel

[Arnd Bergmann]

On Saturday 18 September 2010 21:22:54 Maxim Levitsky wrote:
> On Sat, 2010-09-18 at 11:12 -0600, Joshua Hintze wrote:
> > I'm dedicated to continue working on the problem, I just need to advice from 
> > the linux community. I've tried different I/O schedulers such as noop and
> > anticipatory and I'm studying the ide-dma code but still lost to where this
> > is happening.
> > 
> > Thanks!
> > 
> Maybe you can just bisect this?

That would probably help to pinpoint the problem. A regression somewhere
between 2.6.10 and 2.6.32 is not that interesting when people are working
on stabilizing 2.6.36 and adding features to 2.6.37.

One important question would be if this happens with the libata drivers
in 2.6.36-rc4 or linux-next as well. Note that the IDE subsystem has been
replaced with libata a few years ago, and that is what all development
happens on. If libata is good enough, there is probably no point in
chasing down ide bugs.

	Arnd

Re: PATA IDE is slower in newer versions of kernel

[Alan Cox]

> I'm dedicated to continue working on the problem, I just need to advice from the linux community. I've tried different I/O schedulers such as noop and anticipatory and I'm studying the ide-dma code but still lost to where this is happening.

drivers/ide is pretty much obsolete - so what you are seeing is doubly
odd. Most stuff now is using the newer drivers/ata code which can support
SATA, NCQ etc unlike the old codebase.

One thing to check would be whether using the drivers/ata code shows the
same problem.

Alan

RE: PATA IDE is slower in newer versions of kernel

[Joshua Hintze]

Thank you for the input. The reason why I'm using the old drivers/ide is
because I'm using a TI Davinci ARM processor in an embedded system. They
wrote a PalmChip driver for the older IDE interface and I don't think they
have anything yet for the drivers/ata (out of curiousity is the newer
drivers/ata the reason why PATA devices show up as /dev/sda in newer
kernels?)

After spending most of the day today adding printks and learning about the
old ide driver I'm pretty much convinced that the IDE driver is working
fine. The long delays are actually between block driver request queues. In
the 2.6.10 kernel the time between block writes is less than 1 ms, where as
the 2.6.32 that TI has blessed takes 8+ ms between blocks that are pushed to
the IDE interface.

Like I mentioned earlier I tried changing the I/O scheduler to Noop and as
for the elevator. Anything else I should look to so I get blocks sent out
faster?

Thanks


-----Original Message-----
From: linux-ide-owner@vger.kernel.org
[mailto:linux-ide-owner@vger.kernel.org] On Behalf Of Alan Cox
Sent: Saturday, September 18, 2010 6:59 PM
To: Joshua Hintze
Cc: linux-kernel@vger.kernel.org; linux-ide@vger.kernel.org
Subject: Re: PATA IDE is slower in newer versions of kernel

> I'm dedicated to continue working on the problem, I just need to advice
from the linux community. I've tried different I/O schedulers such as noop
and anticipatory and I'm studying the ide-dma code but still lost to where
this is happening.

drivers/ide is pretty much obsolete - so what you are seeing is doubly
odd. Most stuff now is using the newer drivers/ata code which can support
SATA, NCQ etc unlike the old codebase.

One thing to check would be whether using the drivers/ata code shows the
same problem.

Alan
--
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

No virus found in this incoming message.
Checked by AVG - www.avg.com 
Version: 9.0.851 / Virus Database: 271.1.1/3143 - Release Date: 09/18/10
00:34:00