Best way to segments/requests

Best way to segments/requests

[Adrian McMenamin]

I am working on a driver for the CD Rom drive on the Sega Dreamcast
(the so-called "GD Rom" drive). This device is electorically
compatible with IDE-3 devices and has a pretty good match in terms of
the control block registers but it implements its own packet command
interface.

I now have a working driver but the performance is lousy.

The driver reads data off the disk using DMA and the target for the
DMA has to be a contiguous. Therefore I have set:

	/* using DMA so memory will need to be contiguous */
	blk_queue_max_hw_segments(gd.gdrom_rq, 1);
	/* set a large max size to get most from DMA */
	blk_queue_max_segment_size(gd.gdrom_rq, 0x40000);

ie only one segment per request but a big (for a small device) maximum
size for the segment.

A priori I can see no performance advantage in allowing each request
to include multiple segments because then I'd only have to reshake
them so they went in one at a time. But from the looks of it if I do
set the maximum number of segments to 1 then each request is limited
to the smallest size - ie what I set in blk_queue_hardsect_size.

Is that right? What is the best way to go here?

Re: Best way to segments/requests

[Tejun Heo]

Hello,

Adrian McMenamin wrote:
> I am working on a driver for the CD Rom drive on the Sega Dreamcast
> (the so-called "GD Rom" drive). This device is electorically
> compatible with IDE-3 devices and has a pretty good match in terms of
> the control block registers but it implements its own packet command
> interface.
> 
> I now have a working driver but the performance is lousy.
> 
> The driver reads data off the disk using DMA and the target for the
> DMA has to be a contiguous. Therefore I have set:
> 
> 	/* using DMA so memory will need to be contiguous */
> 	blk_queue_max_hw_segments(gd.gdrom_rq, 1);
> 	/* set a large max size to get most from DMA */
> 	blk_queue_max_segment_size(gd.gdrom_rq, 0x40000);

Ah...

> ie only one segment per request but a big (for a small device) maximum
> size for the segment.
> 
> A priori I can see no performance advantage in allowing each request
> to include multiple segments because then I'd only have to reshake
> them so they went in one at a time. But from the looks of it if I do
> set the maximum number of segments to 1 then each request is limited
> to the smallest size - ie what I set in blk_queue_hardsect_size.

There just isn't much room for maneuver w/ just one segment.  Large
contiguous memory region isn't too common these days.  That said, there
was a bug recently spotted by Mark Lord which made contiguous memory
regions even rarer.  Which kernel version are you using?

> Is that right? What is the best way to go here?

If you can spare some memory and cpu cycles, preparing a contiguous
buffer and staging data there might help.  It will eat up some cpu
cycles but it won't be too much compared to PIO cycles.

-- 
tejun

Re: Best way to segments/requests

[Adrian McMenamin]

On 14/12/2007, Tejun Heo <htejun@gmail.com> wrote:
> Hello,

>
> There just isn't much room for maneuver w/ just one segment.  Large
> contiguous memory region isn't too common these days.  That said, there
> was a bug recently spotted by Mark Lord which made contiguous memory
> regions even rarer.  Which kernel version are you using?
>

Bang up to date latest git, ie -rc5-gitX


> > Is that right? What is the best way to go here?
>
> If you can spare some memory and cpu cycles, preparing a contiguous
> buffer and staging data there might help.  It will eat up some cpu
> cycles but it won't be too much compared to PIO cycles.
>

OK, I'll try it

Re: Best way to segments/requests

[Mark Lord]

[-- Attachment #1: Type: text/plain, Size: 1002 bytes --]

Adrian McMenamin wrote:
> On 14/12/2007, Tejun Heo <htejun@gmail.com> wrote:
>> Hello,
> 
>> There just isn't much room for maneuver w/ just one segment.  Large
>> contiguous memory region isn't too common these days.  That said, there
>> was a bug recently spotted by Mark Lord which made contiguous memory
>> regions even rarer.  Which kernel version are you using?
>>
> 
> Bang up to date latest git, ie -rc5-gitX
..

Not in -git yet, but it is in -mm.
Attached here for your convenience.

>>> Is that right? What is the best way to go here?
>> If you can spare some memory and cpu cycles, preparing a contiguous
>> buffer and staging data there might help.  It will eat up some cpu
>> cycles but it won't be too much compared to PIO cycles.
>>
> 
> OK, I'll try it
..

That's probably your best bet, even though it will mean copying
to/from your big bounce buffer with each I/O.

The code could be clever, I suppose, and only bounce when the supplied
I/O region is smaller than XXX pages.

Cheers


[-- Attachment #2: 13_fix_page_alloc_for_larger_io_segments.patch --]
[-- Type: text/x-patch, Size: 569 bytes --]

"Improved version", more similar to the 2.6.23 code:

Fix page allocator to give better chance of larger contiguous segments (again).

Signed-off-by: Mark Lord <mlord@pobox.com
---

--- old/mm/page_alloc.c	2007-12-13 19:25:15.000000000 -0500
+++ linux-2.6/mm/page_alloc.c	2007-12-13 19:43:07.000000000 -0500
@@ -760,7 +760,7 @@
 		struct page *page = __rmqueue(zone, order, migratetype);
 		if (unlikely(page == NULL))
 			break;
-		list_add(&page->lru, list);
+		list_add_tail(&page->lru, list);
 		set_page_private(page, migratetype);
 	}
 	spin_unlock(&zone->lock);