Chaining sg lists for big I/O commands: Question

Chaining sg lists for big I/O commands: Question

[Justin Piszcz]

http://kerneltrap.org/node/8176

I am a mdadm/disk/hard drive fanatic, I was curious:

> On i386, we can at most fit 256 scatterlist elements into a page,
> and on x86-64 we are stuck with 128. So that puts us somewhere
> between 512kb and 1024kb for a single IO.

How come 32bit is 256 and 64 is only 128?
I am sure it is something very fundamental/simple but I was curious, I 
would think x86_64 would fit/support more scatterlists in a page.

Also, when this patch is implemented for x86_64 and if merged into 
mainline, what does this mean for performance?

I have an mdadm raid5 of 10 raptors and get 434MB/s write and 622MB/s 
read, would I see an increase in performance with this patch?

Justin.

Re: Chaining sg lists for big I/O commands: Question

[Jens Axboe]

On Wed, May 09 2007, Justin Piszcz wrote:
> http://kerneltrap.org/node/8176

Oh

> I am a mdadm/disk/hard drive fanatic, I was curious:
> 
> >On i386, we can at most fit 256 scatterlist elements into a page,
> >and on x86-64 we are stuck with 128. So that puts us somewhere
> >between 512kb and 1024kb for a single IO.
> 
> How come 32bit is 256 and 64 is only 128?
>
> I am sure it is something very fundamental/simple but I was curious, I 
> would think x86_64 would fit/support more scatterlists in a page.

Because of the size of the scatterlist structure. As pointers are bigger
on 64-bit archs, the scatterlist structure ends up being bigger. The
page size on x86-64 is 4kb, hence the number of structures you can fit
in a page is smaller.

> Also, when this patch is implemented for x86_64 and if merged into 
> mainline, what does this mean for performance?

The sglist branch of block repo has x86-64 support now. I'll post a new
patchset tomorrow.

Performance wise, it's meant to help higher end hardware that need 2-4mb
(or bigger) commands to get good performance. That also includes things
like tapes that have big block sizes, getting a command of the right
size there is the difference between good and abysmal performance.

> I have an mdadm raid5 of 10 raptors and get 434MB/s write and 622MB/s 
> read, would I see an increase in performance with this patch?

Perhaps, depends on a lot of factors.

-- 
Jens Axboe

Re: Chaining sg lists for big I/O commands: Question

[Jan Engelhardt]

On May 9 2007 15:38, Jens Axboe wrote:
>> I am a mdadm/disk/hard drive fanatic, I was curious:
>> 
>> >On i386, we can at most fit 256 scatterlist elements into a page,
>> >and on x86-64 we are stuck with 128. So that puts us somewhere
>> >between 512kb and 1024kb for a single IO.
>> 
>> How come 32bit is 256 and 64 is only 128?
>>
>> I am sure it is something very fundamental/simple but I was curious, I 
>> would think x86_64 would fit/support more scatterlists in a page.
>
>Because of the size of the scatterlist structure. As pointers are bigger
>on 64-bit archs, the scatterlist structure ends up being bigger. The
>page size on x86-64 is 4kb, hence the number of structures you can fit
>in a page is smaller.

I take it this problem "goes away" on arches with 8KB page_size?


	Jan
-- 

Re: Chaining sg lists for big I/O commands: Question

[Jens Axboe]

On Thu, May 10 2007, Jan Engelhardt wrote:
> On May 9 2007 15:38, Jens Axboe wrote:
> >> I am a mdadm/disk/hard drive fanatic, I was curious:
> >> 
> >> >On i386, we can at most fit 256 scatterlist elements into a page,
> >> >and on x86-64 we are stuck with 128. So that puts us somewhere
> >> >between 512kb and 1024kb for a single IO.
> >> 
> >> How come 32bit is 256 and 64 is only 128?
> >>
> >> I am sure it is something very fundamental/simple but I was curious, I 
> >> would think x86_64 would fit/support more scatterlists in a page.
> >
> >Because of the size of the scatterlist structure. As pointers are bigger
> >on 64-bit archs, the scatterlist structure ends up being bigger. The
> >page size on x86-64 is 4kb, hence the number of structures you can fit
> >in a page is smaller.
> 
> I take it this problem "goes away" on arches with 8KB page_size?

Not really, the 8kb page size just doubles the sg size. On a 64-bit
arch, that would still only get you 1mb IO size.

-- 
Jens Axboe