submit_bh leaves interrupts on upon return

submit_bh leaves interrupts on upon return

[Jeff V. Merkey]

Any reason why submit_bh should turn on interrupts after being called by 
a process with ints off in 2.4.20?  I see it's possible to sleep during 
elevatoring, but why does it need to leave interrupts on if the calling 
state was with ints off.  

I'm back BTW.  Hope no one missed me.  I can be reached at the email 
address above.

:-)

Jeff

Re: submit_bh leaves interrupts on upon return

[Jens Axboe]

On Wed, Jun 02 2004, Jeff V. Merkey wrote:
> 
> Any reason why submit_bh should turn on interrupts after being called by 
> a process with ints off in 2.4.20?  I see it's possible to sleep during 
> elevatoring, but why does it need to leave interrupts on if the calling 
> state was with ints off.  

It's illegal to call it with interrupts off, so... __make_request()
doesn't save interrupt state, so you will always leave with interrupts
enabled.

-- 
Jens Axboe

Re: submit_bh leaves interrupts on upon return

[Jens Axboe]

On Thu, Jun 03 2004, Jeff V. Merkey wrote:
> Jens Axboe wrote:
> 
> >On Wed, Jun 02 2004, Jeff V. Merkey wrote:
> > 
> >
> >>Any reason why submit_bh should turn on interrupts after being called by 
> >>a process with ints off in 2.4.20?  I see it's possible to sleep during 
> >>elevatoring, but why does it need to leave interrupts on if the calling 
> >>state was with ints off.  
> >>   
> >>
> >
> >It's illegal to call it with interrupts off, so... __make_request()
> >doesn't save interrupt state, so you will always leave with interrupts
> >enabled.
> >
> > 
> >
> Jens
> 
> I noticed in the code it does not check for this when make_request is 
> called, so I altered the calling sequence to call with ints on. I don't 
> see much of a performance difference either way, so calling with ints on 
> was easy to instrument. I am posting about 80,000+ buffer heads per 
> second in with what I am doing, so filling out buffer_head structures 
> and submitting them ad hoc was causing some interrupt windows where the 
> chains were getting corrupted. I altered the calling sequence and added 
> atomic counters so I can submit and call with ints on to avoid the 
> corruption. One of the troublesome aspects of the manner in which 
> make_request is implemented in always needing a context of a thread for 
> sleeping to submit asynch I/O limits the ability to gang schedule large 
> disk I/O from the b_end_io callback. Would make performance a lot more 
> spectacular if it worked this way, but I am seeing good enough 
> performance with it left the way it is. 3Ware's 66Mhz ATA adapter in 
> this implementation is reaching almost 400 MB/S throughput on 2.4.20. I 
> have not tried this on 2.6 yet, but will later this month.

Submitting large numbers of buffer_heads from b_end_io is _nasty_, 2.4
io scheduler runtime isn't exactly world champion and you are doing this
at hard irq time. Not a good idea. Definitely not the true path to
performance, unless you don't care about anything else in the system.

At least in 2.6 you have a much faster io scheduler and the additionally
large bio, so you wont spend nearly as much time there if you are
clever. You still need process context, though, that hasn't changed.

-- 
Jens Axboe

Re: submit_bh leaves interrupts on upon return

[Jens Axboe]

On Thu, Jun 03 2004, Jeff V. Merkey wrote:
> Jens Axboe wrote:
> 
> >Submitting large numbers of buffer_heads from b_end_io is _nasty_, 2.4
> >io scheduler runtime isn't exactly world champion and you are doing this
> >at hard irq time. Not a good idea. Definitely not the true path to
> >performance, unless you don't care about anything else in the system.
> >
> >At least in 2.6 you have a much faster io scheduler and the additionally
> >large bio, so you wont spend nearly as much time there if you are
> >clever. You still need process context, though, that hasn't changed.
> >
> > 
> >
> Sounds like I need to move to 2.6. I noticed the elevator is coalescing 
> quite well, and since I am posting mostly continguous runs of sectors, 
> what ends up at the adapter level would probably not change much much 
> between 2.4 and 2.6 since I am maxing out the driver request queues as 
> it is (255 pending requests of 32 scatter/gather elements of 256 sector 
> runs). 2.6 might help but I suspect it will only help alleviate the 
> submission overhead, and not make much difference on performance since 
> the 3Ware card does have an upward limit on outstanding I/O requests.

That's correct, it just helps you diminish the submission overhead by
pushing down 256 sector entities in one go. So as long as you're io
bound it won't give you better io performance, of course. If you are
doing 400MiB/sec it should help you out, though.

-- 
Jens Axboe

Re: submit_bh leaves interrupts on upon return

[Linus Torvalds]

On Thu, 3 Jun 2004, Jens Axboe wrote:

> On Thu, Jun 03 2004, Jeff V. Merkey wrote:
> >
> > Sounds like I need to move to 2.6. I noticed the elevator is coalescing 
> > quite well, and since I am posting mostly continguous runs of sectors, 
> > what ends up at the adapter level would probably not change much much 
> > between 2.4 and 2.6 since I am maxing out the driver request queues as 
> > it is (255 pending requests of 32 scatter/gather elements of 256 sector 
> > runs). 2.6 might help but I suspect it will only help alleviate the 
> > submission overhead, and not make much difference on performance since 
> > the 3Ware card does have an upward limit on outstanding I/O requests.
> 
> That's correct, it just helps you diminish the submission overhead by
> pushing down 256 sector entities in one go. So as long as you're io
> bound it won't give you better io performance, of course. If you are
> doing 400MiB/sec it should help you out, though.

Well, if Jeff does almost exclusively contiguous stuff and submits them in
order, then the coalescing will make sure that even on 2.4.x the queues
don't get too long, and he probably won't see the pathological cases.

		Linus

Re: submit_bh leaves interrupts on upon return

[Jens Axboe]

On Thu, Jun 03 2004, Linus Torvalds wrote:
> 
> 
> On Thu, 3 Jun 2004, Jens Axboe wrote:
> 
> > On Thu, Jun 03 2004, Jeff V. Merkey wrote:
> > >
> > > Sounds like I need to move to 2.6. I noticed the elevator is coalescing 
> > > quite well, and since I am posting mostly continguous runs of sectors, 
> > > what ends up at the adapter level would probably not change much much 
> > > between 2.4 and 2.6 since I am maxing out the driver request queues as 
> > > it is (255 pending requests of 32 scatter/gather elements of 256 sector 
> > > runs). 2.6 might help but I suspect it will only help alleviate the 
> > > submission overhead, and not make much difference on performance since 
> > > the 3Ware card does have an upward limit on outstanding I/O requests.
> > 
> > That's correct, it just helps you diminish the submission overhead by
> > pushing down 256 sector entities in one go. So as long as you're io
> > bound it won't give you better io performance, of course. If you are
> > doing 400MiB/sec it should help you out, though.
> 
> Well, if Jeff does almost exclusively contiguous stuff and submits them in
> order, then the coalescing will make sure that even on 2.4.x the queues
> don't get too long, and he probably won't see the pathological cases.

At 255 pending requests (see above), it will get somewhat bad. So even
if 2.4 copes, it will spend more sys time than 2.6 for this work load
for sure. The queues will get long even for mainly contig submissions,
but of course the time spent in the io scheduler will not be too bad
since it's scanned backwards. And call frequency will be 32 times as
big...

-- 
Jens Axboe

Re: submit_bh leaves interrupts on upon return

[Jeff V. Merkey]

Jens Axboe wrote:

>On Wed, Jun 02 2004, Jeff V. Merkey wrote:
>  
>
>>Any reason why submit_bh should turn on interrupts after being called by 
>>a process with ints off in 2.4.20?  I see it's possible to sleep during 
>>elevatoring, but why does it need to leave interrupts on if the calling 
>>state was with ints off.  
>>    
>>
>
>It's illegal to call it with interrupts off, so... __make_request()
>doesn't save interrupt state, so you will always leave with interrupts
>enabled.
>
>  
>
Jens

I noticed in the code it does not check for this when make_request is 
called, so I altered the calling sequence to call with ints on. I don't 
see much of a performance difference either way, so calling with ints on 
was easy to instrument. I am posting about 80,000+ buffer heads per 
second in with what I am doing, so filling out buffer_head structures 
and submitting them ad hoc was causing some interrupt windows where the 
chains were getting corrupted. I altered the calling sequence and added 
atomic counters so I can submit and call with ints on to avoid the 
corruption. One of the troublesome aspects of the manner in which 
make_request is implemented in always needing a context of a thread for 
sleeping to submit asynch I/O limits the ability to gang schedule large 
disk I/O from the b_end_io callback. Would make performance a lot more 
spectacular if it worked this way, but I am seeing good enough 
performance with it left the way it is. 3Ware's 66Mhz ATA adapter in 
this implementation is reaching almost 400 MB/S throughput on 2.4.20. I 
have not tried this on 2.6 yet, but will later this month.

Also, I ported the kernel debugger from MANOS to Linux and made a lot of 
significant enhancements and www.devicelogics.com is distributing it 
from their website. If anyone wants a more pleasant debugger for the 
kernel to work with, they are allowing downloads of the modules with a 
patch. Not free (what is in this world) but very nice to work with.

Thanks for the response. Sorry I was out of touch for about a year. I 
was going through a very nasty divorce with my wife of 24 years and I 
discovered when something like that is happening in your life, you don't 
have much attention for much else.

Jeff

Re: submit_bh leaves interrupts on upon return

[Jeff V. Merkey]

Jens Axboe wrote:

>Submitting large numbers of buffer_heads from b_end_io is _nasty_, 2.4
>io scheduler runtime isn't exactly world champion and you are doing this
>at hard irq time. Not a good idea. Definitely not the true path to
>performance, unless you don't care about anything else in the system.
>
>At least in 2.6 you have a much faster io scheduler and the additionally
>large bio, so you wont spend nearly as much time there if you are
>clever. You still need process context, though, that hasn't changed.
>
>  
>
Sounds like I need to move to 2.6. I noticed the elevator is coalescing 
quite well, and since I am posting mostly continguous runs of sectors, 
what ends up at the adapter level would probably not change much much 
between 2.4 and 2.6 since I am maxing out the driver request queues as 
it is (255 pending requests of 32 scatter/gather elements of 256 sector 
runs). 2.6 might help but I suspect it will only help alleviate the 
submission overhead, and not make much difference on performance since 
the 3Ware card does have an upward limit on outstanding I/O requests.

Jeff

Re: submit_bh leaves interrupts on upon return

[Jeff V. Merkey]

Jens Axboe wrote:

>On Thu, Jun 03 2004, Jeff V. Merkey wrote:
>  
>
>>Jens Axboe wrote:
>>
>>    
>>
>>>Submitting large numbers of buffer_heads from b_end_io is _nasty_, 2.4
>>>io scheduler runtime isn't exactly world champion and you are doing this
>>>at hard irq time. Not a good idea. Definitely not the true path to
>>>performance, unless you don't care about anything else in the system.
>>>
>>>At least in 2.6 you have a much faster io scheduler and the additionally
>>>large bio, so you wont spend nearly as much time there if you are
>>>clever. You still need process context, though, that hasn't changed.
>>>
>>>
>>>
>>>      
>>>
>>Sounds like I need to move to 2.6. I noticed the elevator is coalescing 
>>quite well, and since I am posting mostly continguous runs of sectors, 
>>what ends up at the adapter level would probably not change much much 
>>between 2.4 and 2.6 since I am maxing out the driver request queues as 
>>it is (255 pending requests of 32 scatter/gather elements of 256 sector 
>>runs). 2.6 might help but I suspect it will only help alleviate the 
>>submission overhead, and not make much difference on performance since 
>>the 3Ware card does have an upward limit on outstanding I/O requests.
>>    
>>
>
>That's correct, it just helps you diminish the submission overhead by
>pushing down 256 sector entities in one go. So as long as you're io
>bound it won't give you better io performance, of course. If you are
>doing 400MiB/sec it should help you out, though.
>
>  
>
I'll give it a try n 2.6.

:-)

Jeff

Re: submit_bh leaves interrupts on upon return

[Jeff V. Merkey]

Linus Torvalds wrote:

>On Thu, 3 Jun 2004, Jens Axboe wrote:
>
>  
>
>>On Thu, Jun 03 2004, Jeff V. Merkey wrote:
>>    
>>
>>>Sounds like I need to move to 2.6. I noticed the elevator is coalescing 
>>>quite well, and since I am posting mostly continguous runs of sectors, 
>>>what ends up at the adapter level would probably not change much much 
>>>between 2.4 and 2.6 since I am maxing out the driver request queues as 
>>>it is (255 pending requests of 32 scatter/gather elements of 256 sector 
>>>runs). 2.6 might help but I suspect it will only help alleviate the 
>>>submission overhead, and not make much difference on performance since 
>>>the 3Ware card does have an upward limit on outstanding I/O requests.
>>>      
>>>
>>That's correct, it just helps you diminish the submission overhead by
>>pushing down 256 sector entities in one go. So as long as you're io
>>bound it won't give you better io performance, of course. If you are
>>doing 400MiB/sec it should help you out, though.
>>    
>>
>
>Well, if Jeff does almost exclusively contiguous stuff and submits them in
>order, then the coalescing will make sure that even on 2.4.x the queues
>don't get too long, and he probably won't see the pathological cases.
>
>		Linus
>-
>
Linus,

This seems to be the case.

Jeff




>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>  
>