Re: [PATCH (block.git) 0/2] IO CPU affinity update:

[Jens Axboe <jens.axboe@oracle.com>]

On Mon, Mar 17 2008, Alan D. Brunelle wrote:
> Jens Axboe wrote:
> > On Mon, Mar 17 2008, Alan D. Brunelle wrote:
> >> Hi Jens -
> >>
> >> Two patches: 
> >>
> >> 1. Adds in the IRQ saving to generic_smp_call_function_single_interrupt (as you had suggested).
> >> 2. Ensures a single IPI generated to get a remote function call handler going. 
> >>
> >> So far it is working better than before on the 4-way IA64 w/ the mkfs/untar/make test suite - after 22 runs:
> >>
> >> Part  RQ   MIN     AVG     MAX      Dev
> >> ----- --  ------  ------  ------  ------
> >>  mkfs  0  18.786  19.253  19.655   0.241
> >>  mkfs  1  18.639  19.182  19.786   0.293
> >>
> >> untar  0  17.140  17.486  18.250   0.322
> >> untar  1  16.951  17.494  18.274   0.350
> >>
> >>  make  0  22.927  24.310  34.339   2.287
> >>  make  1  22.863  23.788  24.189   0.333
> >>
> >>  comb  0  59.478  61.049  70.320   2.142
> >>  comb  1  59.875  60.463  61.305   0.458
> >>
> >>  psys  0   3.96%   4.14%   4.39%   0.100
> >>  psys  1   3.60%   3.85%   4.19%   0.176
> >>
> >> So we're seeing reduced time (~1.0%) and reduced %sys to do it (7.0%).
> >> The tighter deviations for make with rq=1 may be interesting... :-)
> >>
> >> I've compiled & booted the patches for x86_64 - rq=1 is working on
> >> that platform too.
> > 
> > This is starting to look pretty good! Thanks a lot for these results,
> > and the ->activated optimizations. I had a feeling the unstable results
> > were something like this, missing ipi's.
> > 
> 
> Jens: FYI: I am still seeing infrequent hangs on the x86_64-based
> platform. It happened again today after my patch was added, I did get
> <alt><sysrq><W> to work this time, and the threads that were stuck
> were waiting for IOs to complete. I believe at some point you were
> thinking of hacking in a block IO dump magic key as well - is that
> there yet?

Nope, that is not there yet. My plan was to make it dump block layer
structures, but it would probably not help for this case (as the io
would likely be stuck in the per-cpu queue waiting for completion). I'll
try and add the debug info and include info for the call function
structures.

> I'll get to the ia64 testing tomorrow - have to run now, spent most of
> the day looking at what could cause stuff to be missed on x86_64. The
> code looks solid to me, but this hang needs to be figured out. 

To me as well, but we are not looking at the exact same base (but close
enough). I'll play around with the ppc64 today, it's been very handy for
finding memory ordering problems in the past :-)

> Lastly, I did do a run on a 16-way ia64 (with my patches) and again it
> ran fine.

Good

-- 
Jens Axboe