Re: [PATCH (block.git) 0/2] IO CPU affinity update:

Jens Axboe
2008-03-18T08:41:40Z

On Mon, Mar 17 2008, Alan D. Brunelle wrote:
> Jens Axboe wrote:
> > On Mon, Mar 17 2008, Alan D. Brunelle wrote:
> >> Hi Jens -
> >>
> >> Two patches:
> >>
> >> 1. Adds in the IRQ saving to generic_smp_call_function_single_interrupt (as you had suggested).
> >> 2. Ensures a single IPI generated to get a remote function call handler going.
> >>
> >> So far it is working better than before on the 4-way IA64 w/ the mkfs/untar/make test suite - after 22 runs:
> >>
> >> Part RQ MIN AVG MAX Dev
> >> ----- -- ------ ------ ------ ------
> >> mkfs 0 18.786 19.253 19.655 0.241
> >> mkfs 1 18.639 19.182 19.786 0.293
> >>
> >> untar 0 17.140 17.486 18.250 0.322
> >> untar 1 16.951 17.494 18.274 0.350
> >>
> >> make 0 22.927 24.310 34.339 2.287
> >> make 1 22.863 23.788 24.189 0.333
> >>
> >> comb 0 59.478 61.049 70.320 2.142
> >> comb 1 59.875 60.463 61.305 0.458
> >>
> >> psys 0 3.96% 4.14% 4.39% 0.100
> >> psys 1 3.60% 3.85% 4.19% 0.176
> >>
> >> So we're seeing reduced time (~1.0%) and reduced %sys to do it (7.0%).
> >> The tighter deviations for make with rq=1 may be interesting... :-)
> >>
> >> I've compiled & booted the patches for x86_64 - rq=1 is working on
> >> that platform too.
> >
> > This is starting to look pretty good! Thanks a lot for these results,
> > and the ->activated optimizations. I had a feeling the unstable results
> > were something like this, missing ipi's.
> >
>
> Jens: FYI: I am still seeing infrequent hangs on the x86_64-based
> platform. It happened again today after my patch was added, I did get
> <alt><sysrq><W> to work this time, and the threads that were stuck
> were waiting for IOs to complete. I believe at some point you were
> thinking of hacking in a block IO dump magic key as well - is that
> there yet?

Nope, that is not there yet. My plan was to make it dump block layer
structures, but it would probably not help for this case (as the io
would likely be stuck in the per-cpu queue waiting for completion). I'll
try and add the debug info and include info for the call function
structures.

> I'll get to the ia64 testing tomorrow - have to run now, spent most of
> the day looking at what could cause stuff to be missed on x86_64. The
> code looks solid to me, but this hang needs to be figured out.

To me as well, but we are not looking at the exact same base (but close
enough). I'll play around with the ppc64 today, it's been very handy for
finding memory ordering problems in the past :-)

> Lastly, I did do a run on a 16-way ia64 (with my patches) and again it
> ran fine.

Good

--
Jens Axboe