Re: [Patch V3 0/3] Enable irqs when waiting for rwlocks

Re: [Patch V3 0/3] Enable irqs when waiting for rwlocks

[Andrew Morton]

On Tue, 04 Nov 2008 06:24:05 -0600
holt@sgi.com wrote:

> New in V3:
>  * Handle rearrangement of some arch's include/asm directories.
> 
> New in V2:
>  * get rid of ugly #ifdef's in kernel/spinlock.h
>  * convert __raw_{read|write}_lock_flags to an inline func
> 
> SGI has observed that on large systems, interrupts are not serviced for
> a long period of time when waiting for a rwlock. The following patch
> series re-enables irqs while waiting for the lock, resembling the code
> which is already there for spinlocks.
> 
> I only made the ia64 version, because the patch adds some overhead to
> the fast path. I assume there is currently no demand to have this for
> other architectures, because the systems are not so large. Of course,
> the possibility to implement raw_{read|write}_lock_flags for any
> architecture is still there.
> 

The patches seem reasonable.  I queued all three with the intention of
merging #1 and #2 into 2.6.29.  At that stage, architectures can decide
whether or not they want to do this.  I shall then spam Tony with #3 so
you can duke it out with him.

It's a bit regrettable to have different architectures behaving in
different ways.  It would be interesting to toss an x86_64
implementation into the grinder, see if it causes any problems, see if
it produces any tangible benefits.  Then other architectures might
follow.  Or not, depending on the results ;)

Re: [Patch V3 0/3] Enable irqs when waiting for rwlocks

[Andrew Morton]

On Tue, 04 Nov 2008 06:24:05 -0600
holt@sgi.com wrote:

> New in V3:
>  * Handle rearrangement of some arch's include/asm directories.
> 
> New in V2:
>  * get rid of ugly #ifdef's in kernel/spinlock.h
>  * convert __raw_{read|write}_lock_flags to an inline func
> 
> SGI has observed that on large systems, interrupts are not serviced for
> a long period of time when waiting for a rwlock. The following patch
> series re-enables irqs while waiting for the lock, resembling the code
> which is already there for spinlocks.
> 
> I only made the ia64 version, because the patch adds some overhead to
> the fast path. I assume there is currently no demand to have this for
> other architectures, because the systems are not so large. Of course,
> the possibility to implement raw_{read|write}_lock_flags for any
> architecture is still there.
> 

The patches seem reasonable.  I queued all three with the intention of
merging #1 and #2 into 2.6.29.  At that stage, architectures can decide
whether or not they want to do this.  I shall then spam Tony with #3 so
you can duke it out with him.

It's a bit regrettable to have different architectures behaving in
different ways.  It would be interesting to toss an x86_64
implementation into the grinder, see if it causes any problems, see if
it produces any tangible benefits.  Then other architectures might
follow.  Or not, depending on the results ;)

Re: [Patch V3 0/3] Enable irqs when waiting for rwlocks

[Robin Holt]

> It's a bit regrettable to have different architectures behaving in
> different ways.  It would be interesting to toss an x86_64
> implementation into the grinder, see if it causes any problems, see if
> it produces any tangible benefits.  Then other architectures might
> follow.  Or not, depending on the results ;)

I personally expect SGI to work on this for x86_64 in the future.
Once we actually start testing systems with 128 and above cpus, I
would expect to see these performance issues needing to be addressed.
Until then, it is just a theoretical.

Thank you for your help,
Robin Holt

Re: [Patch V3 0/3] Enable irqs when waiting for rwlocks

[Peter Zijlstra]

On Wed, 2008-12-03 at 05:37 -0600, Robin Holt wrote:
> > It's a bit regrettable to have different architectures behaving in
> > different ways.  It would be interesting to toss an x86_64
> > implementation into the grinder, see if it causes any problems, see if
> > it produces any tangible benefits.  Then other architectures might
> > follow.  Or not, depending on the results ;)
> 
> I personally expect SGI to work on this for x86_64 in the future.
> Once we actually start testing systems with 128 and above cpus, I
> would expect to see these performance issues needing to be addressed.
> Until then, it is just a theoretical.

Personally I consider this a ugly hack and would love to see people
solve the actual problem and move away from rwlock_t, its utter rubbish.

Re: [Patch V3 0/3] Enable irqs when waiting for rwlocks

[Petr Tesarik]

Peter Zijlstra píše v St 03. 12. 2008 v 13:25 +0100:
> On Wed, 2008-12-03 at 05:37 -0600, Robin Holt wrote:
> > > It's a bit regrettable to have different architectures behaving in
> > > different ways.  It would be interesting to toss an x86_64
> > > implementation into the grinder, see if it causes any problems, see if
> > > it produces any tangible benefits.  Then other architectures might
> > > follow.  Or not, depending on the results ;)
> > 
> > I personally expect SGI to work on this for x86_64 in the future.
> > Once we actually start testing systems with 128 and above cpus, I
> > would expect to see these performance issues needing to be addressed.
> > Until then, it is just a theoretical.
> 
> Personally I consider this a ugly hack and would love to see people
> solve the actual problem and move away from rwlock_t, its utter rubbish.

Me too, but we don't have that clean and nice solution today, but what
we _do_ have today are the machines which break badly when interrupts
are disabled for the whole duration of taking a rwlock_t. :(

Feel free to rewrite all users of rwlock_t. I'll appreciate it, oh so
very much.

Petr

--
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Patch V3 0/3] Enable irqs when waiting for rwlocks

[Petr Tesarik]

Peter Zijlstra píše v St 03. 12. 2008 v 13:25 +0100:
> On Wed, 2008-12-03 at 05:37 -0600, Robin Holt wrote:
> > > It's a bit regrettable to have different architectures behaving in
> > > different ways.  It would be interesting to toss an x86_64
> > > implementation into the grinder, see if it causes any problems, see if
> > > it produces any tangible benefits.  Then other architectures might
> > > follow.  Or not, depending on the results ;)
> > 
> > I personally expect SGI to work on this for x86_64 in the future.
> > Once we actually start testing systems with 128 and above cpus, I
> > would expect to see these performance issues needing to be addressed.
> > Until then, it is just a theoretical.
> 
> Personally I consider this a ugly hack and would love to see people
> solve the actual problem and move away from rwlock_t, its utter rubbish.

Me too, but we don't have that clean and nice solution today, but what
we _do_ have today are the machines which break badly when interrupts
are disabled for the whole duration of taking a rwlock_t. :(

Feel free to rewrite all users of rwlock_t. I'll appreciate it, oh so
very much.

Petr

Re: [Patch V3 0/3] Enable irqs when waiting for rwlocks

[Peter Zijlstra]

On Wed, 2008-12-03 at 13:36 +0100, Petr Tesarik wrote:
> Peter Zijlstra píše v St 03. 12. 2008 v 13:25 +0100:
> > On Wed, 2008-12-03 at 05:37 -0600, Robin Holt wrote:
> > > > It's a bit regrettable to have different architectures behaving in
> > > > different ways.  It would be interesting to toss an x86_64
> > > > implementation into the grinder, see if it causes any problems, see if
> > > > it produces any tangible benefits.  Then other architectures might
> > > > follow.  Or not, depending on the results ;)
> > > 
> > > I personally expect SGI to work on this for x86_64 in the future.
> > > Once we actually start testing systems with 128 and above cpus, I
> > > would expect to see these performance issues needing to be addressed.
> > > Until then, it is just a theoretical.
> > 
> > Personally I consider this a ugly hack and would love to see people
> > solve the actual problem and move away from rwlock_t, its utter rubbish.
> 
> Me too, but we don't have that clean and nice solution today, but what
> we _do_ have today are the machines which break badly when interrupts
> are disabled for the whole duration of taking a rwlock_t. :(

Right, which creates an incentive to work on it. So we then either to a
quick and ugly hack and relieve the stress and never again get around to
looking at that particular issue (because management or something
doesn't see the problem anymore) or we do the right thing and fix it
properly.

I prefer the second, but I understand its not always an option.

> Feel free to rewrite all users of rwlock_t. I'll appreciate it, oh so
> very much.

Hehe, I wish! its on my todo list though, along with some other things,
but very unlikely I'll ever get around to it.

The good news is that Eric Dumazet recently got rid of a few prominent
users in the net code.

--
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Patch V3 0/3] Enable irqs when waiting for rwlocks

[Peter Zijlstra]

On Wed, 2008-12-03 at 13:36 +0100, Petr Tesarik wrote:
> Peter Zijlstra píše v St 03. 12. 2008 v 13:25 +0100:
> > On Wed, 2008-12-03 at 05:37 -0600, Robin Holt wrote:
> > > > It's a bit regrettable to have different architectures behaving in
> > > > different ways.  It would be interesting to toss an x86_64
> > > > implementation into the grinder, see if it causes any problems, see if
> > > > it produces any tangible benefits.  Then other architectures might
> > > > follow.  Or not, depending on the results ;)
> > > 
> > > I personally expect SGI to work on this for x86_64 in the future.
> > > Once we actually start testing systems with 128 and above cpus, I
> > > would expect to see these performance issues needing to be addressed.
> > > Until then, it is just a theoretical.
> > 
> > Personally I consider this a ugly hack and would love to see people
> > solve the actual problem and move away from rwlock_t, its utter rubbish.
> 
> Me too, but we don't have that clean and nice solution today, but what
> we _do_ have today are the machines which break badly when interrupts
> are disabled for the whole duration of taking a rwlock_t. :(

Right, which creates an incentive to work on it. So we then either to a
quick and ugly hack and relieve the stress and never again get around to
looking at that particular issue (because management or something
doesn't see the problem anymore) or we do the right thing and fix it
properly.

I prefer the second, but I understand its not always an option.

> Feel free to rewrite all users of rwlock_t. I'll appreciate it, oh so
> very much.

Hehe, I wish! its on my todo list though, along with some other things,
but very unlikely I'll ever get around to it.

The good news is that Eric Dumazet recently got rid of a few prominent
users in the net code.