[Adeos-main] I-pipe for 2.6.32 PPC

[Adeos-main] I-pipe for 2.6.32 PPC

[Bernhard Pfund]

Philippe, all

I'm currently working with a fairly new ppc development board (P2020RDB) by 
Freescale. The board's BSP went mainline with 2.6.32 and I'd like to deploy an 
I-pipe enabled DENX kernel for testing. Now, is there an I-pipe patch for said 
kernel on the release schedule anywhere soon?

TIA::Bernhard

Re: [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Thu, 2009-12-10 at 07:56 +0100, Bernhard Pfund wrote:
> Philippe, all
> 
> I'm currently working with a fairly new ppc development board (P2020RDB) by 
> Freescale. The board's BSP went mainline with 2.6.32 and I'd like to deploy an 
> I-pipe enabled DENX kernel for testing. Now, is there an I-pipe patch for said 
> kernel on the release schedule anywhere soon?

Likely not before January.

> 
> TIA::Bernhard
> 
> _______________________________________________
> Adeos-main mailing list
> Adeos-main@domain.hid
> https://mail.gna.org/listinfo/adeos-main


-- 
Philippe.

Re: [Adeos-main] I-pipe for 2.6.32 PPC

[bernhard]

Zitat von Philippe Gerum <rpm@xenomai.org>:

> On Thu, 2009-12-10 at 07:56 +0100, Bernhard Pfund wrote:
>> Philippe, all
>>
>> I'm currently working with a fairly new ppc development board (P2020RDB) by
>> Freescale. The board's BSP went mainline with 2.6.32 and I'd like   
>> to deploy an
>> I-pipe enabled DENX kernel for testing. Now, is there an I-pipe   
>> patch for said
>> kernel on the release schedule anywhere soon?
>
> Likely not before January.
>
>>
>> TIA::Bernhard
>>
> --
> Philippe.
>

OK, thanks.

Bernhard

Re: [Adeos-main] I-pipe for 2.6.32 PPC

[Richard Cochran]

On Thu, Dec 10, 2009 at 07:56:28AM +0100, Bernhard Pfund wrote:
> I'm currently working with a fairly new ppc development board (P2020RDB) by 
> Freescale. The board's BSP went mainline with 2.6.32 and I'd like to deploy an 
> I-pipe enabled DENX kernel for testing. Now, is there an I-pipe patch for said 
> kernel on the release schedule anywhere soon?

I got ipipe-2.6.30 running the p2020ds with minimal effort. I can post
a diff if that might help you.

I should have my own p2020rdb soon...

I have noticed some unexplained freezes running xenomai programs on
the p2020ds in SMP mode, but I have not yet had time to look into
it. I hope we can work together to get SMP working reliably.

Richard

Re: [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Fri, 2010-01-08 at 11:59 +0100, Richard Cochran wrote:
> On Thu, Dec 10, 2009 at 07:56:28AM +0100, Bernhard Pfund wrote:
> > I'm currently working with a fairly new ppc development board (P2020RDB) by 
> > Freescale. The board's BSP went mainline with 2.6.32 and I'd like to deploy an 
> > I-pipe enabled DENX kernel for testing. Now, is there an I-pipe patch for said 
> > kernel on the release schedule anywhere soon?
> 
> I got ipipe-2.6.30 running the p2020ds with minimal effort. I can post
> a diff if that might help you.
> 
> I should have my own p2020rdb soon...
> 
> I have noticed some unexplained freezes running xenomai programs on
> the p2020ds in SMP mode, but I have not yet had time to look into
> it. I hope we can work together to get SMP working reliably.

I'm working on fixing the pipeline for 2.6.32/ppc64, which includes SMP
mode. 32bit looks ok in UP mode, but I still need to validate SMP there.
So it would be great if you could hammer that patch over your P2020 as
well once it's ready. ETA, early next week. I'll keep you informed about
this.

> 
> Richard
> 
> _______________________________________________
> Adeos-main mailing list
> Adeos-main@domain.hid
> https://mail.gna.org/listinfo/adeos-main


-- 
Philippe.

Re: [Adeos-main] I-pipe for 2.6.32 PPC

[Bernhard Pfund]

Philippe Gerum wrote:
> On Fri, 2010-01-08 at 11:59 +0100, Richard Cochran wrote:
>> On Thu, Dec 10, 2009 at 07:56:28AM +0100, Bernhard Pfund wrote:
>>> I'm currently working with a fairly new ppc development board (P2020RDB) by 
>>> Freescale. The board's BSP went mainline with 2.6.32 and I'd like to deploy an 
>>> I-pipe enabled DENX kernel for testing. Now, is there an I-pipe patch for said 
>>> kernel on the release schedule anywhere soon?
>> I got ipipe-2.6.30 running the p2020ds with minimal effort. I can post
>> a diff if that might help you.
>>
>> I should have my own p2020rdb soon...
>>
>> I have noticed some unexplained freezes running xenomai programs on
>> the p2020ds in SMP mode, but I have not yet had time to look into
>> it. I hope we can work together to get SMP working reliably.
> 
> I'm working on fixing the pipeline for 2.6.32/ppc64, which includes SMP
> mode. 32bit looks ok in UP mode, but I still need to validate SMP there.
> So it would be great if you could hammer that patch over your P2020 as
> well once it's ready. ETA, early next week. I'll keep you informed about
> this.

Philippe, Richard

Due to a change of project priorities I'm currently working on a 
PPC750GX based system (which will require the same kernel and pipeline), 
but I still have the P2020RDB available, so I can (an will) test 
whatever you come up with ;)

Thanx for the ongoing efforts!

Bernhard

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Fri, 2010-01-08 at 12:10 +0100, Philippe Gerum wrote:
> On Fri, 2010-01-08 at 11:59 +0100, Richard Cochran wrote:
> > On Thu, Dec 10, 2009 at 07:56:28AM +0100, Bernhard Pfund wrote:
> > > I'm currently working with a fairly new ppc development board (P2020RDB) by 
> > > Freescale. The board's BSP went mainline with 2.6.32 and I'd like to deploy an 
> > > I-pipe enabled DENX kernel for testing. Now, is there an I-pipe patch for said 
> > > kernel on the release schedule anywhere soon?
> > 
> > I got ipipe-2.6.30 running the p2020ds with minimal effort. I can post
> > a diff if that might help you.
> > 
> > I should have my own p2020rdb soon...
> > 
> > I have noticed some unexplained freezes running xenomai programs on
> > the p2020ds in SMP mode, but I have not yet had time to look into
> > it. I hope we can work together to get SMP working reliably.
> 
> I'm working on fixing the pipeline for 2.6.32/ppc64, which includes SMP
> mode. 32bit looks ok in UP mode, but I still need to validate SMP there.
> So it would be great if you could hammer that patch over your P2020 as
> well once it's ready. ETA, early next week. I'll keep you informed about
> this.

As expected, upgrading from 2.6.30 to 2.6.32 for ppc64 was a real pain.
Anyway, here is a patch which applies against 2.6.32.2 mainline; it is
pretty large because it also includes DENX-originated bits, from v2.6.32
to DENX-v2.6.32.

http://download.gna.org/adeos/patches/tmp/adeos-ipipe-2.6.32.2-powerpc-2.8-00.patch

This is expected to work over Xenomai 2.5.0. Tested over 52xx, 85xx, 4xx
for the 32bit support, pasemi for the 64bit part. I did not manage to
put my hands on a working ppc32/SMP board to give it a shot yet, so,
well, I wish you luck. This said, ppc64/SMP has been validated, so there
is hope.

Sidenote: ftrace may induce a massive stack consumption on ppc64 under
certain circumstances, causing kernel stack overflows. A way to work
around this should be to enable CONFIG_IRQSTACKS, but this won't work
with Xenomai over ppc64 yet, albeit this does work over ppc32. 

This patch is in a staging directory for now; waiting for some feedback
to go further.

HTH,

-- 
Philippe.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Tue, 2010-01-12 at 12:24 -0500, Lennart Sorensen wrote:
> On Tue, Jan 12, 2010 at 05:03:19PM +0100, Philippe Gerum wrote:
> > On Fri, 2010-01-08 at 12:10 +0100, Philippe Gerum wrote:
> > > On Fri, 2010-01-08 at 11:59 +0100, Richard Cochran wrote:
> > > > On Thu, Dec 10, 2009 at 07:56:28AM +0100, Bernhard Pfund wrote:
> > > > > I'm currently working with a fairly new ppc development board (P2020RDB) by 
> > > > > Freescale. The board's BSP went mainline with 2.6.32 and I'd like to deploy an 
> > > > > I-pipe enabled DENX kernel for testing. Now, is there an I-pipe patch for said 
> > > > > kernel on the release schedule anywhere soon?
> > > > 
> > > > I got ipipe-2.6.30 running the p2020ds with minimal effort. I can post
> > > > a diff if that might help you.
> > > > 
> > > > I should have my own p2020rdb soon...
> > > > 
> > > > I have noticed some unexplained freezes running xenomai programs on
> > > > the p2020ds in SMP mode, but I have not yet had time to look into
> > > > it. I hope we can work together to get SMP working reliably.
> > > 
> > > I'm working on fixing the pipeline for 2.6.32/ppc64, which includes SMP
> > > mode. 32bit looks ok in UP mode, but I still need to validate SMP there.
> > > So it would be great if you could hammer that patch over your P2020 as
> > > well once it's ready. ETA, early next week. I'll keep you informed about
> > > this.
> > 
> > As expected, upgrading from 2.6.30 to 2.6.32 for ppc64 was a real pain.
> > Anyway, here is a patch which applies against 2.6.32.2 mainline; it is
> > pretty large because it also includes DENX-originated bits, from v2.6.32
> > to DENX-v2.6.32.
> > 
> > http://download.gna.org/adeos/patches/tmp/adeos-ipipe-2.6.32.2-powerpc-2.8-00.patch
> > 
> > This is expected to work over Xenomai 2.5.0. Tested over 52xx, 85xx, 4xx
> > for the 32bit support, pasemi for the 64bit part. I did not manage to
> > put my hands on a working ppc32/SMP board to give it a shot yet, so,
> > well, I wish you luck. This said, ppc64/SMP has been validated, so there
> > is hope.
> > 
> > Sidenote: ftrace may induce a massive stack consumption on ppc64 under
> > certain circumstances, causing kernel stack overflows. A way to work
> > around this should be to enable CONFIG_IRQSTACKS, but this won't work
> > with Xenomai over ppc64 yet, albeit this does work over ppc32. 
> > 
> > This patch is in a staging directory for now; waiting for some feedback
> > to go further.
> 
> Any chance of getting it without the DENX bits since some of us really
> really don't care for those and want to apply against Linus's releases
> only.  Having the patch always fail to apply is rather annoying.
> 

It turns out that it really, really annoys me not to have the DENX bits
in when working on embedded powerpc targets, but you do have the option
of cloning git://git.denx.de/ipipe-2.6.git, and diff whatever you see
fit with commit 9d81556f or later.

-- 
Philippe.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Tue, 2010-01-12 at 13:50 -0500, Lennart Sorensen wrote:
> On Tue, Jan 12, 2010 at 06:49:11PM +0100, Philippe Gerum wrote:
> > > > As expected, upgrading from 2.6.30 to 2.6.32 for ppc64 was a real pain.
> > > > Anyway, here is a patch which applies against 2.6.32.2 mainline; it is
> > > > pretty large because it also includes DENX-originated bits, from v2.6.32
> > > > to DENX-v2.6.32.
> > > > 
> > > > http://download.gna.org/adeos/patches/tmp/adeos-ipipe-2.6.32.2-powerpc-2.8-00.patch
> > > > 
> > > > This is expected to work over Xenomai 2.5.0. Tested over 52xx, 85xx, 4xx
> > > > for the 32bit support, pasemi for the 64bit part. I did not manage to
> > > > put my hands on a working ppc32/SMP board to give it a shot yet, so,
> > > > well, I wish you luck. This said, ppc64/SMP has been validated, so there
> > > > is hope.
> > > > 
> > > > Sidenote: ftrace may induce a massive stack consumption on ppc64 under
> > > > certain circumstances, causing kernel stack overflows. A way to work
> > > > around this should be to enable CONFIG_IRQSTACKS, but this won't work
> > > > with Xenomai over ppc64 yet, albeit this does work over ppc32. 
> > > > 
> > > > This patch is in a staging directory for now; waiting for some feedback
> > > > to go further.
> > > 
> > > Any chance of getting it without the DENX bits since some of us really
> > > really don't care for those and want to apply against Linus's releases
> > > only.  Having the patch always fail to apply is rather annoying.
> > > 
> > 
> > It turns out that it really, really annoys me not to have the DENX bits
> > in when working on embedded powerpc targets, but you do have the option
> > of cloning git://git.denx.de/ipipe-2.6.git, and diff whatever you see
> > fit with commit 9d81556f or later.
> 
> It seems to me that releasing xenomai and ipipe releases that don't
> apply cleanly to official linux release versions is just rediculous.
> The x86 parts seem to work fine with Linus's releases, why should any
> other architecture be different?  It is rather inconvinient for a lot
> of people.
> 
> I don't believe there is any clean way to generate an ipipe patch for
> Linus's official releases when they contain bits intended to go with
> the DENX tree.  There are just too many changes all over the place.
> 
> 

This is likely because your knowledge overwhelms mine. I'm only a moron,
so git diff with a proper commit number, or even patch -R does work for
me. Simple things always work for simple idiots, it's a fact of life.

> Users of the DENX tree are a tiny minority compared to the users of the
> official kernel releases.  That majority should be the target of releases.
> 

If you do think that the situation for x86 must be applicable to any
other architecture, then I really can't do anything for you, except
perhaps suggest to get your feet a bit more wet with those, and probably
do some reality check against your knowledge of the embedded world.

Aside of this, I don't intend to get dragged into such a discussion,
it's a pure loss of time. So please, take what I can offer, or just
blame me for being an idiot to have a different perspective from yours,
but don't waste my bandwidth. I need it direly to be able to issue
patches you can use, even at the expense of cloning a git tree.

However, if you do think that some good reason might exist to work with
DENX material when it comes to powerpc over embedded platforms, but want
to help in issuing a mainline-based version of the pipeline patches
regularly, then I would agree to ease your task.

So, instead of complaining about how things are not done the way you
want, do you agree to help in changing this situation, or what? 

-- 
Philippe.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Richard Cochran]

On Tue, Jan 12, 2010 at 05:18:02PM -0500, Lennart Sorensen wrote:
> Well I have not been able to find the magic invocation that lets me take
> the DENX tree (which I have had around for a long time just to look at
> occationally, whenever I was trying to get an ipipe patch to apply),
> apply the ipipe patch, revert the DENX changes to get back to a release
> kernel, and generate a diff of the ipipe changes.  It has never worked
> when I tried.

Lennart,

It is not so hard (using git) to remove the Denx patches from the
ipipe tree. I did this myself for 2.6.30 in about a half an hour. If
you don't know how to use git, then you would have to consider the
additional time you need to get to understand it. (For me, it was only
a year or so ;)

In the ipipe tree, the Denx commits have been "squashed" together into
one or two really large commits. So, you can just cherry pick the
adeos commits into a new branch, with a few minor fixups.

Philippe,

I actually agree with Lennart that the Denx stuff is an
annoyance. When considering my "no-denx" branch that I made, I could
not see any significant Denx change that adeos builds upon. There were
a few Denx fixes for one specific board that were close to the adeos
changes, but these were only a few, and easy to fix. So, I could not
understand why Denx is a prerequisite for adeos.

I understand that Denx sponsored the original PowerPC Xenomai port. Is
the reason that ipipe is based on Denx simply to honor that fact? If
so, I would not think it a bad reason at all.

However, I would still prefer the following ordering for the changes:

1. stable linux (2.6.xx.y)
2. adeos arch indepedendent
3. adeos powerpc
4. denx
5. adeos for denx (minimal changes, I expect)

Richard

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Tue, 2010-01-12 at 17:18 -0500, Lennart Sorensen wrote: 
> On Tue, Jan 12, 2010 at 08:23:35PM +0100, Philippe Gerum wrote:
> > This is likely because your knowledge overwhelms mine. I'm only a moron,
> > so git diff with a proper commit number, or even patch -R does work for
> > me. Simple things always work for simple idiots, it's a fact of life.
> 
> Well I have not been able to find the magic invocation that lets me take
> the DENX tree (which I have had around for a long time just to look at
> occationally, whenever I was trying to get an ipipe patch to apply),
> apply the ipipe patch, revert the DENX changes to get back to a release
> kernel, and generate a diff of the ipipe changes.  It has never worked
> when I tried.

To get a complete pipeline, you need the -noarch side, and the -powerpc
side. ipipe-2.6.32-powerpc is based on ipipe-2.6.32-noarch, which is
based on v2.6.32 mainline. From the official I-pipe tree located there:
git://git.denx.de/ipipe-2.6.git

$ git log --abbrev-commit --oneline -4 ipipe-2.6.32-powerpc

gives:

704529e powerpc/ipipe: arch-dependent bits for 2.6.32.2
2cc8efe Merge branch 'DENX-v2.6.32' into ipipe-2.6.32-newppc
01765d4 powerpc: merge DENX-v2.6.32 bits
ea15238 powerpc: merge 2.6.32.2 -stable

So, what you want is a merge between 2.6.32.2 mainline and the powerpc
bits, skipping the DENX bits:

$ git checkout -b ipipe-mainline ipipe-2.6.32-noarch
$ git cherry-pick ea15238
$ git cherry-pick 704529e

This will lead to two trivial rejects, one is in the ppc64 section you
do not care about, the other in process.c, which is a hunk related to
supporting the PA6T 64bit architecture as well, which you don't care
about, either.

# unmerged:   arch/powerpc/kernel/process.c
# unmerged:   arch/powerpc/mm/hash_native_64.c

Basically, it's a 2 minutes work. Granted, it applies to 2.6.32.2 only,
and more conflicts could pop up with different revisions of this work;
but this has never been a massive pain to move from a DENX tree to
mainline, and I did it quite a few times.

> 
> > If you do think that the situation for x86 must be applicable to any
> > other architecture, then I really can't do anything for you, except
> > perhaps suggest to get your feet a bit more wet with those, and probably
> > do some reality check against your knowledge of the embedded world.
> 
> Actually yes I think the x86 situation should be applicable to other
> architectures.  I am currently running embedded x86, powerpc and coldfire,
> all using release kernels.  That to me is a highly desirable goal.
> The coldfire was painful a year ago, but it sure has come a long way in
> that year.
> 

x86 does not qualify for this discussion; with a x86-based system, even
if the kernel at hand does not support the very latest fancy device you
have on board, your kernel will likely still boot and run properly. On
the other hand, you would not even try booting your mpc8360 over a
kernel built for 44x. But given you want to support both, if either of
both happen to be in a poor state in your reference tree, things start
hitting the crapper. The DENX tree saved my day so far, because it
allowed me to work on all powerpc platforms from a single code base.

> > Aside of this, I don't intend to get dragged into such a discussion,
> > it's a pure loss of time. So please, take what I can offer, or just
> > blame me for being an idiot to have a different perspective from yours,
> > but don't waste my bandwidth. I need it direly to be able to issue
> > patches you can use, even at the expense of cloning a git tree.
> 
> Well debian used to try and package xenomai.  They are still on 2.4.8.
> I suspect they may have given up on it given that they only use release
> kernels, and many of the patches are useless in that case.  What's the
> point of trying then.

The Blackfin uClinux distro was still shipping 2.4.7 in their latest
2009R1 release, albeit they merged the Blackfin-specific bits from the
pipeline into their official kernel tree, many moons ago. Incidentally,
I'm maintaining those bits for their development kernel directly feeding
mainline, so I don't think your reasoning should be generalized.

This said, if you have first hand information from the Debian team
regarding this, maybe you should tell them to drop us a note; we would
be happy to help.

> 
> > However, if you do think that some good reason might exist to work with
> > DENX material when it comes to powerpc over embedded platforms, but want
> > to help in issuing a mainline-based version of the pipeline patches
> > regularly, then I would agree to ease your task.
> 
> Well I do work with embedded powerpc, and so far have had no use for
> the DENX tree.  I have no idea what powerpc systems need stuff from the
> DENX tree to work, but certainly not the mpc8360 I deal with.
> 

I understand your point from your perspective. But maintaining our code
base for mpc8360 only is not enough for us.

The reason why the powerpc patches are DENX-based is simply that this
tree worked for me over time, for various platforms (52xx, 4xx/44x, pa6t
come to mind), way before mainline did. Xenomai supports recent
platforms like the 512x series precisely because of that as well.

Does this mean that the pipeline patches will be based on the DENX tree
until hell freezes? No, this does not, because the DENX folks are doing
their job as well, and make their best to close the gap between their
tree and mainline, taking all needed provisions to get their bits merged
upstream sooner.

A sidenote though: you can run Xenomai over mpc8360 because DENX
published this support which was initially based on their tree, on
behalf of a customer, like a bunch of other developments:
http://www.denx.de/en/Software/SoftwareXenomaiProjects
This makes your argument, about not depending on anything being
DENX-originated, sound a bit weird. Make no mistake, you owe them quite
a few kudos. Actually, anyone running Xenomai over powerpc owe them a
lot.

> > So, instead of complaining about how things are not done the way you
> > want, do you agree to help in changing this situation, or what? 
> 
> I did manage to port the 2.6.30-DENX patch to 2.6.32 pure tree rather
> successfully and sent it to the list.  It did take a few hours.
> 

You did port the pipeline to 2.6.32/ppc32 for running on mpc8360, which
fits your needs. But the pipeline patch has to be upgraded for ppc64 as
well. Even for ppc32, a patch is normally validated for a set of 4xx,
512x, 52xx, 82xx, 85xx, and 86xx platforms, by the Xenomai standards:
http://www.xenomai.org/index.php/Embedded_Device_Support#Supported_Evaluation_Boards_3

Btw, your patch never made it to the list, since this is a
subscriber-only list, and last time I checked, you were not subscribed.
Or maybe you posted from a different mail address, but I did not see
your mail though. I usually acknowledge public contributions.

> Having a patch for the DENX tree is useful to some, but having a patch for
> the real Linux tree seems a lot more useful.
>   How about having both then?
> I am willing to help deal with making the patch work on real kernels if
> that makes it easier.

That is good to hear. This said, the DENX tree is based on mainline like
zillions of other trees, and as you certainly know, many of those trees
are feeding mainline with arch-specific improvements and new platform or
driver support. So the concepts of "real Linux tree" or "real kernel",
"or pure tree" bear absolutely no value. The latest "pure" Linux tree
probably dates back to 1994 or so.

>   At least then there would be something that could
> be used by people using a single kernel tree for multiple architectures.
> 

Publishing the DENX-based support on a separate branch for ppc platforms
which have incomplete support upstream, along with a mainline-based
branch for the rest, is something I could handle.

The way for you to help is certainly about testing the mainline-based
branch which will probably appear for 2.6.33, and send feedback/fixes
as/if needed.

-- 
Philippe.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Wolfgang Grandegger]

Hi Philippe,

Philippe Gerum wrote:
[snip]
> On Tue, 2010-01-12 at 17:18 -0500, Lennart Sorensen wrote: 
> You did port the pipeline to 2.6.32/ppc32 for running on mpc8360, which
> fits your needs. But the pipeline patch has to be upgraded for ppc64 as
> well. Even for ppc32, a patch is normally validated for a set of 4xx,
> 512x, 52xx, 82xx, 85xx, and 86xx platforms, by the Xenomai standards:
> http://www.xenomai.org/index.php/Embedded_Device_Support#Supported_Evaluation_Boards_3
> 
> Btw, your patch never made it to the list, since this is a
> subscriber-only list, and last time I checked, you were not subscribed.
> Or maybe you posted from a different mail address, but I did not see
> your mail though. I usually acknowledge public contributions.

Here is the story. Lennart mentioned on the linuxppc-dev mailing list,
that he has a iPipe patch for 2.6.32:
 http://lists.ozlabs.org/pipermail/linuxppc-dev/2009-December/079191.html

I asked him privately, if the patch is available and he sent it to me
with the note, that I can publish it on the mailing list if I feel it's
useful. This patch helped me to go ahead immediately with some related
development work. I also forwarded the patch to you.

Wolfgang.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Fri, 2010-01-15 at 11:14 -0500, Lennart Sorensen wrote:
> On Fri, Jan 15, 2010 at 04:03:31PM +0100, Philippe Gerum wrote:
> > To get a complete pipeline, you need the -noarch side, and the -powerpc
> > side. ipipe-2.6.32-powerpc is based on ipipe-2.6.32-noarch, which is
> > based on v2.6.32 mainline. From the official I-pipe tree located there:
> > git://git.denx.de/ipipe-2.6.git
> > 
> > $ git log --abbrev-commit --oneline -4 ipipe-2.6.32-powerpc
> 
> rceng02:/data/git-trees/ipipe-2.6# git log --abbrev-commit --oneline -4 ipipe-2.6.32-powerpc
> fatal: ambiguous argument 'ipipe-2.6.32-powerpc': unknown revision or path not in the working tree.
> Use '--' to separate paths from revisions
> 

>From your end, the ipipe branches you care of are remote ones available
from the "origin" namespace, so you should try
origin/ipipe-2.6.32-powerpc, and origin/ipipe-2.6.32-noarch is specs.

> Hmm, is my git version too old?
> 

No, I think it's fine.

<snip>

> > x86 does not qualify for this discussion; with a x86-based system, even
> > if the kernel at hand does not support the very latest fancy device you
> > have on board, your kernel will likely still boot and run properly. On
> > the other hand, you would not even try booting your mpc8360 over a
> > kernel built for 44x. But given you want to support both, if either of
> > both happen to be in a poor state in your reference tree, things start
> > hitting the crapper. The DENX tree saved my day so far, because it
> > allowed me to work on all powerpc platforms from a single code base.
> 
> Certainly kernels for embedded systems often are only menat for one
> system.  That doesn't mean you couldn't build a kernel with support for
> multiple systems.
> 

Sure, but for that, you need to have a single source tree that works for
both. This is the gist of the issue I'm facing.

> > The Blackfin uClinux distro was still shipping 2.4.7 in their latest
> > 2009R1 release, albeit they merged the Blackfin-specific bits from the
> > pipeline into their official kernel tree, many moons ago. Incidentally,
> > I'm maintaining those bits for their development kernel directly feeding
> > mainline, so I don't think your reasoning should be generalized.
> > 
> > This said, if you have first hand information from the Debian team
> > regarding this, maybe you should tell them to drop us a note; we would
> > be happy to help.
> 
> No I don't.  Just speculation.  I just remember trying to use the package
> and finding that it simply could not apply to the released kernel sources.
> 
> > I understand your point from your perspective. But maintaining our code
> > base for mpc8360 only is not enough for us.
> > 
> > The reason why the powerpc patches are DENX-based is simply that this
> > tree worked for me over time, for various platforms (52xx, 4xx/44x, pa6t
> > come to mind), way before mainline did. Xenomai supports recent
> > platforms like the 512x series precisely because of that as well.
> > 
> > Does this mean that the pipeline patches will be based on the DENX tree
> > until hell freezes? No, this does not, because the DENX folks are doing
> > their job as well, and make their best to close the gap between their
> > tree and mainline, taking all needed provisions to get their bits merged
> > upstream sooner.
> 
> Well I hope it happens soon.  Things always become much easier when you
> can work with the official kernel releases.  The powerpc support in the
> main kernel tree seem rather good.
> 

Generally yes, it is. We just have to deal with exceptions, and those
exceptions used to be many. I agree that things evolve and the situation
has to be reassessed from time to time, though. It's probably a good
time to do this.

> > A sidenote though: you can run Xenomai over mpc8360 because DENX
> > published this support which was initially based on their tree, on
> > behalf of a customer, like a bunch of other developments:
> > http://www.denx.de/en/Software/SoftwareXenomaiProjects
> > This makes your argument, about not depending on anything being
> > DENX-originated, sound a bit weird. Make no mistake, you owe them quite
> > a few kudos. Actually, anyone running Xenomai over powerpc owe them a
> > lot.
> 
> Oh I know that.  We paid for it.  It was well worth it.  That doesn't
> mean I want to use the DENX kernel tree.

But at least, you know for sure why having a working kernel is
important, specially to those who cannot rely on mainline yet.

> 
> > You did port the pipeline to 2.6.32/ppc32 for running on mpc8360, which
> > fits your needs. But the pipeline patch has to be upgraded for ppc64 as
> > well. Even for ppc32, a patch is normally validated for a set of 4xx,
> > 512x, 52xx, 82xx, 85xx, and 86xx platforms, by the Xenomai standards:
> > http://www.xenomai.org/index.php/Embedded_Device_Support#Supported_Evaluation_Boards_3
> 
> Well I managed to get every piece except a few ppc64 bits (which as
> usual seemed to only apply to parts of the DENX tree.  When will those
> bits go mainline I wonder.  They seem to have been there for a while now).
> 
> I can only test on the 83xx boards we have here, but the person I gave
> a copy to said it worked for him, which was a different board, so it
> can't have been too bad.

It was likely fine, this was not my point. My point was rather that
testing this on all of the hw platforms we are supposed to test it is
much easier from a single code base.

> 
> > Btw, your patch never made it to the list, since this is a
> > subscriber-only list, and last time I checked, you were not subscribed.
> > Or maybe you posted from a different mail address, but I did not see
> > your mail though. I usually acknowledge public contributions.
> 
> I gave a copy to someone else that saked for it who said they would post
> a copy.

For some reason, the patch entered the twilight zone before I could pick
it. Maybe one of my spam filters, I don't know yet, I'll check.

> 
> > That is good to hear. This said, the DENX tree is based on mainline like
> > zillions of other trees, and as you certainly know, many of those trees
> > are feeding mainline with arch-specific improvements and new platform or
> > driver support. So the concepts of "real Linux tree" or "real kernel",
> > "or pure tree" bear absolutely no value. The latest "pure" Linux tree
> > probably dates back to 1994 or so.
> 
> OK, I will call it "Linus's tree from which released kernels are made".
> Specifically patches that apply to those released kernels are nice.
> Of course point release fixes to those release kernels can still break
> the patch.  That's life when working out of tree.

Nope, actually, AFAIC, working out of tree is no life. But someone has
to do it, sometimes.

> 
> > Publishing the DENX-based support on a separate branch for ppc platforms
> > which have incomplete support upstream, along with a mainline-based
> > branch for the rest, is something I could handle.
> 
> Makes sense to me.  Certainly not having support for platforms just
> because the released kernels doesn't support it yet would be silly.
> Just as not having support for people who are running release kernels
> also seems silly.  Both seem very useful.
> 
> > The way for you to help is certainly about testing the mainline-based
> > branch which will probably appear for 2.6.33, and send feedback/fixes
> > as/if needed.
> 
> I will certainly be testing it.
> 

Ok, thanks. I will keep you posted.

-- 
Philippe.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Fri, 2010-01-15 at 16:33 +0100, Wolfgang Grandegger wrote:
> Hi Philippe,
> 
> Philippe Gerum wrote:
> [snip]
> > On Tue, 2010-01-12 at 17:18 -0500, Lennart Sorensen wrote: 
> > You did port the pipeline to 2.6.32/ppc32 for running on mpc8360, which
> > fits your needs. But the pipeline patch has to be upgraded for ppc64 as
> > well. Even for ppc32, a patch is normally validated for a set of 4xx,
> > 512x, 52xx, 82xx, 85xx, and 86xx platforms, by the Xenomai standards:
> > http://www.xenomai.org/index.php/Embedded_Device_Support#Supported_Evaluation_Boards_3
> > 
> > Btw, your patch never made it to the list, since this is a
> > subscriber-only list, and last time I checked, you were not subscribed.
> > Or maybe you posted from a different mail address, but I did not see
> > your mail though. I usually acknowledge public contributions.
> 
> Here is the story. Lennart mentioned on the linuxppc-dev mailing list,
> that he has a iPipe patch for 2.6.32:
>  http://lists.ozlabs.org/pipermail/linuxppc-dev/2009-December/079191.html
> 
> I asked him privately, if the patch is available and he sent it to me
> with the note, that I can publish it on the mailing list if I feel it's
> useful. This patch helped me to go ahead immediately with some related
> development work. I also forwarded the patch to you.

It must be stuck somewhere, I really can't find it in my mailbox. I'll
check on my side again. Thanks.

> 
> Wolfgang.
> 
> _______________________________________________
> Adeos-main mailing list
> Adeos-main@domain.hid
> https://mail.gna.org/listinfo/adeos-main


-- 
Philippe.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Wolfgang Grandegger]

[-- Attachment #1: Type: text/plain, Size: 1652 bytes --]

Philippe Gerum wrote:
> On Fri, 2010-01-15 at 16:33 +0100, Wolfgang Grandegger wrote:
>> Hi Philippe,
>>
>> Philippe Gerum wrote:
>> [snip]
>>> On Tue, 2010-01-12 at 17:18 -0500, Lennart Sorensen wrote: 
>>> You did port the pipeline to 2.6.32/ppc32 for running on mpc8360, which
>>> fits your needs. But the pipeline patch has to be upgraded for ppc64 as
>>> well. Even for ppc32, a patch is normally validated for a set of 4xx,
>>> 512x, 52xx, 82xx, 85xx, and 86xx platforms, by the Xenomai standards:
>>> http://www.xenomai.org/index.php/Embedded_Device_Support#Supported_Evaluation_Boards_3
>>>
>>> Btw, your patch never made it to the list, since this is a
>>> subscriber-only list, and last time I checked, you were not subscribed.
>>> Or maybe you posted from a different mail address, but I did not see
>>> your mail though. I usually acknowledge public contributions.
>> Here is the story. Lennart mentioned on the linuxppc-dev mailing list,
>> that he has a iPipe patch for 2.6.32:
>>  http://lists.ozlabs.org/pipermail/linuxppc-dev/2009-December/079191.html
>>
>> I asked him privately, if the patch is available and he sent it to me
>> with the note, that I can publish it on the mailing list if I feel it's
>> useful. This patch helped me to go ahead immediately with some related
>> development work. I also forwarded the patch to you.
> 
> It must be stuck somewhere, I really can't find it in my mailbox. I'll
> check on my side again. Thanks.

I can't find it either, sorry. Obviously I just put you on CC of the
followup mail and forgot to forward the patch somehow. I have attached
it now.

Sorry again for the confusion.

Wolfgang.

[-- Attachment #2: Attached Message --]
[-- Type: message/rfc822, Size: 335636 bytes --]

From: lsorense@domain.hid (Lennart Sorensen)
To: Wolfgang Grandegger <wg@domain.hid>
Cc: Lennart Sorensen <lsorense@domain.hid>
Subject: Re: [OFFLIST] Re: ucc_geth broken in 2.6.32 by 864fdf884e82bacbe8ca5e93bd43393a61d2e2b4
Date: Tue, 29 Dec 2009 20:25:03 -0500
Message-ID: <20091230012503.GB8605@domain.hid>

On Thu, Dec 24, 2009 at 12:47:45PM +0100, Wolfgang Grandegger wrote:
> Interesting? I wanted to port the Adeos-Ipipe patch for PowerPC to
> 2.6.32 as well, as Philippe will need some more time to catch up. Would
> you dare to send your patch to me or the Xenomai mailing list for
> testing and intermediate solution?

Sure.  I think I may have left out a tiny piece of ppc64 support that
I couldn't understand, and didn't really care about since I use a 32bit
ppc anyhow.

Of course mine is also based on Linus's 2.6.32 release, not the denx
tree that is just different enough to be annoying.

As for the ucc_geth problem, that at least has now been resolved and
was not ipipe related at all.  A few boxes are running stress tests over
the holidays right now.

I think I am missing something from head_64.S but everything else I
believe I have managed to figure out.  It booted on first try, so I was
pretty pleased with it.  I am using xenomai 2.4.10 with it along with
one patch from the git tree for 2.4 as far as I remember.

Feel free to send it to the xenomai list or where it makes sense for
comments or fixes or adoption as desired.

adeos-ipipe-2.6.32.1-powerpc.patch:
diff -urN source_powerpc_none/arch/powerpc/Kconfig source_powerpc_none.ipipe/arch/powerpc/Kconfig
--- source_powerpc_none/arch/powerpc/Kconfig	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/Kconfig	2009-12-22 12:44:08.000000000 -0500
@@ -130,6 +130,7 @@
 	select HAVE_SYSCALL_WRAPPERS if PPC64
 	select GENERIC_ATOMIC64 if PPC32
 	select HAVE_PERF_EVENTS
+	select HAVE_FUNCTION_TRACE_MCOUNT_TEST
 
 config EARLY_PRINTK
 	bool
@@ -145,6 +146,10 @@
 	depends on COMPAT && SYSVIPC
 	default y
 
+config SOFTDISABLE
+	bool
+	default (PPC64 && !IPIPE)
+
 # All PPC32s use generic nvram driver through ppc_md
 config GENERIC_NVRAM
 	bool
@@ -249,6 +254,29 @@
 
 menu "Kernel options"
 
+source "kernel/ipipe/Kconfig"
+
+config IPIPE_HAVE_PREEMPTIBLE_SWITCH
+       bool
+       depends on IPIPE
+       default y
+
+if IPIPE
+config RUNLATCH
+	bool "Enable RUNLATCH support"
+	depends on PPC64
+	default n if IPIPE
+	---help---
+	This option is costly latency-wise, so default is to keep
+	it off when the interrupt pipeline is enabled.
+endif
+if !IPIPE
+config RUNLATCH
+	bool
+	depends on PPC64
+	default y
+endif
+
 config HIGHMEM
 	bool "High memory support"
 	depends on PPC32
diff -urN source_powerpc_none/arch/powerpc/boot/Makefile source_powerpc_none.ipipe/arch/powerpc/boot/Makefile
--- source_powerpc_none/arch/powerpc/boot/Makefile	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/boot/Makefile	2009-12-22 12:44:08.000000000 -0500
@@ -29,6 +29,14 @@
 BOOTCFLAGS	+= -g
 endif
 
+ifdef CONFIG_IPIPE_TRACE
+# do not trace the boot loader
+nullstring :=
+space      := $(nullstring) # end of the line
+pg_flag     = $(nullstring) -pg # end of the line
+BOOTCFLAGS     := $(subst ${pg_flag},${space},${BOOTCFLAGS})
+endif
+
 ifeq ($(call cc-option-yn, -fstack-protector),y)
 BOOTCFLAGS	+= -fno-stack-protector
 endif
diff -urN source_powerpc_none/arch/powerpc/include/asm/exception-64s.h source_powerpc_none.ipipe/arch/powerpc/include/asm/exception-64s.h
--- source_powerpc_none/arch/powerpc/include/asm/exception-64s.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/exception-64s.h	2009-12-22 12:44:08.000000000 -0500
@@ -47,6 +47,30 @@
 #define EX_R3		64
 #define EX_LR		72
 
+#ifdef CONFIG_SOFTDISABLE	
+#define COPY_SOFTISTATE(mreg)			\
+	lbz	mreg,PACASOFTIRQEN(r13);	\
+	std	mreg,SOFTE(r1);
+#define TEST_SOFTISTATE(mreg, dlabel)		\
+	lbz	mreg,PACASOFTIRQEN(r13);	\
+	cmpwi	mreg,0;				\
+	beq-	dlabel;
+#else
+#ifdef CONFIG_IPIPE
+/* Do NOT alter Rc(eq) in this code;  our caller uses it. */
+#define COPY_SOFTISTATE(mreg)			\
+	ld	mreg,PACAROOTPCPU(r13);		\
+	ld	mreg,0(mreg);			\
+	nor	mreg,mreg,mreg;			\
+	clrldi	mreg,mreg,63;			\
+	std	mreg,SOFTE(r1);
+#define TEST_SOFTISTATE(mreg, dlabel)
+#else
+#define COPY_SOFTISTATE(mreg)
+#define TEST_SOFTISTATE(mreg, dlabel)
+#endif
+#endif
+
 /*
  * We're short on space and time in the exception prolog, so we can't
  * use the normal SET_REG_IMMEDIATE macro. Normally we just need the
@@ -128,9 +152,8 @@
 	std	r9,_LINK(r1);						   \
 	mfctr	r10;			/* save CTR in stackframe	*/ \
 	std	r10,_CTR(r1);						   \
-	lbz	r10,PACASOFTIRQEN(r13);				   \
+	COPY_SOFTISTATE(r10);						   \
 	mfspr	r11,SPRN_XER;		/* save XER in stackframe	*/ \
-	std	r10,SOFTE(r1);						   \
 	std	r11,_XER(r1);						   \
 	li	r9,(n)+1;						   \
 	std	r9,_TRAP(r1);		/* set trap number		*/ \
@@ -174,10 +197,8 @@
 	mfspr	r13,SPRN_SPRG_PACA;	/* get paca address into r13 */	\
 	std	r9,PACA_EXGEN+EX_R9(r13);	/* save r9, r10 */	\
 	std	r10,PACA_EXGEN+EX_R10(r13);				\
-	lbz	r10,PACASOFTIRQEN(r13);					\
 	mfcr	r9;							\
-	cmpwi	r10,0;							\
-	beq	masked_interrupt;					\
+	TEST_SOFTISTATE(r10, masked_interrupt);				\
 	mfspr	r10,SPRN_SPRG_SCRATCH0;					\
 	std	r10,PACA_EXGEN+EX_R13(r13);				\
 	std	r11,PACA_EXGEN+EX_R11(r13);				\
@@ -192,6 +213,28 @@
 	rfid;								\
 	b	.	/* prevent speculative execution */
 
+#ifdef CONFIG_IPIPE
+/* IBM legacy I-Series are not supported. */
+#define ENABLE_INTS				\
+	ld	r12,_MSR(r1);			\
+	mfmsr	r11;				\
+	rlwimi	r11,r12,0,MSR_EE;		\
+	mtmsrd	r11,1
+#define DISABLE_INTS	/* We lie, mostly... */	\
+	ld	r11,PACAROOTPCPU(r13);		\
+	ld	r10,0(r11);			\
+	ori	r10,r10,1;			\
+	std	r10,0(r11);			\
+	mfmsr	r10;				\
+	ori	r10,r10,MSR_EE;			\
+	mtmsrd	r10,1;
+#define DISABLE_INTS_REALLY			\
+	mfmsr	r11;				\
+	rldicl	r11,r11,48,1;/* clear MSR_EE */	\
+	rotldi	r11,r11,16;			\
+	mtmsrd	r11,1;
+#else /* !CONFIG_IPIPE */
+
 #ifdef CONFIG_PPC_ISERIES
 #define DISABLE_INTS				\
 	li	r11,0;				\
@@ -212,13 +255,15 @@
 	stb	r11,PACAHARDIRQEN(r13);		\
 	TRACE_DISABLE_INTS
 #endif /* CONFIG_PPC_ISERIES */
-
+		
 #define ENABLE_INTS				\
 	ld	r12,_MSR(r1);			\
 	mfmsr	r11;				\
 	rlwimi	r11,r12,0,MSR_EE;		\
 	mtmsrd	r11,1
 
+#endif /* !CONFIG_IPIPE */
+
 #define STD_EXCEPTION_COMMON(trap, label, hdlr)		\
 	.align	7;					\
 	.globl label##_common;				\
@@ -226,6 +271,7 @@
 	EXCEPTION_PROLOG_COMMON(trap, PACA_EXGEN);	\
 	DISABLE_INTS;					\
 	bl	.save_nvgprs;				\
+	TRACE_DISABLE_INTS;				\
 	addi	r3,r1,STACK_FRAME_OVERHEAD;		\
 	bl	hdlr;					\
 	b	.ret_from_except
@@ -242,6 +288,7 @@
 	FINISH_NAP;					\
 	DISABLE_INTS;					\
 	bl	.save_nvgprs;				\
+	TRACE_DISABLE_INTS;				\
 	addi	r3,r1,STACK_FRAME_OVERHEAD;		\
 	bl	hdlr;					\
 	b	.ret_from_except
@@ -256,10 +303,24 @@
 BEGIN_FTR_SECTION					\
 	bl	.ppc64_runlatch_on;			\
 END_FTR_SECTION_IFSET(CPU_FTR_CTRL)			\
+	TRACE_DISABLE_INTS;				\
 	addi	r3,r1,STACK_FRAME_OVERHEAD;		\
 	bl	hdlr;					\
 	b	.ret_from_except_lite
 
+#ifdef CONFIG_IPIPE
+#define IPIPE_EXCEPTION_COMMON_LITE(trap, label, hdlr)	\
+	.align	7;					\
+	.globl label##_common;				\
+label##_common:						\
+	EXCEPTION_PROLOG_COMMON(trap, PACA_EXGEN);	\
+	DISABLE_INTS_REALLY;				\
+	TRACE_DISABLE_INTS_REALLY;			\
+	addi	r3,r1,STACK_FRAME_OVERHEAD;		\
+	bl	hdlr;					\
+	b	.__ipipe_ret_from_except_lite
+#endif /* CONFIG_IPIPE */
+
 /*
  * When the idle code in power4_idle puts the CPU into NAP mode,
  * it has to do so in a loop, and relies on the external interrupt
diff -urN source_powerpc_none/arch/powerpc/include/asm/ftrace.h source_powerpc_none.ipipe/arch/powerpc/include/asm/ftrace.h
--- source_powerpc_none/arch/powerpc/include/asm/ftrace.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/ftrace.h	2009-12-22 12:44:08.000000000 -0500
@@ -9,9 +9,21 @@
 
 /* Based off of objdump optput from glibc */
 
-#define MCOUNT_SAVE_FRAME			\
-	stwu	r1,-48(r1);			\
-	stw	r3, 12(r1);			\
+#define MCOUNT_SAVE_FRAME			    \
+	stwu	r1,-48(r1);			    \
+	stw	r3, 12(r1);			    \
+        LOAD_REG_IMMEDIATE(r3, function_trace_stop) \
+        lwz     r3, 0(r3);			    \
+        cmpwi   r3, 0;				    \
+	lwz	r3, 12(r1);			    \
+        beq     1f;				    \
+	mflr	r0;				    \
+	mtctr	r0;				    \
+	lwz	r0, 52(r1);			    \
+	mtlr	r0;				    \
+	addi	r1, r1, 48;			    \
+	bctr;					    \
+1:						    \
 	stw	r4, 16(r1);			\
 	stw	r5, 20(r1);			\
 	stw	r6, 24(r1);			\
diff -urN source_powerpc_none/arch/powerpc/include/asm/hw_irq.h source_powerpc_none.ipipe/arch/powerpc/include/asm/hw_irq.h
--- source_powerpc_none/arch/powerpc/include/asm/hw_irq.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/hw_irq.h	2009-12-22 12:44:08.000000000 -0500
@@ -14,6 +14,8 @@
 extern void timer_interrupt(struct pt_regs *);
 
 #ifdef CONFIG_PPC64
+
+#ifdef CONFIG_SOFTDISABLE
 #include <asm/paca.h>
 
 static inline unsigned long local_get_flags(void)
@@ -69,17 +71,173 @@
 	return flags == 0;
 }
 
-#else
+#define local_irq_save_hw(x)	raw_local_irq_save(x)
+#define local_irq_restore_hw(x)	raw_local_irq_restore(x)
+#define local_irq_enable_hw()	raw_local_irq_enable()
+#define local_irq_disable_hw()	raw_local_irq_disable()
+#define irqs_disabled_hw()	raw_irqs_disabled()
+
+#else /* !CONFIG_SOFTDISABLE */
+
+#ifdef CONFIG_IPIPE
+
+#include <linux/ipipe_base.h>
+#include <linux/ipipe_trace.h>
+
+#define raw_local_save_flags(x)		do {			\
+		(x) = (!__ipipe_test_root()) << MSR_EE_LG;	\
+		__asm__ __volatile__("": : :"memory");		\
+	} while(0)
+
+#define raw_local_irq_restore(x)	do {			\
+		__asm__ __volatile__("": : :"memory");		\
+		__ipipe_restore_root(!((x) & MSR_EE));		\
+	} while(0)
+
+static inline void raw_local_irq_save_ptr(unsigned long *x)
+{
+	*x = (!__ipipe_test_and_stall_root()) << MSR_EE_LG;
+	barrier();
+}
+
+#define raw_local_irq_save(x)			\
+do {						\
+	ipipe_check_context(ipipe_root_domain);	\
+	raw_local_irq_save_ptr(&(x));		\
+} while(0)
+
+#define hard_irq_enable()	do {				\
+		barrier();					\
+		__ipipe_unstall_root();				\
+	} while(0)
+
+#define hard_irq_disable()	do {				\
+		ipipe_check_context(ipipe_root_domain);		\
+		__ipipe_stall_root();				\
+		barrier();					\
+	} while(0)
+
+#define raw_local_irq_disable()	hard_irq_disable()
+#define raw_local_irq_enable()	hard_irq_enable()
+#define raw_irqs_disabled()	(__ipipe_test_root() != 0)
+
+static inline int raw_irqs_disabled_flags(unsigned long x)
+{
+	return !(x & MSR_EE);
+}
+
+static inline unsigned long raw_mangle_irq_bits(int virt, unsigned long real)
+{
+	/* Merge virtual and real interrupt mask bits into a single
+	   64bit word. We know MSR_EE will not conflict with 1L<<31. */
+	return (real & ~(1L << 31)) | ((long)virt << 31);
+}
+
+static inline int raw_demangle_irq_bits(unsigned long *x)
+{
+	int virt = (*x & (1L << 31)) != 0;
+	*x &= ~(1L << 31);
+	return virt;
+}
+
+#define local_irq_disable_hw_notrace()		__mtmsrd(mfmsr() & ~MSR_EE, 1)
+#define local_irq_enable_hw_notrace()		__mtmsrd(mfmsr() | MSR_EE, 1)
+#define local_irq_save_hw_notrace(x)		({ (x) = __local_irq_save_hw(); })
+#define local_irq_restore_hw_notrace(x)		__mtmsrd(x, 1)
+
+static inline unsigned long __local_irq_save_hw(void)
+{
+	unsigned long msr = mfmsr();
+	local_irq_disable_hw_notrace();
+	__asm__ __volatile__("": : :"memory");
+	return msr;
+}
+	
+#define local_save_flags_hw(x)		((x) = mfmsr())
+#define irqs_disabled_hw()		((mfmsr() & MSR_EE) == 0)
+
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+#define local_irq_disable_hw() do {			\
+		if (!irqs_disabled_hw()) {		\
+			local_irq_disable_hw_notrace();	\
+			ipipe_trace_begin(0x80000000);	\
+		}					\
+	} while (0)
+#define local_irq_enable_hw() do {			\
+		if (irqs_disabled_hw()) {		\
+			ipipe_trace_end(0x80000000);	\
+			local_irq_enable_hw_notrace();	\
+		}					\
+	} while (0)
+#define local_irq_save_hw(x) do {			\
+		local_save_flags_hw(x);			\
+		if ((x) & MSR_EE) {			\
+			local_irq_disable_hw_notrace();	\
+			ipipe_trace_begin(0x80000001);	\
+		}					\
+	} while (0)
+#define local_irq_restore_hw(x) do {			\
+		if ((x) & MSR_EE)			\
+			ipipe_trace_end(0x80000001);	\
+		local_irq_restore_hw_notrace(x);	\
+	} while (0)
+#else /* !CONFIG_IPIPE_TRACE_IRQSOFF */
+#define local_irq_save_hw(x)	local_irq_save_hw_notrace(x)
+#define local_irq_restore_hw(x)	local_irq_restore_hw_notrace(x)
+#define local_irq_enable_hw()	local_irq_enable_hw_notrace()
+#define local_irq_disable_hw()	local_irq_disable_hw_notrace()
+#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */
+
+#else /* !CONFIG_IPIPE */
+
+#define hard_irq_enable()	__mtmsrd(mfmsr() | MSR_EE, 1)
+#define hard_irq_disable()	__mtmsrd(mfmsr() & ~MSR_EE, 1)
+
+#define raw_local_save_flags(x)		((x) = mfmsr())
+#define raw_local_irq_restore(x) 	__mtmsrd(x, 1)
+#define raw_irqs_disabled()		((mfmsr() & MSR_EE) == 0)
+
+#define local_irq_save_hw(x)	raw_local_irq_save(x)
+#define local_irq_restore_hw(x)	raw_local_irq_restore(x)
+#define local_irq_enable_hw()	hard_irq_enable()
+#define local_irq_disable_hw()	hard_irq_disable()
+#define irqs_disabled_hw()	raw_irqs_disabled()
+
+#endif /* !CONFIG_IPIPE */
+
+#endif /* !CONFIG_SOFTDISABLE */
+
+#else /* !CONFIG_PPC64 */
+
+static inline unsigned long raw_mangle_irq_bits(int virt, unsigned long real)
+{
+	/* Merge virtual and real interrupt mask bits into a single
+	   32bit word. */
+	return (real & ~(1 << 31)) | ((virt != 0) << 31);
+}
+
+static inline int raw_demangle_irq_bits(unsigned long *x)
+{
+	int virt = (*x & (1 << 31)) != 0;
+	*x &= ~(1L << 31);
+	return virt;
+}
+
+#define local_save_flags_hw(x)			((x) = mfmsr())
+#define local_test_iflag_hw(x)			((x) & MSR_EE)
+#define irqs_disabled_hw()			((mfmsr() & MSR_EE) == 0)
+#define local_irq_save_hw_notrace(x)		local_irq_save_ptr_hw(&(x))
+#define raw_irqs_disabled_flags(x)		(!local_test_iflag_hw(x))
 
 #if defined(CONFIG_BOOKE)
-#define SET_MSR_EE(x)	mtmsr(x)
-#define raw_local_irq_restore(flags)	__asm__ __volatile__("wrtee %0" : : "r" (flags) : "memory")
+#define local_irq_restore_hw_notrace(x)	\
+	__asm__ __volatile__("wrtee %0" : : "r" (x) : "memory")
 #else
 #define SET_MSR_EE(x)	mtmsr(x)
-#define raw_local_irq_restore(flags)	mtmsr(flags)
+#define local_irq_restore_hw_notrace(x)	mtmsr(x)
 #endif
 
-static inline void raw_local_irq_disable(void)
+static inline void local_irq_disable_hw_notrace(void)
 {
 #ifdef CONFIG_BOOKE
 	__asm__ __volatile__("wrteei 0": : :"memory");
@@ -91,7 +249,7 @@
 #endif
 }
 
-static inline void raw_local_irq_enable(void)
+static inline void local_irq_enable_hw_notrace(void)
 {
 #ifdef CONFIG_BOOKE
 	__asm__ __volatile__("wrteei 1": : :"memory");
@@ -103,11 +261,11 @@
 #endif
 }
 
-static inline void raw_local_irq_save_ptr(unsigned long *flags)
+static inline void local_irq_save_ptr_hw(unsigned long *x)
 {
 	unsigned long msr;
 	msr = mfmsr();
-	*flags = msr;
+	*x = msr;
 #ifdef CONFIG_BOOKE
 	__asm__ __volatile__("wrteei 0": : :"memory");
 #else
@@ -115,10 +273,107 @@
 #endif
 }
 
-#define raw_local_save_flags(flags)	((flags) = mfmsr())
-#define raw_local_irq_save(flags)	raw_local_irq_save_ptr(&flags)
-#define raw_irqs_disabled()		((mfmsr() & MSR_EE) == 0)
-#define raw_irqs_disabled_flags(flags)	(((flags) & MSR_EE) == 0)
+#ifdef CONFIG_IPIPE
+
+#include <linux/ipipe_base.h>
+#include <linux/ipipe_trace.h>
+
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+
+static inline void local_irq_disable_hw(void)
+{
+	if (!irqs_disabled_hw()) {
+		local_irq_disable_hw_notrace();
+		ipipe_trace_begin(0x80000000);
+	}
+}
+
+static inline void local_irq_enable_hw(void)
+{
+	if (irqs_disabled_hw()) {
+		ipipe_trace_end(0x80000000);
+		local_irq_enable_hw_notrace();
+	}
+}
+
+#define local_irq_save_hw(x) \
+do {						       \
+	local_irq_save_ptr_hw(&(x));		       \
+	if (local_test_iflag_hw(x))		       \
+		ipipe_trace_begin(0x80000001);	       \
+} while(0)
+
+static inline void local_irq_restore_hw(unsigned long x)
+{
+	if (local_test_iflag_hw(x))
+		ipipe_trace_end(0x80000001);
+
+	local_irq_restore_hw_notrace(x);
+}
+
+#else /* !CONFIG_IPIPE_TRACE_IRQSOFF */
+
+#define local_irq_disable_hw    local_irq_disable_hw_notrace
+#define local_irq_enable_hw     local_irq_enable_hw_notrace
+#define local_irq_save_hw       local_irq_save_hw_notrace
+#define local_irq_restore_hw    local_irq_restore_hw_notrace
+
+#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */
+
+static inline void raw_local_irq_disable(void)
+{
+	ipipe_check_context(ipipe_root_domain);
+	__ipipe_stall_root();
+	barrier();
+}
+
+static inline void raw_local_irq_enable(void)
+{
+	barrier();
+	__ipipe_unstall_root();
+}
+
+static inline void raw_local_irq_save_ptr(unsigned long *x)
+{
+	*x = (!__ipipe_test_and_stall_root()) << MSR_EE_LG;
+	barrier();
+}
+
+static inline void raw_local_irq_restore(unsigned long x)
+{
+	barrier();
+	__ipipe_restore_root(!(x & MSR_EE));
+}
+
+#define raw_local_save_flags(x)			\
+do {						\
+	(x) = (!__ipipe_test_root()) << MSR_EE_LG;	\
+	barrier();				\
+} while(0)
+
+#define raw_local_irq_save(x)			\
+do {						\
+	ipipe_check_context(ipipe_root_domain);	\
+	raw_local_irq_save_ptr(&(x));		\
+} while(0)
+
+#define raw_irqs_disabled()		__ipipe_test_root()
+
+#else /* !CONFIG_IPIPE */
+
+#define local_irq_disable_hw    	local_irq_disable_hw_notrace
+#define local_irq_enable_hw     	local_irq_enable_hw_notrace
+#define local_irq_save_hw       	local_irq_save_hw_notrace
+#define local_irq_restore_hw    	local_irq_restore_hw_notrace
+#define raw_local_irq_restore(x)	local_irq_restore_hw(x)
+#define raw_local_irq_disable()	local_irq_disable_hw()
+#define raw_local_irq_enable()	local_irq_enable_hw()
+#define raw_local_irq_save_ptr(x)	local_irq_save_ptr_hw(x)
+#define raw_irqs_disabled()		irqs_disabled_hw()
+#define raw_local_save_flags(x)	local_save_flags_hw(x)
+#define raw_local_irq_save(x)  	local_irq_save_hw(x)
+
+#endif /* !CONFIG_IPIPE */
 
 #define hard_irq_disable()		raw_local_irq_disable()
 
diff -urN source_powerpc_none/arch/powerpc/include/asm/ipipe.h source_powerpc_none.ipipe/arch/powerpc/include/asm/ipipe.h
--- source_powerpc_none/arch/powerpc/include/asm/ipipe.h	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/ipipe.h	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,271 @@
+/*
+ *   include/asm-powerpc/ipipe.h
+ *
+ *   I-pipe 32/64bit merge - Copyright (C) 2007 Philippe Gerum.
+ *   I-pipe PA6T support - Copyright (C) 2007 Philippe Gerum.
+ *   I-pipe 64-bit PowerPC port - Copyright (C) 2005 Heikki Lindholm.
+ *   I-pipe PowerPC support - Copyright (C) 2002-2005 Philippe Gerum.
+ *
+ *   This program is free software; you can redistribute it and/or modify
+ *   it under the terms of the GNU General Public License as published by
+ *   the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139,
+ *   USA; either version 2 of the License, or (at your option) any later
+ *   version.
+ *
+ *   This program is distributed in the hope that it will be useful,
+ *   but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *   GNU General Public License for more details.
+ *
+ *   You should have received a copy of the GNU General Public License
+ *   along with this program; if not, write to the Free Software
+ *   Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ */
+
+#ifndef __ASM_POWERPC_IPIPE_H
+#define __ASM_POWERPC_IPIPE_H
+
+#ifdef CONFIG_IPIPE
+
+#include <asm/ptrace.h>
+#include <asm/hw_irq.h>
+#include <asm/irq.h>
+#include <asm/bitops.h>
+#include <asm/time.h>
+#include <linux/ipipe_percpu.h>
+#include <linux/list.h>
+#include <linux/cpumask.h>
+#include <linux/cache.h>
+#include <linux/threads.h>
+
+#ifdef CONFIG_PPC64
+#ifdef CONFIG_PPC_ISERIES
+#error "I-pipe: IBM I-series not supported, sorry"
+#endif
+#include <asm/paca.h>
+#endif
+
+#define IPIPE_ARCH_STRING	"2.7-03"
+#define IPIPE_MAJOR_NUMBER	2
+#define IPIPE_MINOR_NUMBER	7
+#define IPIPE_PATCH_NUMBER	3
+
+#ifdef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH
+
+#define prepare_arch_switch(next)			\
+	do {						\
+		local_irq_enable_hw();			\
+		ipipe_schedule_notify(current ,next);	\
+	} while(0)
+
+#define task_hijacked(p)						\
+	({								\
+		unsigned long __flags__;				\
+		int __x__;						\
+		local_irq_save_hw_smp(__flags__);			\
+		__x__ = __ipipe_root_domain_p;				\
+		__clear_bit(IPIPE_SYNC_FLAG, &ipipe_root_cpudom_var(status)); \
+		local_irq_restore_hw_smp(__flags__);			\
+		!__x__;							\
+	})
+
+#else /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */
+
+#define prepare_arch_switch(next)			\
+	do {						\
+		ipipe_schedule_notify(current ,next);	\
+		local_irq_disable_hw();			\
+	} while(0)
+
+#define task_hijacked(p)						\
+	({								\
+		int __x__ = __ipipe_root_domain_p;			\
+		__clear_bit(IPIPE_SYNC_FLAG, &ipipe_root_cpudom_var(status)); \
+		if (__x__) local_irq_enable_hw(); !__x__;		\
+	})
+
+#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */
+
+struct ipipe_domain;
+
+struct ipipe_sysinfo {
+
+	int ncpus;			/* Number of CPUs on board */
+	u64 cpufreq;			/* CPU frequency (in Hz) */
+
+	/* Arch-dependent block */
+
+	struct {
+		unsigned tmirq;		/* Decrementer virtual IRQ */
+		u64 tmfreq;		/* Timebase frequency */
+	} archdep;
+};
+
+#ifdef CONFIG_DEBUGGER
+extern cpumask_t __ipipe_dbrk_pending;
+#endif
+
+#ifdef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH
+struct mm;
+DECLARE_PER_CPU(struct mm_struct *, ipipe_active_mm);
+#define ipipe_mm_switch_protect(flags)					\
+	do {								\
+		preempt_disable();					\
+		per_cpu(ipipe_active_mm, smp_processor_id()) = NULL;	\
+		barrier();						\
+		(void)(flags);						\
+	} while(0)
+#define ipipe_mm_switch_unprotect(flags)				\
+	do {								\
+		preempt_enable();					\
+		(void)(flags);						\
+	} while(0)
+#else
+#define ipipe_mm_switch_protect(flags)		local_irq_save_hw_cond(flags)
+#define ipipe_mm_switch_unprotect(flags)	local_irq_restore_hw_cond(flags)
+#endif
+
+#define ipipe_cpu_freq()	ppc_tb_freq
+#ifdef CONFIG_PPC64
+#define ipipe_read_tsc(t)	(t = mftb())
+#define ipipe_tsc2ns(t)		(((t) * 1000UL) / (ipipe_cpu_freq() / 1000000UL))
+#define ipipe_tsc2us(t)		((t) / (ipipe_cpu_freq() / 1000000UL))
+#else
+#define ipipe_read_tsc(t)					\
+	({							\
+		unsigned long __tbu;				\
+		__asm__ __volatile__ ("1: mftbu %0\n"		\
+				      "mftb %1\n"		\
+				      "mftbu %2\n"		\
+				      "cmpw %2,%0\n"		\
+				      "bne- 1b\n"		\
+				      :"=r" (((unsigned long *)&t)[0]),	\
+				       "=r" (((unsigned long *)&t)[1]),	\
+				       "=r" (__tbu));			\
+		t;							\
+	})
+#define ipipe_tsc2ns(t)		((((unsigned long)(t)) * 1000) / (ipipe_cpu_freq() / 1000000))
+#define ipipe_tsc2us(t)						\
+	({							\
+		unsigned long long delta = (t);			\
+		do_div(delta, ipipe_cpu_freq()/1000000+1);	\
+		(unsigned long)delta;				\
+	})
+#endif
+#define __ipipe_read_timebase()					\
+	({							\
+ 	unsigned long long t;					\
+ 	ipipe_read_tsc(t);					\
+ 	t;							\
+ 	})
+
+/* Private interface -- Internal use only */
+
+#define __ipipe_check_platform()		do { } while(0)
+#define __ipipe_enable_irq(irq)			enable_irq(irq)
+#define __ipipe_disable_irq(irq)		disable_irq(irq)
+#define __ipipe_disable_irqdesc(ipd, irq)	do { } while(0)
+
+void __ipipe_enable_irqdesc(struct ipipe_domain *ipd, unsigned irq);
+
+void __ipipe_init_platform(void);
+
+void __ipipe_enable_pipeline(void);
+
+void __ipipe_end_irq(unsigned irq);
+
+static inline int __ipipe_check_tickdev(const char *devname)
+{
+	return 1;
+}
+
+#ifdef CONFIG_SMP
+struct ipipe_ipi_struct {
+	volatile unsigned long value;
+} ____cacheline_aligned;
+
+void __ipipe_hook_critical_ipi(struct ipipe_domain *ipd);
+
+void __ipipe_register_ipi(unsigned int irq);
+#else
+#define __ipipe_hook_critical_ipi(ipd)	do { } while(0)
+#endif /* CONFIG_SMP */
+
+DECLARE_PER_CPU(struct pt_regs, __ipipe_tick_regs);
+
+void __ipipe_handle_irq(int irq, struct pt_regs *regs);
+
+static inline void ipipe_handle_chained_irq(unsigned int irq)
+{
+	struct pt_regs regs;	/* dummy */
+
+	ipipe_trace_irq_entry(irq);
+	__ipipe_handle_irq(irq, &regs);
+	ipipe_trace_irq_exit(irq);
+}
+
+struct irq_desc;
+void __ipipe_ack_level_irq(unsigned irq, struct irq_desc *desc);
+void __ipipe_end_level_irq(unsigned irq, struct irq_desc *desc);
+void __ipipe_ack_edge_irq(unsigned irq, struct irq_desc *desc);
+void __ipipe_end_edge_irq(unsigned irq, struct irq_desc *desc);
+
+void __ipipe_serial_debug(const char *fmt, ...);
+
+#define __ipipe_tick_irq	IPIPE_TIMER_VIRQ
+
+static inline unsigned long __ipipe_ffnz(unsigned long ul)
+{
+#ifdef CONFIG_PPC64
+	__asm__ __volatile__("cntlzd %0, %1":"=r"(ul):"r"(ul & (-ul)));
+	return 63 - ul;
+#else
+	__asm__ __volatile__("cntlzw %0, %1":"=r"(ul):"r"(ul & (-ul)));
+	return 31 - ul;
+#endif
+}
+
+/*
+ * When running handlers, enable hw interrupts for all domains but the
+ * one heading the pipeline, so that IRQs can never be significantly
+ * deferred for the latter.
+ */
+#define __ipipe_run_isr(ipd, irq)					\
+do {									\
+	if (!__ipipe_pipeline_head_p(ipd))				\
+		local_irq_enable_hw();					\
+	if (ipd == ipipe_root_domain)					\
+		if (likely(!ipipe_virtual_irq_p(irq)))			\
+			ipd->irqs[irq].handler(irq, NULL);		\
+		else {							\
+			irq_enter();					\
+			ipd->irqs[irq].handler(irq, ipd->irqs[irq].cookie);\
+			irq_exit();					\
+		}							\
+	else {								\
+		__clear_bit(IPIPE_SYNC_FLAG, &ipipe_cpudom_var(ipd, status)); \
+		ipd->irqs[irq].handler(irq, ipd->irqs[irq].cookie);	\
+		__set_bit(IPIPE_SYNC_FLAG, &ipipe_cpudom_var(ipd, status)); \
+	}								\
+	local_irq_disable_hw();						\
+} while(0)
+
+#define __ipipe_syscall_watched_p(p, sc)	\
+	(((p)->flags & PF_EVNOTIFY) || (unsigned long)sc >= NR_syscalls)
+
+#define __ipipe_root_tick_p(regs)	((regs)->msr & MSR_EE)
+
+#else /* !CONFIG_IPIPE */
+
+#define task_hijacked(p)	0
+
+#define ipipe_handle_chained_irq(irq)	generic_handle_irq(irq)
+
+#define ipipe_mm_switch_protect(flags)		do { (void)(flags); } while(0)
+#define ipipe_mm_switch_unprotect(flags)	do { (void)(flags); } while(0)
+
+#endif /* CONFIG_IPIPE */
+
+#define ipipe_update_tick_evtdev(evtdev)	do { } while (0)
+
+#endif /* !__ASM_POWERPC_IPIPE_H */
diff -urN source_powerpc_none/arch/powerpc/include/asm/ipipe_base.h source_powerpc_none.ipipe/arch/powerpc/include/asm/ipipe_base.h
--- source_powerpc_none/arch/powerpc/include/asm/ipipe_base.h	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/ipipe_base.h	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,154 @@
+/* -*- linux-c -*-
+ * include/asm-powerpc/ipipe_base.h
+ *
+ * Copyright (C) 2007 Philippe Gerum.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139,
+ * USA; either version 2 of the License, or (at your option) any later
+ * version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ */
+
+#ifndef __ASM_POWERPC_IPIPE_BASE_H
+#define __ASM_POWERPC_IPIPE_BASE_H
+
+#ifdef CONFIG_IPIPE
+
+#define IPIPE_NR_XIRQS		NR_IRQS
+#ifdef CONFIG_PPC64
+#define IPIPE_IRQ_ISHIFT	6		/* 64-bit arch. */
+#else
+#define IPIPE_IRQ_ISHIFT	5		/* 32-bit arch. */
+#endif
+
+/* 
+ * The first virtual interrupt is reserved for the timer (see
+ * __ipipe_init_platform). 
+ */
+#define IPIPE_TIMER_VIRQ	IPIPE_VIRQ_BASE
+
+#ifdef CONFIG_SMP
+/* 
+ * These are virtual IPI numbers. The OpenPIC supports only 4 IPIs and
+ * all are already used by Linux. The virtualization layer is
+ * implemented by piggybacking the debugger break IPI 0x3,
+ * which is demultiplexed in __ipipe_ipi_demux().
+ */
+/* these are bit numbers in practice */
+#define IPIPE_MSG_CRITICAL_IPI	0
+#define IPIPE_MSG_SERVICE_IPI0	(IPIPE_MSG_CRITICAL_IPI + 1)
+#define IPIPE_MSG_SERVICE_IPI1	(IPIPE_MSG_CRITICAL_IPI + 2)
+#define IPIPE_MSG_SERVICE_IPI2	(IPIPE_MSG_CRITICAL_IPI + 3)
+#define IPIPE_MSG_SERVICE_IPI3	(IPIPE_MSG_CRITICAL_IPI + 4)
+#define IPIPE_MSG_SERVICE_IPI4	(IPIPE_MSG_CRITICAL_IPI + 5)
+
+#define IPIPE_MSG_IPI_MASK	((1UL << IPIPE_MSG_CRITICAL_IPI) |	\
+				 (1UL << IPIPE_MSG_SERVICE_IPI0) |	\
+				 (1UL << IPIPE_MSG_SERVICE_IPI1) |	\
+				 (1UL << IPIPE_MSG_SERVICE_IPI2) |	\
+				 (1UL << IPIPE_MSG_SERVICE_IPI3) |	\
+				 (1UL << IPIPE_MSG_SERVICE_IPI4))
+
+#define IPIPE_CRITICAL_IPI	(IPIPE_VIRQ_BASE + 1)
+#define IPIPE_SERVICE_IPI0	(IPIPE_CRITICAL_IPI + 1)
+#define IPIPE_SERVICE_IPI1	(IPIPE_CRITICAL_IPI + 2)
+#define IPIPE_SERVICE_IPI2	(IPIPE_CRITICAL_IPI + 3)
+#define IPIPE_SERVICE_IPI3	(IPIPE_CRITICAL_IPI + 4)
+#define IPIPE_SERVICE_IPI4	(IPIPE_CRITICAL_IPI + 5)
+
+#define IPIPE_MSG_IPI_OFFSET	(IPIPE_CRITICAL_IPI)
+
+#define ipipe_processor_id()	raw_smp_processor_id()
+#else  /* !CONFIG_SMP */
+#define ipipe_processor_id()	0
+#endif /* CONFIG_SMP */
+
+/* traps */
+#define IPIPE_TRAP_ACCESS	 0	/* Data or instruction access exception */
+#define IPIPE_TRAP_ALIGNMENT	 1	/* Alignment exception */
+#define IPIPE_TRAP_ALTUNAVAIL	 2	/* Altivec unavailable */
+#define IPIPE_TRAP_PCE		 3	/* Program check exception */
+#define IPIPE_TRAP_MCE		 4	/* Machine check exception */
+#define IPIPE_TRAP_UNKNOWN	 5	/* Unknown exception */
+#define IPIPE_TRAP_IABR		 6	/* Instruction breakpoint */
+#define IPIPE_TRAP_RM		 7	/* Run mode exception */
+#define IPIPE_TRAP_SSTEP	 8	/* Single-step exception */
+#define IPIPE_TRAP_NREC		 9	/* Non-recoverable exception */
+#define IPIPE_TRAP_SOFTEMU	10	/* Software emulation */
+#define IPIPE_TRAP_DEBUG	11	/* Debug exception */
+#define IPIPE_TRAP_SPE		12	/* SPE exception */
+#define IPIPE_TRAP_ALTASSIST	13	/* Altivec assist exception */
+#define IPIPE_TRAP_CACHE	14	/* Cache-locking exception (FSL) */
+#define IPIPE_TRAP_KFPUNAVAIL	15	/* FP unavailable exception */
+#define IPIPE_NR_FAULTS		16
+/* Pseudo-vectors used for kernel events */
+#define IPIPE_FIRST_EVENT		IPIPE_NR_FAULTS
+#define IPIPE_EVENT_SYSCALL		(IPIPE_FIRST_EVENT)
+#define IPIPE_EVENT_SCHEDULE		(IPIPE_FIRST_EVENT + 1)
+#define IPIPE_EVENT_SIGWAKE		(IPIPE_FIRST_EVENT + 2)
+#define IPIPE_EVENT_SETSCHED		(IPIPE_FIRST_EVENT + 3)
+#define IPIPE_EVENT_INIT		(IPIPE_FIRST_EVENT + 4)
+#define IPIPE_EVENT_EXIT		(IPIPE_FIRST_EVENT + 5)
+#define IPIPE_EVENT_CLEANUP		(IPIPE_FIRST_EVENT + 6)
+#define IPIPE_LAST_EVENT		IPIPE_EVENT_CLEANUP
+#define IPIPE_NR_EVENTS			(IPIPE_LAST_EVENT + 1)
+
+#ifndef __ASSEMBLY__
+
+#ifdef CONFIG_SMP
+
+void __ipipe_stall_root(void);
+
+unsigned long __ipipe_test_and_stall_root(void);
+
+unsigned long __ipipe_test_root(void);
+
+#else /* !CONFIG_SMP */
+
+#include <linux/bitops.h>
+
+#if __GNUC__ >= 4
+/* Alias to ipipe_root_cpudom_var(status) */
+extern unsigned long __ipipe_root_status;
+#else
+extern unsigned long *const __ipipe_root_status_addr;
+#define __ipipe_root_status	(*__ipipe_root_status_addr)
+#endif
+
+static __inline__ void __ipipe_stall_root(void)
+{
+	volatile unsigned long *p = &__ipipe_root_status;
+	set_bit(0, p);
+}
+
+static __inline__ unsigned long __ipipe_test_and_stall_root(void)
+{
+	volatile unsigned long *p = &__ipipe_root_status;
+	return test_and_set_bit(0, p);
+}
+
+static __inline__ unsigned long __ipipe_test_root(void)
+{
+	volatile unsigned long *p = &__ipipe_root_status;
+	return test_bit(0, p);
+}
+
+#endif /* !CONFIG_SMP */
+
+#endif /* !__ASSEMBLY__ */
+
+#define __IPIPE_FEATURE_PREEMPTIBLE_SWITCH  1
+
+#endif /* CONFIG_IPIPE */
+
+#endif	/* !__ASM_POWERPC_IPIPE_BASE_H */
diff -urN source_powerpc_none/arch/powerpc/include/asm/irqflags.h source_powerpc_none.ipipe/arch/powerpc/include/asm/irqflags.h
--- source_powerpc_none/arch/powerpc/include/asm/irqflags.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/irqflags.h	2009-12-22 12:44:08.000000000 -0500
@@ -10,7 +10,17 @@
  */
 #include <asm/hw_irq.h>
 
-#else
+#elif CONFIG_IPIPE
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+#define TRACE_DISABLE_INTS		bl	__ipipe_trace_irqson
+#define TRACE_ENABLE_INTS		bl	__ipipe_trace_irqson
+#define TRACE_DISABLE_INTS_REALLY	bl	__ipipe_trace_irqsoff
+#else	
+#define TRACE_DISABLE_INTS
+#define TRACE_ENABLE_INTS
+#define TRACE_DISABLE_INTS_REALLY
+#endif
+#else /* !CONFIG_IPIPE */
 #ifdef CONFIG_TRACE_IRQFLAGS
 /*
  * Most of the CPU's IRQ-state tracing is done from assembly code; we
diff -urN source_powerpc_none/arch/powerpc/include/asm/mmu_context.h source_powerpc_none.ipipe/arch/powerpc/include/asm/mmu_context.h
--- source_powerpc_none/arch/powerpc/include/asm/mmu_context.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/mmu_context.h	2009-12-22 12:44:08.000000000 -0500
@@ -32,11 +32,17 @@
  * switch_mm is the entry point called from the architecture independent
  * code in kernel/sched.c
  */
-static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,
-			     struct task_struct *tsk)
+static inline void __switch_mm(struct mm_struct *prev, struct mm_struct *next,
+			       struct task_struct *tsk)
 {
+	int cpu = smp_processor_id();
+
+#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && \
+	!defined(CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH)
+	WARN_ON_ONCE(!irqs_disabled_hw());
+#endif
 	/* Mark this context has been used on the new CPU */
-	cpumask_set_cpu(smp_processor_id(), mm_cpumask(next));
+	cpumask_set_cpu(cpu, mm_cpumask(next));
 
 	/* 32-bit keeps track of the current PGDIR in the thread struct */
 #ifdef CONFIG_PPC32
@@ -62,6 +68,28 @@
 	/* The actual HW switching method differs between the various
 	 * sub architectures.
 	 */
+#ifdef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH
+#ifdef CONFIG_PPC_STD_MMU_64
+	do {
+		per_cpu(ipipe_active_mm, cpu) = NULL; /* mm state is undefined. */
+		barrier();
+		if (cpu_has_feature(CPU_FTR_SLB))
+			switch_slb(tsk, next);
+		else
+			switch_stab(tsk, next);
+		barrier();
+		per_cpu(ipipe_active_mm, cpu) = next;
+	} while (test_and_clear_thread_flag(TIF_MMSWITCH_INT));
+#else
+	do {
+		per_cpu(ipipe_active_mm, cpu) = NULL; /* mm state is undefined. */
+		barrier();
+		switch_mmu_context(prev, next);
+		barrier();
+		per_cpu(ipipe_active_mm, cpu) = next;
+	} while (test_and_clear_thread_flag(TIF_MMSWITCH_INT));
+#endif
+#else /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */
 #ifdef CONFIG_PPC_STD_MMU_64
 	if (cpu_has_feature(CPU_FTR_SLB))
 		switch_slb(tsk, next);
@@ -71,7 +99,21 @@
 	/* Out of line for now */
 	switch_mmu_context(prev, next);
 #endif
+#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */
+}
 
+static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next,
+			     struct task_struct *tsk)
+{
+#ifndef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH
+	unsigned long flags;
+	local_irq_save_hw(flags);
+#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */
+	__switch_mm(prev, next, tsk);
+#ifndef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH
+	local_irq_restore_hw(flags);
+#endif /* !CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH */
+	return;
 }
 
 #define deactivate_mm(tsk,mm)	do { } while (0)
@@ -85,7 +127,7 @@
 	unsigned long flags;
 
 	local_irq_save(flags);
-	switch_mm(prev, next, current);
+	__switch_mm(prev, next, current);
 	local_irq_restore(flags);
 }
 
diff -urN source_powerpc_none/arch/powerpc/include/asm/mpic.h source_powerpc_none.ipipe/arch/powerpc/include/asm/mpic.h
--- source_powerpc_none/arch/powerpc/include/asm/mpic.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/mpic.h	2009-12-22 12:44:08.000000000 -0500
@@ -289,7 +289,7 @@
 #ifdef CONFIG_MPIC_U3_HT_IRQS
 	/* The fixup table */
 	struct mpic_irq_fixup	*fixups;
-	spinlock_t		fixup_lock;
+	ipipe_spinlock_t	fixup_lock;
 #endif
 
 	/* Register access method */
diff -urN source_powerpc_none/arch/powerpc/include/asm/paca.h source_powerpc_none.ipipe/arch/powerpc/include/asm/paca.h
--- source_powerpc_none/arch/powerpc/include/asm/paca.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/paca.h	2009-12-22 12:44:08.000000000 -0500
@@ -119,8 +119,12 @@
 	u64 saved_r1;			/* r1 save for RTAS calls */
 	u64 saved_msr;			/* MSR saved here by enter_rtas */
 	u16 trap_save;			/* Used when bad stack is encountered */
+#ifdef CONFIG_SOFTDISABLE
 	u8 soft_enabled;		/* irq soft-enable flag */
 	u8 hard_enabled;		/* set if irqs are enabled in MSR */
+#elif CONFIG_IPIPE
+	u64 root_percpu;		/* Address of per_cpu data for the root domain */
+#endif
 	u8 io_sync;			/* writel() needs spin_unlock sync */
 	u8 perf_event_pending;		/* PM interrupt while soft-disabled */
 
diff -urN source_powerpc_none/arch/powerpc/include/asm/ptrace.h source_powerpc_none.ipipe/arch/powerpc/include/asm/ptrace.h
--- source_powerpc_none/arch/powerpc/include/asm/ptrace.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/ptrace.h	2009-12-22 12:44:08.000000000 -0500
@@ -36,7 +36,7 @@
 	unsigned long xer;
 	unsigned long ccr;
 #ifdef __powerpc64__
-	unsigned long softe;		/* Soft enabled/disabled */
+	unsigned long softe;		/* Soft enabled/disabled (CONFIG_SOFTDISABLE || CONFIG_IPIPE) */
 #else
 	unsigned long mq;		/* 601 only (not used at present) */
 					/* Used on APUS to hold IPL value. */
diff -urN source_powerpc_none/arch/powerpc/include/asm/qe_ic.h source_powerpc_none.ipipe/arch/powerpc/include/asm/qe_ic.h
--- source_powerpc_none/arch/powerpc/include/asm/qe_ic.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/qe_ic.h	2009-12-22 12:44:08.000000000 -0500
@@ -74,6 +74,13 @@
 { return 0; }
 #endif /* CONFIG_QUICC_ENGINE */
 
+#ifdef CONFIG_IPIPE
+void __ipipe_qe_ic_cascade_irq(struct qe_ic *qe_ic, unsigned int virq);
+#define qe_ic_cascade_irq(qe_ic, irq)	__ipipe_qe_ic_cascade_irq(qe_ic, irq)
+#else
+#define qe_ic_cascade_irq(qe_ic, irq)	generic_handle_irq(irq)
+#endif
+
 void qe_ic_set_highest_priority(unsigned int virq, int high);
 int qe_ic_set_priority(unsigned int virq, unsigned int priority);
 int qe_ic_set_high_priority(unsigned int virq, unsigned int priority, int high);
@@ -85,7 +92,7 @@
 	unsigned int cascade_irq = qe_ic_get_low_irq(qe_ic);
 
 	if (cascade_irq != NO_IRQ)
-		generic_handle_irq(cascade_irq);
+		qe_ic_cascade_irq(qe_ic, cascade_irq);
 }
 
 static inline void qe_ic_cascade_high_ipic(unsigned int irq,
@@ -95,7 +102,7 @@
 	unsigned int cascade_irq = qe_ic_get_high_irq(qe_ic);
 
 	if (cascade_irq != NO_IRQ)
-		generic_handle_irq(cascade_irq);
+		qe_ic_cascade_irq(qe_ic, cascade_irq);
 }
 
 static inline void qe_ic_cascade_low_mpic(unsigned int irq,
@@ -105,7 +112,7 @@
 	unsigned int cascade_irq = qe_ic_get_low_irq(qe_ic);
 
 	if (cascade_irq != NO_IRQ)
-		generic_handle_irq(cascade_irq);
+		qe_ic_cascade_irq(qe_ic, cascade_irq);
 
 	desc->chip->eoi(irq);
 }
@@ -117,7 +124,7 @@
 	unsigned int cascade_irq = qe_ic_get_high_irq(qe_ic);
 
 	if (cascade_irq != NO_IRQ)
-		generic_handle_irq(cascade_irq);
+		qe_ic_cascade_irq(qe_ic, cascade_irq);
 
 	desc->chip->eoi(irq);
 }
@@ -133,7 +140,7 @@
 		cascade_irq = qe_ic_get_low_irq(qe_ic);
 
 	if (cascade_irq != NO_IRQ)
-		generic_handle_irq(cascade_irq);
+		qe_ic_cascade_irq(qe_ic, cascade_irq);
 
 	desc->chip->eoi(irq);
 }
diff -urN source_powerpc_none/arch/powerpc/include/asm/reg.h source_powerpc_none.ipipe/arch/powerpc/include/asm/reg.h
--- source_powerpc_none/arch/powerpc/include/asm/reg.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/reg.h	2009-12-22 12:44:08.000000000 -0500
@@ -928,7 +928,7 @@
 
 #define proc_trap()	asm volatile("trap")
 
-#ifdef CONFIG_PPC64
+#ifdef CONFIG_RUNLATCH
 
 extern void ppc64_runlatch_on(void);
 extern void ppc64_runlatch_off(void);
diff -urN source_powerpc_none/arch/powerpc/include/asm/smp.h source_powerpc_none.ipipe/arch/powerpc/include/asm/smp.h
--- source_powerpc_none/arch/powerpc/include/asm/smp.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/smp.h	2009-12-22 12:44:08.000000000 -0500
@@ -54,8 +54,14 @@
 /* 32-bit */
 extern int smp_hw_index[];
 
+#ifdef CONFIG_IPIPE
+extern int smp_logical_index[];
+#define raw_smp_processor_id()		(smp_logical_index[mfspr(SPRN_PIR)])
+#define hard_smp_processor_id() 	(smp_hw_index[raw_smp_processor_id()])
+#else
 #define raw_smp_processor_id()	(current_thread_info()->cpu)
 #define hard_smp_processor_id() 	(smp_hw_index[smp_processor_id()])
+#endif
 
 static inline int get_hard_smp_processor_id(int cpu)
 {
@@ -65,6 +71,10 @@
 static inline void set_hard_smp_processor_id(int cpu, int phys)
 {
 	smp_hw_index[cpu] = phys;
+#ifdef CONFIG_IPIPE
+	BUG_ON(phys >= NR_CPUS);
+	smp_logical_index[phys] = cpu;
+#endif
 }
 #endif
 
@@ -80,6 +90,7 @@
 #define PPC_MSG_RESCHEDULE      1
 #define PPC_MSG_CALL_FUNC_SINGLE	2
 #define PPC_MSG_DEBUGGER_BREAK  3
+#define PPC_MSG_IPIPE_DEMUX     PPC_MSG_DEBUGGER_BREAK
 
 /*
  * irq controllers that have dedicated ipis per message and don't
diff -urN source_powerpc_none/arch/powerpc/include/asm/thread_info.h source_powerpc_none.ipipe/arch/powerpc/include/asm/thread_info.h
--- source_powerpc_none/arch/powerpc/include/asm/thread_info.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/thread_info.h	2009-12-22 12:44:08.000000000 -0500
@@ -112,6 +112,7 @@
 #define TIF_FREEZE		14	/* Freezing for suspend */
 #define TIF_RUNLATCH		15	/* Is the runlatch enabled? */
 #define TIF_ABI_PENDING		16	/* 32/64 bit switch needed */
+#define TIF_MMSWITCH_INT	20	/* MMU context switch interrupted */
 
 /* as above, but as bit values */
 #define _TIF_SYSCALL_TRACE	(1<<TIF_SYSCALL_TRACE)
@@ -130,6 +131,7 @@
 #define _TIF_FREEZE		(1<<TIF_FREEZE)
 #define _TIF_RUNLATCH		(1<<TIF_RUNLATCH)
 #define _TIF_ABI_PENDING	(1<<TIF_ABI_PENDING)
+#define _TIF_MMSWITCH_INT	(1<<TIF_MMSWITCH_INT)
 #define _TIF_SYSCALL_T_OR_A	(_TIF_SYSCALL_TRACE|_TIF_SYSCALL_AUDIT|_TIF_SECCOMP)
 
 #define _TIF_USER_WORK_MASK	(_TIF_SIGPENDING | _TIF_NEED_RESCHED | \
diff -urN source_powerpc_none/arch/powerpc/include/asm/time.h source_powerpc_none.ipipe/arch/powerpc/include/asm/time.h
--- source_powerpc_none/arch/powerpc/include/asm/time.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/include/asm/time.h	2009-12-22 12:44:08.000000000 -0500
@@ -30,6 +30,7 @@
 extern unsigned long tb_ticks_per_sec;
 extern u64 tb_to_xs;
 extern unsigned      tb_to_us;
+DECLARE_PER_CPU(int, disarm_decr);
 
 struct rtc_time;
 extern void to_tm(int tim, struct rtc_time * tm);
diff -urN source_powerpc_none/arch/powerpc/kernel/Makefile source_powerpc_none.ipipe/arch/powerpc/kernel/Makefile
--- source_powerpc_none/arch/powerpc/kernel/Makefile	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/Makefile	2009-12-22 12:44:08.000000000 -0500
@@ -39,6 +39,7 @@
 obj-$(CONFIG_PPC_BOOK3E_64)	+= exceptions-64e.o
 obj-$(CONFIG_PPC64)		+= vdso64/
 obj-$(CONFIG_ALTIVEC)		+= vecemu.o
+obj-$(CONFIG_IPIPE)		+= ipipe.o
 obj-$(CONFIG_PPC_970_NAP)	+= idle_power4.o
 obj-$(CONFIG_PPC_OF)		+= of_device.o of_platform.o prom_parse.o
 obj-$(CONFIG_PPC_CLOCK)		+= clock.o
diff -urN source_powerpc_none/arch/powerpc/kernel/asm-offsets.c source_powerpc_none.ipipe/arch/powerpc/kernel/asm-offsets.c
--- source_powerpc_none/arch/powerpc/kernel/asm-offsets.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/asm-offsets.c	2009-12-22 12:44:08.000000000 -0500
@@ -131,8 +131,12 @@
 	DEFINE(PACATOC, offsetof(struct paca_struct, kernel_toc));
 	DEFINE(PACAKBASE, offsetof(struct paca_struct, kernelbase));
 	DEFINE(PACAKMSR, offsetof(struct paca_struct, kernel_msr));
+#ifdef CONFIG_SOFTDISABLE
 	DEFINE(PACASOFTIRQEN, offsetof(struct paca_struct, soft_enabled));
 	DEFINE(PACAHARDIRQEN, offsetof(struct paca_struct, hard_enabled));
+#elif CONFIG_IPIPE
+	DEFINE(PACAROOTPCPU, offsetof(struct paca_struct, root_percpu));
+#endif
 	DEFINE(PACAPERFPEND, offsetof(struct paca_struct, perf_event_pending));
 	DEFINE(PACACONTEXTID, offsetof(struct paca_struct, context.id));
 #ifdef CONFIG_PPC_MM_SLICES
@@ -271,7 +275,9 @@
 	DEFINE(_DEAR, STACK_FRAME_OVERHEAD+offsetof(struct pt_regs, dar));
 	DEFINE(_ESR, STACK_FRAME_OVERHEAD+offsetof(struct pt_regs, dsisr));
 #else /* CONFIG_PPC64 */
+#if defined(CONFIG_SOFTDISABLE) || defined(CONFIG_IPIPE)
 	DEFINE(SOFTE, STACK_FRAME_OVERHEAD+offsetof(struct pt_regs, softe));
+#endif
 
 	/* These _only_ to be used with {PROM,RTAS}_FRAME_SIZE!!! */
 	DEFINE(_SRR0, STACK_FRAME_OVERHEAD+sizeof(struct pt_regs));
diff -urN source_powerpc_none/arch/powerpc/kernel/cputable.c source_powerpc_none.ipipe/arch/powerpc/kernel/cputable.c
--- source_powerpc_none/arch/powerpc/kernel/cputable.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/cputable.c	2009-12-22 12:44:08.000000000 -0500
@@ -1922,7 +1922,7 @@
 #endif /* CONFIG_PPC64 || CONFIG_BOOKE */
 }
 
-struct cpu_spec * __init identify_cpu(unsigned long offset, unsigned int pvr)
+notrace struct cpu_spec * __init identify_cpu(unsigned long offset, unsigned int pvr)
 {
 	struct cpu_spec *s = cpu_specs;
 	int i;
diff -urN source_powerpc_none/arch/powerpc/kernel/entry_32.S source_powerpc_none.ipipe/arch/powerpc/kernel/entry_32.S
--- source_powerpc_none/arch/powerpc/kernel/entry_32.S	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/entry_32.S	2009-12-22 12:44:08.000000000 -0500
@@ -174,8 +174,25 @@
          * check for stack overflow
          */
 	lwz	r9,KSP_LIMIT(r12)
+#ifdef CONFIG_IPIPE
+	/*
+	 * Allow for private kernel-based stacks: those must not cause
+	 * the stack overflow detection to trigger when some activity has
+	 * been preempted over them. We just check if the kernel stack is
+	 * not treading on the memory area ranging from
+	 * &current->thread_info to &current->thread, which is coarser
+	 * than the vanilla implementation, but likely sensitive enough
+	 * to catch overflows soon enough though.
+	 */
+	addi	 r12,r9,THREAD
+	cmplw 0,r1,r9
+	cmplw 1,r1,r12
+	crand 1,1,4
+	bgt-  stack_ovf		/* if r9 < r1 < r9+THREAD */
+#else /* CONFIG_IPIPE */
 	cmplw	r1,r9			/* if r1 <= ksp_limit */
 	ble-	stack_ovf		/* then the kernel stack overflowed */
+#endif /* CONFIG_IPIPE */
 5:
 #if defined(CONFIG_6xx) || defined(CONFIG_E500)
 	rlwinm	r9,r1,0,0,31-THREAD_SHIFT
@@ -286,6 +303,21 @@
 	lwz	r11,_CCR(r1)	/* Clear SO bit in CR */
 	rlwinm	r11,r11,0,4,2
 	stw	r11,_CCR(r1)
+#ifdef CONFIG_IPIPE
+	addi	r3,r1,GPR0
+	bl	__ipipe_syscall_root
+	cmpwi	r3,0
+	lwz	r3,GPR3(r1)
+	lwz	r0,GPR0(r1)
+	lwz	r4,GPR4(r1)
+	lwz	r5,GPR5(r1)
+	lwz	r6,GPR6(r1)
+	lwz	r7,GPR7(r1)
+	lwz	r8,GPR8(r1)
+	lwz	r9,GPR9(r1)
+	bgt	.ipipe_end_syscall
+	blt	ret_from_syscall
+#endif /* CONFIG_IPIPE */
 #ifdef SHOW_SYSCALLS
 	bl	do_show_syscall
 #endif /* SHOW_SYSCALLS */
@@ -402,11 +434,34 @@
 	b	1b
 #endif  /* CONFIG_44x */
 
+#ifdef CONFIG_IPIPE
+.ipipe_end_syscall:
+	LOAD_MSR_KERNEL(r10,MSR_KERNEL)	/* doesn't include MSR_EE */
+	SYNC
+	MTMSRD(r10)
+	b syscall_exit_cont
+#endif /* CONFIG_IPIPE */
+
 66:	li	r3,-ENOSYS
 	b	ret_from_syscall
 
 	.globl	ret_from_fork
 ret_from_fork:
+#ifdef CONFIG_IPIPE
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+	stwu	r1,-4(r1)
+	stw	r3,0(r1)
+	lis	r3,(0x80000000)@h
+	ori	r3,r3,(0x80000000)@l
+	bl	ipipe_trace_end
+	lwz	r3,0(r1)
+	addi	r1,r1,4
+#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */
+	LOAD_MSR_KERNEL(r10,MSR_KERNEL)
+	ori	r10,r10,MSR_EE
+	SYNC
+	MTMSRD(r10)
+#endif /* CONFIG_IPIPE */
 	REST_NVGPRS(r1)
 	bl	schedule_tail
 	li	r3,0
@@ -788,6 +843,12 @@
 	SYNC			/* Some chip revs have problems here... */
 	MTMSRD(r10)		/* disable interrupts */
 
+#ifdef CONFIG_IPIPE
+        bl __ipipe_check_root
+        cmpwi   r3, 0
+	mfmsr	r10		/* this is used later, might be messed */
+        beq- restore
+#endif /* CONFIG_IPIPE */
 	lwz	r3,_MSR(r1)	/* Returning to user mode? */
 	andi.	r0,r3,MSR_PR
 	beq	resume_kernel
@@ -811,6 +872,12 @@
 #ifdef CONFIG_PREEMPT
 	b	restore
 
+#ifdef CONFIG_IPIPE
+#define PREEMPT_SCHEDULE_IRQ	__ipipe_preempt_schedule_irq
+#else
+#define PREEMPT_SCHEDULE_IRQ	preempt_schedule_irq
+#endif
+
 /* N.B. the only way to get here is from the beq following ret_from_except. */
 resume_kernel:
 	/* check current_thread_info->preempt_count */
@@ -830,7 +897,7 @@
 	 */
 	bl	trace_hardirqs_off
 #endif
-1:	bl	preempt_schedule_irq
+1:	bl	PREEMPT_SCHEDULE_IRQ
 	rlwinm	r9,r1,0,0,(31-THREAD_SHIFT)
 	lwz	r3,TI_FLAGS(r9)
 	andi.	r0,r3,_TIF_NEED_RESCHED
@@ -1227,6 +1294,13 @@
 	.space	4
 	.previous
 
+#ifdef CONFIG_IPIPE
+_GLOBAL(__ipipe_ret_from_except)
+        cmpwi   r3, 0
+        bne+ ret_from_except
+        b restore
+#endif /* CONFIG_IPIPE */
+
 /*
  * PROM code for specific machines follows.  Put it
  * here so it's easy to add arch-specific sections later.
diff -urN source_powerpc_none/arch/powerpc/kernel/entry_64.S source_powerpc_none.ipipe/arch/powerpc/kernel/entry_64.S
--- source_powerpc_none/arch/powerpc/kernel/entry_64.S	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/entry_64.S	2009-12-22 12:44:08.000000000 -0500
@@ -105,6 +105,7 @@
 	addi	r9,r1,STACK_FRAME_OVERHEAD
 	ld	r12,_MSR(r1)
 #endif /* CONFIG_TRACE_IRQFLAGS */
+#ifdef CONFIG_SOFTDISABLE
 	li	r10,1
 	stb	r10,PACASOFTIRQEN(r13)
 	stb	r10,PACAHARDIRQEN(r13)
@@ -120,8 +121,25 @@
 2:
 END_FW_FTR_SECTION_IFSET(FW_FEATURE_ISERIES)
 #endif /* CONFIG_PPC_ISERIES */
+#endif /* CONFIG_SOFTDISABLE */
 
 	/* Hard enable interrupts */
+#ifdef CONFIG_IPIPE
+	addi	r3,r1,GPR0
+	bl	.__ipipe_syscall_root
+	cmpwi	r3,0
+	ld	r0,GPR0(r1)
+	ld	r3,GPR3(r1)
+	ld	r4,GPR4(r1)
+	ld	r5,GPR5(r1)
+	ld	r6,GPR6(r1)
+	ld	r7,GPR7(r1)
+	ld	r8,GPR8(r1)
+	ld	r9,GPR9(r1)
+	bgt	ipipe_end_syscall
+	blt	syscall_exit
+	addi	r9,r1,STACK_FRAME_OVERHEAD
+#else	/* !CONFIG_IPIPE */
 #ifdef CONFIG_PPC_BOOK3E
 	wrteei	1
 #else
@@ -129,6 +147,7 @@
 	ori	r11,r11,MSR_EE
 	mtmsrd	r11,1
 #endif /* CONFIG_PPC_BOOK3E */
+#endif /* CONFIG_IPIPE */
 
 #ifdef SHOW_SYSCALLS
 	bl	.do_show_syscall
@@ -188,7 +207,7 @@
 	wrteei	0
 #else
 	mfmsr	r10
-	rldicl	r10,r10,48,1
+	rldicl	r10,r10,48,1		/* clear MSR_EE */
 	rotldi	r10,r10,16
 	mtmsrd	r10,1
 #endif /* CONFIG_PPC_BOOK3E */
@@ -257,6 +276,35 @@
 	ld	r10,TI_FLAGS(r10)
 	b	syscall_dotrace_cont
 
+#ifdef CONFIG_IPIPE
+	.globl  ipipe_end_syscall
+ipipe_end_syscall:
+	mfmsr	r10
+	rldicl	r10,r10,48,1		/* clear MSR_EE - we don't trace this */
+	rotldi	r10,r10,16		/* short IRQs off section, since our caller */
+	mtmsrd	r10,1			/* did branch here with IRQs on. */
+	ld	r5,_CCR(r1)
+	ld	r8,_MSR(r1)
+	ld	r7,_NIP(r1)
+	stdcx.	r0,0,r1			/* to clear pending reservations */
+	andi.	r6,r8,MSR_PR
+	ld	r4,_LINK(r1)
+	beq-	1f			/* only restore r13 if */
+	ld	r13,GPR13(r1)		/* returning to usermode */
+1:	ld	r2,GPR2(r1)
+	li	r12,MSR_RI
+	mfmsr	r10
+	andc	r10,r10,r12
+	mtmsrd	r10,1			/* clear MSR.RI */
+	ld	r1,GPR1(r1)
+	mtlr	r4
+	mtcr	r5
+	mtspr	SPRN_SRR0,r7
+	mtspr	SPRN_SRR1,r8
+	rfid
+	b	.			/* prevent speculative execution */
+#endif /* CONFIG_IPIPE */
+
 syscall_enosys:
 	li	r3,-ENOSYS
 	b	syscall_exit
@@ -296,6 +344,14 @@
 	beq	.ret_from_except_lite
 
 	/* Re-enable interrupts */
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+	bl	.save_nvgprs
+	bl	__ipipe_trace_irqson
+	/* Re-enable interrupts */
+	mfmsr	r10
+	ori	r10,r10,MSR_EE
+	mtmsrd	r10,1
+#else
 #ifdef CONFIG_PPC_BOOK3E
 	wrteei	1
 #else
@@ -305,6 +361,7 @@
 #endif /* CONFIG_PPC_BOOK3E */
 
 	bl	.save_nvgprs
+#endif
 	addi	r3,r1,STACK_FRAME_OVERHEAD
 	bl	.do_syscall_trace_leave
 	b	.ret_from_except
@@ -355,6 +412,18 @@
 	b	syscall_exit
 
 _GLOBAL(ret_from_fork)
+#ifdef CONFIG_IPIPE
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+	stdu	r1,-8(r1)
+	std	r3,0(r1)
+	bl	__ipipe_trace_irqson
+	ld	r3,0(r1)
+	addi	r1,r1,8
+#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */
+	mfmsr	r10
+	ori	r10,r10,MSR_EE
+	mtmsrd	r10,1
+#endif /* CONFIG_IPIPE */
 	bl	.schedule_tail
 	REST_NVGPRS(r1)
 	li	r3,0
@@ -504,6 +573,14 @@
 	blr
 
 	.align	7
+
+#ifdef CONFIG_IPIPE
+_GLOBAL(__ipipe_ret_from_except_lite)
+	cmpwi	r3,0
+	bne+	.ret_from_except_lite /* FIXME:	branching to __ipipe_check_root is useless here */
+	b	restore
+#endif /* CONFIG_IPIPE */
+
 _GLOBAL(ret_from_except)
 	ld	r11,_TRAP(r1)
 	andi.	r0,r11,1
@@ -516,6 +593,9 @@
 	 * can't change between when we test it and when we return
 	 * from the interrupt.
 	 */
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+	bl	__ipipe_trace_irqsoff
+#endif
 #ifdef CONFIG_PPC_BOOK3E
 	wrteei	0
 #else
@@ -525,6 +605,13 @@
 	mtmsrd	r9,1		/* Update machine state */
 #endif /* CONFIG_PPC_BOOK3E */
 
+#ifdef CONFIG_IPIPE
+	bl	.__ipipe_check_root
+	cmpwi	r3,0
+	mfmsr	r10		/* this is used later, might be messed */
+	beq-	restore
+#endif /* CONFIG_IPIPE */
+
 #ifdef CONFIG_PREEMPT
 	clrrdi	r9,r1,THREAD_SHIFT	/* current_thread_info() */
 	li	r0,_TIF_NEED_RESCHED	/* bits to check */
@@ -548,6 +635,7 @@
 #endif
 
 restore:
+#ifdef CONFIG_SOFTDISABLE
 BEGIN_FW_FTR_SECTION
 	ld	r5,SOFTE(r1)
 FW_FTR_SECTION_ELSE
@@ -569,6 +657,22 @@
 	ld	r3,_MSR(r1)
 	rldicl	r4,r3,49,63		/* r0 = (r3 >> 15) & 1 */
 	stb	r4,PACAHARDIRQEN(r13)
+#else	/* !CONFIG_SOFTDISABLE */
+#ifdef CONFIG_IPIPE
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+	ld	r3,_MSR(r1)
+	rldicl	r3,r3,49,63		/* r0 = (r3 >> 15) & 1 */
+	bl	__ipipe_trace_irqsx
+#endif
+	ld	r3,SOFTE(r1)		/* currently hard-disabled, so this is safe */
+	nor	r3,r3,r3
+	ld	r4,PACAROOTPCPU(r13)
+	ld	r5,0(r4)
+	insrdi	r5,r3,1,63
+	std	r5,0(r4)
+	ld	r3,_MSR(r1)
+#endif
+#endif	/* !CONFIG_SOFTDISABLE */
 
 #ifdef CONFIG_PPC_BOOK3E
 	b	.exception_return_book3e
@@ -665,13 +769,29 @@
 	 * the PACA to reflect the fact that they are hard-disabled
 	 * and trace the change
 	 */
+#ifdef CONFIG_SOFTDISABLE
 	li	r0,0
 	stb	r0,PACASOFTIRQEN(r13)
 	stb	r0,PACAHARDIRQEN(r13)
+#else
+#ifdef CONFIG_IPIPE
+	ld      r10,PACAROOTPCPU(r13)
+	ld      r0,0(r10)
+	clrrdi  r0,r0,1
+	std     r0,0(r10)
+#endif /* CONFIG_IPIPE */
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+	bl	__ipipe_trace_irqson
+#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */
+	mfmsr	r10
+#endif /* !CONFIG_SOFTDISABLE */
 	TRACE_DISABLE_INTS
 
 	/* Call the scheduler with soft IRQs off */
 1:	bl	.preempt_schedule_irq
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+	bl	__ipipe_trace_irqsoff
+#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */
 
 	/* Hard-disable interrupts again (and update PACA) */
 #ifdef CONFIG_PPC_BOOK3E
@@ -695,6 +815,13 @@
 user_work:
 #endif /* CONFIG_PREEMPT */
 
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+	bl	__ipipe_trace_irqson
+	clrrdi	r9,r1,THREAD_SHIFT
+	ld	r4,TI_FLAGS(r9)
+	mfmsr	r10
+#endif /* CONFIG_IPIPE_TRACE_IRQSOFF */
+
 	/* Enable interrupts */
 #ifdef CONFIG_PPC_BOOK3E
 	wrteei	1
@@ -758,7 +885,7 @@
 	li	r0,0
 	mtcr	r0
 
-#ifdef CONFIG_BUG	
+#ifdef CONFIG_BUG && CONFIG_SOFTDISABLE
 	/* There is no way it is acceptable to get here with interrupts enabled,
 	 * check it with the asm equivalent of WARN_ON
 	 */
@@ -931,6 +1058,10 @@
 	blr
 
 _GLOBAL(ftrace_caller)
+        LOAD_REG_IMMEDIATE(r3, function_trace_stop)
+        lwz     r3, 0(r3)
+        cmpwi   r3, 0
+        bne     ftrace_stub
 	/* Taken from output of objdump from lib64/glibc */
 	mflr	r3
 	ld	r11, 0(r1)
@@ -958,6 +1089,10 @@
 	blr
 
 _GLOBAL(_mcount)
+        LOAD_REG_IMMEDIATE(r3, function_trace_stop)
+        lwz     r3, 0(r3)
+        cmpwi   r3, 0
+        bne     ftrace_stub
 	/* Taken from output of objdump from lib64/glibc */
 	mflr	r3
 	ld	r11, 0(r1)
diff -urN source_powerpc_none/arch/powerpc/kernel/fpu.S source_powerpc_none.ipipe/arch/powerpc/kernel/fpu.S
--- source_powerpc_none/arch/powerpc/kernel/fpu.S	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/fpu.S	2009-12-22 12:44:08.000000000 -0500
@@ -122,7 +122,17 @@
  * Enables the FPU for use in the kernel on return.
  */
 _GLOBAL(giveup_fpu)
+#ifdef CONFIG_IPIPE
+	mfmsr	r6
+#ifdef CONFIG_PPC64	
+	rldicl	r5,r6,48,1		/* clear MSR_EE */
+	rotldi	r5,r5,16
+#else
+	rlwinm	r5,r6,0,17,15		/* clear MSR_EE */
+#endif	
+#else	
 	mfmsr	r5
+#endif	
 	ori	r5,r5,MSR_FP
 #ifdef CONFIG_VSX
 BEGIN_FTR_SECTION
@@ -135,7 +145,7 @@
 	SYNC_601
 	isync
 	PPC_LCMPI	0,r3,0
-	beqlr-				/* if no previous owner, done */
+	beq-	2f			/* if no previous owner, done */
 	addi	r3,r3,THREAD	        /* want THREAD of task */
 	PPC_LL	r5,PT_REGS(r3)
 	PPC_LCMPI	0,r5,0
@@ -158,6 +168,18 @@
 	LOAD_REG_ADDRBASE(r4,last_task_used_math)
 	PPC_STL	r5,ADDROFF(last_task_used_math)(r4)
 #endif /* CONFIG_SMP */
+2:
+#ifdef CONFIG_IPIPE		/* restore interrupt state */
+	andi.	r6,r6,MSR_EE
+	beqlr
+	mfmsr	r5
+	ori	r5,r5,MSR_EE
+	SYNC_601
+	ISYNC_601
+	MTMSRD(r5)
+	SYNC_601
+	isync
+#endif
 	blr
 
 /*
diff -urN source_powerpc_none/arch/powerpc/kernel/head_32.S source_powerpc_none.ipipe/arch/powerpc/kernel/head_32.S
--- source_powerpc_none/arch/powerpc/kernel/head_32.S	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/head_32.S	2009-12-22 12:44:08.000000000 -0500
@@ -322,6 +322,12 @@
 	EXC_XFER_TEMPLATE(n, hdlr, n, NOCOPY, transfer_to_handler_full,	\
 			  ret_from_except_full)
 
+#ifdef CONFIG_IPIPE
+#define EXC_XFER_IPIPE(n, hdlr)		\
+	EXC_XFER_TEMPLATE(n, hdlr, n+1, NOCOPY, transfer_to_handler, \
+			  __ipipe_ret_from_except)
+#endif /* CONFIG_IPIPE */
+
 #define EXC_XFER_LITE(n, hdlr)		\
 	EXC_XFER_TEMPLATE(n, hdlr, n+1, NOCOPY, transfer_to_handler, \
 			  ret_from_except)
@@ -406,7 +412,11 @@
 	EXC_XFER_EE_LITE(0x400, handle_page_fault)
 
 /* External interrupt */
+#ifdef CONFIG_IPIPE
+	EXCEPTION(0x500, HardwareInterrupt, __ipipe_grab_irq, EXC_XFER_IPIPE)
+#else /* !CONFIG_IPIPE */
 	EXCEPTION(0x500, HardwareInterrupt, do_IRQ, EXC_XFER_LITE)
+#endif /* CONFIG_IPIPE */
 
 /* Alignment exception */
 	. = 0x600
@@ -440,7 +450,11 @@
 	EXC_XFER_EE_LITE(0x800, kernel_fp_unavailable_exception)
 
 /* Decrementer */
+#ifdef CONFIG_IPIPE
+	EXCEPTION(0x900, Decrementer, __ipipe_grab_timer, EXC_XFER_IPIPE)
+#else /* !CONFIG_IPIPE */
 	EXCEPTION(0x900, Decrementer, timer_interrupt, EXC_XFER_LITE)
+#endif /* CONFIG_IPIPE */
 
 	EXCEPTION(0xa00, Trap_0a, unknown_exception, EXC_XFER_EE)
 	EXCEPTION(0xb00, Trap_0b, unknown_exception, EXC_XFER_EE)
@@ -1016,6 +1030,12 @@
 	lwz	r3,MMCONTEXTID(r4)
 	cmpwi	cr0,r3,0
 	blt-	4f
+#ifdef CONFIG_IPIPE
+	mfmsr	r7
+	rlwinm	r0,r7,0,17,15	/* clear MSR_EE in r0 */
+	mtmsr	r0
+	sync
+#endif	
 	mulli	r3,r3,897	/* multiply context by skew factor */
 	rlwinm	r3,r3,4,8,27	/* VSID = (context & 0xfffff) << 4 */
 	addis	r3,r3,0x6000	/* Set Ks, Ku bits */
@@ -1039,6 +1059,9 @@
 	rlwinm	r3,r3,0,8,3	/* clear out any overflow from VSID field */
 	addis	r4,r4,0x1000	/* address of next segment */
 	bdnz	3b
+#ifdef CONFIG_IPIPE
+	mtmsr	r7
+#endif	
 	sync
 	isync
 	blr
diff -urN source_powerpc_none/arch/powerpc/kernel/head_40x.S source_powerpc_none.ipipe/arch/powerpc/kernel/head_40x.S
--- source_powerpc_none/arch/powerpc/kernel/head_40x.S	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/head_40x.S	2009-12-22 12:44:08.000000000 -0500
@@ -234,6 +234,12 @@
 	EXC_XFER_TEMPLATE(hdlr, n, MSR_KERNEL, NOCOPY, transfer_to_handler_full, \
 			  ret_from_except_full)
 
+#ifdef CONFIG_IPIPE
+#define EXC_XFER_IPIPE(n, hdlr)		\
+	EXC_XFER_TEMPLATE(hdlr, n+1, MSR_KERNEL, NOCOPY, transfer_to_handler, \
+			  __ipipe_ret_from_except)
+#endif /* CONFIG_IPIPE */
+
 #define EXC_XFER_LITE(n, hdlr)		\
 	EXC_XFER_TEMPLATE(hdlr, n+1, MSR_KERNEL, NOCOPY, transfer_to_handler, \
 			  ret_from_except)
@@ -402,7 +408,11 @@
 	EXC_XFER_EE_LITE(0x400, handle_page_fault)
 
 /* 0x0500 - External Interrupt Exception */
+#ifdef CONFIG_IPIPE
+	EXCEPTION(0x0500, HardwareInterrupt, __ipipe_grab_irq, EXC_XFER_IPIPE)
+#else /* !CONFIG_IPIPE */
 	EXCEPTION(0x0500, HardwareInterrupt, do_IRQ, EXC_XFER_LITE)
+#endif /* CONFIG_IPIPE */
 
 /* 0x0600 - Alignment Exception */
 	START_EXCEPTION(0x0600, Alignment)
@@ -440,7 +450,11 @@
 	lis	r0,TSR_PIS@h
 	mtspr	SPRN_TSR,r0		/* Clear the PIT exception */
 	addi	r3,r1,STACK_FRAME_OVERHEAD
+#ifdef CONFIG_IPIPE
+	EXC_XFER_IPIPE(0x1000, __ipipe_grab_timer)
+#else /* !CONFIG_IPIPE */
 	EXC_XFER_LITE(0x1000, timer_interrupt)
+#endif /* CONFIG_IPIPE */
 
 #if 0
 /* NOTE:
diff -urN source_powerpc_none/arch/powerpc/kernel/head_44x.S source_powerpc_none.ipipe/arch/powerpc/kernel/head_44x.S
--- source_powerpc_none/arch/powerpc/kernel/head_44x.S	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/head_44x.S	2009-12-22 12:44:08.000000000 -0500
@@ -310,8 +310,11 @@
 		/* Instruction Storage Interrupt */
 	INSTRUCTION_STORAGE_EXCEPTION
 
-	/* External Input Interrupt */
-	EXCEPTION(0x0500, ExternalInput, do_IRQ, EXC_XFER_LITE)
+#ifdef CONFIG_IPIPE
+	EXCEPTION(0x0500, ExternalInput, __ipipe_grab_irq, EXC_XFER_IPIPE)
+#else /* !CONFIG_IPIPE */
+ 	EXCEPTION(0x0500, ExternalInput, do_IRQ, EXC_XFER_LITE)
+#endif /* CONFIG_IPIPE */
 
 	/* Alignment Interrupt */
 	ALIGNMENT_EXCEPTION
diff -urN source_powerpc_none/arch/powerpc/kernel/head_8xx.S source_powerpc_none.ipipe/arch/powerpc/kernel/head_8xx.S
--- source_powerpc_none/arch/powerpc/kernel/head_8xx.S	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/head_8xx.S	2009-12-22 12:44:08.000000000 -0500
@@ -185,6 +185,12 @@
 	EXC_XFER_TEMPLATE(n, hdlr, n, NOCOPY, transfer_to_handler_full,	\
 			  ret_from_except_full)
 
+#ifdef CONFIG_IPIPE
+#define EXC_XFER_IPIPE(n, hdlr)		\
+	EXC_XFER_TEMPLATE(n, hdlr, n+1, NOCOPY, transfer_to_handler, \
+			  __ipipe_ret_from_except)
+#endif /* CONFIG_IPIPE */
+
 #define EXC_XFER_LITE(n, hdlr)		\
 	EXC_XFER_TEMPLATE(n, hdlr, n+1, NOCOPY, transfer_to_handler, \
 			  ret_from_except)
@@ -236,7 +242,11 @@
 	EXC_XFER_EE_LITE(0x400, handle_page_fault)
 
 /* External interrupt */
+#ifdef CONFIG_IPIPE
+	EXCEPTION(0x500, HardwareInterrupt, __ipipe_grab_irq, EXC_XFER_IPIPE)
+#else /* !CONFIG_IPIPE */
 	EXCEPTION(0x500, HardwareInterrupt, do_IRQ, EXC_XFER_LITE)
+#endif /* CONFIG_IPIPE */
 
 /* Alignment exception */
 	. = 0x600
@@ -257,7 +267,11 @@
 	EXCEPTION(0x800, FPUnavailable, unknown_exception, EXC_XFER_STD)
 
 /* Decrementer */
+#ifdef CONFIG_IPIPE
+	EXCEPTION(0x900, Decrementer, __ipipe_grab_timer, EXC_XFER_IPIPE)
+#else /* !CONFIG_IPIPE */
 	EXCEPTION(0x900, Decrementer, timer_interrupt, EXC_XFER_LITE)
+#endif /* CONFIG_IPIPE */
 
 	EXCEPTION(0xa00, Trap_0a, unknown_exception, EXC_XFER_EE)
 	EXCEPTION(0xb00, Trap_0b, unknown_exception, EXC_XFER_EE)
diff -urN source_powerpc_none/arch/powerpc/kernel/head_booke.h source_powerpc_none.ipipe/arch/powerpc/kernel/head_booke.h
--- source_powerpc_none/arch/powerpc/kernel/head_booke.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/head_booke.h	2009-12-22 12:44:08.000000000 -0500
@@ -208,6 +208,12 @@
 	EXC_XFER_TEMPLATE(hdlr, n, MSR_KERNEL, NOCOPY, transfer_to_handler_full, \
 			  ret_from_except_full)
 
+#ifdef CONFIG_IPIPE
+#define EXC_XFER_IPIPE(n, hdlr)		\
+	EXC_XFER_TEMPLATE(hdlr, n+1, MSR_KERNEL, NOCOPY, transfer_to_handler, \
+			  __ipipe_ret_from_except)
+#endif /* CONFIG_IPIPE */
+
 #define EXC_XFER_LITE(n, hdlr)		\
 	EXC_XFER_TEMPLATE(hdlr, n+1, MSR_KERNEL, NOCOPY, transfer_to_handler, \
 			  ret_from_except)
@@ -372,6 +378,15 @@
 	addi	r3,r1,STACK_FRAME_OVERHEAD;				      \
 	EXC_XFER_STD(0x0700, program_check_exception)
 
+#ifdef CONFIG_IPIPE
+#define DECREMENTER_EXCEPTION						      \
+	START_EXCEPTION(Decrementer)					      \
+	NORMAL_EXCEPTION_PROLOG;					      \
+	lis     r0,TSR_DIS@h;           /* Setup the DEC interrupt mask */    \
+	mtspr   SPRN_TSR,r0;		/* Clear the DEC interrupt */	      \
+	addi    r3,r1,STACK_FRAME_OVERHEAD;				      \
+	EXC_XFER_IPIPE(0x0900, __ipipe_grab_timer)
+#else /* !CONFIG_IPIPE */
 #define DECREMENTER_EXCEPTION						      \
 	START_EXCEPTION(Decrementer)					      \
 	NORMAL_EXCEPTION_PROLOG;					      \
@@ -379,6 +394,7 @@
 	mtspr   SPRN_TSR,r0;		/* Clear the DEC interrupt */	      \
 	addi    r3,r1,STACK_FRAME_OVERHEAD;				      \
 	EXC_XFER_LITE(0x0900, timer_interrupt)
+#endif /* CONFIG_IPIPE */
 
 #define FP_UNAVAILABLE_EXCEPTION					      \
 	START_EXCEPTION(FloatingPointUnavailable)			      \
diff -urN source_powerpc_none/arch/powerpc/kernel/head_fsl_booke.S source_powerpc_none.ipipe/arch/powerpc/kernel/head_fsl_booke.S
--- source_powerpc_none/arch/powerpc/kernel/head_fsl_booke.S	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/head_fsl_booke.S	2009-12-22 12:44:08.000000000 -0500
@@ -488,7 +488,11 @@
 	INSTRUCTION_STORAGE_EXCEPTION
 
 	/* External Input Interrupt */
+#ifdef CONFIG_IPIPE
+	EXCEPTION(0x0500, ExternalInput, __ipipe_grab_irq, EXC_XFER_IPIPE)
+#else /* !CONFIG_IPIPE */
 	EXCEPTION(0x0500, ExternalInput, do_IRQ, EXC_XFER_LITE)
+#endif /* CONFIG_IPIPE */
 
 	/* Alignment Interrupt */
 	ALIGNMENT_EXCEPTION
diff -urN source_powerpc_none/arch/powerpc/kernel/idle.c source_powerpc_none.ipipe/arch/powerpc/kernel/idle.c
--- source_powerpc_none/arch/powerpc/kernel/idle.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/idle.c	2009-12-22 12:44:08.000000000 -0500
@@ -59,6 +59,7 @@
 		tick_nohz_stop_sched_tick(1);
 		while (!need_resched() && !cpu_should_die()) {
 			ppc64_runlatch_off();
+			ipipe_suspend_domain();
 
 			if (ppc_md.power_save) {
 				clear_thread_flag(TIF_POLLING_NRFLAG);
@@ -67,7 +68,7 @@
 				 * is ordered w.r.t. need_resched() test.
 				 */
 				smp_mb();
-				local_irq_disable();
+				local_irq_disable_hw();
 
 				/* Don't trace irqs off for idle */
 				stop_critical_timings();
@@ -78,7 +79,7 @@
 
 				start_critical_timings();
 
-				local_irq_enable();
+				local_irq_enable_hw();
 				set_thread_flag(TIF_POLLING_NRFLAG);
 
 			} else {
diff -urN source_powerpc_none/arch/powerpc/kernel/idle_power4.S source_powerpc_none.ipipe/arch/powerpc/kernel/idle_power4.S
--- source_powerpc_none/arch/powerpc/kernel/idle_power4.S	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/idle_power4.S	2009-12-22 12:44:08.000000000 -0500
@@ -34,9 +34,11 @@
 	rldicl	r0,r7,48,1
 	rotldi	r0,r0,16
 	mtmsrd	r0,1			/* hard-disable interrupts */
+#ifdef CONFIG_SOFTDISABLE	
 	li	r0,1
 	stb	r0,PACASOFTIRQEN(r13)	/* we'll hard-enable shortly */
 	stb	r0,PACAHARDIRQEN(r13)
+#endif CONFIG_SOFTDISABLE	
 BEGIN_FTR_SECTION
 	DSSALL
 	sync
@@ -59,10 +61,12 @@
 	rldicl	r0,r7,48,1
 	rotldi	r0,r0,16
 	mtmsrd	r0,1			/* hard-disable interrupts */
+#ifdef CONFIG_SOFTDISABLE	
 	li	r0,1
 	li	r6,0
 	stb	r0,PACAHARDIRQEN(r13)	/* we'll hard-enable shortly */
 	stb	r6,PACASOFTIRQEN(r13)	/* soft-disable irqs */
+#endif
 BEGIN_FTR_SECTION
 	DSSALL
 	sync
diff -urN source_powerpc_none/arch/powerpc/kernel/ipipe.c source_powerpc_none.ipipe/arch/powerpc/kernel/ipipe.c
--- source_powerpc_none/arch/powerpc/kernel/ipipe.c	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/ipipe.c	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,863 @@
+/* -*- linux-c -*-
+ * linux/arch/powerpc/kernel/ipipe.c
+ *
+ * Copyright (C) 2005 Heikki Lindholm (PPC64 port).
+ * Copyright (C) 2004 Wolfgang Grandegger (Adeos/ppc port over 2.4).
+ * Copyright (C) 2002-2007 Philippe Gerum.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139,
+ * USA; either version 2 of the License, or (at your option) any later
+ * version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * Architecture-dependent I-PIPE core support for PowerPC 32/64bit.
+ */
+
+#include <linux/kernel.h>
+#include <linux/smp.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/bitops.h>
+#include <linux/interrupt.h>
+#include <linux/irq.h>
+#include <linux/module.h>
+#include <linux/kernel_stat.h>
+#include <asm/reg.h>
+#include <asm/system.h>
+#include <asm/mmu_context.h>
+#include <asm/unistd.h>
+#include <asm/machdep.h>
+#include <asm/atomic.h>
+#include <asm/hardirq.h>
+#include <asm/io.h>
+#include <asm/time.h>
+
+static void __ipipe_do_IRQ(unsigned irq, void *cookie);
+
+static void __ipipe_do_timer(unsigned irq, void *cookie);
+
+DEFINE_PER_CPU(struct pt_regs, __ipipe_tick_regs);
+#ifdef CONFIG_IPIPE_WANT_PREEMPTIBLE_SWITCH
+DEFINE_PER_CPU(struct mm_struct *, ipipe_active_mm);
+EXPORT_PER_CPU_SYMBOL(ipipe_active_mm);
+#endif
+
+#define DECREMENTER_MAX	0x7fffffff
+
+#ifdef CONFIG_SMP
+
+static cpumask_t __ipipe_cpu_sync_map;
+
+static cpumask_t __ipipe_cpu_lock_map;
+
+static ipipe_spinlock_t __ipipe_cpu_barrier = IPIPE_SPIN_LOCK_UNLOCKED;
+
+static atomic_t __ipipe_critical_count = ATOMIC_INIT(0);
+
+static void (*__ipipe_cpu_sync) (void);
+
+static DEFINE_PER_CPU(struct ipipe_ipi_struct, ipipe_ipi_message);
+
+unsigned int __ipipe_ipi_irq = NR_IRQS + 1; /* dummy value */
+
+#ifdef CONFIG_DEBUGGER
+cpumask_t __ipipe_dbrk_pending;	/* pending debugger break IPIs */
+#endif
+
+/* Always called with hw interrupts off. */
+
+void __ipipe_do_critical_sync(unsigned irq, void *cookie)
+{
+	cpu_set(ipipe_processor_id(), __ipipe_cpu_sync_map);
+
+	/*
+	 * Now we are in sync with the lock requestor running on another
+	 * CPU. Enter a spinning wait until he releases the global
+	 * lock.
+	 */
+	spin_lock(&__ipipe_cpu_barrier);
+
+	/* Got it. Now get out. */
+
+	if (__ipipe_cpu_sync)
+		/* Call the sync routine if any. */
+		__ipipe_cpu_sync();
+
+	spin_unlock(&__ipipe_cpu_barrier);
+
+	cpu_clear(ipipe_processor_id(), __ipipe_cpu_sync_map);
+}
+
+void __ipipe_hook_critical_ipi(struct ipipe_domain *ipd)
+{
+	ipd->irqs[IPIPE_CRITICAL_IPI].acknowledge = NULL;
+	ipd->irqs[IPIPE_CRITICAL_IPI].handler = &__ipipe_do_critical_sync;
+	ipd->irqs[IPIPE_CRITICAL_IPI].cookie = NULL;
+	/* Immediately handle in the current domain but *never* pass */
+	ipd->irqs[IPIPE_CRITICAL_IPI].control =
+		IPIPE_HANDLE_MASK|IPIPE_STICKY_MASK|IPIPE_SYSTEM_MASK;
+}
+
+void __ipipe_register_ipi(unsigned int irq)
+{
+	__ipipe_ipi_irq = irq;
+	mb();
+} 
+
+static void __ipipe_ipi_demux(int irq, struct pt_regs *regs)
+{
+	int ipi, cpu = ipipe_processor_id();
+	struct irq_desc *desc = irq_desc + irq;
+
+	desc->ipipe_ack(irq, desc);
+
+	kstat_incr_irqs_this_cpu(irq, desc);
+
+	while (per_cpu(ipipe_ipi_message, cpu).value & IPIPE_MSG_IPI_MASK) {
+		for (ipi = IPIPE_MSG_CRITICAL_IPI; ipi <= IPIPE_MSG_SERVICE_IPI4; ++ipi) {
+			if (test_and_clear_bit(ipi, &per_cpu(ipipe_ipi_message, cpu).value)) {
+				mb();
+				__ipipe_handle_irq(ipi + IPIPE_MSG_IPI_OFFSET, NULL);
+			}
+		}
+	}
+
+#ifdef CONFIG_DEBUGGER
+	/*
+	 * The debugger IPI handler should be NMI-safe, so let's call
+	 * it immediately in case the IPI is pending.
+	 */
+	if (cpu_isset(cpu, __ipipe_dbrk_pending)) {
+		cpu_clear(cpu, __ipipe_dbrk_pending);
+		debugger_ipi(regs);
+	}
+#endif /* CONFIG_DEBUGGER */
+
+	__ipipe_end_irq(irq);
+}
+
+cpumask_t __ipipe_set_irq_affinity(unsigned irq, cpumask_t cpumask)
+{
+	cpumask_t oldmask;
+
+	if (irq_to_desc(irq)->chip->set_affinity == NULL)
+		return CPU_MASK_NONE;
+
+	if (cpus_empty(cpumask))
+		return CPU_MASK_NONE; /* Return mask value -- no change. */
+
+	cpus_and(cpumask, cpumask, cpu_online_map);
+	if (cpus_empty(cpumask))
+		return CPU_MASK_NONE;	/* Error -- bad mask value or non-routable IRQ. */
+
+	cpumask_copy(&oldmask, irq_to_desc(irq)->affinity);
+	irq_to_desc(irq)->chip->set_affinity(irq, &cpumask);
+
+	return oldmask;
+}
+
+int __ipipe_send_ipi(unsigned ipi, cpumask_t cpumask)
+{
+	extern void mpic_send_ipi(unsigned int ipi_no, unsigned int cpu_mask);
+	unsigned long flags;
+	cpumask_t testmask;
+	int cpu;
+
+	local_irq_save_hw(flags);
+
+	ipi -= IPIPE_MSG_IPI_OFFSET;
+	for_each_online_cpu(cpu) {
+		if (cpu_isset(cpu, cpumask))
+			set_bit(ipi, &per_cpu(ipipe_ipi_message, cpu).value);
+	}
+	mb();	 
+	
+	if (unlikely(cpus_empty(cpumask)))
+		goto out;
+
+	cpus_setall(testmask);
+	cpu_clear(ipipe_processor_id(), testmask);
+	if (likely(cpus_equal(cpumask, testmask)))
+		smp_ops->message_pass(MSG_ALL_BUT_SELF, PPC_MSG_IPIPE_DEMUX);
+	else {
+		/* Long path. */
+		for_each_cpu_mask_nr(cpu, cpumask)
+			smp_ops->message_pass(cpu, PPC_MSG_IPIPE_DEMUX);
+	}
+out:
+	local_irq_restore_hw(flags);
+
+	return 0;
+}
+
+void __ipipe_stall_root(void)
+{
+	unsigned long flags;
+
+	local_irq_save_hw(flags);
+	set_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status));
+	local_irq_restore_hw(flags);
+}
+
+unsigned long __ipipe_test_and_stall_root(void)
+{
+	unsigned long flags;
+	int x;
+
+	local_irq_save_hw(flags);
+	x = test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status));
+	local_irq_restore_hw(flags);
+
+	return x;
+}
+
+unsigned long __ipipe_test_root(void)
+{
+	unsigned long flags;
+	int x;
+
+	local_irq_save_hw(flags);
+	x = test_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status));
+	local_irq_restore_hw(flags);
+
+	return x;
+}
+
+#endif	/* CONFIG_SMP */
+
+/*
+ * ipipe_critical_enter() -- Grab the superlock excluding all CPUs
+ * but the current one from a critical section. This lock is used when
+ * we must enforce a global critical section for a single CPU in a
+ * possibly SMP system whichever context the CPUs are running.
+ */
+unsigned long ipipe_critical_enter(void (*syncfn) (void))
+{
+	unsigned long flags;
+
+	local_irq_save_hw(flags);
+
+#ifdef CONFIG_SMP
+	if (likely(num_online_cpus() > 1)) {
+		/* We might be running a SMP-kernel on a UP box... */
+		int cpu = ipipe_processor_id();
+		cpumask_t lock_map;
+		cpumask_t others;
+
+		if (!cpu_test_and_set(cpu, __ipipe_cpu_lock_map)) {
+			while (cpu_test_and_set(BITS_PER_LONG - 1, __ipipe_cpu_lock_map)) {
+				int n = 0;
+				do {
+					cpu_relax();
+				} while (++n < cpu);
+			}
+
+			spin_lock(&__ipipe_cpu_barrier);
+
+			__ipipe_cpu_sync = syncfn;
+
+			/* Send the sync IPI to all processors but the current one. */
+			cpus_setall(others);
+			cpu_clear(ipipe_processor_id(), others);
+			__ipipe_send_ipi(IPIPE_CRITICAL_IPI, others);
+
+			cpus_andnot(lock_map, cpu_online_map,
+				    __ipipe_cpu_lock_map);
+
+			while (!cpus_equal(__ipipe_cpu_sync_map, lock_map))
+				cpu_relax();
+		}
+
+		atomic_inc(&__ipipe_critical_count);
+	}
+#endif	/* CONFIG_SMP */
+
+	return flags;
+}
+
+/* ipipe_critical_exit() -- Release the superlock. */
+
+void ipipe_critical_exit(unsigned long flags)
+{
+#ifdef CONFIG_SMP
+	if (likely(num_online_cpus() > 1)) {
+		/* We might be running a SMP-kernel on a UP box... */
+		if (atomic_dec_and_test(&__ipipe_critical_count)) {
+			spin_unlock(&__ipipe_cpu_barrier);
+
+			while (!cpus_empty(__ipipe_cpu_sync_map))
+				cpu_relax();
+
+			cpu_clear(ipipe_processor_id(), __ipipe_cpu_lock_map);
+			cpu_clear(BITS_PER_LONG - 1, __ipipe_cpu_lock_map);
+		}
+	}
+#endif	/* CONFIG_SMP */
+
+	local_irq_restore_hw(flags);
+}
+
+void __ipipe_init_platform(void)
+{
+	unsigned int virq;
+
+	/*
+	 * Allocate a virtual IRQ for the decrementer trap early to
+	 * get it mapped to IPIPE_VIRQ_BASE
+	 */
+
+	virq = ipipe_alloc_virq();
+
+	if (virq != IPIPE_TIMER_VIRQ)
+		panic("I-pipe: cannot reserve timer virq #%d (got #%d)",
+		      IPIPE_TIMER_VIRQ, virq);
+
+#ifdef CONFIG_SMP
+	virq = ipipe_alloc_virq();
+	if (virq != IPIPE_CRITICAL_IPI)
+		panic("I-pipe: cannot reserve critical IPI virq #%d (got #%d)",
+		      IPIPE_CRITICAL_IPI, virq);
+	virq = ipipe_alloc_virq();
+	if (virq != IPIPE_SERVICE_IPI0)
+		panic("I-pipe: cannot reserve service IPI 0 virq #%d (got #%d)",
+		      IPIPE_SERVICE_IPI0, virq);
+	virq = ipipe_alloc_virq();
+	if (virq != IPIPE_SERVICE_IPI1)
+		panic("I-pipe: cannot reserve service IPI 1 virq #%d (got #%d)",
+		      IPIPE_SERVICE_IPI1, virq);
+	virq = ipipe_alloc_virq();
+	if (virq != IPIPE_SERVICE_IPI2)
+		panic("I-pipe: cannot reserve service IPI 2 virq #%d (got #%d)",
+		      IPIPE_SERVICE_IPI2, virq);
+	virq = ipipe_alloc_virq();
+	if (virq != IPIPE_SERVICE_IPI3)
+		panic("I-pipe: cannot reserve service IPI 3 virq #%d (got #%d)",
+		      IPIPE_SERVICE_IPI3, virq);
+	virq = ipipe_alloc_virq();
+	if (virq != IPIPE_SERVICE_IPI4)
+		panic("I-pipe: cannot reserve service IPI 4 virq #%d (got #%d)",
+		      IPIPE_SERVICE_IPI4, virq);
+#endif
+}
+
+void __ipipe_end_irq(unsigned irq)
+{
+	struct irq_desc *desc = get_irq_desc(irq);
+	desc->ipipe_end(irq, desc);
+}
+
+void __ipipe_enable_irqdesc(struct ipipe_domain *ipd, unsigned irq)
+{
+	get_irq_desc(irq)->status &= ~IRQ_DISABLED;
+}
+
+static void __ipipe_ack_irq(unsigned irq, struct irq_desc *desc)
+{
+	desc->ipipe_ack(irq, desc);
+}
+
+/*
+ * __ipipe_enable_pipeline() -- We are running on the boot CPU, hw
+ * interrupts are off, and secondary CPUs are still lost in space.
+ */
+void __ipipe_enable_pipeline(void)
+{
+	unsigned long flags;
+	unsigned irq;
+
+	flags = ipipe_critical_enter(NULL);
+
+	/* First, virtualize all interrupts from the root domain. */
+
+	for (irq = 0; irq < NR_IRQS; irq++)
+		ipipe_virtualize_irq(ipipe_root_domain,
+				     irq,
+				     &__ipipe_do_IRQ, NULL,
+				     &__ipipe_ack_irq,
+				     IPIPE_HANDLE_MASK | IPIPE_PASS_MASK);
+	/*
+	 * We use a virtual IRQ to handle the timer irq (decrementer trap)
+	 * which has been allocated early in __ipipe_init_platform().
+	 */
+
+	ipipe_virtualize_irq(ipipe_root_domain,
+			     IPIPE_TIMER_VIRQ,
+			     &__ipipe_do_timer, NULL,
+			     NULL, IPIPE_HANDLE_MASK | IPIPE_PASS_MASK);
+
+	ipipe_critical_exit(flags);
+}
+
+int ipipe_get_sysinfo(struct ipipe_sysinfo *info)
+{
+	info->ncpus = num_online_cpus();
+	info->cpufreq = ipipe_cpu_freq();
+	info->archdep.tmirq = IPIPE_TIMER_VIRQ;
+	info->archdep.tmfreq = info->cpufreq;
+
+	return 0;
+}
+
+/*
+ * ipipe_trigger_irq() -- Push the interrupt at front of the pipeline
+ * just like if it has been actually received from a hw source. Also
+ * works for virtual interrupts.
+ */
+int ipipe_trigger_irq(unsigned irq)
+{
+	unsigned long flags;
+
+#ifdef CONFIG_IPIPE_DEBUG
+	if (irq >= IPIPE_NR_IRQS ||
+	    (ipipe_virtual_irq_p(irq)
+	     && !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map)))
+		return -EINVAL;
+#endif
+	local_irq_save_hw(flags);
+	__ipipe_handle_irq(irq, NULL);
+	local_irq_restore_hw(flags);
+
+	return 1;
+}
+
+/*
+ * __ipipe_handle_irq() -- IPIPE's generic IRQ handler. An optimistic
+ * interrupt protection log is maintained here for each domain. Hw
+ * interrupts are off on entry.
+ */
+void __ipipe_handle_irq(int irq, struct pt_regs *regs)
+{
+	struct ipipe_domain *this_domain, *next_domain;
+	struct list_head *head, *pos;
+	int m_ack;
+
+	/* Software-triggered IRQs do not need any ack. */
+	m_ack = (regs == NULL);
+
+#ifdef CONFIG_IPIPE_DEBUG
+	if (unlikely(irq >= IPIPE_NR_IRQS)) {
+		printk(KERN_ERR "I-pipe: spurious interrupt %d\n", irq);
+		return;
+	}
+#endif
+	this_domain = __ipipe_current_domain;
+
+	if (unlikely(test_bit(IPIPE_STICKY_FLAG, &this_domain->irqs[irq].control)))
+		head = &this_domain->p_link;
+	else {
+		head = __ipipe_pipeline.next;
+		next_domain = list_entry(head, struct ipipe_domain, p_link);
+		if (likely(test_bit(IPIPE_WIRED_FLAG, &next_domain->irqs[irq].control))) {
+			if (!m_ack && next_domain->irqs[irq].acknowledge)
+				next_domain->irqs[irq].acknowledge(irq, irq_desc + irq);
+			__ipipe_dispatch_wired(next_domain, irq);
+			return;
+		}
+	}
+
+	/* Ack the interrupt. */
+
+	pos = head;
+
+	while (pos != &__ipipe_pipeline) {
+		next_domain = list_entry(pos, struct ipipe_domain, p_link);
+		prefetch(next_domain);
+		/*
+		 * For each domain handling the incoming IRQ, mark it as
+		 * pending in its log.
+		 */
+		if (test_bit(IPIPE_HANDLE_FLAG, &next_domain->irqs[irq].control)) {
+			/*
+			 * Domains that handle this IRQ are polled for
+			 * acknowledging it by decreasing priority order. The
+			 * interrupt must be made pending _first_ in the
+			 * domain's status flags before the PIC is unlocked.
+			 */
+			__ipipe_set_irq_pending(next_domain, irq);
+
+			if (!m_ack && next_domain->irqs[irq].acknowledge) {
+				next_domain->irqs[irq].acknowledge(irq, irq_desc + irq);
+				m_ack = 1;
+			}
+		}
+
+		/*
+		 * If the domain does not want the IRQ to be passed down the
+		 * interrupt pipe, exit the loop now.
+		 */
+		if (!test_bit(IPIPE_PASS_FLAG, &next_domain->irqs[irq].control))
+			break;
+
+		pos = next_domain->p_link.next;
+	}
+
+	/*
+	 * If the interrupt preempted the head domain, then do not
+	 * even try to walk the pipeline, unless an interrupt is
+	 * pending for it.
+	 */
+	if (test_bit(IPIPE_AHEAD_FLAG, &this_domain->flags) &&
+	    ipipe_head_cpudom_var(irqpend_himask) == 0)
+		return;
+
+	/*
+	 * Now walk the pipeline, yielding control to the highest
+	 * priority domain that has pending interrupt(s) or
+	 * immediately to the current domain if the interrupt has been
+	 * marked as 'sticky'. This search does not go beyond the
+	 * current domain in the pipeline.
+	 */
+
+	__ipipe_walk_pipeline(head);
+}
+
+int __ipipe_grab_irq(struct pt_regs *regs)
+{
+	extern int ppc_spurious_interrupts;
+	int irq;
+
+	irq = ppc_md.get_irq();
+	if (unlikely(irq == NO_IRQ)) {
+		ppc_spurious_interrupts++;
+		goto root_checks;
+	}
+
+	if (likely(irq != NO_IRQ_IGNORE)) {
+		ipipe_trace_irq_entry(irq);
+#ifdef CONFIG_SMP
+		/* Check for cascaded I-pipe IPIs */
+		if (irq == __ipipe_ipi_irq) {
+			__ipipe_ipi_demux(irq, regs);
+			ipipe_trace_irq_exit(irq);
+			goto root_checks;
+		}
+#endif /* CONFIG_SMP */
+		__ipipe_handle_irq(irq, regs);
+		ipipe_trace_irq_exit(irq);
+	}
+
+root_checks:
+
+	if (__ipipe_root_domain_p) {
+#ifdef CONFIG_PPC_970_NAP
+		struct thread_info *ti = current_thread_info();
+		/* Emulate the napping check when 100% sure we do run
+		 * over the root context. */
+		if (test_and_clear_bit(TLF_NAPPING, &ti->local_flags))
+			regs->nip = regs->link;
+#endif
+#ifdef CONFIG_PPC64
+		ppc64_runlatch_on();
+#endif
+		if (!test_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)))
+			return 1;
+	}
+
+	return 0;
+}
+
+static void __ipipe_do_IRQ(unsigned irq, void *cookie)
+{
+	struct pt_regs *old_regs;
+#ifdef CONFIG_IRQSTACKS
+	struct thread_info *curtp, *irqtp;
+#endif
+
+	/* Provide a valid register frame, even if not the exact one. */
+	old_regs = set_irq_regs(&__raw_get_cpu_var(__ipipe_tick_regs));
+
+	irq_enter();
+
+#ifdef CONFIG_DEBUG_STACKOVERFLOW
+	/* Debugging check for stack overflow: is there less than 2KB free? */
+	{
+		long sp;
+
+		sp = __get_SP() & (THREAD_SIZE-1);
+
+		if (unlikely(sp < (sizeof(struct thread_info) + 2048))) {
+			printk("do_IRQ: stack overflow: %ld\n",
+				sp - sizeof(struct thread_info));
+			dump_stack();
+		}
+	}
+#endif
+
+#ifdef CONFIG_IRQSTACKS
+		/* Switch to the irq stack to handle this */
+		curtp = current_thread_info();
+		irqtp = hardirq_ctx[smp_processor_id()];
+		if (curtp != irqtp) {
+			struct irq_desc *desc = irq_desc + irq;
+			void *handler = desc->handle_irq;
+			if (handler == NULL)
+				handler = &__do_IRQ;
+			irqtp->task = curtp->task;
+			irqtp->flags = 0;
+			call_handle_irq(irq, desc, irqtp, handler);
+			irqtp->task = NULL;
+			if (irqtp->flags)
+				set_bits(irqtp->flags, &curtp->flags);
+		} else
+#endif
+			generic_handle_irq(irq);
+
+	irq_exit();
+
+	set_irq_regs(old_regs);
+}
+
+static void __ipipe_do_timer(unsigned irq, void *cookie)
+{
+	timer_interrupt(&__raw_get_cpu_var(__ipipe_tick_regs));
+}
+
+int __ipipe_grab_timer(struct pt_regs *regs)
+{
+	struct ipipe_domain *ipd, *head;
+
+	ipd = __ipipe_current_domain;
+	head = __ipipe_pipeline_head();
+
+	set_dec(DECREMENTER_MAX);
+
+	ipipe_trace_irq_entry(IPIPE_TIMER_VIRQ);
+
+	__raw_get_cpu_var(__ipipe_tick_regs).msr = regs->msr; /* for timer_interrupt() */
+	__raw_get_cpu_var(__ipipe_tick_regs).nip = regs->nip;
+
+	if (ipd != &ipipe_root)
+		__raw_get_cpu_var(__ipipe_tick_regs).msr &= ~MSR_EE;
+
+	if (test_bit(IPIPE_WIRED_FLAG, &head->irqs[IPIPE_TIMER_VIRQ].control))
+		/*
+		 * Finding a wired IRQ means that we do have a
+		 * registered head domain as well. The decrementer
+		 * interrupt requires no acknowledge, so we may branch
+		 * to the wired IRQ dispatcher directly. Additionally,
+		 * we may bypass checks for locked interrupts or
+		 * stalled stage (the decrementer cannot be locked and
+		 * the head domain is obviously not stalled since we
+		 * got there).
+		 */
+		__ipipe_dispatch_wired_nocheck(head, IPIPE_TIMER_VIRQ);
+	else
+		__ipipe_handle_irq(IPIPE_TIMER_VIRQ, NULL);
+
+	ipipe_trace_irq_exit(IPIPE_TIMER_VIRQ);
+
+	if (ipd == &ipipe_root) {
+#ifdef CONFIG_PPC_970_NAP
+		struct thread_info *ti = current_thread_info();
+		/* Emulate the napping check when 100% sure we do run
+		 * over the root context. */
+		if (test_and_clear_bit(TLF_NAPPING, &ti->local_flags))
+			regs->nip = regs->link;
+#endif
+#ifdef CONFIG_PPC64
+		ppc64_runlatch_on();
+#endif
+		if (!test_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)))
+			return 1;
+	}
+
+	return 0;
+}
+
+notrace int __ipipe_check_root(void) /* hw IRQs off */
+{
+	return __ipipe_root_domain_p;
+}
+
+#ifdef CONFIG_PPC64
+
+#include <asm/firmware.h>
+#include <asm/lv1call.h>
+
+notrace void __ipipe_restore_if_root(unsigned long x) /* hw IRQs off */
+{
+	if (likely(!__ipipe_root_domain_p))
+		return;
+
+	if (x)
+		__set_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status));
+	else
+		__clear_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status));
+
+	if ((int)mfspr(SPRN_DEC) < 0)
+		mtspr(SPRN_DEC, 1);
+
+	/*
+	 * Force the delivery of pending soft-disabled interrupts on
+	 * PS3.  Any HV call will have this side effect.
+	 */
+	if (firmware_has_feature(FW_FEATURE_PS3_LV1)) {
+		u64 tmp;
+		lv1_get_version_info(&tmp);
+	}
+
+	local_irq_enable_hw();
+}
+
+#else
+
+#ifdef CONFIG_PREEMPT
+
+asmlinkage void __sched preempt_schedule_irq(void);
+
+void __sched  __ipipe_preempt_schedule_irq(void)
+{
+	struct ipipe_percpu_domain_data *p; 
+	unsigned long flags;  
+	/*
+	 * We have no IRQ state fixup on entry to exceptions, so we
+	 * have to stall the root stage before rescheduling.
+	 */  
+#ifdef CONFIG_IPIPE_DEBUG
+	BUG_ON(!irqs_disabled_hw());
+#endif
+	local_irq_save(flags);	
+	local_irq_enable_hw();	
+	preempt_schedule_irq(); /* Ok, may reschedule now. */  
+	local_irq_disable_hw(); 
+	/*
+	 * Flush any pending interrupt that may have been logged after
+	 * preempt_schedule_irq() stalled the root stage before
+	 * returning to us, and now.
+	 */
+	p = ipipe_root_cpudom_ptr(); 
+	if (unlikely(p->irqpend_himask != 0)) { 
+		add_preempt_count(PREEMPT_ACTIVE);
+		clear_bit(IPIPE_STALL_FLAG, &p->status); 
+		__ipipe_sync_pipeline(IPIPE_IRQMASK_ANY); 
+		sub_preempt_count(PREEMPT_ACTIVE);
+	} 
+
+	__local_irq_restore_nosync(flags);  
+}
+
+#endif /* CONFIG_PREEMPT */
+
+#endif
+
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+
+notrace void __ipipe_trace_irqsoff(void)
+{
+	ipipe_trace_irqsoff();
+}
+
+notrace void __ipipe_trace_irqson(void)
+{
+	ipipe_trace_irqson();
+}
+
+notrace void __ipipe_trace_irqsx(unsigned long msr_ee)
+{
+	if (msr_ee)
+		ipipe_trace_irqson();
+	else
+		ipipe_trace_irqsoff();
+}
+
+#endif
+
+int __ipipe_syscall_root(struct pt_regs *regs)
+{
+	struct ipipe_percpu_domain_data *p;
+	unsigned long flags;
+        int ret;
+
+#ifdef CONFIG_PPC64
+	/*
+	 * Unlike ppc32, hw interrupts are off on entry here.  We did
+	 * not copy the stall state on entry yet, so do it now.
+	 */
+	p = ipipe_root_cpudom_ptr();
+	regs->softe = !test_bit(IPIPE_STALL_FLAG, &p->status);
+
+	/* We ran DISABLE_INTS before being sent to the syscall
+	 * dispatcher, so we need to unstall the root stage, unless
+	 * the root domain is not current. */
+	if (__ipipe_root_domain_p)
+		__clear_bit(IPIPE_STALL_FLAG, &p->status);
+
+	local_irq_enable_hw();
+#endif
+        /*
+         * This routine either returns:
+         * 0 -- if the syscall is to be passed to Linux;
+         * >0 -- if the syscall should not be passed to Linux, and no
+         * tail work should be performed;
+         * <0 -- if the syscall should not be passed to Linux but the
+         * tail work has to be performed (for handling signals etc).
+         */
+
+        if (!__ipipe_syscall_watched_p(current, regs->gpr[0]) ||
+            !__ipipe_event_monitored_p(IPIPE_EVENT_SYSCALL))
+                return 0;
+
+        ret = __ipipe_dispatch_event(IPIPE_EVENT_SYSCALL, regs);
+
+	local_irq_save_hw(flags);
+
+        if (!__ipipe_root_domain_p) {
+		local_irq_restore_hw(flags);
+		return 1;
+	}
+
+	p = ipipe_root_cpudom_ptr();
+	if ((p->irqpend_himask & IPIPE_IRQMASK_VIRT) != 0)
+		__ipipe_sync_pipeline(IPIPE_IRQMASK_VIRT);
+
+	local_irq_restore_hw(flags);
+
+	return -ret;
+}
+
+void __ipipe_pin_range_globally(unsigned long start, unsigned long end)
+{
+	/* We don't support this. */
+}
+
+#ifdef CONFIG_SMP
+EXPORT_SYMBOL(__ipipe_stall_root);
+EXPORT_SYMBOL(__ipipe_test_root);
+EXPORT_SYMBOL(__ipipe_test_and_stall_root);
+#endif
+
+EXPORT_SYMBOL_GPL(__switch_to);
+EXPORT_SYMBOL_GPL(show_stack);
+EXPORT_SYMBOL_GPL(_switch);
+EXPORT_SYMBOL_GPL(tasklist_lock);
+#ifdef CONFIG_PPC64
+EXPORT_PER_CPU_SYMBOL(ppc64_tlb_batch);
+EXPORT_SYMBOL_GPL(switch_slb);
+EXPORT_SYMBOL_GPL(switch_stab);
+EXPORT_SYMBOL_GPL(__flush_tlb_pending);
+EXPORT_SYMBOL_GPL(mmu_linear_psize);
+EXPORT_SYMBOL_GPL(mmu_psize_defs);
+#else  /* !CONFIG_PPC64 */
+void atomic_set_mask(unsigned long mask, unsigned long *ptr);
+void atomic_clear_mask(unsigned long mask, unsigned long *ptr);
+#ifdef FEW_CONTEXTS
+EXPORT_SYMBOL_GPL(nr_free_contexts);
+EXPORT_SYMBOL_GPL(context_mm);
+EXPORT_SYMBOL_GPL(steal_context);
+#endif	/* !FEW_CONTEXTS */
+EXPORT_SYMBOL_GPL(atomic_set_mask);
+EXPORT_SYMBOL_GPL(atomic_clear_mask);
+#ifndef CONFIG_SMP
+EXPORT_SYMBOL_GPL(last_task_used_math);
+#endif
+#endif	/* !CONFIG_PPC64 */
diff -urN source_powerpc_none/arch/powerpc/kernel/irq.c source_powerpc_none.ipipe/arch/powerpc/kernel/irq.c
--- source_powerpc_none/arch/powerpc/kernel/irq.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/irq.c	2009-12-22 12:44:08.000000000 -0500
@@ -72,7 +72,7 @@
 #endif
 
 int __irq_offset_value;
-static int ppc_spurious_interrupts;
+int ppc_spurious_interrupts;
 
 #ifdef CONFIG_PPC32
 EXPORT_SYMBOL(__irq_offset_value);
@@ -89,6 +89,8 @@
 
 int distribute_irqs = 1;
 
+#ifdef CONFIG_SOFTDISABLE
+
 static inline notrace unsigned long get_hard_enabled(void)
 {
 	unsigned long enabled;
@@ -173,6 +175,9 @@
 	__hard_irq_enable();
 }
 EXPORT_SYMBOL(raw_local_irq_restore);
+
+#endif /* !CONFIG_SOFTDISABLE */
+
 #endif /* CONFIG_PPC64 */
 
 int show_interrupts(struct seq_file *p, void *v)
diff -urN source_powerpc_none/arch/powerpc/kernel/ppc_ksyms.c source_powerpc_none.ipipe/arch/powerpc/kernel/ppc_ksyms.c
--- source_powerpc_none/arch/powerpc/kernel/ppc_ksyms.c	2009-12-21 17:25:55.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/ppc_ksyms.c	2009-12-22 12:44:08.000000000 -0500
@@ -123,6 +123,9 @@
 #ifdef CONFIG_SMP
 #ifdef CONFIG_PPC32
 EXPORT_SYMBOL(smp_hw_index);
+#ifdef CONFIG_IPIPE
+EXPORT_SYMBOL(smp_logical_index);
+#endif
 #endif
 #endif
 
diff -urN source_powerpc_none/arch/powerpc/kernel/process.c source_powerpc_none.ipipe/arch/powerpc/kernel/process.c
--- source_powerpc_none/arch/powerpc/kernel/process.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/process.c	2009-12-22 12:44:08.000000000 -0500
@@ -98,8 +98,12 @@
 
 void enable_kernel_fp(void)
 {
+	unsigned long flags;
+
 	WARN_ON(preemptible());
 
+	local_irq_save_hw_cond(flags);
+
 #ifdef CONFIG_SMP
 	if (current->thread.regs && (current->thread.regs->msr & MSR_FP))
 		giveup_fpu(current);
@@ -108,6 +112,7 @@
 #else
 	giveup_fpu(last_task_used_math);
 #endif /* CONFIG_SMP */
+	local_irq_restore_hw_cond(flags);
 }
 EXPORT_SYMBOL(enable_kernel_fp);
 
@@ -398,7 +403,7 @@
 	}
 #endif
 
-	local_irq_save(flags);
+	local_irq_save_hw(flags);
 
 	account_system_vtime(current);
 	account_process_vtime(current);
@@ -412,7 +417,7 @@
 	hard_irq_disable();
 	last = _switch(old_thread, new_thread);
 
-	local_irq_restore(flags);
+	local_irq_restore_hw(flags);
 
 	return last;
 }
@@ -1084,7 +1089,7 @@
 }
 EXPORT_SYMBOL(dump_stack);
 
-#ifdef CONFIG_PPC64
+#ifdef CONFIG_RUNLATCH
 void ppc64_runlatch_on(void)
 {
 	unsigned long ctrl;
diff -urN source_powerpc_none/arch/powerpc/kernel/prom_init_check.sh source_powerpc_none.ipipe/arch/powerpc/kernel/prom_init_check.sh
--- source_powerpc_none/arch/powerpc/kernel/prom_init_check.sh	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/prom_init_check.sh	2009-12-22 12:44:08.000000000 -0500
@@ -20,7 +20,7 @@
 _end enter_prom memcpy memset reloc_offset __secondary_hold
 __secondary_hold_acknowledge __secondary_hold_spinloop __start
 strcmp strcpy strlcpy strlen strncmp strstr logo_linux_clut224
-reloc_got2 kernstart_addr memstart_addr linux_banner"
+reloc_got2 kernstart_addr memstart_addr linux_banner _mcount"
 
 NM="$1"
 OBJ="$2"
diff -urN source_powerpc_none/arch/powerpc/kernel/setup_32.c source_powerpc_none.ipipe/arch/powerpc/kernel/setup_32.c
--- source_powerpc_none/arch/powerpc/kernel/setup_32.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/setup_32.c	2009-12-22 12:44:08.000000000 -0500
@@ -52,6 +52,9 @@
 int boot_cpuid_phys;
 
 int smp_hw_index[NR_CPUS];
+#ifdef CONFIG_IPIPE
+int smp_logical_index[NR_CPUS];
+#endif
 
 unsigned long ISA_DMA_THRESHOLD;
 unsigned int DMA_MODE_READ;
diff -urN source_powerpc_none/arch/powerpc/kernel/setup_64.c source_powerpc_none.ipipe/arch/powerpc/kernel/setup_64.c
--- source_powerpc_none/arch/powerpc/kernel/setup_64.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/setup_64.c	2009-12-22 12:44:08.000000000 -0500
@@ -224,8 +224,10 @@
 #ifdef CONFIG_SMP
 void early_setup_secondary(void)
 {
+#ifdef CONFIG_SOFTDISABLE
 	/* Mark interrupts enabled in PACA */
 	get_paca()->soft_enabled = 0;
+#endif
 
 	/* Initialize the hash table or TLB handling */
 	early_init_mmu_secondary();
@@ -331,6 +333,12 @@
  */
 void __init setup_system(void)
 {
+#ifdef CONFIG_IPIPE
+	/* Early temporary init, before per-cpu areas are moved to
+	 * their final location. */
+	get_paca()->root_percpu = (u64)&ipipe_percpudom(&ipipe_root, status, 0);
+#endif
+
 	DBG(" -> setup_system()\n");
 
 	/* Apply the CPUs-specific and firmware specific fixups to kernel
@@ -577,6 +585,10 @@
 		snprintf(buf, 128, "%s", msg);
 		ppc_md.progress(buf, 0);
 	}
+#ifdef CONFIG_IPIPE
+	/* Reset pointer to the relocated per-cpu root domain data. */
+	get_paca()->root_percpu = (u64)&ipipe_percpudom(&ipipe_root, status, 0);
+#endif
 }
 
 /* Print a boot progress message. */
diff -urN source_powerpc_none/arch/powerpc/kernel/smp.c source_powerpc_none.ipipe/arch/powerpc/kernel/smp.c
--- source_powerpc_none/arch/powerpc/kernel/smp.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/smp.c	2009-12-22 12:44:08.000000000 -0500
@@ -154,7 +154,7 @@
 	[PPC_MSG_CALL_FUNCTION] =  "ipi call function",
 	[PPC_MSG_RESCHEDULE] = "ipi reschedule",
 	[PPC_MSG_CALL_FUNC_SINGLE] = "ipi call function single",
-	[PPC_MSG_DEBUGGER_BREAK] = "ipi debugger",
+	[PPC_MSG_DEBUGGER_BREAK] = "ipi I-pipe/debugger",
 };
 
 /* optional function to request ipi, for controllers with >= 4 ipis */
@@ -165,11 +165,17 @@
 	if (msg < 0 || msg > PPC_MSG_DEBUGGER_BREAK) {
 		return -EINVAL;
 	}
+#ifdef CONFIG_IPIPE
+	if (msg == PPC_MSG_DEBUGGER_BREAK)
+		/* Piggyback the debugger IPI for the I-pipe. */
+		__ipipe_register_ipi(virq);
+#endif
 #if !defined(CONFIG_DEBUGGER) && !defined(CONFIG_KEXEC)
 	if (msg == PPC_MSG_DEBUGGER_BREAK) {
 		return 1;
 	}
 #endif
+
 	err = request_irq(virq, smp_ipi_action[msg], IRQF_DISABLED|IRQF_PERCPU,
 			  smp_ipi_name[msg], 0);
 	WARN(err < 0, "unable to request_irq %d for %s (rc %d)\n",
@@ -200,8 +206,12 @@
 #ifdef CONFIG_DEBUGGER
 void smp_send_debugger_break(int cpu)
 {
-	if (likely(smp_ops))
+	if (likely(smp_ops)) {
+#ifdef CONFIG_IPIPE
+		cpu_set(cpu, __ipipe_dbrk_pending);
+#endif
 		smp_ops->message_pass(cpu, PPC_MSG_DEBUGGER_BREAK);
+	}
 }
 #endif
 
@@ -210,6 +220,10 @@
 {
 	crash_ipi_function_ptr = crash_ipi_callback;
 	if (crash_ipi_callback && smp_ops) {
+#ifdef CONFIG_IPIPE
+		cpus_setall(__ipipe_dbrk_pending);
+		cpu_clear(ipipe_processor_id(), __ipipe_dbrk_pending);
+#endif
 		mb();
 		smp_ops->message_pass(MSG_ALL_BUT_SELF, PPC_MSG_DEBUGGER_BREAK);
 	}
@@ -488,6 +502,9 @@
 	struct device_node *l2_cache;
 	int i, base;
 
+#if defined(CONFIG_IPIPE) && defined(CONFIG_PPC64)
+	get_paca()->root_percpu = (u64)&ipipe_percpudom(&ipipe_root, status, cpu);
+#endif
 	atomic_inc(&init_mm.mm_count);
 	current->active_mm = &init_mm;
 
diff -urN source_powerpc_none/arch/powerpc/kernel/time.c source_powerpc_none.ipipe/arch/powerpc/kernel/time.c
--- source_powerpc_none/arch/powerpc/kernel/time.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/time.c	2009-12-22 12:44:08.000000000 -0500
@@ -126,6 +126,8 @@
 
 static DEFINE_PER_CPU(struct decrementer_clock, decrementers);
 
+DEFINE_PER_CPU(int, disarm_decr);
+
 #ifdef CONFIG_PPC_ISERIES
 static unsigned long __initdata iSeries_recal_titan;
 static signed long __initdata iSeries_recal_tb;
@@ -569,11 +571,37 @@
 	struct pt_regs *old_regs;
 	struct decrementer_clock *decrementer =  &__get_cpu_var(decrementers);
 	struct clock_event_device *evt = &decrementer->event;
+	int cpu = smp_processor_id();
 	u64 now;
 
 	/* Ensure a positive value is written to the decrementer, or else
 	 * some CPUs will continuue to take decrementer exceptions */
-	set_dec(DECREMENTER_MAX);
+	if (!per_cpu(disarm_decr, cpu))
+		set_dec(DECREMENTER_MAX);
+
+#ifdef CONFIG_PPC_PASEMI_A2_WORKAROUNDS
+	extern ipipe_spinlock_t native_tlbie_lock;
+
+	spin_lock(&native_tlbie_lock);
+	asm("ptesync");
+	spin_unlock(&native_tlbie_lock);
+#endif
+
+#ifdef CONFIG_PPC_PASEMI_A2_WORKAROUNDS
+	extern spinlock_t native_tlbie_lock;
+
+	spin_lock(&native_tlbie_lock);
+	asm("ptesync");
+	spin_unlock(&native_tlbie_lock);
+#endif
+
+#ifdef CONFIG_PPC_PASEMI_A2_WORKAROUNDS
+	extern spinlock_t native_tlbie_lock;
+
+	spin_lock(&native_tlbie_lock);
+	asm("ptesync");
+	spin_unlock(&native_tlbie_lock);
+#endif
 
 #ifdef CONFIG_PPC32
 	if (test_perf_event_pending()) {
@@ -584,16 +612,25 @@
 		do_IRQ(regs);
 #endif
 
-	now = get_tb_or_rtc();
-	if (now < decrementer->next_tb) {
-		/* not time for this event yet */
-		now = decrementer->next_tb - now;
-		if (now <= DECREMENTER_MAX)
-			set_dec((int)now);
-		return;
+ 	if (!per_cpu(disarm_decr, cpu)) {
+		now = get_tb_or_rtc();
+		if (now < decrementer->next_tb) {
+			/* not time for this event yet */
+			now = decrementer->next_tb - now;
+			if (now <= DECREMENTER_MAX)
+				set_dec((int)now);
+			return;
+		}
 	}
 	old_regs = set_irq_regs(regs);
+#ifndef CONFIG_IPIPE
+	/*
+	 * The timer interrupt is a virtual one when the I-pipe is
+	 * active, therefore we already called irq_enter() for it (see
+	 * __ipipe_run_isr).
+	 */
 	irq_enter();
+#endif
 
 	calculate_steal_time();
 
@@ -618,7 +655,9 @@
 	}
 #endif
 
+#ifndef CONFIG_IPIPE
 	irq_exit();
+#endif
 	set_irq_regs(old_regs);
 }
 
diff -urN source_powerpc_none/arch/powerpc/kernel/traps.c source_powerpc_none.ipipe/arch/powerpc/kernel/traps.c
--- source_powerpc_none/arch/powerpc/kernel/traps.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/kernel/traps.c	2009-12-22 12:44:08.000000000 -0500
@@ -491,6 +491,9 @@
 {
 	int recover = 0;
 
+	if (ipipe_trap_notify(IPIPE_TRAP_MCE, regs))
+	    	return;
+
 	/* See if any machine dependent calls. In theory, we would want
 	 * to call the CPU first, and call the ppc_md. one if the CPU
 	 * one returns a positive number. However there is existing code
@@ -549,11 +552,17 @@
 	printk("Bad trap at PC: %lx, SR: %lx, vector=%lx\n",
 	       regs->nip, regs->msr, regs->trap);
 
+	if (ipipe_trap_notify(IPIPE_TRAP_UNKNOWN, regs))
+	    	return;
+
 	_exception(SIGTRAP, regs, 0, 0);
 }
 
 void instruction_breakpoint_exception(struct pt_regs *regs)
 {
+	if (ipipe_trap_notify(IPIPE_TRAP_IABR, regs))
+	    	return;
+
 	if (notify_die(DIE_IABR_MATCH, "iabr_match", regs, 5,
 					5, SIGTRAP) == NOTIFY_STOP)
 		return;
@@ -564,6 +573,8 @@
 
 void RunModeException(struct pt_regs *regs)
 {
+	if (ipipe_trap_notify(IPIPE_TRAP_RM, regs))
+	    	return;
 	_exception(SIGTRAP, regs, 0, 0);
 }
 
@@ -571,6 +582,9 @@
 {
 	regs->msr &= ~(MSR_SE | MSR_BE);  /* Turn off 'trace' bits */
 
+	if (ipipe_trap_notify(IPIPE_TRAP_SSTEP, regs))
+	    return;
+
 	if (notify_die(DIE_SSTEP, "single_step", regs, 5,
 					5, SIGTRAP) == NOTIFY_STOP)
 		return;
@@ -590,6 +604,8 @@
 {
 	if (single_stepping(regs)) {
 		clear_single_step(regs);
+		if (ipipe_trap_notify(IPIPE_TRAP_SSTEP, regs))
+		    return;
 		_exception(SIGTRAP, regs, TRAP_TRACE, 0);
 	}
 }
@@ -816,6 +832,9 @@
 	/* We can now get here via a FP Unavailable exception if the core
 	 * has no FPU, in that case the reason flags will be 0 */
 
+	if (ipipe_trap_notify(IPIPE_TRAP_PCE, regs))
+	    	return;
+
 	if (reason & REASON_FP) {
 		/* IEEE FP exception */
 		parse_fpe(regs);
@@ -888,6 +907,9 @@
 {
 	int sig, code, fixed = 0;
 
+	if (ipipe_trap_notify(IPIPE_TRAP_ALIGNMENT, regs))
+	    	return;
+
 	/* we don't implement logging of alignment exceptions */
 	if (!(current->thread.align_ctl & PR_UNALIGN_SIGBUS))
 		fixed = fix_alignment(regs);
@@ -925,6 +947,8 @@
 {
 	printk(KERN_ERR "Non-recoverable exception at PC=%lx MSR=%lx\n",
 	       regs->nip, regs->msr);
+	if (ipipe_trap_notify(IPIPE_TRAP_NREC, regs))
+	    	return;
 	debugger(regs);
 	die("nonrecoverable exception", regs, SIGKILL);
 }
@@ -940,11 +964,16 @@
 {
 	printk(KERN_EMERG "Unrecoverable FP Unavailable Exception "
 			  "%lx at %lx\n", regs->trap, regs->nip);
+	if (ipipe_trap_notify(IPIPE_TRAP_KFPUNAVAIL, regs))
+	    	return;
 	die("Unrecoverable FP Unavailable Exception", regs, SIGABRT);
 }
 
 void altivec_unavailable_exception(struct pt_regs *regs)
 {
+	if (ipipe_trap_notify(IPIPE_TRAP_ALTUNAVAIL, regs))
+	    	return;
+
 	if (user_mode(regs)) {
 		/* A user program has executed an altivec instruction,
 		   but this kernel doesn't support altivec. */
@@ -985,6 +1014,9 @@
 	int errcode;
 #endif
 
+	if (ipipe_trap_notify(IPIPE_TRAP_SOFTEMU, regs))
+	    	return;
+
 	CHECK_FULL_REGS(regs);
 
 	if (!user_mode(regs)) {
@@ -1046,6 +1078,9 @@
 	 * the server behaviour, we thus restart right away with a single step
 	 * instead of stopping here when hitting a BT
 	 */
+	if (ipipe_trap_notify(IPIPE_TRAP_DEBUG, regs))
+	    	return;
+
 	if (debug_status & DBSR_BT) {
 		regs->msr &= ~MSR_DE;
 
@@ -1121,6 +1156,9 @@
 {
 	int err;
 
+	if (ipipe_trap_notify(IPIPE_TRAP_ALTASSIST, regs))
+	    	return;
+
 	if (!user_mode(regs)) {
 		printk(KERN_EMERG "VMX/Altivec assist exception in kernel mode"
 		       " at %lx\n", regs->nip);
@@ -1192,8 +1230,11 @@
 	 * as priv ops, in the future we could try to do
 	 * something smarter
 	 */
-	if (error_code & (ESR_DLK|ESR_ILK))
+	if (error_code & (ESR_DLK|ESR_ILK)) {
+		if (ipipe_trap_notify(IPIPE_TRAP_CACHE, regs))
+			return;
 		_exception(SIGILL, regs, ILL_PRVOPC, regs->nip);
+	}
 	return;
 }
 #endif /* CONFIG_FSL_BOOKE */
@@ -1207,6 +1248,9 @@
 	int code = 0;
 	int err;
 
+	if (ipipe_trap_notify(IPIPE_TRAP_SPE, regs))
+	    	return;
+
 	preempt_disable();
 	if (regs->msr & MSR_SPE)
 		giveup_spe(current);
@@ -1292,6 +1336,8 @@
 {
 	printk(KERN_EMERG "Unrecoverable exception %lx at %lx\n",
 	       regs->trap, regs->nip);
+	if (ipipe_trap_notify(IPIPE_TRAP_NREC, regs))
+	    	return;
 	die("Unrecoverable exception", regs, SIGABRT);
 }
 
diff -urN source_powerpc_none/arch/powerpc/lib/code-patching.c source_powerpc_none.ipipe/arch/powerpc/lib/code-patching.c
--- source_powerpc_none/arch/powerpc/lib/code-patching.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/lib/code-patching.c	2009-12-22 12:44:08.000000000 -0500
@@ -15,17 +15,20 @@
 #include <asm/code-patching.h>
 
 
+notrace
 void patch_instruction(unsigned int *addr, unsigned int instr)
 {
 	*addr = instr;
 	asm ("dcbst 0, %0; sync; icbi 0,%0; sync; isync" : : "r" (addr));
 }
 
+notrace
 void patch_branch(unsigned int *addr, unsigned long target, int flags)
 {
 	patch_instruction(addr, create_branch(addr, target, flags));
 }
 
+notrace
 unsigned int create_branch(const unsigned int *addr,
 			   unsigned long target, int flags)
 {
@@ -46,6 +49,7 @@
 	return instruction;
 }
 
+notrace
 unsigned int create_cond_branch(const unsigned int *addr,
 				unsigned long target, int flags)
 {
@@ -66,21 +70,25 @@
 	return instruction;
 }
 
+notrace
 static unsigned int branch_opcode(unsigned int instr)
 {
 	return (instr >> 26) & 0x3F;
 }
 
+notrace
 static int instr_is_branch_iform(unsigned int instr)
 {
 	return branch_opcode(instr) == 18;
 }
 
+notrace
 static int instr_is_branch_bform(unsigned int instr)
 {
 	return branch_opcode(instr) == 16;
 }
 
+notrace
 int instr_is_relative_branch(unsigned int instr)
 {
 	if (instr & BRANCH_ABSOLUTE)
@@ -89,6 +97,7 @@
 	return instr_is_branch_iform(instr) || instr_is_branch_bform(instr);
 }
 
+notrace
 static unsigned long branch_iform_target(const unsigned int *instr)
 {
 	signed long imm;
@@ -105,6 +114,7 @@
 	return (unsigned long)imm;
 }
 
+notrace
 static unsigned long branch_bform_target(const unsigned int *instr)
 {
 	signed long imm;
@@ -121,6 +131,7 @@
 	return (unsigned long)imm;
 }
 
+notrace
 unsigned long branch_target(const unsigned int *instr)
 {
 	if (instr_is_branch_iform(*instr))
@@ -131,6 +142,7 @@
 	return 0;
 }
 
+notrace
 int instr_is_branch_to_addr(const unsigned int *instr, unsigned long addr)
 {
 	if (instr_is_branch_iform(*instr) || instr_is_branch_bform(*instr))
@@ -139,6 +151,7 @@
 	return 0;
 }
 
+notrace
 unsigned int translate_branch(const unsigned int *dest, const unsigned int *src)
 {
 	unsigned long target;
diff -urN source_powerpc_none/arch/powerpc/lib/feature-fixups.c source_powerpc_none.ipipe/arch/powerpc/lib/feature-fixups.c
--- source_powerpc_none/arch/powerpc/lib/feature-fixups.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/lib/feature-fixups.c	2009-12-22 12:44:08.000000000 -0500
@@ -28,6 +28,7 @@
 	long		alt_end_off;
 };
 
+notrace
 static unsigned int *calc_addr(struct fixup_entry *fcur, long offset)
 {
 	/*
@@ -38,6 +39,7 @@
 	return (unsigned int *)((unsigned long)fcur + offset);
 }
 
+notrace
 static int patch_alt_instruction(unsigned int *src, unsigned int *dest,
 				 unsigned int *alt_start, unsigned int *alt_end)
 {
@@ -61,6 +63,7 @@
 	return 0;
 }
 
+notrace
 static int patch_feature_section(unsigned long value, struct fixup_entry *fcur)
 {
 	unsigned int *start, *end, *alt_start, *alt_end, *src, *dest;
@@ -90,6 +93,7 @@
 	return 0;
 }
 
+notrace
 void do_feature_fixups(unsigned long value, void *fixup_start, void *fixup_end)
 {
 	struct fixup_entry *fcur, *fend;
@@ -110,6 +114,7 @@
 	}
 }
 
+notrace
 void do_lwsync_fixups(unsigned long value, void *fixup_start, void *fixup_end)
 {
 	unsigned int *start, *end, *dest;
diff -urN source_powerpc_none/arch/powerpc/mm/fault.c source_powerpc_none.ipipe/arch/powerpc/mm/fault.c
--- source_powerpc_none/arch/powerpc/mm/fault.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/mm/fault.c	2009-12-22 12:44:08.000000000 -0500
@@ -119,13 +119,18 @@
 			    unsigned long error_code)
 {
 	struct vm_area_struct * vma;
-	struct mm_struct *mm = current->mm;
+	struct mm_struct *mm;
 	siginfo_t info;
 	int code = SEGV_MAPERR;
 	int is_write = 0, ret;
 	int trap = TRAP(regs);
  	int is_exec = trap == 0x400;
 
+	if (ipipe_trap_notify(IPIPE_TRAP_ACCESS,regs))
+	    	return 0;
+
+	mm = current->mm;
+
 #if !(defined(CONFIG_4xx) || defined(CONFIG_BOOKE))
 	/*
 	 * Fortunately the bit assignments in SRR1 for an instruction
diff -urN source_powerpc_none/arch/powerpc/mm/hash_low_32.S source_powerpc_none.ipipe/arch/powerpc/mm/hash_low_32.S
--- source_powerpc_none/arch/powerpc/mm/hash_low_32.S	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/mm/hash_low_32.S	2009-12-22 12:44:08.000000000 -0500
@@ -496,7 +496,11 @@
  *
  * We assume that there is a hash table in use (Hash != 0).
  */
+#ifdef CONFIG_IPIPE
+_GLOBAL(__flush_hash_pages)
+#else
 _GLOBAL(flush_hash_pages)
+#endif
 	tophys(r7,0)
 
 	/*
@@ -531,18 +535,9 @@
 	addi	r6,r6,-1
 	b	1b
 
-	/* Convert context and va to VSID */
-2:	mulli	r3,r3,897*16		/* multiply context by context skew */
-	rlwinm	r0,r4,4,28,31		/* get ESID (top 4 bits of va) */
-	mulli	r0,r0,0x111		/* multiply by ESID skew */
-	add	r3,r3,r0		/* note code below trims to 24 bits */
-
-	/* Construct the high word of the PPC-style PTE (r11) */
-	rlwinm	r11,r3,7,1,24		/* put VSID in 0x7fffff80 bits */
-	rlwimi	r11,r4,10,26,31		/* put in API (abbrev page index) */
-	SET_V(r11)			/* set V (valid) bit */
-
+2:
 #ifdef CONFIG_SMP
+	li	r11,0
 	addis	r9,r7,mmu_hash_lock@domain.hid
 	addi	r9,r9,mmu_hash_lock@domain.hid
 	rlwinm	r8,r1,0,0,(31-THREAD_SHIFT)
@@ -557,10 +552,36 @@
 11:	lwz	r0,0(r9)
 	cmpi	0,r0,0
 	beq	10b
+	mtmsr	r10
+	SYNC_601
+	isync
+	li	r11,1
+	rlwinm	r0,r10,0,17,15		/* clear bit 16 (MSR_EE) */
+	rlwinm	r0,r0,0,28,26		/* clear MSR_DR */
+	mtmsr	r0
+	SYNC_601
+	isync
 	b	11b
 12:	isync
+	cmpwi	r11,0
+	beq	13f
+	li	r0,0
+	stw	r0,0(r9)		/* clear mmu_hash_lock */
+	b	1b
+13:
 #endif
 
+	/* Convert context and va to VSID */
+	mulli	r3,r3,897*16		/* multiply context by context skew */
+	rlwinm	r0,r4,4,28,31		/* get ESID (top 4 bits of va) */
+	mulli	r0,r0,0x111		/* multiply by ESID skew */
+	add	r3,r3,r0		/* note code below trims to 24 bits */
+
+	/* Construct the high word of the PPC-style PTE (r11) */
+	rlwinm	r11,r3,7,1,24		/* put VSID in 0x7fffff80 bits */
+	rlwimi	r11,r4,10,26,31		/* put in API (abbrev page index) */
+	SET_V(r11)			/* set V (valid) bit */
+
 	/*
 	 * Check the _PAGE_HASHPTE bit in the linux PTE.  If it is
 	 * already clear, we're done (for this pte).  If not,
@@ -631,7 +652,7 @@
 
 19:	mtmsr	r10
 	SYNC_601
-	isync
+	sync
 	blr
 
 /*
diff -urN source_powerpc_none/arch/powerpc/mm/hash_native_64.c source_powerpc_none.ipipe/arch/powerpc/mm/hash_native_64.c
--- source_powerpc_none/arch/powerpc/mm/hash_native_64.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/mm/hash_native_64.c	2009-12-22 12:44:08.000000000 -0500
@@ -37,7 +37,7 @@
 
 #define HPTE_LOCK_BIT 3
 
-static DEFINE_SPINLOCK(native_tlbie_lock);
+static IPIPE_DEFINE_SPINLOCK(native_tlbie_lock);
 
 static inline void __tlbie(unsigned long va, int psize, int ssize)
 {
@@ -142,7 +142,7 @@
 			unsigned long vflags, int psize, int ssize)
 {
 	struct hash_pte *hptep = htab_address + hpte_group;
-	unsigned long hpte_v, hpte_r;
+	unsigned long hpte_v, hpte_r, flags;
 	int i;
 
 	if (!(vflags & HPTE_V_BOLTED)) {
@@ -151,6 +151,8 @@
 			hpte_group, va, pa, rflags, vflags, psize);
 	}
 
+	local_irq_save_hw(flags);
+
 	for (i = 0; i < HPTES_PER_GROUP; i++) {
 		if (! (hptep->v & HPTE_V_VALID)) {
 			/* retry with lock held */
@@ -163,8 +165,28 @@
 		hptep++;
 	}
 
-	if (i == HPTES_PER_GROUP)
+	if (i == HPTES_PER_GROUP) {
+		local_irq_restore_hw(flags);
 		return -1;
+	}
+
+#ifdef CONFIG_PPC_PASEMI_A2_WORKAROUNDS
+	/* Workaround for bug 4910: No non-guarded access over IOB */
+	if (pa >= 0x80000000 && pa < 0x100000000)
+		rflags |= _PAGE_GUARDED;
+#endif
+
+#ifdef CONFIG_PPC_PASEMI_A2_WORKAROUNDS
+	/* Workaround for bug 4910: No non-guarded access over IOB */
+	if (pa >= 0x80000000 && pa < 0x100000000)
+		rflags |= _PAGE_GUARDED;
+#endif
+
+#ifdef CONFIG_PPC_PASEMI_A2_WORKAROUNDS
+	/* Workaround for bug 4910: No non-guarded access over IOB */
+	if (pa >= 0x80000000 && pa < 0x100000000)
+		rflags |= _PAGE_GUARDED;
+#endif
 
 	hpte_v = hpte_encode_v(va, psize, ssize) | vflags | HPTE_V_VALID;
 	hpte_r = hpte_encode_r(pa, psize) | rflags;
@@ -183,6 +205,8 @@
 	 */
 	hptep->v = hpte_v;
 
+	local_irq_restore_hw(flags);
+
 	__asm__ __volatile__ ("ptesync" : : : "memory");
 
 	return i | (!!(vflags & HPTE_V_SECONDARY) << 3);
@@ -193,13 +217,15 @@
 	struct hash_pte *hptep;
 	int i;
 	int slot_offset;
-	unsigned long hpte_v;
+	unsigned long hpte_v, flags;
 
 	DBG_LOW("    remove(group=%lx)\n", hpte_group);
 
 	/* pick a random entry to start at */
 	slot_offset = mftb() & 0x7;
 
+	local_irq_save_hw(flags);
+
 	for (i = 0; i < HPTES_PER_GROUP; i++) {
 		hptep = htab_address + hpte_group + slot_offset;
 		hpte_v = hptep->v;
@@ -218,12 +244,16 @@
 		slot_offset &= 0x7;
 	}
 
-	if (i == HPTES_PER_GROUP)
+	if (i == HPTES_PER_GROUP) {
+		local_irq_restore_hw(flags);
 		return -1;
+	}
 
 	/* Invalidate the hpte. NOTE: this also unlocks it */
 	hptep->v = 0;
 
+	local_irq_restore_hw(flags);
+
 	return i;
 }
 
@@ -232,7 +262,7 @@
 				 int local)
 {
 	struct hash_pte *hptep = htab_address + slot;
-	unsigned long hpte_v, want_v;
+	unsigned long hpte_v, want_v, flags;
 	int ret = 0;
 
 	want_v = hpte_encode_v(va, psize, ssize);
@@ -240,6 +270,8 @@
 	DBG_LOW("    update(va=%016lx, avpnv=%016lx, hash=%016lx, newpp=%x)",
 		va, want_v & HPTE_V_AVPN, slot, newpp);
 
+	local_irq_save_hw(flags);
+
 	native_lock_hpte(hptep);
 
 	hpte_v = hptep->v;
@@ -256,6 +288,8 @@
 	}
 	native_unlock_hpte(hptep);
 
+	local_irq_restore_hw(flags);
+
 	/* Ensure it is out of the tlb too. */
 	tlbie(va, psize, ssize, local);
 
@@ -326,10 +360,10 @@
 	unsigned long want_v;
 	unsigned long flags;
 
-	local_irq_save(flags);
-
 	DBG_LOW("    invalidate(va=%016lx, hash: %x)\n", va, slot);
 
+	local_irq_save(flags);
+
 	want_v = hpte_encode_v(va, psize, ssize);
 	native_lock_hpte(hptep);
 	hpte_v = hptep->v;
diff -urN source_powerpc_none/arch/powerpc/mm/hash_utils_64.c source_powerpc_none.ipipe/arch/powerpc/mm/hash_utils_64.c
--- source_powerpc_none/arch/powerpc/mm/hash_utils_64.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/mm/hash_utils_64.c	2009-12-22 12:44:08.000000000 -0500
@@ -111,7 +111,7 @@
 #ifdef CONFIG_DEBUG_PAGEALLOC
 static u8 *linear_map_hash_slots;
 static unsigned long linear_map_hash_count;
-static DEFINE_SPINLOCK(linear_map_hash_lock);
+static IPIPE_DEFINE_SPINLOCK(linear_map_hash_lock);
 #endif /* CONFIG_DEBUG_PAGEALLOC */
 
 /* There are definitions of page sizes arrays to be used when none
@@ -894,6 +894,7 @@
 	const struct cpumask *tmp;
 	int rc, user_region = 0, local = 0;
 	int psize, ssize;
+	unsigned long flags;
 
 	DBG_LOW("hash_page(ea=%016lx, access=%lx, trap=%lx\n",
 		ea, access, trap);
@@ -1012,6 +1013,9 @@
 #endif
 		}
 	}
+
+	local_irq_save_hw(flags);
+
 	if (user_region) {
 		if (psize != get_paca_psize(ea)) {
 			get_paca()->context = mm->context;
@@ -1023,6 +1027,10 @@
 			mmu_psize_defs[mmu_vmalloc_psize].sllp;
 		slb_vmalloc_update();
 	}
+
+	local_irq_restore_hw(flags);
+#else
+	(void)flags;
 #endif /* CONFIG_PPC_64K_PAGES */
 
 #ifdef CONFIG_PPC_HAS_HASH_64K
@@ -1155,6 +1163,10 @@
  */
 void low_hash_fault(struct pt_regs *regs, unsigned long address, int rc)
 {
+	if (ipipe_trap_notify(IPIPE_TRAP_ACCESS, regs))
+		/* Not all access faults go through do_page_fault(). */
+	    	return;
+
 	if (user_mode(regs)) {
 #ifdef CONFIG_PPC_SUBPAGE_PROT
 		if (rc == -2)
diff -urN source_powerpc_none/arch/powerpc/mm/mmu_context_nohash.c source_powerpc_none.ipipe/arch/powerpc/mm/mmu_context_nohash.c
--- source_powerpc_none/arch/powerpc/mm/mmu_context_nohash.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/mm/mmu_context_nohash.c	2009-12-22 12:44:08.000000000 -0500
@@ -56,7 +56,7 @@
 static unsigned long *context_map;
 static unsigned long *stale_map[NR_CPUS];
 static struct mm_struct **context_mm;
-static DEFINE_SPINLOCK(context_lock);
+static IPIPE_DEFINE_SPINLOCK(context_lock);
 
 #define CTX_MAP_SIZE	\
 	(sizeof(unsigned long) * (last_context / BITS_PER_LONG + 1))
@@ -138,7 +138,7 @@
 static unsigned int steal_context_up(unsigned int id)
 {
 	struct mm_struct *mm;
-	int cpu = smp_processor_id();
+	int cpu = ipipe_processor_id();
 
 	/* Pick up the victim mm */
 	mm = context_mm[id];
@@ -190,9 +190,10 @@
 
 void switch_mmu_context(struct mm_struct *prev, struct mm_struct *next)
 {
-	unsigned int i, id, cpu = smp_processor_id();
-	unsigned long *map;
+	unsigned int i, id, cpu = ipipe_processor_id();
+	unsigned long *map, flags;
 
+	local_irq_save_hw_cond(flags);
 	/* No lockless fast path .. yet */
 	spin_lock(&context_lock);
 
@@ -279,6 +280,7 @@
 	pr_hardcont(" -> %d\n", id);
 	set_context(id, next->pgd);
 	spin_unlock(&context_lock);
+	local_irq_restore_hw_cond(flags);
 }
 
 /*
diff -urN source_powerpc_none/arch/powerpc/mm/slb.c source_powerpc_none.ipipe/arch/powerpc/mm/slb.c
--- source_powerpc_none/arch/powerpc/mm/slb.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/mm/slb.c	2009-12-22 12:44:08.000000000 -0500
@@ -133,16 +133,27 @@
 
 void slb_flush_and_rebolt(void)
 {
+	unsigned long flags;
 
+#ifndef CONFIG_IPIPE
 	WARN_ON(!irqs_disabled());
+#endif
 
 	/*
 	 * We can't take a PMU exception in the following code, so hard
 	 * disable interrupts.
 	 */
+#ifdef CONFIG_IPIPE
+	local_irq_save_hw(flags);
+#else
 	hard_irq_disable();
+	(void)flags;
+#endif
 
 	__slb_flush_and_rebolt();
+#ifdef CONFIG_IPIPE
+	local_irq_restore_hw(flags);
+#endif
 	get_paca()->slb_cache_ptr = 0;
 }
 
diff -urN source_powerpc_none/arch/powerpc/mm/tlb_hash32.c source_powerpc_none.ipipe/arch/powerpc/mm/tlb_hash32.c
--- source_powerpc_none/arch/powerpc/mm/tlb_hash32.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/mm/tlb_hash32.c	2009-12-22 12:44:08.000000000 -0500
@@ -100,6 +100,37 @@
 #define FINISH_FLUSH	do { } while (0)
 #endif
 
+#ifdef CONFIG_IPIPE
+
+int __flush_hash_pages(unsigned context, unsigned long va,
+		       unsigned long pmdval, int count);
+
+int flush_hash_pages(unsigned context, unsigned long va,
+		     unsigned long pmdval, int count)
+{
+	int bulk, ret = 0;
+	/*
+	 * Submitting flush requests on insanely large PTE counts
+	 * (e.g. HIGHMEM) may cause severe latency penalty on high
+	 * priority domains since this must be done with hw interrupts
+	 * off (typically, peaks over 400 us have been observed on
+	 * 864xD). We split flush requests in bulks of 64 PTEs to
+	 * prevent that; the modified assembly helper which performs
+	 * the actual flush (__flush_hash_pages()) will spin on the
+	 * mmu_lock with interrupts enabled to further reduce latency.
+	 */
+	while (count > 0) {
+		bulk = count > 64 ? 64 : count;
+		ret |= __flush_hash_pages(context, va, pmdval, bulk);
+		va += (bulk << PAGE_SHIFT);
+		count -= bulk;
+	}
+
+	return ret;
+}
+
+#endif /* CONFIG_IPIPE */
+
 static void flush_range(struct mm_struct *mm, unsigned long start,
 			unsigned long end)
 {
diff -urN source_powerpc_none/arch/powerpc/platforms/44x/canyonlands-sata.c source_powerpc_none.ipipe/arch/powerpc/platforms/44x/canyonlands-sata.c
--- source_powerpc_none/arch/powerpc/platforms/44x/canyonlands-sata.c	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/platforms/44x/canyonlands-sata.c	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,123 @@
+/*
+ * AMCC Canyonlands SATA wrapper
+ *
+ * Copyright 2008 DENX Software Engineering, Stefan Roese <sr@domain.hid>
+ *
+ * Extract the resources (MEM & IRQ) from the dts file and put them
+ * into the platform-device struct for usage in the platform-device
+ * SATA driver.
+ *
+ */
+
+#include <linux/platform_device.h>
+#include <linux/of_platform.h>
+
+/*
+ * Resource template will be filled dynamically with the values
+ * extracted from the dts file
+ */
+static struct resource sata_resources[] = {
+	[0] = {
+		/* 460EX SATA registers */
+		.flags  = IORESOURCE_MEM,
+	},
+	[1] = {
+		/* 460EX AHBDMA registers */
+		.flags  = IORESOURCE_MEM,
+	},
+	[2] = {
+		/* 460EX SATA IRQ */
+		.flags  = IORESOURCE_IRQ,
+	},
+	[3] = {
+		/* 460EX AHBDMA IRQ */
+		.flags  = IORESOURCE_IRQ,
+	},
+};
+
+static u64 dma_mask = 0xffffffffULL;
+
+static struct platform_device sata_device = {
+        .name = "sata-dwc",
+        .id = 0,
+        .num_resources = ARRAY_SIZE(sata_resources),
+        .resource = sata_resources,
+        .dev = {
+                .dma_mask = &dma_mask,
+                .coherent_dma_mask = 0xffffffffULL,
+        }
+};
+
+static struct platform_device *ppc460ex_devs[] __initdata = {
+        &sata_device,
+};
+
+static int __devinit ppc460ex_sata_probe(struct of_device *ofdev,
+					 const struct of_device_id *match)
+{
+	struct device_node *np = ofdev->node;
+	struct resource res;
+	const char *val;
+
+	/*
+	 * Check if device is enabled
+	 */
+	val = of_get_property(np, "status", NULL);
+	if (val && !strcmp(val, "disabled")) {
+		printk(KERN_INFO "SATA port disabled via device-tree\n");
+		return 0;
+	}
+
+	/*
+	 * Extract register address reange from device tree and put it into
+	 * the platform device structure
+	 */
+	if (of_address_to_resource(np, 0, &res)) {
+		printk(KERN_ERR "%s: Can't get SATA register address\n", __func__);
+		return -ENOMEM;
+	}
+	sata_resources[0].start = res.start;
+	sata_resources[0].end = res.end;
+
+	if (of_address_to_resource(np, 1, &res)) {
+		printk(KERN_ERR "%s: Can't get AHBDMA register address\n", __func__);
+		return -ENOMEM;
+	}
+	sata_resources[1].start = res.start;
+	sata_resources[1].end = res.end;
+
+	/*
+	 * Extract IRQ number(s) from device tree and put them into
+	 * the platform device structure
+	 */
+	sata_resources[2].start = sata_resources[2].end =
+		irq_of_parse_and_map(np, 0);
+	sata_resources[3].start = sata_resources[3].end =
+		irq_of_parse_and_map(np, 1);
+
+	return platform_add_devices(ppc460ex_devs, ARRAY_SIZE(ppc460ex_devs));
+}
+
+static int __devexit ppc460ex_sata_remove(struct of_device *ofdev)
+{
+	/* Nothing to do here */
+	return 0;
+}
+
+static const struct of_device_id ppc460ex_sata_match[] = {
+	{ .compatible = "amcc,sata-460ex", },
+	{}
+};
+
+static struct of_platform_driver ppc460ex_sata_driver = {
+	.name = "sata-460ex",
+	.match_table = ppc460ex_sata_match,
+	.probe = ppc460ex_sata_probe,
+	.remove = ppc460ex_sata_remove,
+};
+
+static int __init ppc460ex_sata_init(void)
+{
+	return of_register_platform_driver(&ppc460ex_sata_driver);
+}
+device_initcall(ppc460ex_sata_init);
diff -urN source_powerpc_none/arch/powerpc/platforms/82xx/pq2ads-pci-pic.c source_powerpc_none.ipipe/arch/powerpc/platforms/82xx/pq2ads-pci-pic.c
--- source_powerpc_none/arch/powerpc/platforms/82xx/pq2ads-pci-pic.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/platforms/82xx/pq2ads-pci-pic.c	2009-12-22 12:44:08.000000000 -0500
@@ -24,7 +24,7 @@
 
 #include "pq2.h"
 
-static DEFINE_SPINLOCK(pci_pic_lock);
+static IPIPE_DEFINE_SPINLOCK(pci_pic_lock);
 
 struct pq2ads_pci_pic {
 	struct device_node *node;
@@ -38,18 +38,42 @@
 
 #define NUM_IRQS 32
 
+static inline void __pq2ads_pci_mask_irq(struct pq2ads_pci_pic *priv,
+					 unsigned int irq)
+{
+	setbits32(&priv->regs->mask, 1 << irq);
+	mb();
+}
+
+static inline void __pq2ads_pci_unmask_irq(struct pq2ads_pci_pic *priv,
+					   unsigned int irq)
+{
+	clrbits32(&priv->regs->mask, 1 << irq);
+}
+
 static void pq2ads_pci_mask_irq(unsigned int virq)
 {
 	struct pq2ads_pci_pic *priv = get_irq_chip_data(virq);
 	int irq = NUM_IRQS - virq_to_hw(virq) - 1;
+	unsigned long flags;
 
 	if (irq != -1) {
-		unsigned long flags;
 		spin_lock_irqsave(&pci_pic_lock, flags);
+		__pq2ads_pci_mask_irq(priv, irq);
+		ipipe_irq_lock(virq);
+		spin_unlock_irqrestore(&pci_pic_lock, flags);
+	}
+}
 
-		setbits32(&priv->regs->mask, 1 << irq);
-		mb();
+static void pq2ads_pci_mask_ack_irq(unsigned int virq)
+{
+	struct pq2ads_pci_pic *priv = get_irq_chip_data(virq);
+	int irq = NUM_IRQS - virq_to_hw(virq) - 1;
 
+	if (irq != -1) {
+		unsigned long flags;
+		spin_lock_irqsave(&pci_pic_lock, flags);
+		__pq2ads_pci_mask_irq(priv, irq);
 		spin_unlock_irqrestore(&pci_pic_lock, flags);
 	}
 }
@@ -58,12 +82,12 @@
 {
 	struct pq2ads_pci_pic *priv = get_irq_chip_data(virq);
 	int irq = NUM_IRQS - virq_to_hw(virq) - 1;
+	unsigned long flags;
 
 	if (irq != -1) {
-		unsigned long flags;
-
 		spin_lock_irqsave(&pci_pic_lock, flags);
-		clrbits32(&priv->regs->mask, 1 << irq);
+		__pq2ads_pci_unmask_irq(priv, irq);
+		ipipe_irq_unlock(virq);
 		spin_unlock_irqrestore(&pci_pic_lock, flags);
 	}
 }
@@ -73,7 +97,7 @@
 	.name = "PQ2 ADS PCI",
 	.end = pq2ads_pci_unmask_irq,
 	.mask = pq2ads_pci_mask_irq,
-	.mask_ack = pq2ads_pci_mask_irq,
+	.mask_ack = pq2ads_pci_mask_ack_irq,
 	.ack = pq2ads_pci_mask_irq,
 	.unmask = pq2ads_pci_unmask_irq,
 	.enable = pq2ads_pci_unmask_irq,
@@ -98,7 +122,7 @@
 		for (bit = 0; pend != 0; ++bit, pend <<= 1) {
 			if (pend & 0x80000000) {
 				int virq = irq_linear_revmap(priv->host, bit);
-				generic_handle_irq(virq);
+				ipipe_handle_chained_irq(virq);
 			}
 		}
 	}
diff -urN source_powerpc_none/arch/powerpc/platforms/85xx/tqm85xx.c source_powerpc_none.ipipe/arch/powerpc/platforms/85xx/tqm85xx.c
--- source_powerpc_none/arch/powerpc/platforms/85xx/tqm85xx.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/platforms/85xx/tqm85xx.c	2009-12-22 12:44:08.000000000 -0500
@@ -46,10 +46,10 @@
 {
 	int cascade_irq;
 
-	while ((cascade_irq = cpm2_get_irq()) >= 0)
-		generic_handle_irq(cascade_irq);
-
 	desc->chip->eoi(irq);
+
+	while ((cascade_irq = cpm2_get_irq()) >= 0)
+		ipipe_handle_chained_irq(cascade_irq);
 }
 #endif /* CONFIG_CPM2 */
 
diff -urN source_powerpc_none/arch/powerpc/platforms/cell/spu_base.c source_powerpc_none.ipipe/arch/powerpc/platforms/cell/spu_base.c
--- source_powerpc_none/arch/powerpc/platforms/cell/spu_base.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/platforms/cell/spu_base.c	2009-12-22 12:44:08.000000000 -0500
@@ -57,7 +57,7 @@
 /*
  * Protects cbe_spu_info and spu->number.
  */
-static DEFINE_SPINLOCK(spu_lock);
+static IPIPE_DEFINE_SPINLOCK(spu_lock);
 
 /*
  * List of all spus in the system.
diff -urN source_powerpc_none/arch/powerpc/platforms/iseries/irq.c source_powerpc_none.ipipe/arch/powerpc/platforms/iseries/irq.c
--- source_powerpc_none/arch/powerpc/platforms/iseries/irq.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/platforms/iseries/irq.c	2009-12-22 12:44:08.000000000 -0500
@@ -80,7 +80,7 @@
 	} data;
 };
 
-static DEFINE_SPINLOCK(pending_irqs_lock);
+static IPIPE_DEFINE_SPINLOCK(pending_irqs_lock);
 static int num_pending_irqs;
 static int pending_irqs[NR_IRQS];
 
diff -urN source_powerpc_none/arch/powerpc/platforms/powermac/pic.c source_powerpc_none.ipipe/arch/powerpc/platforms/powermac/pic.c
--- source_powerpc_none/arch/powerpc/platforms/powermac/pic.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/platforms/powermac/pic.c	2009-12-22 12:44:08.000000000 -0500
@@ -57,7 +57,7 @@
 static int max_real_irqs;
 static u32 level_mask[4];
 
-static DEFINE_SPINLOCK(pmac_pic_lock);
+static IPIPE_DEFINE_SPINLOCK(pmac_pic_lock);
 
 #define NR_MASK_WORDS	((NR_IRQS + 31) / 32)
 static unsigned long ppc_lost_interrupts[NR_MASK_WORDS];
diff -urN source_powerpc_none/arch/powerpc/platforms/ps3/htab.c source_powerpc_none.ipipe/arch/powerpc/platforms/ps3/htab.c
--- source_powerpc_none/arch/powerpc/platforms/ps3/htab.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/platforms/ps3/htab.c	2009-12-22 12:44:08.000000000 -0500
@@ -41,7 +41,7 @@
 };
 
 
-static DEFINE_SPINLOCK(ps3_htab_lock);
+static IPIPE_DEFINE_SPINLOCK(ps3_htab_lock);
 
 static long ps3_hpte_insert(unsigned long hpte_group, unsigned long va,
 	unsigned long pa, unsigned long rflags, unsigned long vflags,
diff -urN source_powerpc_none/arch/powerpc/platforms/ps3/interrupt.c source_powerpc_none.ipipe/arch/powerpc/platforms/ps3/interrupt.c
--- source_powerpc_none/arch/powerpc/platforms/ps3/interrupt.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/platforms/ps3/interrupt.c	2009-12-22 12:44:08.000000000 -0500
@@ -74,7 +74,7 @@
 		u64 unused_2[3];
 	};
 	u64 ipi_debug_brk_mask;
-	spinlock_t lock;
+	ipipe_spinlock_t lock;
 };
 
 /**
diff -urN source_powerpc_none/arch/powerpc/platforms/pseries/lpar.c source_powerpc_none.ipipe/arch/powerpc/platforms/pseries/lpar.c
--- source_powerpc_none/arch/powerpc/platforms/pseries/lpar.c	2009-12-21 17:25:55.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/platforms/pseries/lpar.c	2009-12-22 12:44:08.000000000 -0500
@@ -339,7 +339,7 @@
 	return (slot & 7) | (!!(vflags & HPTE_V_SECONDARY) << 3);
 }
 
-static DEFINE_SPINLOCK(pSeries_lpar_tlbie_lock);
+static IPIPE_DEFINE_SPINLOCK(pSeries_lpar_tlbie_lock);
 
 static long pSeries_lpar_hpte_remove(unsigned long hpte_group)
 {
diff -urN source_powerpc_none/arch/powerpc/sysdev/cpm2_pic.c source_powerpc_none.ipipe/arch/powerpc/sysdev/cpm2_pic.c
--- source_powerpc_none/arch/powerpc/sysdev/cpm2_pic.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/sysdev/cpm2_pic.c	2009-12-22 12:44:08.000000000 -0500
@@ -82,44 +82,61 @@
 {
 	int	bit, word;
 	unsigned int irq_nr = virq_to_hw(virq);
+	unsigned long flags;
 
 	bit = irq_to_siubit[irq_nr];
 	word = irq_to_siureg[irq_nr];
 
+	local_irq_save_hw_cond(flags);
+	ipipe_irq_lock(virq);
 	ppc_cached_irq_mask[word] &= ~(1 << bit);
 	out_be32(&cpm2_intctl->ic_simrh + word, ppc_cached_irq_mask[word]);
+	local_irq_restore_hw_cond(flags);
 }
 
 static void cpm2_unmask_irq(unsigned int virq)
 {
 	int	bit, word;
 	unsigned int irq_nr = virq_to_hw(virq);
+	unsigned long flags;
 
 	bit = irq_to_siubit[irq_nr];
 	word = irq_to_siureg[irq_nr];
 
+	local_irq_save_hw_cond(flags);
 	ppc_cached_irq_mask[word] |= 1 << bit;
 	out_be32(&cpm2_intctl->ic_simrh + word, ppc_cached_irq_mask[word]);
+	ipipe_irq_unlock(virq);
+	local_irq_restore_hw_cond(flags);
 }
 
-static void cpm2_ack(unsigned int virq)
+static void cpm2_mask_ack(unsigned int virq)
 {
 	int	bit, word;
 	unsigned int irq_nr = virq_to_hw(virq);
+	unsigned long flags;
 
 	bit = irq_to_siubit[irq_nr];
 	word = irq_to_siureg[irq_nr];
 
+	local_irq_save_hw_cond(flags);
+	ppc_cached_irq_mask[word] &= ~(1 << bit);
+	out_be32(&cpm2_intctl->ic_simrh + word, ppc_cached_irq_mask[word]);
 	out_be32(&cpm2_intctl->ic_sipnrh + word, 1 << bit);
+	local_irq_restore_hw_cond(flags);
 }
 
 static void cpm2_end_irq(unsigned int virq)
 {
 	int	bit, word;
 	unsigned int irq_nr = virq_to_hw(virq);
+	unsigned long flags;
 
-	if (!(irq_desc[irq_nr].status & (IRQ_DISABLED|IRQ_INPROGRESS))
-			&& irq_desc[irq_nr].action) {
+	local_irq_save_hw_cond(flags);
+	
+	if (!__ipipe_root_domain_p ||
+	    (!(irq_desc[irq_nr].status & (IRQ_DISABLED|IRQ_INPROGRESS))
+	     && irq_desc[irq_nr].action)) {
 
 		bit = irq_to_siubit[irq_nr];
 		word = irq_to_siureg[irq_nr];
@@ -133,6 +150,8 @@
 		 */
 		mb();
 	}
+
+	local_irq_restore_hw_cond(flags);
 }
 
 static int cpm2_set_irq_type(unsigned int virq, unsigned int flow_type)
@@ -185,7 +204,7 @@
 	.typename = " CPM2 SIU ",
 	.mask = cpm2_mask_irq,
 	.unmask = cpm2_unmask_irq,
-	.ack = cpm2_ack,
+	.mask_ack = cpm2_mask_ack,
 	.eoi = cpm2_end_irq,
 	.set_type = cpm2_set_irq_type,
 };
diff -urN source_powerpc_none/arch/powerpc/sysdev/i8259.c source_powerpc_none.ipipe/arch/powerpc/sysdev/i8259.c
--- source_powerpc_none/arch/powerpc/sysdev/i8259.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/sysdev/i8259.c	2009-12-22 12:44:08.000000000 -0500
@@ -23,7 +23,7 @@
 #define cached_A1 (cached_8259[0])
 #define cached_21 (cached_8259[1])
 
-static DEFINE_SPINLOCK(i8259_lock);
+static IPIPE_DEFINE_SPINLOCK(i8259_lock);
 
 static struct irq_host *i8259_host;
 
diff -urN source_powerpc_none/arch/powerpc/sysdev/ipic.c source_powerpc_none.ipipe/arch/powerpc/sysdev/ipic.c
--- source_powerpc_none/arch/powerpc/sysdev/ipic.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/sysdev/ipic.c	2009-12-22 12:44:08.000000000 -0500
@@ -32,7 +32,7 @@
 
 static struct ipic * primary_ipic;
 static struct irq_chip ipic_level_irq_chip, ipic_edge_irq_chip;
-static DEFINE_SPINLOCK(ipic_lock);
+static IPIPE_DEFINE_SPINLOCK(ipic_lock);
 
 static struct ipic_info ipic_info[] = {
 	[1] = {
diff -urN source_powerpc_none/arch/powerpc/sysdev/mpc8xx_pic.c source_powerpc_none.ipipe/arch/powerpc/sysdev/mpc8xx_pic.c
--- source_powerpc_none/arch/powerpc/sysdev/mpc8xx_pic.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/sysdev/mpc8xx_pic.c	2009-12-22 12:44:08.000000000 -0500
@@ -29,24 +29,30 @@
 {
 	int	bit, word;
 	unsigned int irq_nr = (unsigned int)irq_map[virq].hwirq;
+	unsigned long flags;
 
 	bit = irq_nr & 0x1f;
 	word = irq_nr >> 5;
 
+	local_irq_save_hw_cond(flags);
 	ppc_cached_irq_mask[word] |= (1 << (31-bit));
 	out_be32(&siu_reg->sc_simask, ppc_cached_irq_mask[word]);
+	local_irq_restore_hw_cond(flags);
 }
 
 static void mpc8xx_mask_irq(unsigned int virq)
 {
 	int	bit, word;
 	unsigned int irq_nr = (unsigned int)irq_map[virq].hwirq;
+	unsigned long flags;
 
 	bit = irq_nr & 0x1f;
 	word = irq_nr >> 5;
 
+	local_irq_save_hw_cond(flags);
 	ppc_cached_irq_mask[word] &= ~(1 << (31-bit));
 	out_be32(&siu_reg->sc_simask, ppc_cached_irq_mask[word]);
+	local_irq_save_hw_cond(flags);
 }
 
 static void mpc8xx_ack(unsigned int virq)
diff -urN source_powerpc_none/arch/powerpc/sysdev/mpic.c source_powerpc_none.ipipe/arch/powerpc/sysdev/mpic.c
--- source_powerpc_none/arch/powerpc/sysdev/mpic.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/sysdev/mpic.c	2009-12-22 12:44:08.000000000 -0500
@@ -46,7 +46,7 @@
 
 static struct mpic *mpics;
 static struct mpic *mpic_primary;
-static DEFINE_SPINLOCK(mpic_lock);
+static IPIPE_DEFINE_SPINLOCK(mpic_lock);
 
 #ifdef CONFIG_PPC32	/* XXX for now */
 #ifdef CONFIG_IRQ_ALL_CPUS
@@ -670,33 +670,44 @@
  */
 
 
-void mpic_unmask_irq(unsigned int irq)
+void __mpic_unmask_irq(unsigned int irq)
 {
 	unsigned int loops = 100000;
 	struct mpic *mpic = mpic_from_irq(irq);
 	unsigned int src = mpic_irq_to_hw(irq);
 
-	DBG("%p: %s: enable_irq: %d (src %d)\n", mpic, mpic->name, irq, src);
-
 	mpic_irq_write(src, MPIC_INFO(IRQ_VECTOR_PRI),
 		       mpic_irq_read(src, MPIC_INFO(IRQ_VECTOR_PRI)) &
 		       ~MPIC_VECPRI_MASK);
 	/* make sure mask gets to controller before we return to user */
 	do {
 		if (!loops--) {
-			printk(KERN_ERR "mpic_enable_irq timeout\n");
+			printk(KERN_ERR "mpic_unmask_irq timeout\n");
 			break;
 		}
 	} while(mpic_irq_read(src, MPIC_INFO(IRQ_VECTOR_PRI)) & MPIC_VECPRI_MASK);
 }
 
-void mpic_mask_irq(unsigned int irq)
+void mpic_unmask_irq(unsigned int irq)
 {
-	unsigned int loops = 100000;
+#ifdef DEBUG
 	struct mpic *mpic = mpic_from_irq(irq);
-	unsigned int src = mpic_irq_to_hw(irq);
+#endif
+	unsigned long flags;
+
+	DBG("%p: %s: unmask_irq: %d (src %d)\n", mpic, mpic->name, irq, src);
+
+	spin_lock_irqsave(&mpic_lock, flags);
+	__mpic_unmask_irq(irq);
+	ipipe_irq_unlock(irq);
+	spin_unlock_irqrestore(&mpic_lock, flags);
+}
 
-	DBG("%s: disable_irq: %d (src %d)\n", mpic->name, irq, src);
+static inline void __mpic_mask_irq(unsigned int irq)
+{
+	struct mpic *mpic = mpic_from_irq(irq);
+	unsigned int src = mpic_irq_to_hw(irq);
+	unsigned int loops = 100000;
 
 	mpic_irq_write(src, MPIC_INFO(IRQ_VECTOR_PRI),
 		       mpic_irq_read(src, MPIC_INFO(IRQ_VECTOR_PRI)) |
@@ -705,15 +716,31 @@
 	/* make sure mask gets to controller before we return to user */
 	do {
 		if (!loops--) {
-			printk(KERN_ERR "mpic_enable_irq timeout\n");
+			printk(KERN_ERR "mpic_mask_irq timeout, irq %u\n", irq);
 			break;
 		}
 	} while(!(mpic_irq_read(src, MPIC_INFO(IRQ_VECTOR_PRI)) & MPIC_VECPRI_MASK));
 }
 
+void mpic_mask_irq(unsigned int irq)
+{
+#ifdef DEBUG
+	struct mpic *mpic = mpic_from_irq(irq);
+#endif
+	unsigned long flags;
+
+	DBG("%s: mask_irq: irq %u (src %d)\n", mpic->name, irq, mpic_irq_to_hw(irq));
+
+	spin_lock_irqsave(&mpic_lock, flags);
+	__mpic_mask_irq(irq);
+	ipipe_irq_lock(irq);
+	spin_unlock_irqrestore(&mpic_lock, flags);
+}
+
 void mpic_end_irq(unsigned int irq)
 {
 	struct mpic *mpic = mpic_from_irq(irq);
+	unsigned long flags;
 
 #ifdef DEBUG_IRQ
 	DBG("%s: end_irq: %d\n", mpic->name, irq);
@@ -723,6 +750,14 @@
 	 * latched another edge interrupt coming in anyway
 	 */
 
+#ifdef CONFIG_IPIPE
+	spin_lock_irqsave(&mpic_lock, flags);
+	if (!(irq_desc[irq].status & IRQ_NOREQUEST))
+		__mpic_mask_irq(irq);
+	spin_unlock_irqrestore(&mpic_lock, flags);
+#else
+	(void)flags;
+#endif
 	mpic_eoi(mpic);
 }
 
@@ -732,8 +767,11 @@
 {
 	struct mpic *mpic = mpic_from_irq(irq);
 	unsigned int src = mpic_irq_to_hw(irq);
+	unsigned long flags;
 
-	mpic_unmask_irq(irq);
+	spin_lock_irqsave(&mpic_lock, flags);
+	__mpic_unmask_irq(irq);
+	spin_unlock_irqrestore(&mpic_lock, flags);
 
 	if (irq_desc[irq].status & IRQ_LEVEL)
 		mpic_ht_end_irq(mpic, src);
@@ -763,9 +801,18 @@
 {
 	struct mpic *mpic = mpic_from_irq(irq);
 	unsigned int src = mpic_irq_to_hw(irq);
+	unsigned long flags;
 
 #ifdef DEBUG_IRQ
-	DBG("%s: end_irq: %d\n", mpic->name, irq);
+	DBG("%s: end_ht_irq: %d\n", mpic->name, irq);
+#endif
+
+#ifdef CONFIG_IPIPE
+	spin_lock_irqsave(&mpic_lock, flags);
+	__mpic_mask_irq(irq);
+	spin_unlock_irqrestore(&mpic_lock, flags);
+#else
+	(void)flags;
 #endif
 	/* We always EOI on end_irq() even for edge interrupts since that
 	 * should only lower the priority, the MPIC should have properly
@@ -784,9 +831,12 @@
 {
 	struct mpic *mpic = mpic_from_ipi(irq);
 	unsigned int src = mpic_irq_to_hw(irq) - mpic->ipi_vecs[0];
+	unsigned long flags;
 
-	DBG("%s: enable_ipi: %d (ipi %d)\n", mpic->name, irq, src);
+	DBG("%s: unmask_ipi: %d (ipi %d)\n", mpic->name, irq, src);
+	spin_lock_irqsave(&mpic_lock, flags);
 	mpic_ipi_write(src, mpic_ipi_read(src) & ~MPIC_VECPRI_MASK);
+	spin_unlock_irqrestore(&mpic_lock, flags);
 }
 
 static void mpic_mask_ipi(unsigned int irq)
@@ -858,6 +908,7 @@
 	unsigned int src = mpic_irq_to_hw(virq);
 	struct irq_desc *desc = get_irq_desc(virq);
 	unsigned int vecpri, vold, vnew;
+	unsigned long flags;
 
 	DBG("mpic: set_irq_type(mpic:@%p,virq:%d,src:0x%x,type:0x%x)\n",
 	    mpic, virq, src, flow_type);
@@ -882,6 +933,8 @@
 	else
 		vecpri = mpic_type_to_vecpri(mpic, flow_type);
 
+	local_irq_save_hw_cond(flags);
+
 	vold = mpic_irq_read(src, MPIC_INFO(IRQ_VECTOR_PRI));
 	vnew = vold & ~(MPIC_INFO(VECPRI_POLARITY_MASK) |
 			MPIC_INFO(VECPRI_SENSE_MASK));
@@ -889,6 +942,8 @@
 	if (vold != vnew)
 		mpic_irq_write(src, MPIC_INFO(IRQ_VECTOR_PRI), vnew);
 
+	local_irq_restore_hw_cond(flags);
+
 	return 0;
 }
 
@@ -1576,6 +1631,7 @@
 }
 
 #ifdef CONFIG_SMP
+
 void mpic_request_ipis(void)
 {
 	struct mpic *mpic = mpic_primary;
diff -urN source_powerpc_none/arch/powerpc/sysdev/qe_lib/qe_ic.c source_powerpc_none.ipipe/arch/powerpc/sysdev/qe_lib/qe_ic.c
--- source_powerpc_none/arch/powerpc/sysdev/qe_lib/qe_ic.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/sysdev/qe_lib/qe_ic.c	2009-12-22 12:44:08.000000000 -0500
@@ -33,7 +33,7 @@
 
 #include "qe_ic.h"
 
-static DEFINE_SPINLOCK(qe_ic_lock);
+static IPIPE_DEFINE_SPINLOCK(qe_ic_lock);
 
 static struct qe_ic_info qe_ic_info[] = {
 	[1] = {
@@ -236,6 +236,20 @@
 	spin_unlock_irqrestore(&qe_ic_lock, flags);
 }
 
+#ifdef CONFIG_IPIPE
+
+void __ipipe_qe_ic_cascade_irq(struct qe_ic *qe_ic, unsigned int virq)
+{
+	
+	struct pt_regs regs;    /* Contents not used. */
+
+	ipipe_trace_irq_entry(virq);
+        __ipipe_handle_irq(virq, &regs);
+        ipipe_trace_irq_exit(virq);
+}
+
+#endif
+
 static struct irq_chip qe_ic_irq_chip = {
 	.typename = " QEIC  ",
 	.unmask = qe_ic_unmask_irq,
diff -urN source_powerpc_none/arch/powerpc/sysdev/tsi108_pci.c source_powerpc_none.ipipe/arch/powerpc/sysdev/tsi108_pci.c
--- source_powerpc_none/arch/powerpc/sysdev/tsi108_pci.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/sysdev/tsi108_pci.c	2009-12-22 12:44:08.000000000 -0500
@@ -250,7 +250,9 @@
 {
 	u_int irp_cfg;
 	int int_line = (irq - IRQ_PCI_INTAD_BASE);
+	unsigned long flags;
 
+	local_irq_save_hw_cond(flags);
 	irp_cfg = tsi108_read_reg(TSI108_PCI_OFFSET + TSI108_PCI_IRP_CFG_CTL);
 	mb();
 	irp_cfg |= (1 << int_line);	/* INTx_DIR = output */
@@ -258,19 +260,23 @@
 	tsi108_write_reg(TSI108_PCI_OFFSET + TSI108_PCI_IRP_CFG_CTL, irp_cfg);
 	mb();
 	irp_cfg = tsi108_read_reg(TSI108_PCI_OFFSET + TSI108_PCI_IRP_CFG_CTL);
+	local_irq_restore_hw_cond(flags);
 }
 
 static void tsi108_pci_int_unmask(u_int irq)
 {
 	u_int irp_cfg;
 	int int_line = (irq - IRQ_PCI_INTAD_BASE);
+	unsigned long flags;
 
+	local_irq_save_hw_cond(flags);
 	irp_cfg = tsi108_read_reg(TSI108_PCI_OFFSET + TSI108_PCI_IRP_CFG_CTL);
 	mb();
 	irp_cfg &= ~(1 << int_line);
 	irp_cfg |= (3 << (8 + (int_line * 2)));
 	tsi108_write_reg(TSI108_PCI_OFFSET + TSI108_PCI_IRP_CFG_CTL, irp_cfg);
 	mb();
+	local_irq_restore_hw_cond(flags);
 }
 
 static void init_pci_source(void)
@@ -361,6 +367,9 @@
 
 static void tsi108_pci_irq_end(u_int irq)
 {
+	unsigned long flags;
+
+	local_irq_save_hw_cond(flags);
 	tsi108_pci_int_unmask(irq);
 
 	/* Enable interrupts from PCI block */
@@ -368,6 +377,7 @@
 			 tsi108_read_reg(TSI108_PCI_OFFSET +
 					 TSI108_PCI_IRP_ENABLE) |
 			 TSI108_PCI_IRP_ENABLE_P_INT);
+	local_irq_restore_hw_cond(flags);
 	mb();
 }
 
diff -urN source_powerpc_none/arch/powerpc/sysdev/uic.c source_powerpc_none.ipipe/arch/powerpc/sysdev/uic.c
--- source_powerpc_none/arch/powerpc/sysdev/uic.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/arch/powerpc/sysdev/uic.c	2009-12-22 12:44:08.000000000 -0500
@@ -49,7 +49,7 @@
 	int index;
 	int dcrbase;
 
-	spinlock_t lock;
+	ipipe_spinlock_t lock;
 
 	/* The remapper for this UIC */
 	struct irq_host	*irqhost;
@@ -71,6 +71,7 @@
 	er = mfdcr(uic->dcrbase + UIC_ER);
 	er |= sr;
 	mtdcr(uic->dcrbase + UIC_ER, er);
+	ipipe_irq_unlock(virq);
 	spin_unlock_irqrestore(&uic->lock, flags);
 }
 
@@ -82,6 +83,7 @@
 	u32 er;
 
 	spin_lock_irqsave(&uic->lock, flags);
+	ipipe_irq_lock(virq);
 	er = mfdcr(uic->dcrbase + UIC_ER);
 	er &= ~(1 << (31 - src));
 	mtdcr(uic->dcrbase + UIC_ER, er);
@@ -239,7 +241,16 @@
 	src = 32 - ffs(msr);
 
 	subvirq = irq_linear_revmap(uic->irqhost, src);
+#ifdef CONFIG_IPIPE
+	{
+		struct pt_regs regs;    /* Contents not used. */
+		ipipe_trace_irq_entry(subvirq);
+		__ipipe_handle_irq(subvirq, &regs);
+		ipipe_trace_irq_exit(subvirq);
+	}
+#else
 	generic_handle_irq(subvirq);
+#endif
 
 uic_irq_ret:
 	spin_lock(&desc->lock);
diff -urN source_powerpc_none/fs/exec.c source_powerpc_none.ipipe/fs/exec.c
--- source_powerpc_none/fs/exec.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/fs/exec.c	2009-12-22 12:44:08.000000000 -0500
@@ -699,6 +699,7 @@
 {
 	struct task_struct *tsk;
 	struct mm_struct * old_mm, *active_mm;
+	unsigned long flags;
 
 	/* Notify parent that we're no longer interested in the old VM */
 	tsk = current;
@@ -721,8 +722,10 @@
 	task_lock(tsk);
 	active_mm = tsk->active_mm;
 	tsk->mm = mm;
+	ipipe_mm_switch_protect(flags);
 	tsk->active_mm = mm;
 	activate_mm(active_mm, mm);
+	ipipe_mm_switch_unprotect(flags);
 	task_unlock(tsk);
 	arch_pick_mmap_layout(mm);
 	if (old_mm) {
diff -urN source_powerpc_none/include/asm-generic/bitops/atomic.h source_powerpc_none.ipipe/include/asm-generic/bitops/atomic.h
--- source_powerpc_none/include/asm-generic/bitops/atomic.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/asm-generic/bitops/atomic.h	2009-12-22 12:44:08.000000000 -0500
@@ -21,20 +21,20 @@
  * this is the substitute */
 #define _atomic_spin_lock_irqsave(l,f) do {	\
 	raw_spinlock_t *s = ATOMIC_HASH(l);	\
-	local_irq_save(f);			\
+	local_irq_save_hw(f);			\
 	__raw_spin_lock(s);			\
 } while(0)
 
 #define _atomic_spin_unlock_irqrestore(l,f) do {	\
 	raw_spinlock_t *s = ATOMIC_HASH(l);		\
 	__raw_spin_unlock(s);				\
-	local_irq_restore(f);				\
+	local_irq_restore_hw(f);			\
 } while(0)
 
 
 #else
-#  define _atomic_spin_lock_irqsave(l,f) do { local_irq_save(f); } while (0)
-#  define _atomic_spin_unlock_irqrestore(l,f) do { local_irq_restore(f); } while (0)
+#  define _atomic_spin_lock_irqsave(l,f) do { local_irq_save_hw(f); } while (0)
+#  define _atomic_spin_unlock_irqrestore(l,f) do { local_irq_restore_hw(f); } while (0)
 #endif
 
 /*
diff -urN source_powerpc_none/include/asm-generic/cmpxchg-local.h source_powerpc_none.ipipe/include/asm-generic/cmpxchg-local.h
--- source_powerpc_none/include/asm-generic/cmpxchg-local.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/asm-generic/cmpxchg-local.h	2009-12-22 12:44:08.000000000 -0500
@@ -20,7 +20,7 @@
 	if (size == 8 && sizeof(unsigned long) != 8)
 		wrong_size_cmpxchg(ptr);
 
-	local_irq_save(flags);
+	local_irq_save_hw(flags);
 	switch (size) {
 	case 1: prev = *(u8 *)ptr;
 		if (prev == old)
@@ -41,7 +41,7 @@
 	default:
 		wrong_size_cmpxchg(ptr);
 	}
-	local_irq_restore(flags);
+	local_irq_restore_hw(flags);
 	return prev;
 }
 
@@ -54,11 +54,11 @@
 	u64 prev;
 	unsigned long flags;
 
-	local_irq_save(flags);
+	local_irq_save_hw(flags);
 	prev = *(u64 *)ptr;
 	if (prev == old)
 		*(u64 *)ptr = new;
-	local_irq_restore(flags);
+	local_irq_restore_hw(flags);
 	return prev;
 }
 
diff -urN source_powerpc_none/include/asm-generic/percpu.h source_powerpc_none.ipipe/include/asm-generic/percpu.h
--- source_powerpc_none/include/asm-generic/percpu.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/asm-generic/percpu.h	2009-12-22 12:44:08.000000000 -0500
@@ -56,6 +56,20 @@
 #define __raw_get_cpu_var(var) \
 	(*SHIFT_PERCPU_PTR(&per_cpu_var(var), __my_cpu_offset))
 
+#ifdef CONFIG_IPIPE
+#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP)
+extern int __ipipe_check_percpu_access(void);
+#define __ipipe_local_cpu_offset				\
+	({							\
+		WARN_ON_ONCE(__ipipe_check_percpu_access());	\
+		__my_cpu_offset;				\
+	})
+#else
+#define __ipipe_local_cpu_offset  __my_cpu_offset
+#endif
+#define __ipipe_get_cpu_var(var) \
+	(*SHIFT_PERCPU_PTR(&per_cpu_var(var), __ipipe_local_cpu_offset))
+#endif /* CONFIG_IPIPE */
 
 #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
 extern void setup_per_cpu_areas(void);
@@ -66,6 +80,7 @@
 #define per_cpu(var, cpu)			(*((void)(cpu), &per_cpu_var(var)))
 #define __get_cpu_var(var)			per_cpu_var(var)
 #define __raw_get_cpu_var(var)			per_cpu_var(var)
+#define __ipipe_get_cpu_var(var)		__raw_get_cpu_var(var)
 
 #endif	/* SMP */
 
diff -urN source_powerpc_none/include/linux/fb.h source_powerpc_none.ipipe/include/linux/fb.h
--- source_powerpc_none/include/linux/fb.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/fb.h	2009-12-22 12:44:08.000000000 -0500
@@ -911,6 +911,53 @@
 #define fb_writeq __raw_writeq
 #define fb_memset memset_io
 
+static inline void fb_writel_swapped(u32 data, volatile void __iomem *addr) {
+#if 0
+#if defined(__powerpc__)
+	volatile u16 *sp = (volatile u16 *)&data;
+
+	st_le16((volatile u16*)addr, *sp);
+	st_le16(((volatile u16*)addr + 1), *(sp+1));
+#else
+#endif
+#endif
+	volatile u32 rd_l,val_l;
+	volatile u8 *rd_b, *val_b;
+
+	val_b=(u8*)&val_l;
+	rd_b=(u8*)&rd_l;
+	rd_l=data;
+	val_b[0]=rd_b[2];
+	val_b[1]=rd_b[3];
+	val_b[2]=rd_b[0];
+	val_b[3]=rd_b[1];
+	__raw_writel(val_l,addr);
+/* #endif */
+};
+
+static inline u32 fb_readl_swapped(const volatile void __iomem *addr) {
+#if 0
+#if defined(__powerpc__)
+	volatile u16 *sp = (volatile u16 *)addr;
+
+	return (ld_le16(sp) << 16 | ld_le16(sp+1));
+#else
+#endif
+#endif
+	volatile u32 rd_l,val_l;
+	volatile u8 *rd_b, *val_b;
+
+	val_b=(u8*)&val_l;
+	rd_b=(u8*)&rd_l;
+	rd_l=__raw_readl(addr);
+	val_b[0]=rd_b[2];
+	val_b[1]=rd_b[3];
+	val_b[2]=rd_b[0];
+	val_b[3]=rd_b[1];
+	return val_l;
+/* #endif */
+};
+
 #else
 
 #define fb_readb(addr) (*(volatile u8 *) (addr))
diff -urN source_powerpc_none/include/linux/hardirq.h source_powerpc_none.ipipe/include/linux/hardirq.h
--- source_powerpc_none/include/linux/hardirq.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/hardirq.h	2009-12-22 12:44:08.000000000 -0500
@@ -183,24 +183,28 @@
  */
 extern void irq_exit(void);
 
-#define nmi_enter()						\
-	do {							\
-		ftrace_nmi_enter();				\
-		BUG_ON(in_nmi());				\
-		add_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET);	\
-		lockdep_off();					\
-		rcu_nmi_enter();				\
-		trace_hardirq_enter();				\
+#define nmi_enter()							\
+	do {								\
+		if (likely(!ipipe_test_foreign_stack())) {		\
+			ftrace_nmi_enter();				\
+			BUG_ON(in_nmi());				\
+			add_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET);	\
+			lockdep_off();					\
+			rcu_nmi_enter();				\
+			trace_hardirq_enter();				\
+		}							\
 	} while (0)
 
-#define nmi_exit()						\
-	do {							\
-		trace_hardirq_exit();				\
-		rcu_nmi_exit();					\
-		lockdep_on();					\
-		BUG_ON(!in_nmi());				\
-		sub_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET);	\
-		ftrace_nmi_exit();				\
+#define nmi_exit()							\
+	do {								\
+		if (likely(!ipipe_test_foreign_stack())) {		\
+			trace_hardirq_exit();				\
+			rcu_nmi_exit();					\
+			lockdep_on();					\
+			BUG_ON(!in_nmi());				\
+			sub_preempt_count(NMI_OFFSET + HARDIRQ_OFFSET);	\
+			ftrace_nmi_exit();				\
+		}							\
 	} while (0)
 
 #endif /* LINUX_HARDIRQ_H */
diff -urN source_powerpc_none/include/linux/ipipe.h source_powerpc_none.ipipe/include/linux/ipipe.h
--- source_powerpc_none/include/linux/ipipe.h	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/ipipe.h	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,688 @@
+/* -*- linux-c -*-
+ * include/linux/ipipe.h
+ *
+ * Copyright (C) 2002-2007 Philippe Gerum.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139,
+ * USA; either version 2 of the License, or (at your option) any later
+ * version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ */
+
+#ifndef __LINUX_IPIPE_H
+#define __LINUX_IPIPE_H
+
+#include <linux/spinlock.h>
+#include <linux/cache.h>
+#include <linux/percpu.h>
+#include <linux/mutex.h>
+#include <linux/linkage.h>
+#include <linux/ipipe_base.h>
+#include <linux/ipipe_compat.h>
+#include <asm/ipipe.h>
+
+#ifdef CONFIG_IPIPE_DEBUG_CONTEXT
+
+#include <linux/cpumask.h>
+#include <asm/system.h>
+
+static inline int ipipe_disable_context_check(int cpu)
+{
+	return xchg(&per_cpu(ipipe_percpu_context_check, cpu), 0);
+}
+
+static inline void ipipe_restore_context_check(int cpu, int old_state)
+{
+	per_cpu(ipipe_percpu_context_check, cpu) = old_state;
+}
+
+static inline void ipipe_context_check_off(void)
+{
+	int cpu;
+	for_each_online_cpu(cpu)
+		per_cpu(ipipe_percpu_context_check, cpu) = 0;
+}
+
+#else	/* !CONFIG_IPIPE_DEBUG_CONTEXT */
+
+static inline int ipipe_disable_context_check(int cpu)
+{
+	return 0;
+}
+
+static inline void ipipe_restore_context_check(int cpu, int old_state) { }
+
+static inline void ipipe_context_check_off(void) { }
+
+#endif	/* !CONFIG_IPIPE_DEBUG_CONTEXT */
+
+#ifdef CONFIG_IPIPE
+
+/*
+ * Sanity check: IPIPE_VIRQ_BASE depends on CONFIG_NR_CPUS, and if the
+ * latter gets too large, we fail to map the virtual interrupts.
+ */
+#if IPIPE_VIRQ_BASE / BITS_PER_LONG > BITS_PER_LONG
+#error "CONFIG_NR_CPUS is too large, please lower it."
+#endif
+
+#define IPIPE_VERSION_STRING	IPIPE_ARCH_STRING
+#define IPIPE_RELEASE_NUMBER	((IPIPE_MAJOR_NUMBER << 16) | \
+				 (IPIPE_MINOR_NUMBER <<  8) | \
+				 (IPIPE_PATCH_NUMBER))
+
+#ifndef BROKEN_BUILTIN_RETURN_ADDRESS
+#define __BUILTIN_RETURN_ADDRESS0 ((unsigned long)__builtin_return_address(0))
+#define __BUILTIN_RETURN_ADDRESS1 ((unsigned long)__builtin_return_address(1))
+#endif /* !BUILTIN_RETURN_ADDRESS */
+
+#define IPIPE_ROOT_PRIO		100
+#define IPIPE_ROOT_ID		0
+#define IPIPE_ROOT_NPTDKEYS	4	/* Must be <= BITS_PER_LONG */
+
+#define IPIPE_RESET_TIMER	0x1
+#define IPIPE_GRAB_TIMER	0x2
+
+/* Global domain flags */
+#define IPIPE_SPRINTK_FLAG	0	/* Synchronous printk() allowed */
+#define IPIPE_AHEAD_FLAG	1	/* Domain always heads the pipeline */
+
+/* Interrupt control bits */
+#define IPIPE_HANDLE_FLAG	0
+#define IPIPE_PASS_FLAG		1
+#define IPIPE_ENABLE_FLAG	2
+#define IPIPE_DYNAMIC_FLAG	IPIPE_HANDLE_FLAG
+#define IPIPE_STICKY_FLAG	3
+#define IPIPE_SYSTEM_FLAG	4
+#define IPIPE_LOCK_FLAG		5
+#define IPIPE_WIRED_FLAG	6
+#define IPIPE_EXCLUSIVE_FLAG	7
+
+#define IPIPE_HANDLE_MASK	(1 << IPIPE_HANDLE_FLAG)
+#define IPIPE_PASS_MASK		(1 << IPIPE_PASS_FLAG)
+#define IPIPE_ENABLE_MASK	(1 << IPIPE_ENABLE_FLAG)
+#define IPIPE_DYNAMIC_MASK	IPIPE_HANDLE_MASK
+#define IPIPE_STICKY_MASK	(1 << IPIPE_STICKY_FLAG)
+#define IPIPE_SYSTEM_MASK	(1 << IPIPE_SYSTEM_FLAG)
+#define IPIPE_LOCK_MASK		(1 << IPIPE_LOCK_FLAG)
+#define IPIPE_WIRED_MASK	(1 << IPIPE_WIRED_FLAG)
+#define IPIPE_EXCLUSIVE_MASK	(1 << IPIPE_EXCLUSIVE_FLAG)
+
+#define IPIPE_DEFAULT_MASK	(IPIPE_HANDLE_MASK|IPIPE_PASS_MASK)
+#define IPIPE_STDROOT_MASK	(IPIPE_HANDLE_MASK|IPIPE_PASS_MASK|IPIPE_SYSTEM_MASK)
+
+#define IPIPE_EVENT_SELF        0x80000000
+
+#define IPIPE_NR_CPUS		NR_CPUS
+
+/* This accessor assumes hw IRQs are off on SMP; allows assignment. */
+#define __ipipe_current_domain	__ipipe_get_cpu_var(ipipe_percpu_domain)
+/* This read-only accessor makes sure that hw IRQs are off on SMP. */
+#define ipipe_current_domain				\
+	({						\
+		struct ipipe_domain *__ipd__;		\
+		unsigned long __flags__;		\
+		local_irq_save_hw_smp(__flags__);	\
+		__ipd__ = __ipipe_current_domain;	\
+		local_irq_restore_hw_smp(__flags__);	\
+		__ipd__;				\
+	})
+
+#define ipipe_virtual_irq_p(irq)	((irq) >= IPIPE_VIRQ_BASE && \
+					 (irq) < IPIPE_NR_IRQS)
+
+#define IPIPE_SAME_HANDLER	((ipipe_irq_handler_t)(-1))
+
+struct irq_desc;
+
+typedef void (*ipipe_irq_ackfn_t)(unsigned irq, struct irq_desc *desc);
+
+typedef int (*ipipe_event_handler_t)(unsigned event,
+				     struct ipipe_domain *from,
+				     void *data);
+struct ipipe_domain {
+
+	int slot;			/* Slot number in percpu domain data array. */
+	struct list_head p_link;	/* Link in pipeline */
+	ipipe_event_handler_t evhand[IPIPE_NR_EVENTS]; /* Event handlers. */
+	unsigned long long evself;	/* Self-monitored event bits. */
+
+	struct {
+		unsigned long control;
+		ipipe_irq_ackfn_t acknowledge;
+		ipipe_irq_handler_t handler;
+		void *cookie;
+	} ____cacheline_aligned irqs[IPIPE_NR_IRQS];
+
+	int priority;
+	void *pdd;
+	unsigned long flags;
+	unsigned domid;
+	const char *name;
+	struct mutex mutex;
+};
+
+#define IPIPE_HEAD_PRIORITY	(-1) /* For domains always heading the pipeline */
+
+struct ipipe_domain_attr {
+
+	unsigned domid;		/* Domain identifier -- Magic value set by caller */
+	const char *name;	/* Domain name -- Warning: won't be dup'ed! */
+	int priority;		/* Priority in interrupt pipeline */
+	void (*entry) (void);	/* Domain entry point */
+	void *pdd;		/* Per-domain (opaque) data pointer */
+};
+
+#define __ipipe_irq_cookie(ipd, irq)		(ipd)->irqs[irq].cookie
+#define __ipipe_irq_handler(ipd, irq)		(ipd)->irqs[irq].handler
+#define __ipipe_cpudata_irq_hits(ipd, cpu, irq)	ipipe_percpudom(ipd, irqall, cpu)[irq]
+
+extern unsigned __ipipe_printk_virq;
+
+extern unsigned long __ipipe_virtual_irq_map;
+
+extern struct list_head __ipipe_pipeline;
+
+extern int __ipipe_event_monitors[];
+
+/* Private interface */
+
+void ipipe_init(void);
+
+#ifdef CONFIG_PROC_FS
+void ipipe_init_proc(void);
+
+#ifdef CONFIG_IPIPE_TRACE
+void __ipipe_init_tracer(void);
+#else /* !CONFIG_IPIPE_TRACE */
+#define __ipipe_init_tracer()       do { } while(0)
+#endif /* CONFIG_IPIPE_TRACE */
+
+#else	/* !CONFIG_PROC_FS */
+#define ipipe_init_proc()	do { } while(0)
+#endif	/* CONFIG_PROC_FS */
+
+void __ipipe_init_stage(struct ipipe_domain *ipd);
+
+void __ipipe_cleanup_domain(struct ipipe_domain *ipd);
+
+void __ipipe_add_domain_proc(struct ipipe_domain *ipd);
+
+void __ipipe_remove_domain_proc(struct ipipe_domain *ipd);
+
+void __ipipe_flush_printk(unsigned irq, void *cookie);
+
+void __ipipe_walk_pipeline(struct list_head *pos);
+
+void __ipipe_pend_irq(unsigned irq, struct list_head *head);
+
+int __ipipe_dispatch_event(unsigned event, void *data);
+
+void __ipipe_dispatch_wired_nocheck(struct ipipe_domain *head, unsigned irq);
+
+void __ipipe_dispatch_wired(struct ipipe_domain *head, unsigned irq);
+
+void __ipipe_sync_stage(unsigned long syncmask);
+
+void __ipipe_set_irq_pending(struct ipipe_domain *ipd, unsigned irq);
+
+void __ipipe_lock_irq(struct ipipe_domain *ipd, int cpu, unsigned irq);
+
+void __ipipe_unlock_irq(struct ipipe_domain *ipd, unsigned irq);
+
+void __ipipe_pin_range_globally(unsigned long start, unsigned long end);
+
+/* Must be called hw IRQs off. */
+static inline void ipipe_irq_lock(unsigned irq)
+{
+	__ipipe_lock_irq(__ipipe_current_domain, ipipe_processor_id(), irq);
+}
+
+/* Must be called hw IRQs off. */
+static inline void ipipe_irq_unlock(unsigned irq)
+{
+	__ipipe_unlock_irq(__ipipe_current_domain, irq);
+}
+
+#ifndef __ipipe_sync_pipeline
+#define __ipipe_sync_pipeline(syncmask) __ipipe_sync_stage(syncmask)
+#endif
+
+#ifndef __ipipe_run_irqtail
+#define __ipipe_run_irqtail() do { } while(0)
+#endif
+
+#define __ipipe_pipeline_head_p(ipd) (&(ipd)->p_link == __ipipe_pipeline.next)
+
+/*
+ * Keep the following as a macro, so that client code could check for
+ * the support of the invariant pipeline head optimization.
+ */
+#define __ipipe_pipeline_head() \
+	list_entry(__ipipe_pipeline.next, struct ipipe_domain, p_link)
+
+#define local_irq_enable_hw_cond()		local_irq_enable_hw()
+#define local_irq_disable_hw_cond()		local_irq_disable_hw()
+#define local_irq_save_hw_cond(flags)		local_irq_save_hw(flags)
+#define local_irq_restore_hw_cond(flags)	local_irq_restore_hw(flags)
+
+#ifdef CONFIG_SMP
+cpumask_t __ipipe_set_irq_affinity(unsigned irq, cpumask_t cpumask);
+int __ipipe_send_ipi(unsigned ipi, cpumask_t cpumask);
+#define local_irq_save_hw_smp(flags)		local_irq_save_hw(flags)
+#define local_irq_restore_hw_smp(flags)		local_irq_restore_hw(flags)
+#else /* !CONFIG_SMP */
+#define local_irq_save_hw_smp(flags)		do { (void)(flags); } while(0)
+#define local_irq_restore_hw_smp(flags)		do { } while(0)
+#endif /* CONFIG_SMP */
+
+#define local_irq_save_full(vflags, rflags)		\
+	do {						\
+		local_irq_save(vflags);			\
+		local_irq_save_hw(rflags);		\
+	} while(0)
+
+#define local_irq_restore_full(vflags, rflags)		\
+	do {						\
+		local_irq_restore_hw(rflags);		\
+		local_irq_restore(vflags);		\
+	} while(0)
+
+static inline void __local_irq_restore_nosync(unsigned long x)
+{
+	struct ipipe_percpu_domain_data *p = ipipe_root_cpudom_ptr();
+
+	if (raw_irqs_disabled_flags(x))
+		set_bit(IPIPE_STALL_FLAG, &p->status);
+	else
+		clear_bit(IPIPE_STALL_FLAG, &p->status);
+}
+
+static inline void local_irq_restore_nosync(unsigned long x)
+{
+	unsigned long flags;
+	local_irq_save_hw_smp(flags);
+	__local_irq_restore_nosync(x);
+	local_irq_restore_hw_smp(flags);
+}
+
+#define __ipipe_root_domain_p	(__ipipe_current_domain == ipipe_root_domain)
+#define ipipe_root_domain_p	(ipipe_current_domain == ipipe_root_domain)
+
+static inline int __ipipe_event_monitored_p(int ev)
+{
+	if (__ipipe_event_monitors[ev] > 0)
+		return 1;
+
+	return (ipipe_current_domain->evself & (1LL << ev)) != 0;
+}
+
+#define ipipe_sigwake_notify(p)	\
+do {					\
+	if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_SIGWAKE)) \
+		__ipipe_dispatch_event(IPIPE_EVENT_SIGWAKE, p);		\
+} while(0)
+
+#define ipipe_exit_notify(p)	\
+do {				\
+	if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_EXIT)) \
+		__ipipe_dispatch_event(IPIPE_EVENT_EXIT, p);		\
+} while(0)
+
+#define ipipe_setsched_notify(p)	\
+do {					\
+	if (((p)->flags & PF_EVNOTIFY) && __ipipe_event_monitored_p(IPIPE_EVENT_SETSCHED)) \
+		__ipipe_dispatch_event(IPIPE_EVENT_SETSCHED, p);	\
+} while(0)
+
+#define ipipe_schedule_notify(prev, next)				\
+do {									\
+	if ((((prev)->flags|(next)->flags) & PF_EVNOTIFY) &&		\
+	    __ipipe_event_monitored_p(IPIPE_EVENT_SCHEDULE))		\
+		__ipipe_dispatch_event(IPIPE_EVENT_SCHEDULE,next);	\
+} while(0)
+
+#define ipipe_trap_notify(ex, regs)					\
+({									\
+	unsigned long __flags__;					\
+	int __ret__ = 0;						\
+	local_irq_save_hw_smp(__flags__);				\
+	if ((test_bit(IPIPE_NOSTACK_FLAG, &ipipe_this_cpudom_var(status)) || \
+	     ((current)->flags & PF_EVNOTIFY)) &&			\
+	    __ipipe_event_monitored_p(ex)) {				\
+		local_irq_restore_hw_smp(__flags__);			\
+		__ret__ = __ipipe_dispatch_event(ex, regs);		\
+	} else								\
+		local_irq_restore_hw_smp(__flags__);			\
+	__ret__;							\
+})
+
+static inline void ipipe_init_notify(struct task_struct *p)
+{
+	if (__ipipe_event_monitored_p(IPIPE_EVENT_INIT))
+		__ipipe_dispatch_event(IPIPE_EVENT_INIT, p);
+}
+
+struct mm_struct;
+
+static inline void ipipe_cleanup_notify(struct mm_struct *mm)
+{
+	if (__ipipe_event_monitored_p(IPIPE_EVENT_CLEANUP))
+		__ipipe_dispatch_event(IPIPE_EVENT_CLEANUP, mm);
+}
+
+/* Public interface */
+
+int ipipe_register_domain(struct ipipe_domain *ipd,
+			  struct ipipe_domain_attr *attr);
+
+int ipipe_unregister_domain(struct ipipe_domain *ipd);
+
+void ipipe_suspend_domain(void);
+
+int ipipe_virtualize_irq(struct ipipe_domain *ipd,
+			 unsigned irq,
+			 ipipe_irq_handler_t handler,
+			 void *cookie,
+			 ipipe_irq_ackfn_t acknowledge,
+			 unsigned modemask);
+
+int ipipe_control_irq(unsigned irq,
+		      unsigned clrmask,
+		      unsigned setmask);
+
+unsigned ipipe_alloc_virq(void);
+
+int ipipe_free_virq(unsigned virq);
+
+int ipipe_trigger_irq(unsigned irq);
+
+static inline void __ipipe_propagate_irq(unsigned irq)
+{
+	struct list_head *next = __ipipe_current_domain->p_link.next;
+	if (next == &ipipe_root.p_link) {
+		/* Fast path: root must handle all interrupts. */
+		__ipipe_set_irq_pending(&ipipe_root, irq);
+		return;
+	}
+	__ipipe_pend_irq(irq, next);
+}
+
+static inline void __ipipe_schedule_irq(unsigned irq)
+{
+	__ipipe_pend_irq(irq, &__ipipe_current_domain->p_link);
+}
+
+static inline void __ipipe_schedule_irq_head(unsigned irq)
+{
+	__ipipe_set_irq_pending(__ipipe_pipeline_head(), irq);
+}
+
+static inline void __ipipe_schedule_irq_root(unsigned irq)
+{
+	__ipipe_set_irq_pending(&ipipe_root, irq);
+}
+
+static inline void ipipe_propagate_irq(unsigned irq)
+{
+	unsigned long flags;
+
+	local_irq_save_hw(flags);
+	__ipipe_propagate_irq(irq);
+	local_irq_restore_hw(flags);
+}
+
+static inline void ipipe_schedule_irq(unsigned irq)
+{
+	unsigned long flags;
+
+	local_irq_save_hw(flags);
+	__ipipe_schedule_irq(irq);
+	local_irq_restore_hw(flags);
+}
+
+static inline void ipipe_schedule_irq_head(unsigned irq)
+{
+	unsigned long flags;
+
+	local_irq_save_hw(flags);
+	__ipipe_schedule_irq_head(irq);
+	local_irq_restore_hw(flags);
+}
+
+static inline void ipipe_schedule_irq_root(unsigned irq)
+{
+	unsigned long flags;
+
+	local_irq_save_hw(flags);
+	__ipipe_schedule_irq_root(irq);
+	local_irq_restore_hw(flags);
+}
+
+void ipipe_stall_pipeline_from(struct ipipe_domain *ipd);
+
+unsigned long ipipe_test_and_stall_pipeline_from(struct ipipe_domain *ipd);
+
+unsigned long ipipe_test_and_unstall_pipeline_from(struct ipipe_domain *ipd);
+
+static inline void ipipe_unstall_pipeline_from(struct ipipe_domain *ipd)
+{
+	ipipe_test_and_unstall_pipeline_from(ipd);
+}
+
+void ipipe_restore_pipeline_from(struct ipipe_domain *ipd,
+					  unsigned long x);
+
+static inline unsigned long ipipe_test_pipeline_from(struct ipipe_domain *ipd)
+{
+	return test_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status));
+}
+
+static inline void ipipe_stall_pipeline_head(void)
+{
+	local_irq_disable_hw();
+	__set_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status));
+}
+
+static inline unsigned long ipipe_test_and_stall_pipeline_head(void)
+{
+	local_irq_disable_hw();
+	return __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status));
+}
+
+void ipipe_unstall_pipeline_head(void);
+
+void __ipipe_restore_pipeline_head(unsigned long x);
+
+static inline void ipipe_restore_pipeline_head(unsigned long x)
+{
+	/* On some archs, __test_and_set_bit() might return different
+	 * truth value than test_bit(), so we test the exclusive OR of
+	 * both statuses, assuming that the lowest bit is always set in
+	 * the truth value (if this is wrong, the failed optimization will
+	 * be caught in __ipipe_restore_pipeline_head() if
+	 * CONFIG_DEBUG_KERNEL is set). */
+	if ((x ^ test_bit(IPIPE_STALL_FLAG, &ipipe_head_cpudom_var(status))) & 1)
+		__ipipe_restore_pipeline_head(x);
+}
+
+#define ipipe_unstall_pipeline() \
+	ipipe_unstall_pipeline_from(ipipe_current_domain)
+
+#define ipipe_test_and_unstall_pipeline() \
+	ipipe_test_and_unstall_pipeline_from(ipipe_current_domain)
+
+#define ipipe_test_pipeline() \
+	ipipe_test_pipeline_from(ipipe_current_domain)
+
+#define ipipe_test_and_stall_pipeline() \
+	ipipe_test_and_stall_pipeline_from(ipipe_current_domain)
+
+#define ipipe_stall_pipeline() \
+	ipipe_stall_pipeline_from(ipipe_current_domain)
+
+#define ipipe_restore_pipeline(x) \
+	ipipe_restore_pipeline_from(ipipe_current_domain, (x))
+
+void ipipe_init_attr(struct ipipe_domain_attr *attr);
+
+int ipipe_get_sysinfo(struct ipipe_sysinfo *sysinfo);
+
+unsigned long ipipe_critical_enter(void (*syncfn) (void));
+
+void ipipe_critical_exit(unsigned long flags);
+
+static inline void ipipe_set_printk_sync(struct ipipe_domain *ipd)
+{
+	set_bit(IPIPE_SPRINTK_FLAG, &ipd->flags);
+}
+
+static inline void ipipe_set_printk_async(struct ipipe_domain *ipd)
+{
+	clear_bit(IPIPE_SPRINTK_FLAG, &ipd->flags);
+}
+
+static inline void ipipe_set_foreign_stack(struct ipipe_domain *ipd)
+{
+	/* Must be called hw interrupts off. */
+	__set_bit(IPIPE_NOSTACK_FLAG, &ipipe_cpudom_var(ipd, status));
+}
+
+static inline void ipipe_clear_foreign_stack(struct ipipe_domain *ipd)
+{
+	/* Must be called hw interrupts off. */
+	__clear_bit(IPIPE_NOSTACK_FLAG, &ipipe_cpudom_var(ipd, status));
+}
+
+static inline int ipipe_test_foreign_stack(void)
+{
+	/* Must be called hw interrupts off. */
+	return test_bit(IPIPE_NOSTACK_FLAG, &ipipe_this_cpudom_var(status));
+}
+
+#ifndef ipipe_safe_current
+#define ipipe_safe_current()					\
+({								\
+	struct task_struct *p;					\
+	unsigned long flags;					\
+	local_irq_save_hw_smp(flags);				\
+	p = ipipe_test_foreign_stack() ? &init_task : current;	\
+	local_irq_restore_hw_smp(flags);			\
+	p; \
+})
+#endif
+
+ipipe_event_handler_t ipipe_catch_event(struct ipipe_domain *ipd,
+					unsigned event,
+					ipipe_event_handler_t handler);
+
+cpumask_t ipipe_set_irq_affinity(unsigned irq,
+				 cpumask_t cpumask);
+
+int ipipe_send_ipi(unsigned ipi,
+		   cpumask_t cpumask);
+
+int ipipe_setscheduler_root(struct task_struct *p,
+			    int policy,
+			    int prio);
+
+int ipipe_reenter_root(struct task_struct *prev,
+		       int policy,
+		       int prio);
+
+int ipipe_alloc_ptdkey(void);
+
+int ipipe_free_ptdkey(int key);
+
+int ipipe_set_ptd(int key,
+		  void *value);
+
+void *ipipe_get_ptd(int key);
+
+int ipipe_disable_ondemand_mappings(struct task_struct *tsk);
+
+static inline void ipipe_nmi_enter(void)
+{
+	int cpu = ipipe_processor_id();
+
+	per_cpu(ipipe_nmi_saved_root, cpu) = ipipe_root_cpudom_var(status);
+	__set_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status));
+
+#ifdef CONFIG_IPIPE_DEBUG_CONTEXT
+	per_cpu(ipipe_saved_context_check_state, cpu) =
+		ipipe_disable_context_check(cpu);
+#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */
+}
+
+static inline void ipipe_nmi_exit(void)
+{
+	int cpu = ipipe_processor_id();
+
+#ifdef CONFIG_IPIPE_DEBUG_CONTEXT
+	ipipe_restore_context_check
+		(cpu, per_cpu(ipipe_saved_context_check_state, cpu));
+#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */
+
+	if (!test_bit(IPIPE_STALL_FLAG, &per_cpu(ipipe_nmi_saved_root, cpu)))
+		__clear_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status));
+}
+
+#else	/* !CONFIG_IPIPE */
+
+#define ipipe_init()			do { } while(0)
+#define ipipe_suspend_domain()		do { } while(0)
+#define ipipe_sigwake_notify(p)		do { } while(0)
+#define ipipe_setsched_notify(p)	do { } while(0)
+#define ipipe_init_notify(p)		do { } while(0)
+#define ipipe_exit_notify(p)		do { } while(0)
+#define ipipe_cleanup_notify(mm)	do { } while(0)
+#define ipipe_trap_notify(t,r)		0
+#define ipipe_init_proc()		do { } while(0)
+
+static inline void __ipipe_pin_range_globally(unsigned long start,
+					      unsigned long end)
+{
+}
+
+static inline int ipipe_test_foreign_stack(void)
+{
+	return 0;
+}
+
+#define local_irq_enable_hw_cond()		do { } while(0)
+#define local_irq_disable_hw_cond()		do { } while(0)
+#define local_irq_save_hw_cond(flags)		do { (void)(flags); } while(0)
+#define local_irq_restore_hw_cond(flags)	do { } while(0)
+#define local_irq_save_hw_smp(flags)		do { (void)(flags); } while(0)
+#define local_irq_restore_hw_smp(flags)		do { } while(0)
+
+#define ipipe_irq_lock(irq)		do { } while(0)
+#define ipipe_irq_unlock(irq)		do { } while(0)
+
+#define __ipipe_root_domain_p		1
+#define ipipe_root_domain_p		1
+#define ipipe_safe_current		current
+#define ipipe_processor_id()		smp_processor_id()
+
+#define ipipe_nmi_enter()		do { } while (0)
+#define ipipe_nmi_exit()		do { } while (0)
+
+#define local_irq_disable_head()	local_irq_disable()
+
+#define local_irq_save_full(vflags, rflags)	do { (void)(vflags); local_irq_save(rflags); } while(0)
+#define local_irq_restore_full(vflags, rflags)	do { (void)(vflags); local_irq_restore(rflags); } while(0)
+#define local_irq_restore_nosync(vflags)	local_irq_restore(vflags)
+
+#endif	/* CONFIG_IPIPE */
+
+#endif	/* !__LINUX_IPIPE_H */
diff -urN source_powerpc_none/include/linux/ipipe_base.h source_powerpc_none.ipipe/include/linux/ipipe_base.h
--- source_powerpc_none/include/linux/ipipe_base.h	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/ipipe_base.h	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,102 @@
+/* -*- linux-c -*-
+ * include/linux/ipipe_base.h
+ *
+ * Copyright (C) 2002-2007 Philippe Gerum.
+ *               2007 Jan Kiszka.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139,
+ * USA; either version 2 of the License, or (at your option) any later
+ * version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ */
+
+#ifndef __LINUX_IPIPE_BASE_H
+#define __LINUX_IPIPE_BASE_H
+
+#ifdef CONFIG_IPIPE
+
+#include <asm/ipipe_base.h>
+
+/* Number of virtual IRQs */
+#define IPIPE_NR_VIRQS		BITS_PER_LONG
+/* First virtual IRQ # */
+#define IPIPE_VIRQ_BASE		(((IPIPE_NR_XIRQS + BITS_PER_LONG - 1) / BITS_PER_LONG) * BITS_PER_LONG)
+/* Total number of IRQ slots */
+#define IPIPE_NR_IRQS		(IPIPE_VIRQ_BASE + IPIPE_NR_VIRQS)
+/* Number of indirect words needed to map the whole IRQ space. */
+#define IPIPE_IRQ_IWORDS	((IPIPE_NR_IRQS + BITS_PER_LONG - 1) / BITS_PER_LONG)
+#define IPIPE_IRQ_IMASK		(BITS_PER_LONG - 1)
+#define IPIPE_IRQMASK_ANY	(~0L)
+#define IPIPE_IRQMASK_VIRT	(IPIPE_IRQMASK_ANY << (IPIPE_VIRQ_BASE / BITS_PER_LONG))
+
+/* Per-cpu pipeline status */
+#define IPIPE_STALL_FLAG	0	/* Stalls a pipeline stage -- guaranteed at bit #0 */
+#define IPIPE_SYNC_FLAG		1	/* The interrupt syncer is running for the domain */
+#define IPIPE_NOSTACK_FLAG	2	/* Domain currently runs on a foreign stack */
+
+#define IPIPE_STALL_MASK	(1L << IPIPE_STALL_FLAG)
+#define IPIPE_SYNC_MASK		(1L << IPIPE_SYNC_FLAG)
+#define IPIPE_NOSTACK_MASK	(1L << IPIPE_NOSTACK_FLAG)
+
+typedef void (*ipipe_irq_handler_t)(unsigned irq,
+				    void *cookie);
+
+extern struct ipipe_domain ipipe_root;
+
+#define ipipe_root_domain (&ipipe_root)
+
+void __ipipe_unstall_root(void);
+
+void __ipipe_restore_root(unsigned long x);
+
+#define ipipe_preempt_disable(flags)		\
+	do {					\
+		local_irq_save_hw(flags);	\
+		if (__ipipe_root_domain_p)	\
+			preempt_disable();	\
+	} while (0)
+#define ipipe_preempt_enable(flags)			\
+	do {						\
+		if (__ipipe_root_domain_p) {		\
+			preempt_enable_no_resched();	\
+			local_irq_restore_hw(flags);	\
+			preempt_check_resched();	\
+		} else					\
+			local_irq_restore_hw(flags);	\
+	} while (0)
+ 
+#ifdef CONFIG_IPIPE_DEBUG_CONTEXT
+void ipipe_check_context(struct ipipe_domain *border_ipd);
+#else /* !CONFIG_IPIPE_DEBUG_CONTEXT */
+static inline void ipipe_check_context(struct ipipe_domain *border_ipd) { }
+#endif /* !CONFIG_IPIPE_DEBUG_CONTEXT */
+
+/* Generic features */
+
+#ifdef CONFIG_GENERIC_CLOCKEVENTS
+#define __IPIPE_FEATURE_REQUEST_TICKDEV    1
+#endif
+#define __IPIPE_FEATURE_DELAYED_ATOMICSW   1
+#define __IPIPE_FEATURE_FASTPEND_IRQ       1
+#define __IPIPE_FEATURE_TRACE_EVENT	   1
+
+#else /* !CONFIG_IPIPE */
+#define ipipe_preempt_disable(flags)	do { \
+						preempt_disable(); \
+						(void)(flags); \
+					} while (0)
+#define ipipe_preempt_enable(flags)	preempt_enable()
+#define ipipe_check_context(ipd)	do { } while(0)
+#endif	/* CONFIG_IPIPE */
+
+#endif	/* !__LINUX_IPIPE_BASE_H */
diff -urN source_powerpc_none/include/linux/ipipe_compat.h source_powerpc_none.ipipe/include/linux/ipipe_compat.h
--- source_powerpc_none/include/linux/ipipe_compat.h	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/ipipe_compat.h	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,54 @@
+/* -*- linux-c -*-
+ * include/linux/ipipe_compat.h
+ *
+ * Copyright (C) 2007 Philippe Gerum.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139,
+ * USA; either version 2 of the License, or (at your option) any later
+ * version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ */
+
+#ifndef __LINUX_IPIPE_COMPAT_H
+#define __LINUX_IPIPE_COMPAT_H
+
+#ifdef CONFIG_IPIPE_COMPAT
+/*
+ * OBSOLETE: defined only for backward compatibility. Will be removed
+ * in future releases, please update client code accordingly.
+ */
+
+#ifdef CONFIG_SMP
+#define ipipe_declare_cpuid	int cpuid
+#define ipipe_load_cpuid()	do { \
+					cpuid = ipipe_processor_id();	\
+				} while(0)
+#define ipipe_lock_cpu(flags)	do { \
+					local_irq_save_hw(flags); \
+					cpuid = ipipe_processor_id(); \
+				} while(0)
+#define ipipe_unlock_cpu(flags)	local_irq_restore_hw(flags)
+#define ipipe_get_cpu(flags)	ipipe_lock_cpu(flags)
+#define ipipe_put_cpu(flags)	ipipe_unlock_cpu(flags)
+#else /* !CONFIG_SMP */
+#define ipipe_declare_cpuid	const int cpuid = 0
+#define ipipe_load_cpuid()	do { } while(0)
+#define ipipe_lock_cpu(flags)	local_irq_save_hw(flags)
+#define ipipe_unlock_cpu(flags)	local_irq_restore_hw(flags)
+#define ipipe_get_cpu(flags)	do { (void)(flags); } while(0)
+#define ipipe_put_cpu(flags)	do { } while(0)
+#endif /* CONFIG_SMP */
+
+#endif /* CONFIG_IPIPE_COMPAT */
+
+#endif	/* !__LINUX_IPIPE_COMPAT_H */
diff -urN source_powerpc_none/include/linux/ipipe_percpu.h source_powerpc_none.ipipe/include/linux/ipipe_percpu.h
--- source_powerpc_none/include/linux/ipipe_percpu.h	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/ipipe_percpu.h	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,86 @@
+/*   -*- linux-c -*-
+ *   include/linux/ipipe_percpu.h
+ *
+ *   Copyright (C) 2007 Philippe Gerum.
+ *
+ *   This program is free software; you can redistribute it and/or modify
+ *   it under the terms of the GNU General Public License as published by
+ *   the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139,
+ *   USA; either version 2 of the License, or (at your option) any later
+ *   version.
+ *
+ *   This program is distributed in the hope that it will be useful,
+ *   but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *   GNU General Public License for more details.
+ *
+ *   You should have received a copy of the GNU General Public License
+ *   along with this program; if not, write to the Free Software
+ *   Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ */
+
+#ifndef __LINUX_IPIPE_PERCPU_H
+#define __LINUX_IPIPE_PERCPU_H
+
+#include <asm/percpu.h>
+#include <asm/ptrace.h>
+
+struct ipipe_domain;
+
+struct ipipe_percpu_domain_data {
+	unsigned long status;	/* <= Must be first in struct. */
+	unsigned long irqpend_himask;
+	unsigned long irqpend_lomask[IPIPE_IRQ_IWORDS];
+	unsigned long irqheld_mask[IPIPE_IRQ_IWORDS];
+	unsigned long irqall[IPIPE_NR_IRQS];
+	u64 evsync;
+};
+
+/*
+ * CAREFUL: all accessors based on __raw_get_cpu_var() you may find in
+ * this file should be used only while hw interrupts are off, to
+ * prevent from CPU migration regardless of the running domain.
+ */
+#ifdef CONFIG_SMP
+#define ipipe_percpudom_ptr(ipd, cpu)	\
+	(&per_cpu(ipipe_percpu_darray, cpu)[(ipd)->slot])
+#define ipipe_cpudom_ptr(ipd)	\
+	(&__ipipe_get_cpu_var(ipipe_percpu_darray)[(ipd)->slot])
+#else
+DECLARE_PER_CPU(struct ipipe_percpu_domain_data *, ipipe_percpu_daddr[CONFIG_IPIPE_DOMAINS]);
+#define ipipe_percpudom_ptr(ipd, cpu)	\
+	(per_cpu(ipipe_percpu_daddr, cpu)[(ipd)->slot])
+#define ipipe_cpudom_ptr(ipd)	\
+	(__ipipe_get_cpu_var(ipipe_percpu_daddr)[(ipd)->slot])
+#endif
+#define ipipe_percpudom(ipd, var, cpu)	(ipipe_percpudom_ptr(ipd, cpu)->var)
+#define ipipe_cpudom_var(ipd, var)	(ipipe_cpudom_ptr(ipd)->var)
+
+#define IPIPE_ROOT_SLOT			0
+#define IPIPE_HEAD_SLOT			(CONFIG_IPIPE_DOMAINS - 1)
+
+DECLARE_PER_CPU(struct ipipe_percpu_domain_data, ipipe_percpu_darray[CONFIG_IPIPE_DOMAINS]);
+
+DECLARE_PER_CPU(struct ipipe_domain *, ipipe_percpu_domain);
+
+DECLARE_PER_CPU(unsigned long, ipipe_nmi_saved_root);
+
+#ifdef CONFIG_IPIPE_DEBUG_CONTEXT
+DECLARE_PER_CPU(int, ipipe_percpu_context_check);
+DECLARE_PER_CPU(int, ipipe_saved_context_check_state);
+#endif
+
+#define ipipe_root_cpudom_ptr(var)	\
+	(&__ipipe_get_cpu_var(ipipe_percpu_darray)[IPIPE_ROOT_SLOT])
+
+#define ipipe_root_cpudom_var(var)	ipipe_root_cpudom_ptr()->var
+
+#define ipipe_this_cpudom_var(var)	\
+	ipipe_cpudom_var(__ipipe_current_domain, var)
+
+#define ipipe_head_cpudom_ptr()		\
+	(&__ipipe_get_cpu_var(ipipe_percpu_darray)[IPIPE_HEAD_SLOT])
+
+#define ipipe_head_cpudom_var(var)	ipipe_head_cpudom_ptr()->var
+
+#endif	/* !__LINUX_IPIPE_PERCPU_H */
diff -urN source_powerpc_none/include/linux/ipipe_tickdev.h source_powerpc_none.ipipe/include/linux/ipipe_tickdev.h
--- source_powerpc_none/include/linux/ipipe_tickdev.h	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/ipipe_tickdev.h	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,58 @@
+/* -*- linux-c -*-
+ * include/linux/ipipe_tickdev.h
+ *
+ * Copyright (C) 2007 Philippe Gerum.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139,
+ * USA; either version 2 of the License, or (at your option) any later
+ * version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ */
+
+#ifndef __LINUX_IPIPE_TICKDEV_H
+#define __LINUX_IPIPE_TICKDEV_H
+
+#if defined(CONFIG_IPIPE) && defined(CONFIG_GENERIC_CLOCKEVENTS)
+
+#include <linux/clockchips.h>
+
+struct tick_device;
+
+struct ipipe_tick_device {
+
+	void (*emul_set_mode)(enum clock_event_mode,
+			      struct clock_event_device *cdev);
+	int (*emul_set_tick)(unsigned long delta,
+			     struct clock_event_device *cdev);
+	void (*real_set_mode)(enum clock_event_mode mode,
+			      struct clock_event_device *cdev);
+	int (*real_set_tick)(unsigned long delta,
+			     struct clock_event_device *cdev);
+	struct tick_device *slave;
+	unsigned long real_max_delta_ns;
+	unsigned long real_mult;
+	int real_shift;
+};
+
+int ipipe_request_tickdev(const char *devname,
+			  void (*emumode)(enum clock_event_mode mode,
+					  struct clock_event_device *cdev),
+			  int (*emutick)(unsigned long evt,
+					 struct clock_event_device *cdev),
+			  int cpu, unsigned long *tmfreq);
+
+void ipipe_release_tickdev(int cpu);
+
+#endif /* CONFIG_IPIPE && CONFIG_GENERIC_CLOCKEVENTS */
+
+#endif /* !__LINUX_IPIPE_TICKDEV_H */
diff -urN source_powerpc_none/include/linux/ipipe_trace.h source_powerpc_none.ipipe/include/linux/ipipe_trace.h
--- source_powerpc_none/include/linux/ipipe_trace.h	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/ipipe_trace.h	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,72 @@
+/* -*- linux-c -*-
+ * include/linux/ipipe_trace.h
+ *
+ * Copyright (C) 2005 Luotao Fu.
+ *               2005-2007 Jan Kiszka.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139,
+ * USA; either version 2 of the License, or (at your option) any later
+ * version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ */
+
+#ifndef _LINUX_IPIPE_TRACE_H
+#define _LINUX_IPIPE_TRACE_H
+
+#ifdef CONFIG_IPIPE_TRACE
+
+#include <linux/types.h>
+
+void ipipe_trace_begin(unsigned long v);
+void ipipe_trace_end(unsigned long v);
+void ipipe_trace_freeze(unsigned long v);
+void ipipe_trace_special(unsigned char special_id, unsigned long v);
+void ipipe_trace_pid(pid_t pid, short prio);
+void ipipe_trace_event(unsigned char id, unsigned long delay_tsc);
+int ipipe_trace_max_reset(void);
+int ipipe_trace_frozen_reset(void);
+
+#else /* !CONFIG_IPIPE_TRACE */
+
+#define ipipe_trace_begin(v)			do { (void)(v); } while(0)
+#define ipipe_trace_end(v)			do { (void)(v); } while(0)
+#define ipipe_trace_freeze(v)			do { (void)(v); } while(0)
+#define ipipe_trace_special(id, v)		do { (void)(id); (void)(v); } while(0)
+#define ipipe_trace_pid(pid, prio)		do { (void)(pid); (void)(prio); } while(0)
+#define ipipe_trace_event(id, delay_tsc)	do { (void)(id); (void)(delay_tsc); } while(0)
+#define ipipe_trace_max_reset()			do { } while(0)
+#define ipipe_trace_froze_reset()		do { } while(0)
+
+#endif /* !CONFIG_IPIPE_TRACE */
+
+#ifdef CONFIG_IPIPE_TRACE_PANIC
+void ipipe_trace_panic_freeze(void);
+void ipipe_trace_panic_dump(void);
+#else
+static inline void ipipe_trace_panic_freeze(void) { }
+static inline void ipipe_trace_panic_dump(void) { }
+#endif
+
+#ifdef CONFIG_IPIPE_TRACE_IRQSOFF
+#define ipipe_trace_irq_entry(irq)	ipipe_trace_begin(irq)
+#define ipipe_trace_irq_exit(irq)	ipipe_trace_end(irq)
+#define ipipe_trace_irqsoff()		ipipe_trace_begin(0x80000000UL)
+#define ipipe_trace_irqson()		ipipe_trace_end(0x80000000UL)
+#else
+#define ipipe_trace_irq_entry(irq)	do { (void)(irq);} while(0)
+#define ipipe_trace_irq_exit(irq)	do { (void)(irq);} while(0)
+#define ipipe_trace_irqsoff()		do { } while(0)
+#define ipipe_trace_irqson()		do { } while(0)
+#endif
+
+#endif	/* !__LINUX_IPIPE_TRACE_H */
diff -urN source_powerpc_none/include/linux/irq.h source_powerpc_none.ipipe/include/linux/irq.h
--- source_powerpc_none/include/linux/irq.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/irq.h	2009-12-22 12:44:08.000000000 -0500
@@ -124,6 +124,9 @@
 	void		(*end)(unsigned int irq);
 	int		(*set_affinity)(unsigned int irq,
 					const struct cpumask *dest);
+#ifdef CONFIG_IPIPE
+	void		(*move)(unsigned int irq);
+#endif /* CONFIG_IPIPE */
 	int		(*retrigger)(unsigned int irq);
 	int		(*set_type)(unsigned int irq, unsigned int flow_type);
 	int		(*set_wake)(unsigned int irq, unsigned int on);
@@ -173,6 +176,12 @@
  * @name:		flow handler name for /proc/interrupts output
  */
 struct irq_desc {
+#ifdef CONFIG_IPIPE
+	void			(*ipipe_ack)(unsigned int irq,
+					     struct irq_desc *desc);
+	void			(*ipipe_end)(unsigned int irq,
+					     struct irq_desc *desc);
+#endif /* CONFIG_IPIPE */
 	unsigned int		irq;
 	struct timer_rand_state *timer_rand_state;
 	unsigned int            *kstat_irqs;
@@ -347,6 +356,10 @@
 			      irq_flow_handler_t handle, const char *name);
 
 extern void
+___set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained,
+		   const char *name);
+
+extern void
 __set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained,
 		  const char *name);
 
@@ -361,6 +374,15 @@
 }
 
 /*
+ * Same, but without holding the descriptor lock.
+ */
+static inline void
+_set_irq_handler(unsigned int irq, irq_flow_handler_t handle)
+{
+	___set_irq_handler(irq, handle, 0, NULL);
+}
+
+/*
  * Set a highlevel flow handler for a given IRQ:
  */
 static inline void
diff -urN source_powerpc_none/include/linux/kernel.h source_powerpc_none.ipipe/include/linux/kernel.h
--- source_powerpc_none/include/linux/kernel.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/kernel.h	2009-12-22 12:44:08.000000000 -0500
@@ -14,6 +14,7 @@
 #include <linux/compiler.h>
 #include <linux/bitops.h>
 #include <linux/log2.h>
+#include <linux/ipipe_base.h>
 #include <linux/typecheck.h>
 #include <linux/ratelimit.h>
 #include <linux/dynamic_debug.h>
@@ -119,9 +120,12 @@
 
 #ifdef CONFIG_PREEMPT_VOLUNTARY
 extern int _cond_resched(void);
-# define might_resched() _cond_resched()
+# define might_resched() do { \
+		ipipe_check_context(ipipe_root_domain); \
+		_cond_resched(); \
+	} while (0)
 #else
-# define might_resched() do { } while (0)
+# define might_resched() ipipe_check_context(ipipe_root_domain)
 #endif
 
 #ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
diff -urN source_powerpc_none/include/linux/lockdep.h source_powerpc_none.ipipe/include/linux/lockdep.h
--- source_powerpc_none/include/linux/lockdep.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/lockdep.h	2009-12-22 12:44:08.000000000 -0500
@@ -395,7 +395,7 @@
 
 #endif /* CONFIG_LOCK_STAT */
 
-#ifdef CONFIG_LOCKDEP
+#if defined(CONFIG_LOCKDEP) || defined(CONFIG_IPIPE)
 
 /*
  * On lockdep we dont want the hand-coded irq-enable of
diff -urN source_powerpc_none/include/linux/mm.h source_powerpc_none.ipipe/include/linux/mm.h
--- source_powerpc_none/include/linux/mm.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/mm.h	2009-12-22 12:44:08.000000000 -0500
@@ -106,6 +106,8 @@
 #define VM_PFN_AT_MMAP	0x40000000	/* PFNMAP vma that is fully mapped at mmap time */
 #define VM_MERGEABLE	0x80000000	/* KSM may merge identical pages */
 
+#define VM_PINNED	0x80000000	/* Disable faults for the vma */
+
 #ifndef VM_STACK_DEFAULT_FLAGS		/* arch can override this */
 #define VM_STACK_DEFAULT_FLAGS VM_DATA_DEFAULT_FLAGS
 #endif
diff -urN source_powerpc_none/include/linux/preempt.h source_powerpc_none.ipipe/include/linux/preempt.h
--- source_powerpc_none/include/linux/preempt.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/preempt.h	2009-12-22 12:44:08.000000000 -0500
@@ -9,13 +9,20 @@
 #include <linux/thread_info.h>
 #include <linux/linkage.h>
 #include <linux/list.h>
+#include <linux/ipipe_base.h>
 
 #if defined(CONFIG_DEBUG_PREEMPT) || defined(CONFIG_PREEMPT_TRACER)
   extern void add_preempt_count(int val);
   extern void sub_preempt_count(int val);
 #else
-# define add_preempt_count(val)	do { preempt_count() += (val); } while (0)
-# define sub_preempt_count(val)	do { preempt_count() -= (val); } while (0)
+# define add_preempt_count(val)	do {		\
+    ipipe_check_context(ipipe_root_domain);	\
+    preempt_count() += (val);			\
+  } while (0)
+# define sub_preempt_count(val)	do {		\
+    ipipe_check_context(ipipe_root_domain);	\
+    preempt_count() -= (val);			\
+  } while (0)
 #endif
 
 #define inc_preempt_count() add_preempt_count(1)
diff -urN source_powerpc_none/include/linux/sched.h source_powerpc_none.ipipe/include/linux/sched.h
--- source_powerpc_none/include/linux/sched.h	2009-12-21 17:25:55.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/sched.h	2009-12-22 12:44:08.000000000 -0500
@@ -61,6 +61,7 @@
 #include <linux/errno.h>
 #include <linux/nodemask.h>
 #include <linux/mm_types.h>
+#include <linux/ipipe.h>
 
 #include <asm/system.h>
 #include <asm/page.h>
@@ -195,6 +196,13 @@
 #define TASK_DEAD		64
 #define TASK_WAKEKILL		128
 #define TASK_WAKING		256
+#ifdef CONFIG_IPIPE
+#define TASK_ATOMICSWITCH	512
+#define TASK_NOWAKEUP		1024
+#else  /* !CONFIG_IPIPE */
+#define TASK_ATOMICSWITCH	0
+#define TASK_NOWAKEUP		0
+#endif /* CONFIG_IPIPE */
 
 /* Convenience macros for the sake of set_task_state */
 #define TASK_KILLABLE		(TASK_WAKEKILL | TASK_UNINTERRUPTIBLE)
@@ -302,6 +310,15 @@
 extern void update_process_times(int user);
 extern void scheduler_tick(void);
 
+#ifdef CONFIG_IPIPE
+void update_root_process_times(struct pt_regs *regs);
+#else  /* !CONFIG_IPIPE */
+static inline void update_root_process_times(struct pt_regs *regs)
+{
+	update_process_times(user_mode(regs));
+}
+#endif /* CONFIG_IPIPE */
+
 extern void sched_show_task(struct task_struct *p);
 
 #ifdef CONFIG_DETECT_SOFTLOCKUP
@@ -349,7 +366,7 @@
 extern signed long schedule_timeout_interruptible(signed long timeout);
 extern signed long schedule_timeout_killable(signed long timeout);
 extern signed long schedule_timeout_uninterruptible(signed long timeout);
-asmlinkage void __schedule(void);
+asmlinkage int __schedule(void);
 asmlinkage void schedule(void);
 extern int mutex_spin_on_owner(struct mutex *lock, struct thread_info *owner);
 
@@ -1493,6 +1510,9 @@
 #endif
 	atomic_t fs_excl;	/* holding fs exclusive resources */
 	struct rcu_head rcu;
+#ifdef CONFIG_IPIPE
+	void *ptd[IPIPE_ROOT_NPTDKEYS];
+#endif
 
 	/*
 	 * cache last used pipe for splice
@@ -1733,6 +1753,11 @@
 #define PF_EXITING	0x00000004	/* getting shut down */
 #define PF_EXITPIDONE	0x00000008	/* pi exit done on shut down */
 #define PF_VCPU		0x00000010	/* I'm a virtual CPU */
+#ifdef CONFIG_IPIPE
+#define PF_EVNOTIFY	0x00000020	/* Notify other domains about internal events */
+#else
+#define PF_EVNOTIFY	0
+#endif /* CONFIG_IPIPE */
 #define PF_FORKNOEXEC	0x00000040	/* forked but didn't exec */
 #define PF_MCE_PROCESS  0x00000080      /* process policy on mce errors */
 #define PF_SUPERPRIV	0x00000100	/* used super-user privileges */
diff -urN source_powerpc_none/include/linux/spinlock.h source_powerpc_none.ipipe/include/linux/spinlock.h
--- source_powerpc_none/include/linux/spinlock.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/spinlock.h	2009-12-22 12:44:08.000000000 -0500
@@ -90,10 +90,14 @@
 # include <linux/spinlock_up.h>
 #endif
 
+#undef TYPE_EQUAL
+#define TYPE_EQUAL(lock, type) \
+	__builtin_types_compatible_p(typeof(lock), type *)
+
 #ifdef CONFIG_DEBUG_SPINLOCK
   extern void __spin_lock_init(spinlock_t *lock, const char *name,
 			       struct lock_class_key *key);
-# define spin_lock_init(lock)					\
+# define _spin_lock_init(lock)				\
 do {								\
 	static struct lock_class_key __key;			\
 								\
@@ -101,10 +105,21 @@
 } while (0)
 
 #else
-# define spin_lock_init(lock)					\
+# define _spin_lock_init(lock)				\
 	do { *(lock) = SPIN_LOCK_UNLOCKED; } while (0)
 #endif
 
+# define spin_lock_init(lock)						\
+	do {								\
+		if (TYPE_EQUAL((lock), __ipipe_spinlock_t))		\
+			do {						\
+				IPIPE_DEFINE_SPINLOCK(__lock__);	\
+				*((ipipe_spinlock_t *)lock) = __lock__; \
+			} while(0);					\
+		else							\
+			_spin_lock_init((spinlock_t *)lock);		\
+	} while(0)
+
 #ifdef CONFIG_DEBUG_SPINLOCK
   extern void __rwlock_init(rwlock_t *lock, const char *name,
 			    struct lock_class_key *key);
@@ -186,7 +201,94 @@
 #define read_trylock(lock)		__cond_lock(lock, _read_trylock(lock))
 #define write_trylock(lock)		__cond_lock(lock, _write_trylock(lock))
 
-#define spin_lock(lock)			_spin_lock(lock)
+#define PICK_SPINOP(op, lock)						\
+do {									\
+	if (TYPE_EQUAL((lock), __ipipe_spinlock_t))			\
+		__raw_spin##op(&((__ipipe_spinlock_t *)(lock))->__raw_lock); \
+	else if (TYPE_EQUAL(lock, spinlock_t))				\
+		_spin##op((spinlock_t *)(lock));			\
+} while (0)
+
+#define PICK_SPINOP_RAW(op, lock)					\
+do {									\
+	if (TYPE_EQUAL((lock), __ipipe_spinlock_t))			\
+		__raw_spin##op(&((__ipipe_spinlock_t *)(lock))->__raw_lock); \
+	else if (TYPE_EQUAL(lock, spinlock_t))				\
+		__raw_spin##op(&((spinlock_t *)(lock))->raw_lock);	\
+} while (0)
+
+#define PICK_SPINLOCK_IRQ(lock)						\
+do {									\
+	if (TYPE_EQUAL((lock), __ipipe_spinlock_t)) {			\
+		__ipipe_spin_lock_irq(&((__ipipe_spinlock_t *)(lock))->__raw_lock); \
+	} else if (TYPE_EQUAL(lock, spinlock_t))			\
+		_spin_lock_irq((spinlock_t *)(lock));			\
+} while (0)
+
+#define PICK_SPINUNLOCK_IRQ(lock)					\
+do {									\
+	if (TYPE_EQUAL((lock), __ipipe_spinlock_t)) {			\
+		__ipipe_spin_unlock_irq(&((__ipipe_spinlock_t *)(lock))->__raw_lock); \
+	} else if (TYPE_EQUAL(lock, spinlock_t))			\
+		_spin_unlock_irq((spinlock_t *)(lock));			\
+} while (0)
+
+#define PICK_SPINLOCK_IRQ_RAW(lock)					\
+do {									\
+	if (TYPE_EQUAL((lock), __ipipe_spinlock_t)) {			\
+		__ipipe_spin_lock_irq(&((__ipipe_spinlock_t *)(lock))->__raw_lock); \
+	} else if (TYPE_EQUAL(lock, spinlock_t))			\
+		local_irq_disable();					\
+		__raw_spin_lock(&((spinlock_t *)(lock))->raw_lock);	\
+} while (0)
+
+#define PICK_SPINUNLOCK_IRQ_RAW(lock)					\
+do {									\
+	if (TYPE_EQUAL((lock), __ipipe_spinlock_t)) {			\
+		__ipipe_spin_unlock_irq(&((__ipipe_spinlock_t *)(lock))->__raw_lock); \
+	} else if (TYPE_EQUAL(lock, spinlock_t))			\
+		__raw_spin_unlock(&((spinlock_t *)(lock))->raw_lock);	\
+		local_irq_enable();					\
+} while (0)
+
+#if defined(CONFIG_SMP) || defined(CONFIG_DEBUG_SPINLOCK)
+extern int __bad_spinlock_type(void);
+
+#define PICK_SPINLOCK_IRQSAVE(lock, flags)				\
+do {									\
+	if (TYPE_EQUAL((lock), __ipipe_spinlock_t)) {			\
+		(flags) = __ipipe_spin_lock_irqsave(&((__ipipe_spinlock_t *)(lock))->__raw_lock); \
+	} else if (TYPE_EQUAL(lock, spinlock_t))			\
+		flags = _spin_lock_irqsave((spinlock_t *)(lock));	\
+	else __bad_spinlock_type();					\
+} while (0)
+#define PICK_SPINLOCK_IRQSAVE_NESTED(lock, flags, subclass)		\
+do {									\
+	if (TYPE_EQUAL((lock), __ipipe_spinlock_t)) {			\
+		(flags) = __ipipe_spin_lock_irqsave(&((__ipipe_spinlock_t *)(lock))->__raw_lock); \
+	} else if (TYPE_EQUAL(lock, spinlock_t))			\
+		flags = _spin_lock_irqsave_nested((spinlock_t *)(lock), subclass); \
+	else __bad_spinlock_type();					\
+} while (0)
+#else
+#define PICK_SPINLOCK_IRQSAVE(lock, flags)				\
+do {									\
+	if (TYPE_EQUAL((lock), __ipipe_spinlock_t)) {			\
+		(flags) = __ipipe_spin_lock_irqsave(&((__ipipe_spinlock_t *)(lock))->__raw_lock); \
+	} else if (TYPE_EQUAL(lock, spinlock_t))			\
+		_spin_lock_irqsave((spinlock_t *)(lock), flags);	\
+} while (0)
+#endif
+
+#define PICK_SPINUNLOCK_IRQRESTORE(lock, flags)				\
+	do {								\
+	if (TYPE_EQUAL((lock), __ipipe_spinlock_t)) {			\
+		__ipipe_spin_unlock_irqrestore(&((__ipipe_spinlock_t *)(lock))->__raw_lock, flags); \
+	} else if (TYPE_EQUAL(lock, spinlock_t))			\
+		_spin_unlock_irqrestore((spinlock_t *)(lock), flags);	\
+} while (0)
+
+#define spin_lock(lock)	PICK_SPINOP(_lock, lock)
 
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 # define spin_lock_nested(lock, subclass) _spin_lock_nested(lock, subclass)
@@ -208,7 +310,7 @@
 #define spin_lock_irqsave(lock, flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
-		flags = _spin_lock_irqsave(lock);	\
+		PICK_SPINLOCK_IRQSAVE(lock, flags);	\
 	} while (0)
 #define read_lock_irqsave(lock, flags)			\
 	do {						\
@@ -225,13 +327,13 @@
 #define spin_lock_irqsave_nested(lock, flags, subclass)			\
 	do {								\
 		typecheck(unsigned long, flags);			\
-		flags = _spin_lock_irqsave_nested(lock, subclass);	\
+		PICK_SPINLOCK_IRQSAVE_NESTED(lock, flags, subclass);	\
 	} while (0)
 #else
 #define spin_lock_irqsave_nested(lock, flags, subclass)			\
 	do {								\
 		typecheck(unsigned long, flags);			\
-		flags = _spin_lock_irqsave(lock);			\
+		PICK_SPINLOCK_IRQSAVE(lock, flags);			\
 	} while (0)
 #endif
 
@@ -240,7 +342,7 @@
 #define spin_lock_irqsave(lock, flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
-		_spin_lock_irqsave(lock, flags);	\
+		PICK_SPINLOCK_IRQSAVE(lock, flags);	\
 	} while (0)
 #define read_lock_irqsave(lock, flags)			\
 	do {						\
@@ -257,23 +359,23 @@
 
 #endif
 
-#define spin_lock_irq(lock)		_spin_lock_irq(lock)
+#define spin_lock_irq(lock)		PICK_SPINLOCK_IRQ(lock)
 #define spin_lock_bh(lock)		_spin_lock_bh(lock)
 #define read_lock_irq(lock)		_read_lock_irq(lock)
 #define read_lock_bh(lock)		_read_lock_bh(lock)
 #define write_lock_irq(lock)		_write_lock_irq(lock)
 #define write_lock_bh(lock)		_write_lock_bh(lock)
-#define spin_unlock(lock)		_spin_unlock(lock)
+#define spin_unlock(lock)		PICK_SPINOP(_unlock, lock)
 #define read_unlock(lock)		_read_unlock(lock)
 #define write_unlock(lock)		_write_unlock(lock)
-#define spin_unlock_irq(lock)		_spin_unlock_irq(lock)
+#define spin_unlock_irq(lock)		PICK_SPINUNLOCK_IRQ(lock)
 #define read_unlock_irq(lock)		_read_unlock_irq(lock)
 #define write_unlock_irq(lock)		_write_unlock_irq(lock)
 
 #define spin_unlock_irqrestore(lock, flags)		\
 	do {						\
 		typecheck(unsigned long, flags);	\
-		_spin_unlock_irqrestore(lock, flags);	\
+		PICK_SPINUNLOCK_IRQRESTORE(lock, flags);	\
 	} while (0)
 #define spin_unlock_bh(lock)		_spin_unlock_bh(lock)
 
@@ -346,4 +448,29 @@
 # include <linux/spinlock_api_up.h>
 #endif
 
+#ifdef CONFIG_IPIPE
+void __ipipe_spin_lock_irq(raw_spinlock_t *lock);
+void __ipipe_spin_unlock_irq(raw_spinlock_t *lock);
+unsigned long __ipipe_spin_lock_irqsave(raw_spinlock_t *lock);
+void __ipipe_spin_unlock_irqrestore(raw_spinlock_t *lock,
+					     unsigned long x);
+void __ipipe_spin_unlock_irqbegin(ipipe_spinlock_t *lock);
+void __ipipe_spin_unlock_irqcomplete(unsigned long x);
+#define spin_lock_irqsave_cond(lock, flags) \
+	spin_lock_irqsave(lock, flags)
+#define spin_unlock_irqrestore_cond(lock, flags) \
+	spin_unlock_irqrestore(lock, flags)
+#else
+#define spin_lock_irqsave_cond(lock, flags) \
+	do { (void)(flags); spin_lock(lock); } while(0)
+#define spin_unlock_irqrestore_cond(lock, flags) \
+	spin_unlock(lock)
+#define __ipipe_spin_lock_irq(lock)		do { } while(0)
+#define __ipipe_spin_unlock_irq(lock)		do { } while(0)
+#define __ipipe_spin_lock_irqsave(lock)		0
+#define __ipipe_spin_unlock_irqrestore(lock, x)	do { (void)(x); } while(0)
+#define __ipipe_spin_unlock_irqbegin(lock)	do { } while(0)
+#define __ipipe_spin_unlock_irqcomplete(x)	do { (void)(x); } while(0)
+#endif
+
 #endif /* __LINUX_SPINLOCK_H */
diff -urN source_powerpc_none/include/linux/spinlock_types.h source_powerpc_none.ipipe/include/linux/spinlock_types.h
--- source_powerpc_none/include/linux/spinlock_types.h	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/include/linux/spinlock_types.h	2009-12-22 12:44:08.000000000 -0500
@@ -31,6 +31,10 @@
 #endif
 } spinlock_t;
 
+typedef struct {
+	raw_spinlock_t __raw_lock;
+} __ipipe_spinlock_t;
+
 #define SPINLOCK_MAGIC		0xdead4ead
 
 typedef struct {
@@ -92,9 +96,21 @@
  * __SPIN_LOCK_UNLOCKED()/__RW_LOCK_UNLOCKED() as appropriate.
  */
 #define SPIN_LOCK_UNLOCKED	__SPIN_LOCK_UNLOCKED(old_style_spin_init)
+#define IPIPE_SPIN_LOCK_UNLOCKED					\
+	(__ipipe_spinlock_t) {	.__raw_lock = __RAW_SPIN_LOCK_UNLOCKED }
 #define RW_LOCK_UNLOCKED	__RW_LOCK_UNLOCKED(old_style_rw_init)
 
 #define DEFINE_SPINLOCK(x)	spinlock_t x = __SPIN_LOCK_UNLOCKED(x)
 #define DEFINE_RWLOCK(x)	rwlock_t x = __RW_LOCK_UNLOCKED(x)
 
+#ifdef CONFIG_IPIPE
+# define ipipe_spinlock_t		__ipipe_spinlock_t
+# define IPIPE_DEFINE_SPINLOCK(x)	ipipe_spinlock_t x = IPIPE_SPIN_LOCK_UNLOCKED
+# define IPIPE_DECLARE_SPINLOCK(x)	extern ipipe_spinlock_t x
+#else
+# define ipipe_spinlock_t		spinlock_t
+# define IPIPE_DEFINE_SPINLOCK(x)	DEFINE_SPINLOCK(x)
+# define IPIPE_DECLARE_SPINLOCK(x)	extern spinlock_t x
+#endif
+
 #endif /* __LINUX_SPINLOCK_TYPES_H */
diff -urN source_powerpc_none/init/Kconfig source_powerpc_none.ipipe/init/Kconfig
--- source_powerpc_none/init/Kconfig	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/init/Kconfig	2009-12-22 12:44:08.000000000 -0500
@@ -78,6 +78,7 @@
 
 config LOCALVERSION
 	string "Local version - append to kernel release"
+	default "-ipipe"
 	help
 	  Append an extra string to the end of your kernel version.
 	  This will show up when you type uname, for example.
diff -urN source_powerpc_none/init/main.c source_powerpc_none.ipipe/init/main.c
--- source_powerpc_none/init/main.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/init/main.c	2009-12-22 12:44:08.000000000 -0500
@@ -535,7 +535,7 @@
 
 	cgroup_init_early();
 
-	local_irq_disable();
+	local_irq_disable_hw();
 	early_boot_irqs_off();
 	early_init_irq_lock_class();
 
@@ -598,6 +598,11 @@
 	softirq_init();
 	timekeeping_init();
 	time_init();
+	/*
+	 * We need to wait for the interrupt and time subsystems to be
+	 * initialized before enabling the pipeline.
+	 */
+	ipipe_init();
 	profile_init();
 	if (!irqs_disabled())
 		printk(KERN_CRIT "start_kernel(): bug: interrupts were "
@@ -779,6 +784,7 @@
 	init_tmpfs();
 	driver_init();
 	init_irq_proc();
+ 	ipipe_init_proc();
 	do_ctors();
 	do_initcalls();
 }
diff -urN source_powerpc_none/kernel/Makefile source_powerpc_none.ipipe/kernel/Makefile
--- source_powerpc_none/kernel/Makefile	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/Makefile	2009-12-22 12:44:08.000000000 -0500
@@ -83,6 +83,7 @@
 obj-$(CONFIG_TREE_PREEMPT_RCU) += rcutree.o
 obj-$(CONFIG_TREE_RCU_TRACE) += rcutree_trace.o
 obj-$(CONFIG_RELAY) += relay.o
+obj-$(CONFIG_IPIPE) += ipipe/
 obj-$(CONFIG_SYSCTL) += utsname_sysctl.o
 obj-$(CONFIG_TASK_DELAY_ACCT) += delayacct.o
 obj-$(CONFIG_TASKSTATS) += taskstats.o tsacct.o
diff -urN source_powerpc_none/kernel/exit.c source_powerpc_none.ipipe/kernel/exit.c
--- source_powerpc_none/kernel/exit.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/exit.c	2009-12-22 12:44:08.000000000 -0500
@@ -963,6 +963,7 @@
 		acct_process();
 	trace_sched_process_exit(tsk);
 
+  	ipipe_exit_notify(tsk);
 	exit_sem(tsk);
 	exit_files(tsk);
 	exit_fs(tsk);
diff -urN source_powerpc_none/kernel/fork.c source_powerpc_none.ipipe/kernel/fork.c
--- source_powerpc_none/kernel/fork.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/fork.c	2009-12-22 12:44:08.000000000 -0500
@@ -511,6 +511,7 @@
 		exit_aio(mm);
 		ksm_exit(mm);
 		exit_mmap(mm);
+ 		ipipe_cleanup_notify(mm);
 		set_mm_exe_file(mm, NULL);
 		if (!list_empty(&mm->mmlist)) {
 			spin_lock(&mmlist_lock);
@@ -918,7 +919,7 @@
 {
 	unsigned long new_flags = p->flags;
 
-	new_flags &= ~PF_SUPERPRIV;
+ 	new_flags &= ~(PF_SUPERPRIV | PF_EVNOTIFY);
 	new_flags |= PF_FORKNOEXEC;
 	new_flags |= PF_STARTING;
 	p->flags = new_flags;
@@ -1304,6 +1305,9 @@
 	proc_fork_connector(p);
 	cgroup_post_fork(p);
 	perf_event_fork(p);
+#ifdef CONFIG_IPIPE
+	memset(p->ptd, 0, sizeof(p->ptd));
+#endif /* CONFIG_IPIPE */
 	return p;
 
 bad_fork_free_pid:
@@ -1700,11 +1704,14 @@
 		}
 
 		if (new_mm) {
+			unsigned long flags;
 			mm = current->mm;
 			active_mm = current->active_mm;
 			current->mm = new_mm;
+			ipipe_mm_switch_protect(flags);
 			current->active_mm = new_mm;
 			activate_mm(active_mm, new_mm);
+			ipipe_mm_switch_unprotect(flags);
 			new_mm = mm;
 		}
 
diff -urN source_powerpc_none/kernel/ipipe/Kconfig source_powerpc_none.ipipe/kernel/ipipe/Kconfig
--- source_powerpc_none/kernel/ipipe/Kconfig	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/ipipe/Kconfig	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,30 @@
+config IPIPE
+	bool "Interrupt pipeline"
+	default y
+	---help---
+	  Activate this option if you want the interrupt pipeline to be
+	  compiled in.
+
+config IPIPE_DOMAINS
+	int "Max domains"
+	depends on IPIPE
+	default 4
+	---help---
+	The maximum number of I-pipe domains to run concurrently.
+
+config IPIPE_COMPAT
+	bool "Maintain code compatibility with older releases"
+	depends on IPIPE
+	default y
+	---help---
+	Activate this option if you want the compatibility code to be
+	defined, so that older I-pipe clients may use obsolete
+	constructs. WARNING: obsolete code will be eventually
+	deprecated in future I-pipe releases, and removed from the
+	compatibility support as time passes. Please fix I-pipe
+	clients to get rid of such uses as soon as possible.
+
+config IPIPE_DELAYED_ATOMICSW
+       bool
+       depends on IPIPE
+       default n
diff -urN source_powerpc_none/kernel/ipipe/Kconfig.debug source_powerpc_none.ipipe/kernel/ipipe/Kconfig.debug
--- source_powerpc_none/kernel/ipipe/Kconfig.debug	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/ipipe/Kconfig.debug	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,96 @@
+config IPIPE_DEBUG
+	bool "I-pipe debugging"
+	depends on IPIPE
+
+config IPIPE_DEBUG_CONTEXT
+	bool "Check for illicit cross-domain calls"
+	depends on IPIPE_DEBUG
+	default y
+	---help---
+	  Enable this feature to arm checkpoints in the kernel that
+	  verify the correct invocation context. On entry of critical
+	  Linux services a warning is issued if the caller is not
+	  running over the root domain.
+
+config IPIPE_DEBUG_INTERNAL
+	bool "Enable internal debug checks"
+	depends on IPIPE_DEBUG
+	default y
+	---help---
+	  When this feature is enabled, I-pipe will perform internal
+	  consistency checks of its subsystems, e.g. on per-cpu variable
+	  access.
+
+config IPIPE_TRACE
+	bool "Latency tracing"
+	depends on IPIPE_DEBUG
+	select FRAME_POINTER
+	select KALLSYMS
+	select PROC_FS
+	---help---
+	  Activate this option if you want to use per-function tracing of
+	  the kernel. The tracer will collect data via instrumentation
+	  features like the one below or with the help of explicite calls
+	  of ipipe_trace_xxx(). See include/linux/ipipe_trace.h for the
+	  in-kernel tracing API. The collected data and runtime control
+	  is available via /proc/ipipe/trace/*.
+
+if IPIPE_TRACE
+
+config IPIPE_TRACE_ENABLE
+	bool "Enable tracing on boot"
+	default y
+	---help---
+	  Disable this option if you want to arm the tracer after booting
+	  manually ("echo 1 > /proc/ipipe/tracer/enable"). This can reduce
+	  boot time on slow embedded devices due to the tracer overhead.
+
+config IPIPE_TRACE_MCOUNT
+	bool "Instrument function entries"
+	default y
+	select FUNCTION_TRACER
+	select TRACING
+	select CONTEXT_SWITCH_TRACER
+ 	select DYNAMIC_FTRACE
+	---help---
+	  When enabled, records every kernel function entry in the tracer
+	  log. While this slows down the system noticeably, it provides
+	  the highest level of information about the flow of events.
+	  However, it can be switch off in order to record only explicit
+	  I-pipe trace points.
+
+config IPIPE_TRACE_IRQSOFF
+	bool "Trace IRQs-off times"
+	default y
+	---help---
+	  Activate this option if I-pipe shall trace the longest path
+	  with hard-IRQs switched off.
+
+config IPIPE_TRACE_SHIFT
+	int "Depth of trace log (14 => 16Kpoints, 15 => 32Kpoints)"
+	range 10 18
+	default 14
+	---help---
+	  The number of trace points to hold tracing data for each
+	  trace path, as a power of 2.
+
+config IPIPE_TRACE_VMALLOC
+	bool "Use vmalloc'ed trace buffer"
+	default y if EMBEDDED
+	---help---
+	  Instead of reserving static kernel data, the required buffer
+	  is allocated via vmalloc during boot-up when this option is
+	  enabled. This can help to start systems that are low on memory,
+	  but it slightly degrades overall performance. Try this option
+	  when a traced kernel hangs unexpectedly at boot time.
+
+config IPIPE_TRACE_PANIC
+	bool "Enable panic back traces"
+	default y
+	---help---
+	  Provides services to freeze and dump a back trace on panic
+	  situations. This is used on IPIPE_DEBUG_CONTEXT exceptions
+	  as well as ordinary kernel oopses. You can control the number
+	  of printed back trace points via /proc/ipipe/trace.
+
+endif
diff -urN source_powerpc_none/kernel/ipipe/Makefile source_powerpc_none.ipipe/kernel/ipipe/Makefile
--- source_powerpc_none/kernel/ipipe/Makefile	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/ipipe/Makefile	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,3 @@
+
+obj-$(CONFIG_IPIPE)	+= core.o
+obj-$(CONFIG_IPIPE_TRACE) += tracer.o
diff -urN source_powerpc_none/kernel/ipipe/core.c source_powerpc_none.ipipe/kernel/ipipe/core.c
--- source_powerpc_none/kernel/ipipe/core.c	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/ipipe/core.c	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,1794 @@
+/* -*- linux-c -*-
+ * linux/kernel/ipipe/core.c
+ *
+ * Copyright (C) 2002-2005 Philippe Gerum.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139,
+ * USA; either version 2 of the License, or (at your option) any later
+ * version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * Architecture-independent I-PIPE core support.
+ */
+
+#include <linux/version.h>
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/sched.h>
+#include <linux/sched.h>
+#include <linux/kallsyms.h>
+#include <linux/interrupt.h>
+#include <linux/bitops.h>
+#include <linux/tick.h>
+#include <linux/prefetch.h>
+#ifdef CONFIG_PROC_FS
+#include <linux/proc_fs.h>
+#include <linux/seq_file.h>
+#endif	/* CONFIG_PROC_FS */
+#include <linux/ipipe_trace.h>
+#include <linux/ipipe_tickdev.h>
+#include <linux/irq.h>
+
+static int __ipipe_ptd_key_count;
+
+static unsigned long __ipipe_ptd_key_map;
+
+static unsigned long __ipipe_domain_slot_map;
+
+struct ipipe_domain ipipe_root;
+
+#ifndef CONFIG_SMP
+/*
+ * Create an alias to the unique root status, so that arch-dep code
+ * may get simple and easy access to this percpu variable.  We also
+ * create an array of pointers to the percpu domain data; this tends
+ * to produce a better code when reaching non-root domains. We make
+ * sure that the early boot code would be able to dereference the
+ * pointer to the root domain data safely by statically initializing
+ * its value (local_irq*() routines depend on this).
+ */
+#if __GNUC__ >= 4
+extern unsigned long __ipipe_root_status
+__attribute__((alias(__stringify(__raw_get_cpu_var(ipipe_percpu_darray)))));
+EXPORT_SYMBOL(__ipipe_root_status);
+#else /* __GNUC__ < 4 */
+/*
+ * Work around a GCC 3.x issue making alias symbols unusable as
+ * constant initializers.
+ */
+unsigned long *const __ipipe_root_status_addr =
+	&__raw_get_cpu_var(ipipe_percpu_darray)[IPIPE_ROOT_SLOT].status;
+EXPORT_SYMBOL(__ipipe_root_status_addr);
+#endif /* __GNUC__ < 4 */
+
+DEFINE_PER_CPU(struct ipipe_percpu_domain_data *, ipipe_percpu_daddr[CONFIG_IPIPE_DOMAINS]) =
+{ [IPIPE_ROOT_SLOT] = (struct ipipe_percpu_domain_data *)&__raw_get_cpu_var(ipipe_percpu_darray) };
+EXPORT_PER_CPU_SYMBOL(ipipe_percpu_daddr);
+#endif /* !CONFIG_SMP */
+
+DEFINE_PER_CPU(struct ipipe_percpu_domain_data, ipipe_percpu_darray[CONFIG_IPIPE_DOMAINS]) =
+{ [IPIPE_ROOT_SLOT] = { .status = IPIPE_STALL_MASK } }; /* Root domain stalled on each CPU at startup. */
+
+DEFINE_PER_CPU(struct ipipe_domain *, ipipe_percpu_domain) = { &ipipe_root };
+
+DEFINE_PER_CPU(unsigned long, ipipe_nmi_saved_root); /* Copy of root status during NMI */
+
+static IPIPE_DEFINE_SPINLOCK(__ipipe_pipelock);
+
+LIST_HEAD(__ipipe_pipeline);
+
+unsigned long __ipipe_virtual_irq_map;
+
+#ifdef CONFIG_PRINTK
+unsigned __ipipe_printk_virq;
+#endif /* CONFIG_PRINTK */
+
+int __ipipe_event_monitors[IPIPE_NR_EVENTS];
+
+#ifdef CONFIG_GENERIC_CLOCKEVENTS
+
+DECLARE_PER_CPU(struct tick_device, tick_cpu_device);
+
+static DEFINE_PER_CPU(struct ipipe_tick_device, ipipe_tick_cpu_device);
+
+int ipipe_request_tickdev(const char *devname,
+			  void (*emumode)(enum clock_event_mode mode,
+					  struct clock_event_device *cdev),
+			  int (*emutick)(unsigned long delta,
+					 struct clock_event_device *cdev),
+			  int cpu, unsigned long *tmfreq)
+{
+	struct ipipe_tick_device *itd;
+	struct tick_device *slave;
+	struct clock_event_device *evtdev;
+	unsigned long long freq;
+	unsigned long flags;
+	int status;
+
+	flags = ipipe_critical_enter(NULL);
+
+	itd = &per_cpu(ipipe_tick_cpu_device, cpu);
+
+	if (itd->slave != NULL) {
+		status = -EBUSY;
+		goto out;
+	}
+
+	slave = &per_cpu(tick_cpu_device, cpu);
+
+	if (strcmp(slave->evtdev->name, devname)) {
+		/*
+		 * No conflict so far with the current tick device,
+		 * check whether the requested device is sane and has
+		 * been blessed by the kernel.
+		 */
+		status = __ipipe_check_tickdev(devname) ?
+			CLOCK_EVT_MODE_UNUSED : CLOCK_EVT_MODE_SHUTDOWN;
+		goto out;
+	}
+
+	/*
+	 * Our caller asks for using the same clock event device for
+	 * ticking than we do, let's create a tick emulation device to
+	 * interpose on the set_next_event() method, so that we may
+	 * both manage the device in oneshot mode. Only the tick
+	 * emulation code will actually program the clockchip hardware
+	 * for the next shot, though.
+	 *
+	 * CAUTION: we still have to grab the tick device even when it
+	 * current runs in periodic mode, since the kernel may switch
+	 * to oneshot dynamically (highres/no_hz tick mode).
+	 */
+
+	evtdev = slave->evtdev;
+	status = evtdev->mode;
+
+        if (status == CLOCK_EVT_MODE_SHUTDOWN)
+                goto out;
+
+	itd->slave = slave;
+	itd->emul_set_mode = emumode;
+	itd->emul_set_tick = emutick;
+	itd->real_set_mode = evtdev->set_mode;
+	itd->real_set_tick = evtdev->set_next_event;
+	itd->real_max_delta_ns = evtdev->max_delta_ns;
+	itd->real_mult = evtdev->mult;
+	itd->real_shift = evtdev->shift;
+	freq = (1000000000ULL * evtdev->mult) >> evtdev->shift;
+	*tmfreq = (unsigned long)freq;
+	evtdev->set_mode = emumode;
+	evtdev->set_next_event = emutick;
+	evtdev->max_delta_ns = ULONG_MAX;
+	evtdev->mult = 1;
+	evtdev->shift = 0;
+out:
+	ipipe_critical_exit(flags);
+
+	return status;
+}
+
+void ipipe_release_tickdev(int cpu)
+{
+	struct ipipe_tick_device *itd;
+	struct tick_device *slave;
+	struct clock_event_device *evtdev;
+	unsigned long flags;
+
+	flags = ipipe_critical_enter(NULL);
+
+	itd = &per_cpu(ipipe_tick_cpu_device, cpu);
+
+	if (itd->slave != NULL) {
+		slave = &per_cpu(tick_cpu_device, cpu);
+		evtdev = slave->evtdev;
+		evtdev->set_mode = itd->real_set_mode;
+		evtdev->set_next_event = itd->real_set_tick;
+		evtdev->max_delta_ns = itd->real_max_delta_ns;
+		evtdev->mult = itd->real_mult;
+		evtdev->shift = itd->real_shift;
+		itd->slave = NULL;
+	}
+
+	ipipe_critical_exit(flags);
+}
+
+#endif /* CONFIG_GENERIC_CLOCKEVENTS */
+
+/*
+ * ipipe_init() -- Initialization routine of the IPIPE layer. Called
+ * by the host kernel early during the boot procedure.
+ */
+void __init ipipe_init(void)
+{
+	struct ipipe_domain *ipd = &ipipe_root;
+
+	__ipipe_check_platform();	/* Do platform dependent checks first. */
+
+	/*
+	 * A lightweight registration code for the root domain. We are
+	 * running on the boot CPU, hw interrupts are off, and
+	 * secondary CPUs are still lost in space.
+	 */
+
+	/* Reserve percpu data slot #0 for the root domain. */
+	ipd->slot = 0;
+	set_bit(0, &__ipipe_domain_slot_map);
+
+	ipd->name = "Linux";
+	ipd->domid = IPIPE_ROOT_ID;
+	ipd->priority = IPIPE_ROOT_PRIO;
+
+	__ipipe_init_stage(ipd);
+
+	INIT_LIST_HEAD(&ipd->p_link);
+	list_add_tail(&ipd->p_link, &__ipipe_pipeline);
+
+	__ipipe_init_platform();
+
+#ifdef CONFIG_PRINTK
+	__ipipe_printk_virq = ipipe_alloc_virq();	/* Cannot fail here. */
+	ipd->irqs[__ipipe_printk_virq].handler = &__ipipe_flush_printk;
+	ipd->irqs[__ipipe_printk_virq].cookie = NULL;
+	ipd->irqs[__ipipe_printk_virq].acknowledge = NULL;
+	ipd->irqs[__ipipe_printk_virq].control = IPIPE_HANDLE_MASK;
+#endif /* CONFIG_PRINTK */
+
+	__ipipe_enable_pipeline();
+
+	printk(KERN_INFO "I-pipe %s: pipeline enabled.\n",
+	       IPIPE_VERSION_STRING);
+}
+
+void __ipipe_init_stage(struct ipipe_domain *ipd)
+{
+	int cpu, n;
+
+	for_each_online_cpu(cpu) {
+
+		ipipe_percpudom(ipd, irqpend_himask, cpu) = 0;
+
+		for (n = 0; n < IPIPE_IRQ_IWORDS; n++) {
+			ipipe_percpudom(ipd, irqpend_lomask, cpu)[n] = 0;
+			ipipe_percpudom(ipd, irqheld_mask, cpu)[n] = 0;
+		}
+
+		for (n = 0; n < IPIPE_NR_IRQS; n++)
+			ipipe_percpudom(ipd, irqall, cpu)[n] = 0;
+
+		ipipe_percpudom(ipd, evsync, cpu) = 0;
+	}
+
+	for (n = 0; n < IPIPE_NR_IRQS; n++) {
+		ipd->irqs[n].acknowledge = NULL;
+		ipd->irqs[n].handler = NULL;
+		ipd->irqs[n].control = IPIPE_PASS_MASK;	/* Pass but don't handle */
+	}
+
+	for (n = 0; n < IPIPE_NR_EVENTS; n++)
+		ipd->evhand[n] = NULL;
+
+	ipd->evself = 0LL;
+	mutex_init(&ipd->mutex);
+
+	__ipipe_hook_critical_ipi(ipd);
+}
+
+void __ipipe_cleanup_domain(struct ipipe_domain *ipd)
+{
+	ipipe_unstall_pipeline_from(ipd);
+
+#ifdef CONFIG_SMP
+	{
+		int cpu;
+
+		for_each_online_cpu(cpu) {
+			while (ipipe_percpudom(ipd, irqpend_himask, cpu) != 0)
+				cpu_relax();
+		}
+	}
+#else
+	__raw_get_cpu_var(ipipe_percpu_daddr)[ipd->slot] = NULL;
+#endif
+
+	clear_bit(ipd->slot, &__ipipe_domain_slot_map);
+}
+
+void __ipipe_unstall_root(void)
+{
+	struct ipipe_percpu_domain_data *p;
+
+        local_irq_disable_hw();
+
+#ifdef CONFIG_IPIPE_DEBUG_INTERNAL
+	/* This helps catching bad usage from assembly call sites. */
+	BUG_ON(!__ipipe_root_domain_p);
+#endif
+
+	p = ipipe_root_cpudom_ptr();
+
+        __clear_bit(IPIPE_STALL_FLAG, &p->status);
+
+        if (unlikely(p->irqpend_himask != 0))
+                __ipipe_sync_pipeline(IPIPE_IRQMASK_ANY);
+
+        local_irq_enable_hw();
+}
+
+void __ipipe_restore_root(unsigned long x)
+{
+#ifdef CONFIG_IPIPE_DEBUG_INTERNAL
+	BUG_ON(!ipipe_root_domain_p);
+#endif
+
+	if (x)
+		__ipipe_stall_root();
+	else
+		__ipipe_unstall_root();
+}
+
+void ipipe_stall_pipeline_from(struct ipipe_domain *ipd)
+{
+	unsigned long flags;
+	/*
+	 * We have to prevent against race on updating the status
+	 * variable _and_ CPU migration at the same time, so disable
+	 * hw IRQs here.
+	 */
+	local_irq_save_hw(flags);
+
+	__set_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status));
+
+	if (!__ipipe_pipeline_head_p(ipd))
+		local_irq_restore_hw(flags);
+}
+
+unsigned long ipipe_test_and_stall_pipeline_from(struct ipipe_domain *ipd)
+{
+	unsigned long flags, x;
+
+	/* See ipipe_stall_pipeline_from() */
+	local_irq_save_hw(flags);
+
+	x = __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status));
+
+	if (!__ipipe_pipeline_head_p(ipd))
+		local_irq_restore_hw(flags);
+
+	return x;
+}
+
+unsigned long ipipe_test_and_unstall_pipeline_from(struct ipipe_domain *ipd)
+{
+	unsigned long flags, x;
+	struct list_head *pos;
+
+	local_irq_save_hw(flags);
+
+	x = __test_and_clear_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status));
+
+	if (ipd == __ipipe_current_domain)
+		pos = &ipd->p_link;
+	else
+		pos = __ipipe_pipeline.next;
+
+	__ipipe_walk_pipeline(pos);
+
+	if (likely(__ipipe_pipeline_head_p(ipd)))
+		local_irq_enable_hw();
+	else
+		local_irq_restore_hw(flags);
+
+	return x;
+}
+
+void ipipe_restore_pipeline_from(struct ipipe_domain *ipd,
+					  unsigned long x)
+{
+	if (x)
+		ipipe_stall_pipeline_from(ipd);
+	else
+		ipipe_unstall_pipeline_from(ipd);
+}
+
+void ipipe_unstall_pipeline_head(void)
+{
+	struct ipipe_percpu_domain_data *p = ipipe_head_cpudom_ptr();
+
+	local_irq_disable_hw();
+
+	__clear_bit(IPIPE_STALL_FLAG, &p->status);
+
+	if (unlikely(p->irqpend_himask != 0)) {
+		struct ipipe_domain *head_domain = __ipipe_pipeline_head();
+		if (likely(head_domain == __ipipe_current_domain))
+			__ipipe_sync_pipeline(IPIPE_IRQMASK_ANY);
+		else
+			__ipipe_walk_pipeline(&head_domain->p_link);
+        }
+
+	local_irq_enable_hw();
+}
+
+void __ipipe_restore_pipeline_head(unsigned long x)
+{
+	struct ipipe_percpu_domain_data *p = ipipe_head_cpudom_ptr();
+
+	local_irq_disable_hw();
+
+	if (x) {
+#ifdef CONFIG_DEBUG_KERNEL
+		static int warned;
+		if (!warned && test_and_set_bit(IPIPE_STALL_FLAG, &p->status)) {
+			/*
+			 * Already stalled albeit ipipe_restore_pipeline_head()
+			 * should have detected it? Send a warning once.
+			 */
+			warned = 1;
+			printk(KERN_WARNING
+				   "I-pipe: ipipe_restore_pipeline_head() optimization failed.\n");
+			dump_stack();
+		}
+#else /* !CONFIG_DEBUG_KERNEL */
+		set_bit(IPIPE_STALL_FLAG, &p->status);
+#endif /* CONFIG_DEBUG_KERNEL */
+	}
+	else {
+		__clear_bit(IPIPE_STALL_FLAG, &p->status);
+		if (unlikely(p->irqpend_himask != 0)) {
+			struct ipipe_domain *head_domain = __ipipe_pipeline_head();
+			if (likely(head_domain == __ipipe_current_domain))
+				__ipipe_sync_pipeline(IPIPE_IRQMASK_ANY);
+			else
+				__ipipe_walk_pipeline(&head_domain->p_link);
+		}
+		local_irq_enable_hw();
+	}
+}
+
+void __ipipe_spin_lock_irq(raw_spinlock_t *lock)
+{
+	local_irq_disable_hw();
+	__raw_spin_lock(lock);
+	__set_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status));
+}
+
+void __ipipe_spin_unlock_irq(raw_spinlock_t *lock)
+{
+	__raw_spin_unlock(lock);
+	__clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status));
+	local_irq_enable_hw();
+}
+
+unsigned long __ipipe_spin_lock_irqsave(raw_spinlock_t *lock)
+{
+	unsigned long flags;
+	int s;
+
+	local_irq_save_hw(flags);
+	__raw_spin_lock(lock);
+	s = __test_and_set_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status));
+
+	return raw_mangle_irq_bits(s, flags);
+}
+
+void __ipipe_spin_unlock_irqrestore(raw_spinlock_t *lock, unsigned long x)
+{
+	__raw_spin_unlock(lock);
+	if (!raw_demangle_irq_bits(&x))
+		__clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status));
+	local_irq_restore_hw(x);
+}
+
+void __ipipe_spin_unlock_irqbegin(ipipe_spinlock_t *lock)
+{
+	__raw_spin_unlock(&lock->__raw_lock);
+}
+
+void __ipipe_spin_unlock_irqcomplete(unsigned long x)
+{
+	if (!raw_demangle_irq_bits(&x))
+		__clear_bit(IPIPE_STALL_FLAG, &ipipe_this_cpudom_var(status));
+	local_irq_restore_hw(x);
+}
+
+/* Must be called hw IRQs off. */
+void __ipipe_set_irq_pending(struct ipipe_domain *ipd, unsigned irq)
+{
+	int level = irq >> IPIPE_IRQ_ISHIFT, rank = irq & IPIPE_IRQ_IMASK;
+	struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(ipd);
+
+	prefetchw(p);
+	
+	if (likely(!test_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control))) {
+		__set_bit(rank, &p->irqpend_lomask[level]);
+		__set_bit(level, &p->irqpend_himask);
+	} else
+		__set_bit(rank, &p->irqheld_mask[level]);
+
+	p->irqall[irq]++;
+}
+
+/* Must be called hw IRQs off. */
+void __ipipe_lock_irq(struct ipipe_domain *ipd, int cpu, unsigned irq)
+{
+	struct ipipe_percpu_domain_data *p;
+	int level, rank;
+
+	if (unlikely(test_and_set_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control)))
+		return;
+
+	level = irq >> IPIPE_IRQ_ISHIFT;
+	rank = irq & IPIPE_IRQ_IMASK;
+	p = ipipe_percpudom_ptr(ipd, cpu);
+
+	if (__test_and_clear_bit(rank, &p->irqpend_lomask[level]))
+		__set_bit(rank, &p->irqheld_mask[level]);
+	if (p->irqpend_lomask[level] == 0)
+		__clear_bit(level, &p->irqpend_himask);
+}
+
+/* Must be called hw IRQs off. */
+void __ipipe_unlock_irq(struct ipipe_domain *ipd, unsigned irq)
+{
+	struct ipipe_percpu_domain_data *p;
+	int cpu, level, rank;
+
+	if (unlikely(!test_and_clear_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control)))
+		return;
+
+	level = irq >> IPIPE_IRQ_ISHIFT, rank = irq & IPIPE_IRQ_IMASK;
+	for_each_online_cpu(cpu) {
+		p = ipipe_percpudom_ptr(ipd, cpu);
+		if (test_and_clear_bit(rank, &p->irqheld_mask[level])) {
+			/* We need atomic ops here: */
+			set_bit(rank, &p->irqpend_lomask[level]);
+			set_bit(level, &p->irqpend_himask);
+		}
+	}
+}
+
+/*
+ * __ipipe_walk_pipeline(): Plays interrupts pending in the log. Must
+ * be called with local hw interrupts disabled.
+ */
+void __ipipe_walk_pipeline(struct list_head *pos)
+{
+	struct ipipe_domain *this_domain = __ipipe_current_domain, *next_domain;
+	struct ipipe_percpu_domain_data *p, *np;
+
+	p = ipipe_cpudom_ptr(this_domain);
+
+	while (pos != &__ipipe_pipeline) {
+
+		next_domain = list_entry(pos, struct ipipe_domain, p_link);
+		np = ipipe_cpudom_ptr(next_domain);
+
+		if (test_bit(IPIPE_STALL_FLAG, &np->status))
+			break;	/* Stalled stage -- do not go further. */
+
+		if (np->irqpend_himask) {
+			if (next_domain == this_domain)
+				__ipipe_sync_pipeline(IPIPE_IRQMASK_ANY);
+			else {
+
+				p->evsync = 0;
+				__ipipe_current_domain = next_domain;
+				ipipe_suspend_domain();	/* Sync stage and propagate interrupts. */
+
+				if (__ipipe_current_domain == next_domain)
+					__ipipe_current_domain = this_domain;
+				/*
+				 * Otherwise, something changed the current domain under our
+				 * feet recycling the register set; do not override the new
+				 * domain.
+				 */
+
+				if (p->irqpend_himask &&
+				    !test_bit(IPIPE_STALL_FLAG, &p->status))
+					__ipipe_sync_pipeline(IPIPE_IRQMASK_ANY);
+			}
+			break;
+		} else if (next_domain == this_domain)
+			break;
+
+		pos = next_domain->p_link.next;
+	}
+}
+
+/*
+ * ipipe_suspend_domain() -- Suspend the current domain, switching to
+ * the next one which has pending work down the pipeline.
+ */
+void ipipe_suspend_domain(void)
+{
+	struct ipipe_domain *this_domain, *next_domain;
+	struct ipipe_percpu_domain_data *p;
+	struct list_head *ln;
+	unsigned long flags;
+
+	local_irq_save_hw(flags);
+
+	this_domain = next_domain = __ipipe_current_domain;
+	p = ipipe_cpudom_ptr(this_domain);
+	p->status &= ~(IPIPE_STALL_MASK|IPIPE_SYNC_MASK);
+
+	if (p->irqpend_himask != 0)
+		goto sync_stage;
+
+	for (;;) {
+		ln = next_domain->p_link.next;
+
+		if (ln == &__ipipe_pipeline)
+			break;
+
+		next_domain = list_entry(ln, struct ipipe_domain, p_link);
+		p = ipipe_cpudom_ptr(next_domain);
+
+		if (p->status & IPIPE_STALL_MASK)
+			break;
+
+		if (p->irqpend_himask == 0)
+			continue;
+
+		__ipipe_current_domain = next_domain;
+sync_stage:
+		__ipipe_sync_pipeline(IPIPE_IRQMASK_ANY);
+
+		if (__ipipe_current_domain != next_domain)
+			/*
+			 * Something has changed the current domain under our
+			 * feet, recycling the register set; take note.
+			 */
+			this_domain = __ipipe_current_domain;
+	}
+
+	__ipipe_current_domain = this_domain;
+
+	local_irq_restore_hw(flags);
+}
+
+
+/* ipipe_alloc_virq() -- Allocate a pipelined virtual/soft interrupt.
+ * Virtual interrupts are handled in exactly the same way than their
+ * hw-generated counterparts wrt pipelining.
+ */
+unsigned ipipe_alloc_virq(void)
+{
+	unsigned long flags, irq = 0;
+	int ipos;
+
+	spin_lock_irqsave(&__ipipe_pipelock, flags);
+
+	if (__ipipe_virtual_irq_map != ~0) {
+		ipos = ffz(__ipipe_virtual_irq_map);
+		set_bit(ipos, &__ipipe_virtual_irq_map);
+		irq = ipos + IPIPE_VIRQ_BASE;
+	}
+
+	spin_unlock_irqrestore(&__ipipe_pipelock, flags);
+
+	return irq;
+}
+
+/* ipipe_virtualize_irq() -- Attach a handler (and optionally a hw
+   acknowledge routine) to an interrupt for a given domain. */
+
+int ipipe_virtualize_irq(struct ipipe_domain *ipd,
+			 unsigned irq,
+			 ipipe_irq_handler_t handler,
+			 void *cookie,
+			 ipipe_irq_ackfn_t acknowledge,
+			 unsigned modemask)
+{
+	ipipe_irq_handler_t old_handler;
+	unsigned long flags;
+	int err;
+
+	if (irq >= IPIPE_NR_IRQS)
+		return -EINVAL;
+
+	if (ipd->irqs[irq].control & IPIPE_SYSTEM_MASK)
+		return -EPERM;
+
+	if (!test_bit(IPIPE_AHEAD_FLAG, &ipd->flags))
+		/* Silently unwire interrupts for non-heading domains. */
+		modemask &= ~IPIPE_WIRED_MASK;
+
+	spin_lock_irqsave(&__ipipe_pipelock, flags);
+
+	old_handler = ipd->irqs[irq].handler;
+
+	if (handler != NULL) {
+		if (handler == IPIPE_SAME_HANDLER) {
+			handler = old_handler;
+			cookie = ipd->irqs[irq].cookie;
+
+			if (handler == NULL) {
+				err = -EINVAL;
+				goto unlock_and_exit;
+			}
+		} else if ((modemask & IPIPE_EXCLUSIVE_MASK) != 0 &&
+			   old_handler != NULL) {
+			err = -EBUSY;
+			goto unlock_and_exit;
+		}
+
+		/* Wired interrupts can only be delivered to domains
+		 * always heading the pipeline, and using dynamic
+		 * propagation. */
+
+		if ((modemask & IPIPE_WIRED_MASK) != 0) {
+			if ((modemask & (IPIPE_PASS_MASK | IPIPE_STICKY_MASK)) != 0) {
+				err = -EINVAL;
+				goto unlock_and_exit;
+			}
+			modemask |= (IPIPE_HANDLE_MASK);
+		}
+
+		if ((modemask & IPIPE_STICKY_MASK) != 0)
+			modemask |= IPIPE_HANDLE_MASK;
+	} else
+		modemask &=
+		    ~(IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK |
+		      IPIPE_EXCLUSIVE_MASK | IPIPE_WIRED_MASK);
+
+	if (acknowledge == NULL && !ipipe_virtual_irq_p(irq))
+		/*
+		 * Acknowledge handler unspecified for a hw interrupt:
+		 * use the Linux-defined handler instead.
+		 */
+		acknowledge = ipipe_root_domain->irqs[irq].acknowledge;
+
+	ipd->irqs[irq].handler = handler;
+	ipd->irqs[irq].cookie = cookie;
+	ipd->irqs[irq].acknowledge = acknowledge;
+	ipd->irqs[irq].control = modemask;
+
+	if (irq < NR_IRQS && !ipipe_virtual_irq_p(irq)) {
+		if (handler != NULL) {
+			__ipipe_enable_irqdesc(ipd, irq);
+
+			if ((modemask & IPIPE_ENABLE_MASK) != 0) {
+				if (ipd != __ipipe_current_domain) {
+		/*
+		 * IRQ enable/disable state is domain-sensitive, so we
+		 * may not change it for another domain. What is
+		 * allowed however is forcing some domain to handle an
+		 * interrupt source, by passing the proper 'ipd'
+		 * descriptor which thus may be different from
+		 * __ipipe_current_domain.
+		 */
+					err = -EPERM;
+					goto unlock_and_exit;
+				}
+				__ipipe_enable_irq(irq);
+			}
+		} else if (old_handler != NULL)
+				__ipipe_disable_irqdesc(ipd, irq);
+	}
+
+	err = 0;
+
+      unlock_and_exit:
+
+	spin_unlock_irqrestore(&__ipipe_pipelock, flags);
+
+	return err;
+}
+
+/* ipipe_control_irq() -- Change modes of a pipelined interrupt for
+ * the current domain. */
+
+int ipipe_control_irq(unsigned irq, unsigned clrmask, unsigned setmask)
+{
+	struct ipipe_domain *ipd;
+	unsigned long flags;
+
+	if (irq >= IPIPE_NR_IRQS)
+		return -EINVAL;
+
+	spin_lock_irqsave(&__ipipe_pipelock, flags);
+
+	ipd = __ipipe_current_domain;
+
+	if (ipd->irqs[irq].control & IPIPE_SYSTEM_MASK) {
+		spin_unlock_irqrestore(&__ipipe_pipelock, flags);
+		return -EPERM;
+	}
+
+	if (ipd->irqs[irq].handler == NULL)
+		setmask &= ~(IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK);
+
+	if ((setmask & IPIPE_STICKY_MASK) != 0)
+		setmask |= IPIPE_HANDLE_MASK;
+
+	if ((clrmask & (IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK)) != 0)	/* If one goes, both go. */
+		clrmask |= (IPIPE_HANDLE_MASK | IPIPE_STICKY_MASK);
+
+	ipd->irqs[irq].control &= ~clrmask;
+	ipd->irqs[irq].control |= setmask;
+
+	if ((setmask & IPIPE_ENABLE_MASK) != 0)
+		__ipipe_enable_irq(irq);
+	else if ((clrmask & IPIPE_ENABLE_MASK) != 0)
+		__ipipe_disable_irq(irq);
+
+	spin_unlock_irqrestore(&__ipipe_pipelock, flags);
+
+	return 0;
+}
+
+/* __ipipe_dispatch_event() -- Low-level event dispatcher. */
+
+int __ipipe_dispatch_event (unsigned event, void *data)
+{
+	struct ipipe_domain *start_domain, *this_domain, *next_domain;
+	ipipe_event_handler_t evhand;
+	struct list_head *pos, *npos;
+	unsigned long flags;
+	int propagate = 1;
+
+	local_irq_save_hw(flags);
+
+	start_domain = this_domain = __ipipe_current_domain;
+
+	list_for_each_safe(pos, npos, &__ipipe_pipeline) {
+		/*
+		 * Note: Domain migration may occur while running
+		 * event or interrupt handlers, in which case the
+		 * current register set is going to be recycled for a
+		 * different domain than the initiating one. We do
+		 * care for that, always tracking the current domain
+		 * descriptor upon return from those handlers.
+		 */
+		next_domain = list_entry(pos, struct ipipe_domain, p_link);
+
+		/*
+		 * Keep a cached copy of the handler's address since
+		 * ipipe_catch_event() may clear it under our feet.
+		 */
+		evhand = next_domain->evhand[event];
+
+		if (evhand != NULL) {
+			__ipipe_current_domain = next_domain;
+			ipipe_cpudom_var(next_domain, evsync) |= (1LL << event);
+			local_irq_restore_hw(flags);
+			propagate = !evhand(event, start_domain, data);
+			local_irq_save_hw(flags);
+			/*
+			 * We may have a migration issue here, if the
+			 * current task is migrated to another CPU on
+			 * behalf of the invoked handler, usually when
+			 * a syscall event is processed. However,
+			 * ipipe_catch_event() will make sure that a
+			 * CPU that clears a handler for any given
+			 * event will not attempt to wait for itself
+			 * to clear the evsync bit for that event,
+			 * which practically plugs the hole, without
+			 * resorting to a much more complex strategy.
+			 */
+			ipipe_cpudom_var(next_domain, evsync) &= ~(1LL << event);
+			if (__ipipe_current_domain != next_domain)
+				this_domain = __ipipe_current_domain;
+		}
+
+		if (next_domain != ipipe_root_domain &&	/* NEVER sync the root stage here. */
+		    ipipe_cpudom_var(next_domain, irqpend_himask) != 0 &&
+		    !test_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(next_domain, status))) {
+			__ipipe_current_domain = next_domain;
+			__ipipe_sync_pipeline(IPIPE_IRQMASK_ANY);
+			if (__ipipe_current_domain != next_domain)
+				this_domain = __ipipe_current_domain;
+		}
+
+		__ipipe_current_domain = this_domain;
+
+		if (next_domain == this_domain || !propagate)
+			break;
+	}
+
+	local_irq_restore_hw(flags);
+
+	return !propagate;
+}
+
+/*
+ * __ipipe_dispatch_wired -- Wired interrupt dispatcher. Wired
+ * interrupts are immediately and unconditionally delivered to the
+ * domain heading the pipeline upon receipt, and such domain must have
+ * been registered as an invariant head for the system (priority ==
+ * IPIPE_HEAD_PRIORITY). The motivation for using wired interrupts is
+ * to get an extra-fast dispatching path for those IRQs, by relying on
+ * a straightforward logic based on assumptions that must always be
+ * true for invariant head domains.  The following assumptions are
+ * made when dealing with such interrupts:
+ *
+ * 1- Wired interrupts are purely dynamic, i.e. the decision to
+ * propagate them down the pipeline must be done from the head domain
+ * ISR.
+ * 2- Wired interrupts cannot be shared or sticky.
+ * 3- The root domain cannot be an invariant pipeline head, in
+ * consequence of what the root domain cannot handle wired
+ * interrupts.
+ * 4- Wired interrupts must have a valid acknowledge handler for the
+ * head domain (if needed, see __ipipe_handle_irq).
+ *
+ * Called with hw interrupts off.
+ */
+
+void __ipipe_dispatch_wired(struct ipipe_domain *head, unsigned irq)
+{
+	struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(head);
+
+	prefetchw(p);
+
+	if (unlikely(test_bit(IPIPE_LOCK_FLAG, &head->irqs[irq].control))) {
+		/*
+		 * If we can't process this IRQ right now, we must
+		 * mark it as held, so that it will get played during
+		 * normal log sync when the corresponding interrupt
+		 * source is eventually unlocked.
+		 */
+		p->irqall[irq]++;
+		__set_bit(irq & IPIPE_IRQ_IMASK, &p->irqheld_mask[irq >> IPIPE_IRQ_ISHIFT]);
+		return;
+	}
+
+	if (test_bit(IPIPE_STALL_FLAG, &p->status)) {
+		__ipipe_set_irq_pending(head, irq);
+		return;
+	}
+
+	__ipipe_dispatch_wired_nocheck(head, irq);
+}
+
+void __ipipe_dispatch_wired_nocheck(struct ipipe_domain *head, unsigned irq) /* hw interrupts off */
+{
+	struct ipipe_percpu_domain_data *p = ipipe_cpudom_ptr(head);
+	struct ipipe_domain *old;
+
+	prefetchw(p);
+
+	old = __ipipe_current_domain;
+	__ipipe_current_domain = head; /* Switch to the head domain. */
+
+	p->irqall[irq]++;
+	__set_bit(IPIPE_STALL_FLAG, &p->status);
+	head->irqs[irq].handler(irq, head->irqs[irq].cookie); /* Call the ISR. */
+	__ipipe_run_irqtail();
+	__clear_bit(IPIPE_STALL_FLAG, &p->status);
+
+	if (__ipipe_current_domain == head) {
+		__ipipe_current_domain = old;
+		if (old == head) {
+			if (p->irqpend_himask)
+				__ipipe_sync_pipeline(IPIPE_IRQMASK_ANY);
+			return;
+		}
+	}
+
+	__ipipe_walk_pipeline(&head->p_link);
+}
+
+/*
+ * __ipipe_sync_stage() -- Flush the pending IRQs for the current
+ * domain (and processor). This routine flushes the interrupt log
+ * (see "Optimistic interrupt protection" from D. Stodolsky et al. for
+ * more on the deferred interrupt scheme). Every interrupt that
+ * occurred while the pipeline was stalled gets played. WARNING:
+ * callers on SMP boxen should always check for CPU migration on
+ * return of this routine. One can control the kind of interrupts
+ * which are going to be sync'ed using the syncmask
+ * parameter. IPIPE_IRQMASK_ANY plays them all, IPIPE_IRQMASK_VIRT
+ * plays virtual interrupts only.
+ *
+ * This routine must be called with hw interrupts off.
+ */
+void __ipipe_sync_stage(unsigned long syncmask)
+{
+	struct ipipe_percpu_domain_data *p;
+	unsigned long mask, submask;
+	struct ipipe_domain *ipd;
+	int level, rank, cpu;
+	unsigned irq;
+
+	ipd = __ipipe_current_domain;
+	p = ipipe_cpudom_ptr(ipd);
+
+	if (__test_and_set_bit(IPIPE_SYNC_FLAG, &p->status))
+		return;
+
+	cpu = ipipe_processor_id();
+
+	/*
+	 * The policy here is to keep the dispatching code interrupt-free
+	 * by stalling the current stage. If the upper domain handler
+	 * (which we call) wants to re-enable interrupts while in a safe
+	 * portion of the code (e.g. SA_INTERRUPT flag unset for Linux's
+	 * sigaction()), it will have to unstall (then stall again before
+	 * returning to us!) the stage when it sees fit.
+	 */
+	while ((mask = (p->irqpend_himask & syncmask)) != 0) {
+		level = __ipipe_ffnz(mask);
+
+		while ((submask = p->irqpend_lomask[level]) != 0) {
+			rank = __ipipe_ffnz(submask);
+			irq = (level << IPIPE_IRQ_ISHIFT) + rank;
+
+			__clear_bit(rank, &p->irqpend_lomask[level]);
+
+			if (p->irqpend_lomask[level] == 0)
+				__clear_bit(level, &p->irqpend_himask);
+			/*
+			 * Make sure the compiler will not postpone
+			 * the pending bitmask updates before calling
+			 * the interrupt handling routine. Otherwise,
+			 * those late updates could overwrite any
+			 * change to irqpend_hi/lomask due to a nested
+			 * interrupt, leaving the latter unprocessed
+			 * (seen on mpc836x).
+			 */
+			barrier();
+
+			if (test_bit(IPIPE_LOCK_FLAG, &ipd->irqs[irq].control))
+				continue;
+
+			__set_bit(IPIPE_STALL_FLAG, &p->status);
+			barrier();
+
+			if (ipd == ipipe_root_domain)
+				trace_hardirqs_off();
+
+			__ipipe_run_isr(ipd, irq);
+			barrier();
+			p = ipipe_cpudom_ptr(__ipipe_current_domain);
+#ifdef CONFIG_SMP
+			{
+				int newcpu = ipipe_processor_id();
+
+				if (newcpu != cpu) {	/* Handle CPU migration. */
+					/*
+					 * We expect any domain to clear the SYNC bit each
+					 * time it switches in a new task, so that preemptions
+					 * and/or CPU migrations (in the SMP case) over the
+					 * ISR do not lock out the log syncer for some
+					 * indefinite amount of time. In the Linux case,
+					 * schedule() handles this (see kernel/sched.c). For
+					 * this reason, we don't bother clearing it here for
+					 * the source CPU in the migration handling case,
+					 * since it must have scheduled another task in by
+					 * now.
+					 */
+					__set_bit(IPIPE_SYNC_FLAG, &p->status);
+					cpu = newcpu;
+				}
+			}
+#endif	/* CONFIG_SMP */
+#ifdef CONFIG_TRACE_IRQFLAGS
+			if (__ipipe_root_domain_p &&
+			    test_bit(IPIPE_STALL_FLAG, &p->status))
+				trace_hardirqs_on();
+#endif
+			__clear_bit(IPIPE_STALL_FLAG, &p->status);
+		}
+	}
+
+	__clear_bit(IPIPE_SYNC_FLAG, &p->status);
+}
+
+/* ipipe_register_domain() -- Link a new domain to the pipeline. */
+
+int ipipe_register_domain(struct ipipe_domain *ipd,
+			  struct ipipe_domain_attr *attr)
+{
+	struct ipipe_domain *_ipd;
+	struct list_head *pos = NULL;
+	unsigned long flags;
+
+	if (!ipipe_root_domain_p) {
+		printk(KERN_WARNING
+		       "I-pipe: Only the root domain may register a new domain.\n");
+		return -EPERM;
+	}
+
+	flags = ipipe_critical_enter(NULL);
+
+	if (attr->priority == IPIPE_HEAD_PRIORITY) {
+		if (test_bit(IPIPE_HEAD_SLOT, &__ipipe_domain_slot_map)) {
+			ipipe_critical_exit(flags);
+			return -EAGAIN;	/* Cannot override current head. */
+		}
+		ipd->slot = IPIPE_HEAD_SLOT;
+	} else
+		ipd->slot = ffz(__ipipe_domain_slot_map);
+
+	if (ipd->slot < CONFIG_IPIPE_DOMAINS) {
+		set_bit(ipd->slot, &__ipipe_domain_slot_map);
+		list_for_each(pos, &__ipipe_pipeline) {
+			_ipd = list_entry(pos, struct ipipe_domain, p_link);
+			if (_ipd->domid == attr->domid)
+				break;
+		}
+	}
+
+	ipipe_critical_exit(flags);
+
+	if (pos != &__ipipe_pipeline) {
+		if (ipd->slot < CONFIG_IPIPE_DOMAINS)
+			clear_bit(ipd->slot, &__ipipe_domain_slot_map);
+		return -EBUSY;
+	}
+
+#ifndef CONFIG_SMP
+	/*
+	 * Set up the perdomain pointers for direct access to the
+	 * percpu domain data. This saves a costly multiply each time
+	 * we need to refer to the contents of the percpu domain data
+	 * array.
+	 */
+	__raw_get_cpu_var(ipipe_percpu_daddr)[ipd->slot] = &__raw_get_cpu_var(ipipe_percpu_darray)[ipd->slot];
+#endif
+
+	ipd->name = attr->name;
+	ipd->domid = attr->domid;
+	ipd->pdd = attr->pdd;
+	ipd->flags = 0;
+
+	if (attr->priority == IPIPE_HEAD_PRIORITY) {
+		ipd->priority = INT_MAX;
+		__set_bit(IPIPE_AHEAD_FLAG,&ipd->flags);
+	}
+	else
+		ipd->priority = attr->priority;
+
+	__ipipe_init_stage(ipd);
+
+	INIT_LIST_HEAD(&ipd->p_link);
+
+#ifdef CONFIG_PROC_FS
+	__ipipe_add_domain_proc(ipd);
+#endif /* CONFIG_PROC_FS */
+
+	flags = ipipe_critical_enter(NULL);
+
+	list_for_each(pos, &__ipipe_pipeline) {
+		_ipd = list_entry(pos, struct ipipe_domain, p_link);
+		if (ipd->priority > _ipd->priority)
+			break;
+	}
+
+	list_add_tail(&ipd->p_link, pos);
+
+	ipipe_critical_exit(flags);
+
+	printk(KERN_INFO "I-pipe: Domain %s registered.\n", ipd->name);
+
+	/*
+	 * Finally, allow the new domain to perform its initialization
+	 * chores.
+	 */
+
+	if (attr->entry != NULL) {
+		local_irq_save_hw_smp(flags);
+		__ipipe_current_domain = ipd;
+		local_irq_restore_hw_smp(flags);
+		attr->entry();
+		local_irq_save_hw(flags);
+		__ipipe_current_domain = ipipe_root_domain;
+
+		if (ipipe_root_cpudom_var(irqpend_himask) != 0 &&
+		    !test_bit(IPIPE_STALL_FLAG, &ipipe_root_cpudom_var(status)))
+			__ipipe_sync_pipeline(IPIPE_IRQMASK_ANY);
+
+		local_irq_restore_hw(flags);
+	}
+
+	return 0;
+}
+
+/* ipipe_unregister_domain() -- Remove a domain from the pipeline. */
+
+int ipipe_unregister_domain(struct ipipe_domain *ipd)
+{
+	unsigned long flags;
+
+	if (!ipipe_root_domain_p) {
+		printk(KERN_WARNING
+		       "I-pipe: Only the root domain may unregister a domain.\n");
+		return -EPERM;
+	}
+
+	if (ipd == ipipe_root_domain) {
+		printk(KERN_WARNING
+		       "I-pipe: Cannot unregister the root domain.\n");
+		return -EPERM;
+	}
+#ifdef CONFIG_SMP
+	{
+		unsigned irq;
+		int cpu;
+
+		/*
+		 * In the SMP case, wait for the logged events to drain on
+		 * other processors before eventually removing the domain
+		 * from the pipeline.
+		 */
+
+		ipipe_unstall_pipeline_from(ipd);
+
+		flags = ipipe_critical_enter(NULL);
+
+		for (irq = 0; irq < IPIPE_NR_IRQS; irq++) {
+			clear_bit(IPIPE_HANDLE_FLAG, &ipd->irqs[irq].control);
+			clear_bit(IPIPE_STICKY_FLAG, &ipd->irqs[irq].control);
+			set_bit(IPIPE_PASS_FLAG, &ipd->irqs[irq].control);
+		}
+
+		ipipe_critical_exit(flags);
+
+		for_each_online_cpu(cpu) {
+			while (ipipe_percpudom(ipd, irqpend_himask, cpu) > 0)
+				cpu_relax();
+		}
+	}
+#endif	/* CONFIG_SMP */
+
+	mutex_lock(&ipd->mutex);
+
+#ifdef CONFIG_PROC_FS
+	__ipipe_remove_domain_proc(ipd);
+#endif /* CONFIG_PROC_FS */
+
+	/*
+	 * Simply remove the domain from the pipeline and we are almost done.
+	 */
+
+	flags = ipipe_critical_enter(NULL);
+	list_del_init(&ipd->p_link);
+	ipipe_critical_exit(flags);
+
+	__ipipe_cleanup_domain(ipd);
+
+	mutex_unlock(&ipd->mutex);
+
+	printk(KERN_INFO "I-pipe: Domain %s unregistered.\n", ipd->name);
+
+	return 0;
+}
+
+/*
+ * ipipe_propagate_irq() -- Force a given IRQ propagation on behalf of
+ * a running interrupt handler to the next domain down the pipeline.
+ * ipipe_schedule_irq() -- Does almost the same as above, but attempts
+ * to pend the interrupt for the current domain first.
+ * Must be called hw IRQs off.
+ */
+void __ipipe_pend_irq(unsigned irq, struct list_head *head)
+{
+	struct ipipe_domain *ipd;
+	struct list_head *ln;
+
+#ifdef CONFIG_IPIPE_DEBUG
+	BUG_ON(irq >= IPIPE_NR_IRQS ||
+	       (ipipe_virtual_irq_p(irq)
+		&& !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map)));
+#endif
+	for (ln = head; ln != &__ipipe_pipeline; ln = ipd->p_link.next) {
+		ipd = list_entry(ln, struct ipipe_domain, p_link);
+		if (test_bit(IPIPE_HANDLE_FLAG, &ipd->irqs[irq].control)) {
+			__ipipe_set_irq_pending(ipd, irq);
+			return;
+		}
+	}
+}
+
+/* ipipe_free_virq() -- Release a virtual/soft interrupt. */
+
+int ipipe_free_virq(unsigned virq)
+{
+	if (!ipipe_virtual_irq_p(virq))
+		return -EINVAL;
+
+	clear_bit(virq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map);
+
+	return 0;
+}
+
+void ipipe_init_attr(struct ipipe_domain_attr *attr)
+{
+	attr->name = "anon";
+	attr->domid = 1;
+	attr->entry = NULL;
+	attr->priority = IPIPE_ROOT_PRIO;
+	attr->pdd = NULL;
+}
+
+/*
+ * ipipe_catch_event() -- Interpose or remove an event handler for a
+ * given domain.
+ */
+ipipe_event_handler_t ipipe_catch_event(struct ipipe_domain *ipd,
+					unsigned event,
+					ipipe_event_handler_t handler)
+{
+	ipipe_event_handler_t old_handler;
+	unsigned long flags;
+	int self = 0, cpu;
+
+	if (event & IPIPE_EVENT_SELF) {
+		event &= ~IPIPE_EVENT_SELF;
+		self = 1;
+	}
+
+	if (event >= IPIPE_NR_EVENTS)
+		return NULL;
+
+	flags = ipipe_critical_enter(NULL);
+
+	if (!(old_handler = xchg(&ipd->evhand[event],handler)))	{
+		if (handler) {
+			if (self)
+				ipd->evself |= (1LL << event);
+			else
+				__ipipe_event_monitors[event]++;
+		}
+	}
+	else if (!handler) {
+		if (ipd->evself & (1LL << event))
+			ipd->evself &= ~(1LL << event);
+		else
+			__ipipe_event_monitors[event]--;
+	} else if ((ipd->evself & (1LL << event)) && !self) {
+			__ipipe_event_monitors[event]++;
+			ipd->evself &= ~(1LL << event);
+	} else if (!(ipd->evself & (1LL << event)) && self) {
+			__ipipe_event_monitors[event]--;
+			ipd->evself |= (1LL << event);
+	}
+
+	ipipe_critical_exit(flags);
+
+	if (!handler && ipipe_root_domain_p) {
+		/*
+		 * If we cleared a handler on behalf of the root
+		 * domain, we have to wait for any current invocation
+		 * to drain, since our caller might subsequently unmap
+		 * the target domain. To this aim, this code
+		 * synchronizes with __ipipe_dispatch_event(),
+		 * guaranteeing that either the dispatcher sees a null
+		 * handler in which case it discards the invocation
+		 * (which also prevents from entering a livelock), or
+		 * finds a valid handler and calls it. Symmetrically,
+		 * ipipe_catch_event() ensures that the called code
+		 * won't be unmapped under our feet until the event
+		 * synchronization flag is cleared for the given event
+		 * on all CPUs.
+		 */
+		preempt_disable();
+		cpu = smp_processor_id();
+		/*
+		 * Hack: this solves the potential migration issue
+		 * raised in __ipipe_dispatch_event(). This is a
+		 * work-around which makes the assumption that other
+		 * CPUs will subsequently, either process at least one
+		 * interrupt for the target domain, or call
+		 * __ipipe_dispatch_event() without going through a
+		 * migration while running the handler at least once;
+		 * practically, this is safe on any normally running
+		 * system.
+		 */
+		ipipe_percpudom(ipd, evsync, cpu) &= ~(1LL << event);
+		preempt_enable();
+
+		for_each_online_cpu(cpu) {
+			while (ipipe_percpudom(ipd, evsync, cpu) & (1LL << event))
+				schedule_timeout_interruptible(HZ / 50);
+		}
+	}
+
+	return old_handler;
+}
+
+cpumask_t ipipe_set_irq_affinity (unsigned irq, cpumask_t cpumask)
+{
+#ifdef CONFIG_SMP
+	if (irq >= IPIPE_NR_XIRQS)
+		/* Allow changing affinity of external IRQs only. */
+		return CPU_MASK_NONE;
+
+	if (num_online_cpus() > 1)
+		return __ipipe_set_irq_affinity(irq,cpumask);
+#endif /* CONFIG_SMP */
+
+	return CPU_MASK_NONE;
+}
+
+int ipipe_send_ipi (unsigned ipi, cpumask_t cpumask)
+
+{
+#ifdef CONFIG_SMP
+	return __ipipe_send_ipi(ipi,cpumask);
+#else /* !CONFIG_SMP */
+	return -EINVAL;
+#endif /* CONFIG_SMP */
+}
+
+int ipipe_alloc_ptdkey (void)
+{
+	unsigned long flags;
+	int key = -1;
+
+	spin_lock_irqsave(&__ipipe_pipelock,flags);
+
+	if (__ipipe_ptd_key_count < IPIPE_ROOT_NPTDKEYS) {
+		key = ffz(__ipipe_ptd_key_map);
+		set_bit(key,&__ipipe_ptd_key_map);
+		__ipipe_ptd_key_count++;
+	}
+
+	spin_unlock_irqrestore(&__ipipe_pipelock,flags);
+
+	return key;
+}
+
+int ipipe_free_ptdkey (int key)
+{
+	unsigned long flags;
+
+	if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS)
+		return -EINVAL;
+
+	spin_lock_irqsave(&__ipipe_pipelock,flags);
+
+	if (test_and_clear_bit(key,&__ipipe_ptd_key_map))
+		__ipipe_ptd_key_count--;
+
+	spin_unlock_irqrestore(&__ipipe_pipelock,flags);
+
+	return 0;
+}
+
+int ipipe_set_ptd (int key, void *value)
+
+{
+	if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS)
+		return -EINVAL;
+
+	current->ptd[key] = value;
+
+	return 0;
+}
+
+void *ipipe_get_ptd (int key)
+
+{
+	if (key < 0 || key >= IPIPE_ROOT_NPTDKEYS)
+		return NULL;
+
+	return current->ptd[key];
+}
+
+#ifdef CONFIG_PROC_FS
+
+struct proc_dir_entry *ipipe_proc_root;
+
+static int __ipipe_version_info_proc(char *page,
+				     char **start,
+				     off_t off, int count, int *eof, void *data)
+{
+	int len = sprintf(page, "%s\n", IPIPE_VERSION_STRING);
+
+	len -= off;
+
+	if (len <= off + count)
+		*eof = 1;
+
+	*start = page + off;
+
+	if(len > count)
+		len = count;
+
+	if(len < 0)
+		len = 0;
+
+	return len;
+}
+
+static int __ipipe_common_info_show(struct seq_file *p, void *data)
+{
+	struct ipipe_domain *ipd = (struct ipipe_domain *)p->private;
+	char handling, stickiness, lockbit, exclusive, virtuality;
+
+	unsigned long ctlbits;
+	unsigned irq;
+
+	seq_printf(p, "       +----- Handling ([A]ccepted, [G]rabbed, [W]ired, [D]iscarded)\n");
+	seq_printf(p, "       |+---- Sticky\n");
+	seq_printf(p, "       ||+--- Locked\n");
+	seq_printf(p, "       |||+-- Exclusive\n");
+	seq_printf(p, "       ||||+- Virtual\n");
+	seq_printf(p, "[IRQ]  |||||\n");
+
+	mutex_lock(&ipd->mutex);
+
+	for (irq = 0; irq < IPIPE_NR_IRQS; irq++) {
+		/* Remember to protect against
+		 * ipipe_virtual_irq/ipipe_control_irq if more fields
+		 * get involved. */
+		ctlbits = ipd->irqs[irq].control;
+
+		if (irq >= IPIPE_NR_XIRQS && !ipipe_virtual_irq_p(irq))
+			/*
+			 * There might be a hole between the last external
+			 * IRQ and the first virtual one; skip it.
+			 */
+			continue;
+
+		if (ipipe_virtual_irq_p(irq)
+		    && !test_bit(irq - IPIPE_VIRQ_BASE, &__ipipe_virtual_irq_map))
+			/* Non-allocated virtual IRQ; skip it. */
+			continue;
+
+		/*
+		 * Statuses are as follows:
+		 * o "accepted" means handled _and_ passed down the pipeline.
+		 * o "grabbed" means handled, but the interrupt might be
+		 * terminated _or_ passed down the pipeline depending on
+		 * what the domain handler asks for to the I-pipe.
+		 * o "wired" is basically the same as "grabbed", except that
+		 * the interrupt is unconditionally delivered to an invariant
+		 * pipeline head domain.
+		 * o "passed" means unhandled by the domain but passed
+		 * down the pipeline.
+		 * o "discarded" means unhandled and _not_ passed down the
+		 * pipeline. The interrupt merely disappears from the
+		 * current domain down to the end of the pipeline.
+		 */
+		if (ctlbits & IPIPE_HANDLE_MASK) {
+			if (ctlbits & IPIPE_PASS_MASK)
+				handling = 'A';
+			else if (ctlbits & IPIPE_WIRED_MASK)
+				handling = 'W';
+			else
+				handling = 'G';
+		} else if (ctlbits & IPIPE_PASS_MASK)
+			/* Do not output if no major action is taken. */
+			continue;
+		else
+			handling = 'D';
+
+		if (ctlbits & IPIPE_STICKY_MASK)
+			stickiness = 'S';
+		else
+			stickiness = '.';
+
+		if (ctlbits & IPIPE_LOCK_MASK)
+			lockbit = 'L';
+		else
+			lockbit = '.';
+
+		if (ctlbits & IPIPE_EXCLUSIVE_MASK)
+			exclusive = 'X';
+		else
+			exclusive = '.';
+
+		if (ipipe_virtual_irq_p(irq))
+			virtuality = 'V';
+		else
+			virtuality = '.';
+
+		seq_printf(p, " %3u:  %c%c%c%c%c\n",
+			     irq, handling, stickiness, lockbit, exclusive, virtuality);
+	}
+
+	seq_printf(p, "[Domain info]\n");
+
+	seq_printf(p, "id=0x%.8x\n", ipd->domid);
+
+	if (test_bit(IPIPE_AHEAD_FLAG,&ipd->flags))
+		seq_printf(p, "priority=topmost\n");
+	else
+		seq_printf(p, "priority=%d\n", ipd->priority);
+
+	mutex_unlock(&ipd->mutex);
+
+	return 0;
+}
+
+static int __ipipe_common_info_open(struct inode *inode, struct file *file)
+{
+	return single_open(file, __ipipe_common_info_show, PROC_I(inode)->pde->data);
+}
+
+static struct file_operations __ipipe_info_proc_ops = {
+	.owner		= THIS_MODULE,
+	.open		= __ipipe_common_info_open,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= single_release,
+};
+
+void __ipipe_add_domain_proc(struct ipipe_domain *ipd)
+{
+	struct proc_dir_entry *e = create_proc_entry(ipd->name, 0444, ipipe_proc_root);
+	if (e) {
+		e->proc_fops = &__ipipe_info_proc_ops;
+		e->data = (void*) ipd;
+	}
+}
+
+void __ipipe_remove_domain_proc(struct ipipe_domain *ipd)
+{
+	remove_proc_entry(ipd->name,ipipe_proc_root);
+}
+
+void __init ipipe_init_proc(void)
+{
+	ipipe_proc_root = create_proc_entry("ipipe",S_IFDIR, 0);
+	create_proc_read_entry("version",0444,ipipe_proc_root,&__ipipe_version_info_proc,NULL);
+	__ipipe_add_domain_proc(ipipe_root_domain);
+
+	__ipipe_init_tracer();
+}
+
+#endif	/* CONFIG_PROC_FS */
+
+#ifdef CONFIG_IPIPE_DEBUG_CONTEXT
+
+DEFINE_PER_CPU(int, ipipe_percpu_context_check) = { 1 };
+DEFINE_PER_CPU(int, ipipe_saved_context_check_state);
+
+void ipipe_check_context(struct ipipe_domain *border_domain)
+{
+        struct ipipe_percpu_domain_data *p; 
+        struct ipipe_domain *this_domain; 
+        unsigned long flags;
+	int cpu;
+ 
+        local_irq_save_hw_smp(flags); 
+
+        this_domain = __ipipe_current_domain; 
+        p = ipipe_head_cpudom_ptr(); 
+        if (likely(this_domain->priority <= border_domain->priority && 
+		   !test_bit(IPIPE_STALL_FLAG, &p->status))) { 
+                local_irq_restore_hw_smp(flags); 
+                return; 
+        } 
+ 
+	cpu = ipipe_processor_id();
+        if (!per_cpu(ipipe_percpu_context_check, cpu)) { 
+                local_irq_restore_hw_smp(flags); 
+                return; 
+        } 
+ 
+        local_irq_restore_hw_smp(flags); 
+
+	ipipe_context_check_off();
+	ipipe_trace_panic_freeze();
+	ipipe_set_printk_sync(__ipipe_current_domain);
+
+	if (this_domain->priority > border_domain->priority)
+		printk(KERN_ERR "I-pipe: Detected illicit call from domain "
+				"'%s'\n"
+		       KERN_ERR "        into a service reserved for domain "
+				"'%s' and below.\n",
+		       this_domain->name, border_domain->name);
+	else
+		printk(KERN_ERR "I-pipe: Detected stalled topmost domain, "
+				"probably caused by a bug.\n"
+				"        A critical section may have been "
+				"left unterminated.\n");
+	dump_stack();
+	ipipe_trace_panic_dump();
+}
+
+EXPORT_SYMBOL(ipipe_check_context);
+
+#endif /* CONFIG_IPIPE_DEBUG_CONTEXT */
+
+#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP)
+
+int notrace __ipipe_check_percpu_access(void)
+{
+	struct ipipe_percpu_domain_data *p;
+	struct ipipe_domain *this_domain;
+	unsigned long flags;
+	int ret = 0;
+
+	local_irq_save_hw_notrace(flags);
+
+	this_domain = __raw_get_cpu_var(ipipe_percpu_domain);
+
+	/*
+	 * Only the root domain may implement preemptive CPU migration
+	 * of tasks, so anything above in the pipeline should be fine.
+	 */
+	if (this_domain->priority > IPIPE_ROOT_PRIO)
+		goto out;
+
+	if (raw_irqs_disabled_flags(flags))
+		goto out;
+
+	/*
+	 * Last chance: hw interrupts were enabled on entry while
+	 * running over the root domain, but the root stage might be
+	 * currently stalled, in which case preemption would be
+	 * disabled, and no migration could occur.
+	 */
+	if (this_domain == ipipe_root_domain) {
+		p = ipipe_root_cpudom_ptr(); 
+		if (test_bit(IPIPE_STALL_FLAG, &p->status))
+			goto out;
+	}
+	/*
+	 * Our caller may end up accessing the wrong per-cpu variable
+	 * instance due to CPU migration; tell it to complain about
+	 * this.
+	 */
+	ret = 1;
+out:
+	local_irq_restore_hw_notrace(flags);
+
+	return ret;
+}
+
+#endif /* CONFIG_IPIPE_DEBUG_INTERNAL && CONFIG_SMP */
+
+EXPORT_SYMBOL(ipipe_virtualize_irq);
+EXPORT_SYMBOL(ipipe_control_irq);
+EXPORT_SYMBOL(ipipe_suspend_domain);
+EXPORT_SYMBOL(ipipe_alloc_virq);
+EXPORT_PER_CPU_SYMBOL(ipipe_percpu_domain);
+EXPORT_PER_CPU_SYMBOL(ipipe_percpu_darray);
+EXPORT_SYMBOL(ipipe_root);
+EXPORT_SYMBOL(ipipe_stall_pipeline_from);
+EXPORT_SYMBOL(ipipe_test_and_stall_pipeline_from);
+EXPORT_SYMBOL(ipipe_test_and_unstall_pipeline_from);
+EXPORT_SYMBOL(ipipe_restore_pipeline_from);
+EXPORT_SYMBOL(ipipe_unstall_pipeline_head);
+EXPORT_SYMBOL(__ipipe_restore_pipeline_head);
+EXPORT_SYMBOL(__ipipe_unstall_root);
+EXPORT_SYMBOL(__ipipe_restore_root);
+EXPORT_SYMBOL(__ipipe_spin_lock_irq);
+EXPORT_SYMBOL(__ipipe_spin_unlock_irq);
+EXPORT_SYMBOL(__ipipe_spin_lock_irqsave);
+EXPORT_SYMBOL(__ipipe_spin_unlock_irqrestore);
+EXPORT_SYMBOL(__ipipe_pipeline);
+EXPORT_SYMBOL(__ipipe_lock_irq);
+EXPORT_SYMBOL(__ipipe_unlock_irq);
+EXPORT_SYMBOL(ipipe_register_domain);
+EXPORT_SYMBOL(ipipe_unregister_domain);
+EXPORT_SYMBOL(ipipe_free_virq);
+EXPORT_SYMBOL(ipipe_init_attr);
+EXPORT_SYMBOL(ipipe_catch_event);
+EXPORT_SYMBOL(ipipe_alloc_ptdkey);
+EXPORT_SYMBOL(ipipe_free_ptdkey);
+EXPORT_SYMBOL(ipipe_set_ptd);
+EXPORT_SYMBOL(ipipe_get_ptd);
+EXPORT_SYMBOL(ipipe_set_irq_affinity);
+EXPORT_SYMBOL(ipipe_send_ipi);
+EXPORT_SYMBOL(__ipipe_pend_irq);
+EXPORT_SYMBOL(__ipipe_set_irq_pending);
+#if defined(CONFIG_IPIPE_DEBUG_INTERNAL) && defined(CONFIG_SMP)
+EXPORT_SYMBOL(__ipipe_check_percpu_access);
+#endif
+#ifdef CONFIG_GENERIC_CLOCKEVENTS
+EXPORT_SYMBOL(ipipe_request_tickdev);
+EXPORT_SYMBOL(ipipe_release_tickdev);
+#endif
+
+EXPORT_SYMBOL(ipipe_critical_enter);
+EXPORT_SYMBOL(ipipe_critical_exit);
+EXPORT_SYMBOL(ipipe_trigger_irq);
+EXPORT_SYMBOL(ipipe_get_sysinfo);
diff -urN source_powerpc_none/kernel/ipipe/tracer.c source_powerpc_none.ipipe/kernel/ipipe/tracer.c
--- source_powerpc_none/kernel/ipipe/tracer.c	1969-12-31 19:00:00.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/ipipe/tracer.c	2009-12-22 12:44:08.000000000 -0500
@@ -0,0 +1,1441 @@
+/* -*- linux-c -*-
+ * kernel/ipipe/tracer.c
+ *
+ * Copyright (C) 2005 Luotao Fu.
+ *               2005-2008 Jan Kiszka.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation, Inc., 675 Mass Ave, Cambridge MA 02139,
+ * USA; either version 2 of the License, or (at your option) any later
+ * version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/version.h>
+#include <linux/kallsyms.h>
+#include <linux/seq_file.h>
+#include <linux/proc_fs.h>
+#include <linux/ctype.h>
+#include <linux/vmalloc.h>
+#include <linux/pid.h>
+#include <linux/utsrelease.h>
+#include <linux/sched.h>
+#include <linux/ipipe.h>
+#include <linux/ftrace.h>
+#include <asm/uaccess.h>
+
+#define IPIPE_TRACE_PATHS           4 /* <!> Do not lower below 3 */
+#define IPIPE_DEFAULT_ACTIVE        0
+#define IPIPE_DEFAULT_MAX           1
+#define IPIPE_DEFAULT_FROZEN        2
+
+#define IPIPE_TRACE_POINTS          (1 << CONFIG_IPIPE_TRACE_SHIFT)
+#define WRAP_POINT_NO(point)        ((point) & (IPIPE_TRACE_POINTS-1))
+
+#define IPIPE_DEFAULT_PRE_TRACE     10
+#define IPIPE_DEFAULT_POST_TRACE    10
+#define IPIPE_DEFAULT_BACK_TRACE    100
+
+#define IPIPE_DELAY_NOTE            1000  /* in nanoseconds */
+#define IPIPE_DELAY_WARN            10000 /* in nanoseconds */
+
+#define IPIPE_TFLG_NMI_LOCK         0x0001
+#define IPIPE_TFLG_NMI_HIT          0x0002
+#define IPIPE_TFLG_NMI_FREEZE_REQ   0x0004
+
+#define IPIPE_TFLG_HWIRQ_OFF        0x0100
+#define IPIPE_TFLG_FREEZING         0x0200
+#define IPIPE_TFLG_CURRDOM_SHIFT    10   /* bits 10..11: current domain */
+#define IPIPE_TFLG_CURRDOM_MASK     0x0C00
+#define IPIPE_TFLG_DOMSTATE_SHIFT   12   /* bits 12..15: domain stalled? */
+#define IPIPE_TFLG_DOMSTATE_BITS    3
+
+#define IPIPE_TFLG_DOMAIN_STALLED(point, n) \
+	(point->flags & (1 << (n + IPIPE_TFLG_DOMSTATE_SHIFT)))
+#define IPIPE_TFLG_CURRENT_DOMAIN(point) \
+	((point->flags & IPIPE_TFLG_CURRDOM_MASK) >> IPIPE_TFLG_CURRDOM_SHIFT)
+
+struct ipipe_trace_point {
+	short type;
+	short flags;
+	unsigned long eip;
+	unsigned long parent_eip;
+	unsigned long v;
+	unsigned long long timestamp;
+};
+
+struct ipipe_trace_path {
+	volatile int flags;
+	int dump_lock; /* separated from flags due to cross-cpu access */
+	int trace_pos; /* next point to fill */
+	int begin, end; /* finalised path begin and end */
+	int post_trace; /* non-zero when in post-trace phase */
+	unsigned long long length; /* max path length in cycles */
+	unsigned long nmi_saved_eip; /* for deferred requests from NMIs */
+	unsigned long nmi_saved_parent_eip;
+	unsigned long nmi_saved_v;
+	struct ipipe_trace_point point[IPIPE_TRACE_POINTS];
+} ____cacheline_aligned_in_smp;
+
+enum ipipe_trace_type
+{
+	IPIPE_TRACE_FUNC = 0,
+	IPIPE_TRACE_BEGIN,
+	IPIPE_TRACE_END,
+	IPIPE_TRACE_FREEZE,
+	IPIPE_TRACE_SPECIAL,
+	IPIPE_TRACE_PID,
+	IPIPE_TRACE_EVENT,
+};
+
+#define IPIPE_TYPE_MASK             0x0007
+#define IPIPE_TYPE_BITS             3
+
+#ifdef CONFIG_IPIPE_TRACE_VMALLOC
+static DEFINE_PER_CPU(struct ipipe_trace_path *, trace_path);
+#else /* !CONFIG_IPIPE_TRACE_VMALLOC */
+static DEFINE_PER_CPU(struct ipipe_trace_path, trace_path[IPIPE_TRACE_PATHS]) =
+	{ [0 ... IPIPE_TRACE_PATHS-1] = { .begin = -1, .end = -1 } };
+#endif /* CONFIG_IPIPE_TRACE_VMALLOC */
+
+int ipipe_trace_enable = 0;
+
+static DEFINE_PER_CPU(int, active_path) = { IPIPE_DEFAULT_ACTIVE };
+static DEFINE_PER_CPU(int, max_path) = { IPIPE_DEFAULT_MAX };
+static DEFINE_PER_CPU(int, frozen_path) = { IPIPE_DEFAULT_FROZEN };
+static IPIPE_DEFINE_SPINLOCK(global_path_lock);
+static int pre_trace = IPIPE_DEFAULT_PRE_TRACE;
+static int post_trace = IPIPE_DEFAULT_POST_TRACE;
+static int back_trace = IPIPE_DEFAULT_BACK_TRACE;
+static int verbose_trace = 1;
+static unsigned long trace_overhead;
+
+static unsigned long trigger_begin;
+static unsigned long trigger_end;
+
+static DEFINE_MUTEX(out_mutex);
+static struct ipipe_trace_path *print_path;
+#ifdef CONFIG_IPIPE_TRACE_PANIC
+static struct ipipe_trace_path *panic_path;
+#endif /* CONFIG_IPIPE_TRACE_PANIC */
+static int print_pre_trace;
+static int print_post_trace;
+
+
+static long __ipipe_signed_tsc2us(long long tsc);
+static void
+__ipipe_trace_point_type(char *buf, struct ipipe_trace_point *point);
+static void __ipipe_print_symname(struct seq_file *m, unsigned long eip);
+
+
+static notrace void
+__ipipe_store_domain_states(struct ipipe_trace_point *point)
+{
+	struct ipipe_domain *ipd;
+	struct list_head *pos;
+	int i = 0;
+
+	list_for_each_prev(pos, &__ipipe_pipeline) {
+		ipd = list_entry(pos, struct ipipe_domain, p_link);
+
+		if (test_bit(IPIPE_STALL_FLAG, &ipipe_cpudom_var(ipd, status)))
+			point->flags |= 1 << (i + IPIPE_TFLG_DOMSTATE_SHIFT);
+
+		if (ipd == __ipipe_current_domain)
+			point->flags |= i << IPIPE_TFLG_CURRDOM_SHIFT;
+
+		if (++i > IPIPE_TFLG_DOMSTATE_BITS)
+			break;
+	}
+}
+
+static notrace int __ipipe_get_free_trace_path(int old, int cpu)
+{
+	int new_active = old;
+	struct ipipe_trace_path *tp;
+
+	do {
+		if (++new_active == IPIPE_TRACE_PATHS)
+			new_active = 0;
+		tp = &per_cpu(trace_path, cpu)[new_active];
+	} while (new_active == per_cpu(max_path, cpu) ||
+	         new_active == per_cpu(frozen_path, cpu) ||
+	         tp->dump_lock);
+
+	return new_active;
+}
+
+static notrace void
+__ipipe_migrate_pre_trace(struct ipipe_trace_path *new_tp,
+                          struct ipipe_trace_path *old_tp, int old_pos)
+{
+	int i;
+
+	new_tp->trace_pos = pre_trace+1;
+
+	for (i = new_tp->trace_pos; i > 0; i--)
+		memcpy(&new_tp->point[WRAP_POINT_NO(new_tp->trace_pos-i)],
+		       &old_tp->point[WRAP_POINT_NO(old_pos-i)],
+		       sizeof(struct ipipe_trace_point));
+
+	/* mark the end (i.e. the point before point[0]) invalid */
+	new_tp->point[IPIPE_TRACE_POINTS-1].eip = 0;
+}
+
+static notrace struct ipipe_trace_path *
+__ipipe_trace_end(int cpu, struct ipipe_trace_path *tp, int pos)
+{
+	struct ipipe_trace_path *old_tp = tp;
+	long active = per_cpu(active_path, cpu);
+	unsigned long long length;
+
+	/* do we have a new worst case? */
+	length = tp->point[tp->end].timestamp -
+	         tp->point[tp->begin].timestamp;
+	if (length > per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)].length) {
+		/* we need protection here against other cpus trying
+		   to start a proc dump */
+		spin_lock(&global_path_lock);
+
+		/* active path holds new worst case */
+		tp->length = length;
+		per_cpu(max_path, cpu) = active;
+
+		/* find next unused trace path */
+		active = __ipipe_get_free_trace_path(active, cpu);
+
+		spin_unlock(&global_path_lock);
+
+		tp = &per_cpu(trace_path, cpu)[active];
+
+		/* migrate last entries for pre-tracing */
+		__ipipe_migrate_pre_trace(tp, old_tp, pos);
+	}
+
+	return tp;
+}
+
+static notrace struct ipipe_trace_path *
+__ipipe_trace_freeze(int cpu, struct ipipe_trace_path *tp, int pos)
+{
+	struct ipipe_trace_path *old_tp = tp;
+	long active = per_cpu(active_path, cpu);
+	int n;
+
+	/* frozen paths have no core (begin=end) */
+	tp->begin = tp->end;
+
+	/* we need protection here against other cpus trying
+	 * to set their frozen path or to start a proc dump */
+	spin_lock(&global_path_lock);
+
+	per_cpu(frozen_path, cpu) = active;
+
+	/* find next unused trace path */
+	active = __ipipe_get_free_trace_path(active, cpu);
+
+	/* check if this is the first frozen path */
+	for_each_possible_cpu(n) {
+		if (n != cpu &&
+		    per_cpu(trace_path, n)[per_cpu(frozen_path, n)].end >= 0)
+			tp->end = -1;
+	}
+
+	spin_unlock(&global_path_lock);
+
+	tp = &per_cpu(trace_path, cpu)[active];
+
+	/* migrate last entries for pre-tracing */
+	__ipipe_migrate_pre_trace(tp, old_tp, pos);
+
+	return tp;
+}
+
+void notrace
+__ipipe_trace(enum ipipe_trace_type type, unsigned long eip,
+              unsigned long parent_eip, unsigned long v)
+{
+	struct ipipe_trace_path *tp, *old_tp;
+	int pos, next_pos, begin;
+	struct ipipe_trace_point *point;
+	unsigned long flags;
+	int cpu;
+
+	local_irq_save_hw_notrace(flags);
+
+	cpu = ipipe_processor_id();
+ restart:
+	tp = old_tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)];
+
+	/* here starts a race window with NMIs - catched below */
+
+	/* check for NMI recursion */
+	if (unlikely(tp->flags & IPIPE_TFLG_NMI_LOCK)) {
+		tp->flags |= IPIPE_TFLG_NMI_HIT;
+
+		/* first freeze request from NMI context? */
+		if ((type == IPIPE_TRACE_FREEZE) &&
+		    !(tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ)) {
+			/* save arguments and mark deferred freezing */
+			tp->flags |= IPIPE_TFLG_NMI_FREEZE_REQ;
+			tp->nmi_saved_eip = eip;
+			tp->nmi_saved_parent_eip = parent_eip;
+			tp->nmi_saved_v = v;
+		}
+		return; /* no need for restoring flags inside IRQ */
+	}
+
+	/* clear NMI events and set lock (atomically per cpu) */
+	tp->flags = (tp->flags & ~(IPIPE_TFLG_NMI_HIT |
+	                           IPIPE_TFLG_NMI_FREEZE_REQ))
+	                       | IPIPE_TFLG_NMI_LOCK;
+
+	/* check active_path again - some nasty NMI may have switched
+	 * it meanwhile */
+	if (unlikely(tp !=
+		     &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)])) {
+		/* release lock on wrong path and restart */
+		tp->flags &= ~IPIPE_TFLG_NMI_LOCK;
+
+		/* there is no chance that the NMI got deferred
+		 * => no need to check for pending freeze requests */
+		goto restart;
+	}
+
+	/* get the point buffer */
+	pos = tp->trace_pos;
+	point = &tp->point[pos];
+
+	/* store all trace point data */
+	point->type = type;
+	point->flags = raw_irqs_disabled_flags(flags) ? IPIPE_TFLG_HWIRQ_OFF : 0;
+	point->eip = eip;
+	point->parent_eip = parent_eip;
+	point->v = v;
+	ipipe_read_tsc(point->timestamp);
+
+	__ipipe_store_domain_states(point);
+
+	/* forward to next point buffer */
+	next_pos = WRAP_POINT_NO(pos+1);
+	tp->trace_pos = next_pos;
+
+	/* only mark beginning if we haven't started yet */
+	begin = tp->begin;
+	if (unlikely(type == IPIPE_TRACE_BEGIN) && (begin < 0))
+		tp->begin = pos;
+
+	/* end of critical path, start post-trace if not already started */
+	if (unlikely(type == IPIPE_TRACE_END) &&
+	    (begin >= 0) && !tp->post_trace)
+		tp->post_trace = post_trace + 1;
+
+	/* freeze only if the slot is free and we are not already freezing */
+	if ((unlikely(type == IPIPE_TRACE_FREEZE) ||
+	     (unlikely(eip >= trigger_begin && eip <= trigger_end) &&
+	     type == IPIPE_TRACE_FUNC)) &&
+	    per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)].begin < 0 &&
+	    !(tp->flags & IPIPE_TFLG_FREEZING)) {
+		tp->post_trace = post_trace + 1;
+		tp->flags |= IPIPE_TFLG_FREEZING;
+	}
+
+	/* enforce end of trace in case of overflow */
+	if (unlikely(WRAP_POINT_NO(next_pos + 1) == begin)) {
+		tp->end = pos;
+		goto enforce_end;
+	}
+
+	/* stop tracing this path if we are in post-trace and
+	 *  a) that phase is over now or
+	 *  b) a new TRACE_BEGIN came in but we are not freezing this path */
+	if (unlikely((tp->post_trace > 0) && ((--tp->post_trace == 0) ||
+	             ((type == IPIPE_TRACE_BEGIN) &&
+	              !(tp->flags & IPIPE_TFLG_FREEZING))))) {
+		/* store the path's end (i.e. excluding post-trace) */
+		tp->end = WRAP_POINT_NO(pos - post_trace + tp->post_trace);
+
+ enforce_end:
+		if (tp->flags & IPIPE_TFLG_FREEZING)
+			tp = __ipipe_trace_freeze(cpu, tp, pos);
+		else
+			tp = __ipipe_trace_end(cpu, tp, pos);
+
+		/* reset the active path, maybe already start a new one */
+		tp->begin = (type == IPIPE_TRACE_BEGIN) ?
+			WRAP_POINT_NO(tp->trace_pos - 1) : -1;
+		tp->end = -1;
+		tp->post_trace = 0;
+		tp->flags = 0;
+
+		/* update active_path not earlier to avoid races with NMIs */
+		per_cpu(active_path, cpu) = tp - per_cpu(trace_path, cpu);
+	}
+
+	/* we still have old_tp and point,
+	 * let's reset NMI lock and check for catches */
+	old_tp->flags &= ~IPIPE_TFLG_NMI_LOCK;
+	if (unlikely(old_tp->flags & IPIPE_TFLG_NMI_HIT)) {
+		/* well, this late tagging may not immediately be visible for
+		 * other cpus already dumping this path - a minor issue */
+		point->flags |= IPIPE_TFLG_NMI_HIT;
+
+		/* handle deferred freezing from NMI context */
+		if (old_tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ)
+			__ipipe_trace(IPIPE_TRACE_FREEZE, old_tp->nmi_saved_eip,
+			              old_tp->nmi_saved_parent_eip,
+			              old_tp->nmi_saved_v);
+	}
+
+	local_irq_restore_hw_notrace(flags);
+}
+
+static unsigned long __ipipe_global_path_lock(void)
+{
+	unsigned long flags;
+	int cpu;
+	struct ipipe_trace_path *tp;
+
+	spin_lock_irqsave(&global_path_lock, flags);
+
+	cpu = ipipe_processor_id();
+ restart:
+	tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)];
+
+	/* here is small race window with NMIs - catched below */
+
+	/* clear NMI events and set lock (atomically per cpu) */
+	tp->flags = (tp->flags & ~(IPIPE_TFLG_NMI_HIT |
+	                           IPIPE_TFLG_NMI_FREEZE_REQ))
+	                       | IPIPE_TFLG_NMI_LOCK;
+
+	/* check active_path again - some nasty NMI may have switched
+	 * it meanwhile */
+	if (tp != &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)]) {
+		/* release lock on wrong path and restart */
+		tp->flags &= ~IPIPE_TFLG_NMI_LOCK;
+
+		/* there is no chance that the NMI got deferred
+		 * => no need to check for pending freeze requests */
+		goto restart;
+	}
+
+	return flags;
+}
+
+static void __ipipe_global_path_unlock(unsigned long flags)
+{
+	int cpu;
+	struct ipipe_trace_path *tp;
+
+	/* release spinlock first - it's not involved in the NMI issue */
+	__ipipe_spin_unlock_irqbegin(&global_path_lock);
+
+	cpu = ipipe_processor_id();
+	tp = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)];
+
+	tp->flags &= ~IPIPE_TFLG_NMI_LOCK;
+
+	/* handle deferred freezing from NMI context */
+	if (tp->flags & IPIPE_TFLG_NMI_FREEZE_REQ)
+		__ipipe_trace(IPIPE_TRACE_FREEZE, tp->nmi_saved_eip,
+		              tp->nmi_saved_parent_eip, tp->nmi_saved_v);
+
+	/* See __ipipe_spin_lock_irqsave() and friends. */
+	__ipipe_spin_unlock_irqcomplete(flags);
+}
+
+void notrace ipipe_trace_begin(unsigned long v)
+{
+	if (!ipipe_trace_enable)
+		return;
+	__ipipe_trace(IPIPE_TRACE_BEGIN, __BUILTIN_RETURN_ADDRESS0,
+	              __BUILTIN_RETURN_ADDRESS1, v);
+}
+EXPORT_SYMBOL(ipipe_trace_begin);
+
+void notrace ipipe_trace_end(unsigned long v)
+{
+	if (!ipipe_trace_enable)
+		return;
+	__ipipe_trace(IPIPE_TRACE_END, __BUILTIN_RETURN_ADDRESS0,
+	              __BUILTIN_RETURN_ADDRESS1, v);
+}
+EXPORT_SYMBOL(ipipe_trace_end);
+
+void notrace ipipe_trace_freeze(unsigned long v)
+{
+	if (!ipipe_trace_enable)
+		return;
+	__ipipe_trace(IPIPE_TRACE_FREEZE, __BUILTIN_RETURN_ADDRESS0,
+	              __BUILTIN_RETURN_ADDRESS1, v);
+}
+EXPORT_SYMBOL(ipipe_trace_freeze);
+
+void notrace ipipe_trace_special(unsigned char id, unsigned long v)
+{
+	if (!ipipe_trace_enable)
+		return;
+	__ipipe_trace(IPIPE_TRACE_SPECIAL | (id << IPIPE_TYPE_BITS),
+	              __BUILTIN_RETURN_ADDRESS0,
+	              __BUILTIN_RETURN_ADDRESS1, v);
+}
+EXPORT_SYMBOL(ipipe_trace_special);
+
+void notrace ipipe_trace_pid(pid_t pid, short prio)
+{
+	if (!ipipe_trace_enable)
+		return;
+	__ipipe_trace(IPIPE_TRACE_PID | (prio << IPIPE_TYPE_BITS),
+	              __BUILTIN_RETURN_ADDRESS0,
+	              __BUILTIN_RETURN_ADDRESS1, pid);
+}
+EXPORT_SYMBOL(ipipe_trace_pid);
+
+void notrace ipipe_trace_event(unsigned char id, unsigned long delay_tsc)
+{
+	if (!ipipe_trace_enable)
+		return;
+	__ipipe_trace(IPIPE_TRACE_EVENT | (id << IPIPE_TYPE_BITS),
+	              __BUILTIN_RETURN_ADDRESS0,
+	              __BUILTIN_RETURN_ADDRESS1, delay_tsc);
+}
+EXPORT_SYMBOL(ipipe_trace_event);
+
+int ipipe_trace_max_reset(void)
+{
+	int cpu;
+	unsigned long flags;
+	struct ipipe_trace_path *path;
+	int ret = 0;
+
+	flags = __ipipe_global_path_lock();
+
+	for_each_possible_cpu(cpu) {
+		path = &per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)];
+
+		if (path->dump_lock) {
+			ret = -EBUSY;
+			break;
+		}
+
+		path->begin     = -1;
+		path->end       = -1;
+		path->trace_pos = 0;
+		path->length    = 0;
+	}
+
+	__ipipe_global_path_unlock(flags);
+
+	return ret;
+}
+EXPORT_SYMBOL(ipipe_trace_max_reset);
+
+int ipipe_trace_frozen_reset(void)
+{
+	int cpu;
+	unsigned long flags;
+	struct ipipe_trace_path *path;
+	int ret = 0;
+
+	flags = __ipipe_global_path_lock();
+
+	for_each_online_cpu(cpu) {
+		path = &per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)];
+
+		if (path->dump_lock) {
+			ret = -EBUSY;
+			break;
+		}
+
+		path->begin = -1;
+		path->end = -1;
+		path->trace_pos = 0;
+		path->length    = 0;
+	}
+
+	__ipipe_global_path_unlock(flags);
+
+	return ret;
+}
+EXPORT_SYMBOL(ipipe_trace_frozen_reset);
+
+static void
+__ipipe_get_task_info(char *task_info, struct ipipe_trace_point *point,
+                      int trylock)
+{
+	struct task_struct *task = NULL;
+	char buf[8];
+	int i;
+	int locked = 1;
+
+	if (trylock) {
+		if (!read_trylock(&tasklist_lock))
+			locked = 0;
+	} else
+		read_lock(&tasklist_lock);
+
+	if (locked)
+		task = find_task_by_pid_type_ns(PIDTYPE_PID, (pid_t)point->v, &init_pid_ns);
+
+	if (task)
+		strncpy(task_info, task->comm, 11);
+	else
+		strcpy(task_info, "-<?>-");
+
+	if (locked)
+		read_unlock(&tasklist_lock);
+
+	for (i = strlen(task_info); i < 11; i++)
+		task_info[i] = ' ';
+
+	sprintf(buf, " %d ", point->type >> IPIPE_TYPE_BITS);
+	strcpy(task_info + (11 - strlen(buf)), buf);
+}
+
+static void
+__ipipe_get_event_date(char *buf,struct ipipe_trace_path *path,
+		       struct ipipe_trace_point *point)
+{
+	long time;
+	int type;
+
+	time = __ipipe_signed_tsc2us(point->timestamp -
+				     path->point[path->begin].timestamp + point->v);
+	type = point->type >> IPIPE_TYPE_BITS;
+
+	if (type == 0)
+		/*
+		 * Event type #0 is predefined, stands for the next
+		 * timer tick.
+		 */
+		sprintf(buf, "tick@domain.hid", time);
+	else
+		sprintf(buf, "%3d@domain.hid", type, time);
+}
+
+#ifdef CONFIG_IPIPE_TRACE_PANIC
+void ipipe_trace_panic_freeze(void)
+{
+	unsigned long flags;
+	int cpu;
+
+	if (!ipipe_trace_enable)
+		return;
+
+	ipipe_trace_enable = 0;
+	local_irq_save_hw_notrace(flags);
+
+	cpu = ipipe_processor_id();
+
+	panic_path = &per_cpu(trace_path, cpu)[per_cpu(active_path, cpu)];
+
+	local_irq_restore_hw(flags);
+}
+EXPORT_SYMBOL(ipipe_trace_panic_freeze);
+
+void ipipe_trace_panic_dump(void)
+{
+	int cnt = back_trace;
+	int start, pos;
+	char buf[16];
+
+	if (!panic_path)
+		return;
+
+	ipipe_context_check_off();
+
+	printk("I-pipe tracer log (%d points):\n", cnt);
+
+	start = pos = WRAP_POINT_NO(panic_path->trace_pos-1);
+
+	while (cnt-- > 0) {
+		struct ipipe_trace_point *point = &panic_path->point[pos];
+		long time;
+		char info[16];
+		int i;
+
+		printk(" %c",
+		       (point->flags & IPIPE_TFLG_HWIRQ_OFF) ? '|' : ' ');
+
+		for (i = IPIPE_TFLG_DOMSTATE_BITS; i >= 0; i--)
+			printk("%c",
+			       (IPIPE_TFLG_CURRENT_DOMAIN(point) == i) ?
+				(IPIPE_TFLG_DOMAIN_STALLED(point, i) ?
+					'#' : '+') :
+				(IPIPE_TFLG_DOMAIN_STALLED(point, i) ?
+					'*' : ' '));
+
+		if (!point->eip)
+			printk("-<invalid>-\n");
+		else {
+			__ipipe_trace_point_type(buf, point);
+			printk("%s", buf);
+
+			switch (point->type & IPIPE_TYPE_MASK) {
+				case IPIPE_TRACE_FUNC:
+					printk("           ");
+					break;
+
+				case IPIPE_TRACE_PID:
+					__ipipe_get_task_info(info,
+							      point, 1);
+					printk("%s", info);
+					break;
+
+				case IPIPE_TRACE_EVENT:
+					__ipipe_get_event_date(info,
+							       panic_path, point);
+					printk("%s", info);
+					break;
+
+				default:
+					printk("0x%08lx ", point->v);
+			}
+
+			time = __ipipe_signed_tsc2us(point->timestamp -
+				panic_path->point[start].timestamp);
+			printk(" %5ld ", time);
+
+			__ipipe_print_symname(NULL, point->eip);
+			printk(" (");
+			__ipipe_print_symname(NULL, point->parent_eip);
+			printk(")\n");
+		}
+		pos = WRAP_POINT_NO(pos - 1);
+	}
+
+	panic_path = NULL;
+}
+EXPORT_SYMBOL(ipipe_trace_panic_dump);
+#endif /* CONFIG_IPIPE_TRACE_PANIC */
+
+
+/* --- /proc output --- */
+
+static notrace int __ipipe_in_critical_trpath(long point_no)
+{
+	return ((WRAP_POINT_NO(point_no-print_path->begin) <
+	         WRAP_POINT_NO(print_path->end-print_path->begin)) ||
+	        ((print_path->end == print_path->begin) &&
+	         (WRAP_POINT_NO(point_no-print_path->end) >
+	          print_post_trace)));
+}
+
+static long __ipipe_signed_tsc2us(long long tsc)
+{
+        unsigned long long abs_tsc;
+        long us;
+
+	/* ipipe_tsc2us works on unsigned => handle sign separately */
+        abs_tsc = (tsc >= 0) ? tsc : -tsc;
+        us = ipipe_tsc2us(abs_tsc);
+        if (tsc < 0)
+                return -us;
+        else
+                return us;
+}
+
+static void
+__ipipe_trace_point_type(char *buf, struct ipipe_trace_point *point)
+{
+	switch (point->type & IPIPE_TYPE_MASK) {
+		case IPIPE_TRACE_FUNC:
+			strcpy(buf, "func    ");
+			break;
+
+		case IPIPE_TRACE_BEGIN:
+			strcpy(buf, "begin   ");
+			break;
+
+		case IPIPE_TRACE_END:
+			strcpy(buf, "end     ");
+			break;
+
+		case IPIPE_TRACE_FREEZE:
+			strcpy(buf, "freeze  ");
+			break;
+
+		case IPIPE_TRACE_SPECIAL:
+			sprintf(buf, "(0x%02x)  ",
+				point->type >> IPIPE_TYPE_BITS);
+			break;
+
+		case IPIPE_TRACE_PID:
+			sprintf(buf, "[%5d] ", (pid_t)point->v);
+			break;
+
+		case IPIPE_TRACE_EVENT:
+			sprintf(buf, "event   ");
+			break;
+	}
+}
+
+static void
+__ipipe_print_pathmark(struct seq_file *m, struct ipipe_trace_point *point)
+{
+	char mark = ' ';
+	int point_no = point - print_path->point;
+	int i;
+
+	if (print_path->end == point_no)
+		mark = '<';
+	else if (print_path->begin == point_no)
+		mark = '>';
+	else if (__ipipe_in_critical_trpath(point_no))
+		mark = ':';
+	seq_printf(m, "%c%c", mark,
+	           (point->flags & IPIPE_TFLG_HWIRQ_OFF) ? '|' : ' ');
+
+	if (!verbose_trace)
+		return;
+
+	for (i = IPIPE_TFLG_DOMSTATE_BITS; i >= 0; i--)
+		seq_printf(m, "%c",
+			(IPIPE_TFLG_CURRENT_DOMAIN(point) == i) ?
+			    (IPIPE_TFLG_DOMAIN_STALLED(point, i) ?
+				'#' : '+') :
+			(IPIPE_TFLG_DOMAIN_STALLED(point, i) ? '*' : ' '));
+}
+
+static void
+__ipipe_print_delay(struct seq_file *m, struct ipipe_trace_point *point)
+{
+	unsigned long delay = 0;
+	int next;
+	char *mark = "  ";
+
+	next = WRAP_POINT_NO(point+1 - print_path->point);
+
+	if (next != print_path->trace_pos)
+		delay = ipipe_tsc2ns(print_path->point[next].timestamp -
+		                     point->timestamp);
+
+	if (__ipipe_in_critical_trpath(point - print_path->point)) {
+		if (delay > IPIPE_DELAY_WARN)
+			mark = "! ";
+		else if (delay > IPIPE_DELAY_NOTE)
+			mark = "+ ";
+	}
+	seq_puts(m, mark);
+
+	if (verbose_trace)
+		seq_printf(m, "%3lu.%03lu%c ", delay/1000, delay%1000,
+		           (point->flags & IPIPE_TFLG_NMI_HIT) ? 'N' : ' ');
+	else
+		seq_puts(m, " ");
+}
+
+static void __ipipe_print_symname(struct seq_file *m, unsigned long eip)
+{
+	char namebuf[KSYM_NAME_LEN+1];
+	unsigned long size, offset;
+	const char *sym_name;
+	char *modname;
+
+	sym_name = kallsyms_lookup(eip, &size, &offset, &modname, namebuf);
+
+#ifdef CONFIG_IPIPE_TRACE_PANIC
+	if (!m) {
+		/* panic dump */
+		if (sym_name) {
+			printk("%s+0x%lx", sym_name, offset);
+			if (modname)
+				printk(" [%s]", modname);
+		}
+	} else
+#endif /* CONFIG_IPIPE_TRACE_PANIC */
+	{
+		if (sym_name) {
+			if (verbose_trace) {
+				seq_printf(m, "%s+0x%lx", sym_name, offset);
+				if (modname)
+					seq_printf(m, " [%s]", modname);
+			} else
+				seq_puts(m, sym_name);
+		} else
+			seq_printf(m, "<%08lx>", eip);
+	}
+}
+
+static void __ipipe_print_headline(struct seq_file *m)
+{
+	seq_printf(m, "Calibrated minimum trace-point overhead: %lu.%03lu "
+		   "us\n\n", trace_overhead/1000, trace_overhead%1000);
+
+	if (verbose_trace) {
+		const char *name[4] = { [0 ... 3] = "<unused>" };
+		struct list_head *pos;
+		int i = 0;
+
+		list_for_each_prev(pos, &__ipipe_pipeline) {
+			struct ipipe_domain *ipd =
+				list_entry(pos, struct ipipe_domain, p_link);
+
+			name[i] = ipd->name;
+			if (++i > 3)
+				break;
+		}
+
+		seq_printf(m,
+		           " +----- Hard IRQs ('|': locked)\n"
+		           " |+---- %s\n"
+		           " ||+--- %s\n"
+		           " |||+-- %s\n"
+		           " ||||+- %s%s\n"
+		           " |||||                        +---------- "
+		               "Delay flag ('+': > %d us, '!': > %d us)\n"
+		           " |||||                        |        +- "
+		               "NMI noise ('N')\n"
+		           " |||||                        |        |\n"
+		           "      Type    User Val.   Time    Delay  Function "
+		               "(Parent)\n",
+		           name[3], name[2], name[1], name[0],
+		           name[0] ? " ('*': domain stalled, '+': current, "
+		               "'#': current+stalled)" : "",
+		           IPIPE_DELAY_NOTE/1000, IPIPE_DELAY_WARN/1000);
+	} else
+		seq_printf(m,
+		           " +--------------- Hard IRQs ('|': locked)\n"
+		           " |             +- Delay flag "
+		               "('+': > %d us, '!': > %d us)\n"
+		           " |             |\n"
+		           "  Type     Time   Function (Parent)\n",
+		           IPIPE_DELAY_NOTE/1000, IPIPE_DELAY_WARN/1000);
+}
+
+static void *__ipipe_max_prtrace_start(struct seq_file *m, loff_t *pos)
+{
+	loff_t n = *pos;
+
+	mutex_lock(&out_mutex);
+
+	if (!n) {
+		struct ipipe_trace_path *tp;
+		unsigned long length_usecs;
+		int points, cpu;
+		unsigned long flags;
+
+		/* protect against max_path/frozen_path updates while we
+		 * haven't locked our target path, also avoid recursively
+		 * taking global_path_lock from NMI context */
+		flags = __ipipe_global_path_lock();
+
+		/* find the longest of all per-cpu paths */
+		print_path = NULL;
+		for_each_online_cpu(cpu) {
+			tp = &per_cpu(trace_path, cpu)[per_cpu(max_path, cpu)];
+			if ((print_path == NULL) ||
+			    (tp->length > print_path->length)) {
+				print_path = tp;
+				break;
+			}
+		}
+		print_path->dump_lock = 1;
+
+		__ipipe_global_path_unlock(flags);
+
+		/* does this path actually contain data? */
+		if (print_path->end == print_path->begin)
+			return NULL;
+
+		/* number of points inside the critical path */
+		points = WRAP_POINT_NO(print_path->end-print_path->begin+1);
+
+		/* pre- and post-tracing length, post-trace length was frozen
+		   in __ipipe_trace, pre-trace may have to be reduced due to
+		   buffer overrun */
+		print_pre_trace  = pre_trace;
+		print_post_trace = WRAP_POINT_NO(print_path->trace_pos -
+		                                 print_path->end - 1);
+		if (points+pre_trace+print_post_trace > IPIPE_TRACE_POINTS - 1)
+			print_pre_trace = IPIPE_TRACE_POINTS - 1 - points -
+				print_post_trace;
+
+		length_usecs = ipipe_tsc2us(print_path->length);
+		seq_printf(m, "I-pipe worst-case tracing service on %s/ipipe-%s\n"
+			"------------------------------------------------------------\n",
+			UTS_RELEASE, IPIPE_ARCH_STRING);
+		seq_printf(m, "CPU: %d, Begin: %lld cycles, Trace Points: "
+			"%d (-%d/+%d), Length: %lu us\n",
+			cpu, print_path->point[print_path->begin].timestamp,
+			points, print_pre_trace, print_post_trace, length_usecs);
+		__ipipe_print_headline(m);
+	}
+
+	/* check if we are inside the trace range */
+	if (n >= WRAP_POINT_NO(print_path->end - print_path->begin + 1 +
+	                       print_pre_trace + print_post_trace))
+		return NULL;
+
+	/* return the next point to be shown */
+	return &print_path->point[WRAP_POINT_NO(print_path->begin -
+	                                        print_pre_trace + n)];
+}
+
+static void *__ipipe_prtrace_next(struct seq_file *m, void *p, loff_t *pos)
+{
+	loff_t n = ++*pos;
+
+	/* check if we are inside the trace range with the next entry */
+	if (n >= WRAP_POINT_NO(print_path->end - print_path->begin + 1 +
+	                       print_pre_trace + print_post_trace))
+		return NULL;
+
+	/* return the next point to be shown */
+	return &print_path->point[WRAP_POINT_NO(print_path->begin -
+	                                        print_pre_trace + *pos)];
+}
+
+static void __ipipe_prtrace_stop(struct seq_file *m, void *p)
+{
+	if (print_path)
+		print_path->dump_lock = 0;
+	mutex_unlock(&out_mutex);
+}
+
+static int __ipipe_prtrace_show(struct seq_file *m, void *p)
+{
+	long time;
+	struct ipipe_trace_point *point = p;
+	char buf[16];
+
+	if (!point->eip) {
+		seq_puts(m, "-<invalid>-\n");
+		return 0;
+	}
+
+	__ipipe_print_pathmark(m, point);
+	__ipipe_trace_point_type(buf, point);
+	seq_puts(m, buf);
+	if (verbose_trace)
+		switch (point->type & IPIPE_TYPE_MASK) {
+			case IPIPE_TRACE_FUNC:
+				seq_puts(m, "           ");
+				break;
+
+			case IPIPE_TRACE_PID:
+				__ipipe_get_task_info(buf, point, 0);
+				seq_puts(m, buf);
+				break;
+
+			case IPIPE_TRACE_EVENT:
+				__ipipe_get_event_date(buf, print_path, point);
+				seq_puts(m, buf);
+				break;
+
+			default:
+				seq_printf(m, "0x%08lx ", point->v);
+		}
+
+	time = __ipipe_signed_tsc2us(point->timestamp -
+		print_path->point[print_path->begin].timestamp);
+	seq_printf(m, "%5ld", time);
+
+	__ipipe_print_delay(m, point);
+	__ipipe_print_symname(m, point->eip);
+	seq_puts(m, " (");
+	__ipipe_print_symname(m, point->parent_eip);
+	seq_puts(m, ")\n");
+
+	return 0;
+}
+
+static struct seq_operations __ipipe_max_ptrace_ops = {
+	.start = __ipipe_max_prtrace_start,
+	.next  = __ipipe_prtrace_next,
+	.stop  = __ipipe_prtrace_stop,
+	.show  = __ipipe_prtrace_show
+};
+
+static int __ipipe_max_prtrace_open(struct inode *inode, struct file *file)
+{
+	return seq_open(file, &__ipipe_max_ptrace_ops);
+}
+
+static ssize_t
+__ipipe_max_reset(struct file *file, const char __user *pbuffer,
+                  size_t count, loff_t *data)
+{
+	mutex_lock(&out_mutex);
+	ipipe_trace_max_reset();
+	mutex_unlock(&out_mutex);
+
+	return count;
+}
+
+struct file_operations __ipipe_max_prtrace_fops = {
+	.open       = __ipipe_max_prtrace_open,
+	.read       = seq_read,
+	.write      = __ipipe_max_reset,
+	.llseek     = seq_lseek,
+	.release    = seq_release,
+};
+
+static void *__ipipe_frozen_prtrace_start(struct seq_file *m, loff_t *pos)
+{
+	loff_t n = *pos;
+
+	mutex_lock(&out_mutex);
+
+	if (!n) {
+		struct ipipe_trace_path *tp;
+		int cpu;
+		unsigned long flags;
+
+		/* protect against max_path/frozen_path updates while we
+		 * haven't locked our target path, also avoid recursively
+		 * taking global_path_lock from NMI context */
+		flags = __ipipe_global_path_lock();
+
+		/* find the first of all per-cpu frozen paths */
+		print_path = NULL;
+		for_each_online_cpu(cpu) {
+			tp = &per_cpu(trace_path, cpu)[per_cpu(frozen_path, cpu)];
+			if (tp->end >= 0) {
+				print_path = tp;
+				break;
+			}
+		}
+		if (print_path)
+			print_path->dump_lock = 1;
+
+		__ipipe_global_path_unlock(flags);
+
+		if (!print_path)
+			return NULL;
+
+		/* back- and post-tracing length, post-trace length was frozen
+		   in __ipipe_trace, back-trace may have to be reduced due to
+		   buffer overrun */
+		print_pre_trace  = back_trace-1; /* substract freeze point */
+		print_post_trace = WRAP_POINT_NO(print_path->trace_pos -
+		                                 print_path->end - 1);
+		if (1+pre_trace+print_post_trace > IPIPE_TRACE_POINTS - 1)
+			print_pre_trace = IPIPE_TRACE_POINTS - 2 -
+				print_post_trace;
+
+		seq_printf(m, "I-pipe frozen back-tracing service on %s/ipipe-%s\n"
+			"------------------------------------------------------"
+			"------\n",
+			UTS_RELEASE, IPIPE_ARCH_STRING);
+		seq_printf(m, "CPU: %d, Freeze: %lld cycles, Trace Points: %d (+%d)\n",
+			cpu, print_path->point[print_path->begin].timestamp,
+			print_pre_trace+1, print_post_trace);
+		__ipipe_print_headline(m);
+	}
+
+	/* check if we are inside the trace range */
+	if (n >= print_pre_trace + 1 + print_post_trace)
+		return NULL;
+
+	/* return the next point to be shown */
+	return &print_path->point[WRAP_POINT_NO(print_path->begin-
+	                                        print_pre_trace+n)];
+}
+
+static struct seq_operations __ipipe_frozen_ptrace_ops = {
+	.start = __ipipe_frozen_prtrace_start,
+	.next  = __ipipe_prtrace_next,
+	.stop  = __ipipe_prtrace_stop,
+	.show  = __ipipe_prtrace_show
+};
+
+static int __ipipe_frozen_prtrace_open(struct inode *inode, struct file *file)
+{
+	return seq_open(file, &__ipipe_frozen_ptrace_ops);
+}
+
+static ssize_t
+__ipipe_frozen_ctrl(struct file *file, const char __user *pbuffer,
+                    size_t count, loff_t *data)
+{
+	char *end, buf[16];
+	int val;
+	int n;
+
+	n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count;
+
+	if (copy_from_user(buf, pbuffer, n))
+		return -EFAULT;
+
+	buf[n] = '\0';
+	val = simple_strtol(buf, &end, 0);
+
+	if (((*end != '\0') && !isspace(*end)) || (val < 0))
+		return -EINVAL;
+
+	mutex_lock(&out_mutex);
+	ipipe_trace_frozen_reset();
+	if (val > 0)
+		ipipe_trace_freeze(-1);
+	mutex_unlock(&out_mutex);
+
+	return count;
+}
+
+struct file_operations __ipipe_frozen_prtrace_fops = {
+	.open       = __ipipe_frozen_prtrace_open,
+	.read       = seq_read,
+	.write      = __ipipe_frozen_ctrl,
+	.llseek     = seq_lseek,
+	.release    = seq_release,
+};
+
+static int __ipipe_rd_proc_val(char *page, char **start, off_t off,
+                               int count, int *eof, void *data)
+{
+	int len;
+
+	len = sprintf(page, "%u\n", *(int *)data);
+	len -= off;
+	if (len <= off + count)
+		*eof = 1;
+	*start = page + off;
+	if (len > count)
+		len = count;
+	if (len < 0)
+		len = 0;
+
+	return len;
+}
+
+static int __ipipe_wr_proc_val(struct file *file, const char __user *buffer,
+                               unsigned long count, void *data)
+{
+	char *end, buf[16];
+	int val;
+	int n;
+
+	n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count;
+
+	if (copy_from_user(buf, buffer, n))
+		return -EFAULT;
+
+	buf[n] = '\0';
+	val = simple_strtol(buf, &end, 0);
+
+	if (((*end != '\0') && !isspace(*end)) || (val < 0))
+		return -EINVAL;
+
+	mutex_lock(&out_mutex);
+	*(int *)data = val;
+	mutex_unlock(&out_mutex);
+
+	return count;
+}
+
+static int __ipipe_rd_trigger(char *page, char **start, off_t off, int count,
+			      int *eof, void *data)
+{
+	int len;
+
+	if (!trigger_begin)
+		return 0;
+
+	len = sprint_symbol(page, trigger_begin);
+	page[len++] = '\n';
+
+	len -= off;
+	if (len <= off + count)
+		*eof = 1;
+	*start = page + off;
+	if (len > count)
+		len = count;
+	if (len < 0)
+		len = 0;
+
+	return len;
+}
+
+static int __ipipe_wr_trigger(struct file *file, const char __user *buffer,
+			      unsigned long count, void *data)
+{
+	char buf[KSYM_SYMBOL_LEN];
+	unsigned long begin, end;
+
+	if (count > sizeof(buf) - 1)
+		count = sizeof(buf) - 1;
+	if (copy_from_user(buf, buffer, count))
+		return -EFAULT;
+	buf[count] = 0;
+	if (buf[count-1] == '\n')
+		buf[count-1] = 0;
+
+	begin = kallsyms_lookup_name(buf);
+	if (!begin || !kallsyms_lookup_size_offset(begin, &end, NULL))
+		return -ENOENT;
+	end += begin - 1;
+
+	mutex_lock(&out_mutex);
+	/* invalidate the current range before setting a new one */
+	trigger_end = 0;
+	wmb();
+	ipipe_trace_frozen_reset();
+
+	/* set new range */
+	trigger_begin = begin;
+	wmb();
+	trigger_end = end;
+	mutex_unlock(&out_mutex);
+
+	return count;
+}
+
+#ifdef CONFIG_IPIPE_TRACE_MCOUNT
+static void notrace
+ipipe_trace_function(unsigned long ip, unsigned long parent_ip)
+{
+	if (!ipipe_trace_enable)
+		return;
+	__ipipe_trace(IPIPE_TRACE_FUNC, ip, parent_ip, 0);
+}
+
+static struct ftrace_ops ipipe_trace_ops = {
+	.func = ipipe_trace_function
+};
+
+static int __ipipe_wr_enable(struct file *file, const char __user *buffer,
+			     unsigned long count, void *data)
+{
+	char *end, buf[16];
+	int val;
+	int n;
+
+	n = (count > sizeof(buf) - 1) ? sizeof(buf) - 1 : count;
+
+	if (copy_from_user(buf, buffer, n))
+		return -EFAULT;
+
+	buf[n] = '\0';
+	val = simple_strtol(buf, &end, 0);
+
+	if (((*end != '\0') && !isspace(*end)) || (val < 0))
+		return -EINVAL;
+
+	mutex_lock(&out_mutex);
+
+	if (ipipe_trace_enable) {
+		if (!val)
+			unregister_ftrace_function(&ipipe_trace_ops);
+	} else if (val)
+		register_ftrace_function(&ipipe_trace_ops);
+
+	ipipe_trace_enable = val;
+
+	mutex_unlock(&out_mutex);
+
+	return count;
+}
+#endif /* CONFIG_IPIPE_TRACE_MCOUNT */
+
+extern struct proc_dir_entry *ipipe_proc_root;
+
+static struct proc_dir_entry * __init
+__ipipe_create_trace_proc_val(struct proc_dir_entry *trace_dir,
+                              const char *name, int *value_ptr)
+{
+	struct proc_dir_entry *entry;
+
+	entry = create_proc_entry(name, 0644, trace_dir);
+	if (entry) {
+		entry->data = value_ptr;
+		entry->read_proc = __ipipe_rd_proc_val;
+		entry->write_proc = __ipipe_wr_proc_val;
+	}
+	return entry;
+}
+
+void __init __ipipe_init_tracer(void)
+{
+	struct proc_dir_entry *trace_dir;
+	struct proc_dir_entry *entry;
+	unsigned long long start, end, min = ULLONG_MAX;
+	int i;
+#ifdef CONFIG_IPIPE_TRACE_VMALLOC
+	int cpu, path;
+
+	for_each_possible_cpu(cpu) {
+		struct ipipe_trace_path *tp_buf;
+
+		tp_buf = vmalloc_node(sizeof(struct ipipe_trace_path) *
+				      IPIPE_TRACE_PATHS, cpu_to_node(cpu));
+		if (!tp_buf) {
+			printk(KERN_ERR "I-pipe: "
+			       "insufficient memory for trace buffer.\n");
+			return;
+		}
+		memset(tp_buf, 0,
+		       sizeof(struct ipipe_trace_path) * IPIPE_TRACE_PATHS);
+		for (path = 0; path < IPIPE_TRACE_PATHS; path++) {
+			tp_buf[path].begin = -1;
+			tp_buf[path].end   = -1;
+		}
+		per_cpu(trace_path, cpu) = tp_buf;
+	}
+#endif /* CONFIG_IPIPE_TRACE_VMALLOC */
+
+	/* Calculate minimum overhead of __ipipe_trace() */
+	local_irq_disable_hw();
+	for (i = 0; i < 100; i++) {
+		ipipe_read_tsc(start);
+		__ipipe_trace(IPIPE_TRACE_FUNC, __BUILTIN_RETURN_ADDRESS0,
+			      __BUILTIN_RETURN_ADDRESS1, 0);
+		ipipe_read_tsc(end);
+
+		end -= start;
+		if (end < min)
+			min = end;
+	}
+	local_irq_enable_hw();
+	trace_overhead = ipipe_tsc2ns(min);
+
+#ifdef CONFIG_IPIPE_TRACE_ENABLE
+	ipipe_trace_enable = 1;
+#ifdef CONFIG_IPIPE_TRACE_MCOUNT
+	register_ftrace_function(&ipipe_trace_ops);
+#endif /* CONFIG_IPIPE_TRACE_MCOUNT */
+#endif /* CONFIG_IPIPE_TRACE_ENABLE */
+
+	trace_dir = create_proc_entry("trace", S_IFDIR, ipipe_proc_root);
+
+	entry = create_proc_entry("max", 0644, trace_dir);
+	if (entry)
+		entry->proc_fops = &__ipipe_max_prtrace_fops;
+
+	entry = create_proc_entry("frozen", 0644, trace_dir);
+	if (entry)
+		entry->proc_fops = &__ipipe_frozen_prtrace_fops;
+
+	entry = create_proc_entry("trigger", 0644, trace_dir);
+	if (entry) {
+		entry->read_proc = __ipipe_rd_trigger;
+		entry->write_proc = __ipipe_wr_trigger;
+	}
+
+	__ipipe_create_trace_proc_val(trace_dir, "pre_trace_points",
+	                              &pre_trace);
+	__ipipe_create_trace_proc_val(trace_dir, "post_trace_points",
+	                              &post_trace);
+	__ipipe_create_trace_proc_val(trace_dir, "back_trace_points",
+	                              &back_trace);
+	__ipipe_create_trace_proc_val(trace_dir, "verbose",
+	                              &verbose_trace);
+	entry = __ipipe_create_trace_proc_val(trace_dir, "enable",
+					      &ipipe_trace_enable);
+#ifdef CONFIG_IPIPE_TRACE_MCOUNT
+	if (entry)
+		entry->write_proc = __ipipe_wr_enable;
+#endif /* CONFIG_IPIPE_TRACE_MCOUNT */
+}
diff -urN source_powerpc_none/kernel/irq/chip.c source_powerpc_none.ipipe/kernel/irq/chip.c
--- source_powerpc_none/kernel/irq/chip.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/irq/chip.c	2009-12-22 12:44:08.000000000 -0500
@@ -425,7 +425,9 @@
 	irqreturn_t action_ret;
 
 	spin_lock(&desc->lock);
+#ifndef CONFIG_IPIPE
 	mask_ack_irq(desc, irq);
+#endif /* CONFIG_IPIPE */
 
 	if (unlikely(desc->status & IRQ_INPROGRESS))
 		goto out_unlock;
@@ -505,8 +507,13 @@
 
 	spin_lock(&desc->lock);
 	desc->status &= ~IRQ_INPROGRESS;
+#ifdef CONFIG_IPIPE
+	desc->chip->unmask(irq);
+out:
+#else
 out:
 	desc->chip->eoi(irq);
+#endif
 
 	spin_unlock(&desc->lock);
 }
@@ -548,8 +555,10 @@
 	kstat_incr_irqs_this_cpu(irq, desc);
 
 	/* Start handling the irq */
+#ifndef CONFIG_IPIPE
 	if (desc->chip->ack)
 		desc->chip->ack(irq);
+#endif /* CONFIG_IPIPE */
 
 	/* Mark the IRQ currently in progress.*/
 	desc->status |= IRQ_INPROGRESS;
@@ -589,6 +598,85 @@
 	spin_unlock(&desc->lock);
 }
 
+#ifdef CONFIG_IPIPE
+
+void __ipipe_ack_simple_irq(unsigned irq, struct irq_desc *desc)
+{
+}
+
+void __ipipe_end_simple_irq(unsigned irq, struct irq_desc *desc)
+{
+}
+
+void __ipipe_ack_level_irq(unsigned irq, struct irq_desc *desc)
+{
+	mask_ack_irq(desc, irq);
+}
+
+void __ipipe_end_level_irq(unsigned irq, struct irq_desc *desc)
+{
+	if (desc->chip->unmask)
+		desc->chip->unmask(irq);
+}
+
+void __ipipe_ack_fasteoi_irq(unsigned irq, struct irq_desc *desc)
+{
+	desc->chip->eoi(irq);
+}
+
+void __ipipe_end_fasteoi_irq(unsigned irq, struct irq_desc *desc)
+{
+	/*
+	 * Non-requestable IRQs should not be masked in EOI handler.
+	 */
+	if (!(desc->status & IRQ_NOREQUEST))
+		desc->chip->unmask(irq);
+}
+
+void __ipipe_ack_edge_irq(unsigned irq, struct irq_desc *desc)
+{
+	desc->chip->ack(irq);
+}
+
+void __ipipe_ack_percpu_irq(unsigned irq, struct irq_desc *desc)
+{
+	if (desc->chip->ack)
+		desc->chip->ack(irq);
+}
+
+void __ipipe_end_percpu_irq(unsigned irq, struct irq_desc *desc)
+{
+	if (desc->chip->eoi)
+		desc->chip->eoi(irq);
+}
+
+void __ipipe_end_edge_irq(unsigned irq, struct irq_desc *desc)
+{
+}
+
+void __ipipe_ack_bad_irq(unsigned irq, struct irq_desc *desc)
+{
+	static int done;
+
+	handle_bad_irq(irq, desc);
+
+	if (!done) {
+		printk(KERN_WARNING "%s: unknown flow handler for IRQ %d\n",
+		       __FUNCTION__, irq);
+		done = 1;
+	}
+}
+
+void __ipipe_noack_irq(unsigned irq, struct irq_desc *desc)
+{
+}
+
+void __ipipe_noend_irq(unsigned irq, struct irq_desc *desc)
+{
+}
+
+#endif /* CONFIG_IPIPE */
+
 /**
  *	handle_percpu_IRQ - Per CPU local irq handler
  *	@irq:	the interrupt number
@@ -603,8 +691,10 @@
 
 	kstat_incr_irqs_this_cpu(irq, desc);
 
+#ifndef CONFIG_IPIPE
 	if (desc->chip->ack)
 		desc->chip->ack(irq);
+#endif /* CONFIG_IPIPE */
 
 	action_ret = handle_IRQ_event(irq, desc->action);
 	if (!noirqdebug)
@@ -615,20 +705,35 @@
 }
 
 void
-__set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained,
-		  const char *name)
+___set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained,
+		   const char *name)
 {
 	struct irq_desc *desc = irq_to_desc(irq);
-	unsigned long flags;
-
-	if (!desc) {
-		printk(KERN_ERR
-		       "Trying to install type control for IRQ%d\n", irq);
-		return;
-	}
 
 	if (!handle)
 		handle = handle_bad_irq;
+#ifdef CONFIG_IPIPE
+	else if (handle == &handle_simple_irq) {
+		desc->ipipe_ack = &__ipipe_ack_simple_irq;
+		desc->ipipe_end = &__ipipe_end_simple_irq;
+	}
+	else if (handle == &handle_level_irq) {
+		desc->ipipe_ack = &__ipipe_ack_level_irq;
+		desc->ipipe_end = &__ipipe_end_level_irq;
+	}
+	else if (handle == &handle_edge_irq) {
+		desc->ipipe_ack = &__ipipe_ack_edge_irq;
+		desc->ipipe_end = &__ipipe_end_edge_irq;
+	}
+	else if (handle == &handle_fasteoi_irq) {
+		desc->ipipe_ack = &__ipipe_ack_fasteoi_irq;
+		desc->ipipe_end = &__ipipe_end_fasteoi_irq;
+	}
+	else if (handle == &handle_percpu_irq) {
+		desc->ipipe_ack = &__ipipe_ack_percpu_irq;
+		desc->ipipe_end = &__ipipe_end_percpu_irq;
+	}
+#endif /* CONFIG_IPIPE */
 	else if (desc->chip == &no_irq_chip) {
 		printk(KERN_WARNING "Trying to install %sinterrupt handler "
 		       "for IRQ%d\n", is_chained ? "chained " : "", irq);
@@ -640,10 +745,21 @@
 		 * dummy_irq_chip for easy transition.
 		 */
 		desc->chip = &dummy_irq_chip;
+#ifdef CONFIG_IPIPE
+		desc->ipipe_ack = &__ipipe_noack_irq;
+		desc->ipipe_end = &__ipipe_noend_irq;
+#endif /* CONFIG_IPIPE */
 	}
-
-	chip_bus_lock(irq, desc);
-	spin_lock_irqsave(&desc->lock, flags);
+#ifdef CONFIG_IPIPE
+ 	else if (is_chained) {
+ 		desc->ipipe_ack = handle;
+ 		desc->ipipe_end = &__ipipe_noend_irq;
+		handle = &__ipipe_noack_irq;
+ 	} else {
+ 		desc->ipipe_ack = &__ipipe_ack_bad_irq;
+ 		desc->ipipe_end = &__ipipe_noend_irq;
+	}
+#endif /* CONFIG_IPIPE */
 
 	/* Uninstall? */
 	if (handle == handle_bad_irq) {
@@ -651,9 +767,17 @@
 			mask_ack_irq(desc, irq);
 		desc->status |= IRQ_DISABLED;
 		desc->depth = 1;
+#ifdef CONFIG_IPIPE
+		desc->ipipe_ack = &__ipipe_ack_bad_irq;
+		desc->ipipe_end = &__ipipe_noend_irq;
+#endif /* CONFIG_IPIPE */
 	}
 	desc->handle_irq = handle;
 	desc->name = name;
+#ifdef CONFIG_IPIPE
+	/* Suppress intermediate trampoline routine. */
+	ipipe_root_domain->irqs[irq].acknowledge = desc->ipipe_ack;
+#endif /* CONFIG_IPIPE */
 
 	if (handle != handle_bad_irq && is_chained) {
 		desc->status &= ~IRQ_DISABLED;
@@ -661,6 +785,24 @@
 		desc->depth = 0;
 		desc->chip->startup(irq);
 	}
+}
+
+void
+__set_irq_handler(unsigned int irq, irq_flow_handler_t handle, int is_chained,
+		  const char *name)
+{
+ 	struct irq_desc *desc = irq_to_desc(irq);
+	unsigned long flags;
+
+	if (!desc) {
+		printk(KERN_ERR
+		       "Trying to install type control for IRQ%d\n", irq);
+		return;
+	}
+
+	chip_bus_lock(irq, desc);
+	spin_lock_irqsave(&desc->lock, flags);
+	___set_irq_handler(irq, handle, is_chained, name);
 	spin_unlock_irqrestore(&desc->lock, flags);
 	chip_bus_sync_unlock(irq, desc);
 }
diff -urN source_powerpc_none/kernel/irq/handle.c source_powerpc_none.ipipe/kernel/irq/handle.c
--- source_powerpc_none/kernel/irq/handle.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/irq/handle.c	2009-12-22 12:44:08.000000000 -0500
@@ -462,8 +462,10 @@
 		/*
 		 * No locking required for CPU-local interrupts:
 		 */
+#ifndef CONFIG_IPIPE
 		if (desc->chip->ack)
 			desc->chip->ack(irq);
+#endif
 		if (likely(!(desc->status & IRQ_DISABLED))) {
 			action_ret = handle_IRQ_event(irq, desc->action);
 			if (!noirqdebug)
@@ -474,8 +476,10 @@
 	}
 
 	spin_lock(&desc->lock);
+#ifndef CONFIG_IPIPE
 	if (desc->chip->ack)
 		desc->chip->ack(irq);
+#endif
 	/*
 	 * REPLAY is when Linux resends an IRQ that was dropped earlier
 	 * WAITING is used by probe to mark irqs that are being tested
diff -urN source_powerpc_none/kernel/lockdep.c source_powerpc_none.ipipe/kernel/lockdep.c
--- source_powerpc_none/kernel/lockdep.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/lockdep.c	2009-12-22 12:44:08.000000000 -0500
@@ -2318,7 +2318,7 @@
 	/* we'll do an OFF -> ON transition: */
 	curr->hardirqs_enabled = 1;
 
-	if (DEBUG_LOCKS_WARN_ON(!irqs_disabled()))
+	if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw()))
 		return;
 	if (DEBUG_LOCKS_WARN_ON(current->hardirq_context))
 		return;
@@ -2361,7 +2361,7 @@
 	if (unlikely(!debug_locks || current->lockdep_recursion))
 		return;
 
-	if (DEBUG_LOCKS_WARN_ON(!irqs_disabled()))
+	if (DEBUG_LOCKS_WARN_ON(!irqs_disabled() && !irqs_disabled_hw()))
 		return;
 
 	if (curr->hardirqs_enabled) {
diff -urN source_powerpc_none/kernel/panic.c source_powerpc_none.ipipe/kernel/panic.c
--- source_powerpc_none/kernel/panic.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/panic.c	2009-12-22 12:44:08.000000000 -0500
@@ -22,6 +22,7 @@
 #include <linux/init.h>
 #include <linux/nmi.h>
 #include <linux/dmi.h>
+#include <linux/ipipe_trace.h>
 
 int panic_on_oops;
 static unsigned long tainted_mask;
@@ -304,6 +305,8 @@
 {
 	tracing_off();
 	/* can't trust the integrity of the kernel anymore: */
+	ipipe_trace_panic_freeze();
+	ipipe_disable_context_check(ipipe_processor_id());
 	debug_locks_off();
 	do_oops_enter_exit();
 }
diff -urN source_powerpc_none/kernel/power/hibernate.c source_powerpc_none.ipipe/kernel/power/hibernate.c
--- source_powerpc_none/kernel/power/hibernate.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/power/hibernate.c	2009-12-22 12:44:08.000000000 -0500
@@ -238,6 +238,7 @@
 		goto Enable_cpus;
 
 	local_irq_disable();
+	local_irq_disable_hw_cond();
 
 	error = sysdev_suspend(PMSG_FREEZE);
 	if (error) {
@@ -267,6 +268,7 @@
 	 */
 
  Enable_irqs:
+	local_irq_enable_hw_cond();
 	local_irq_enable();
 
  Enable_cpus:
@@ -359,6 +361,7 @@
 		goto Enable_cpus;
 
 	local_irq_disable();
+	local_irq_disable_hw_cond();
 
 	error = sysdev_suspend(PMSG_QUIESCE);
 	if (error)
@@ -390,6 +393,7 @@
 	sysdev_resume();
 
  Enable_irqs:
+	local_irq_enable_hw_cond();
 	local_irq_enable();
 
  Enable_cpus:
diff -urN source_powerpc_none/kernel/printk.c source_powerpc_none.ipipe/kernel/printk.c
--- source_powerpc_none/kernel/printk.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/printk.c	2009-12-22 12:44:08.000000000 -0500
@@ -564,6 +564,41 @@
 	return 0;
 }
 
+#ifdef CONFIG_IPIPE
+
+static ipipe_spinlock_t __ipipe_printk_lock = IPIPE_SPIN_LOCK_UNLOCKED;
+
+static int __ipipe_printk_fill;
+
+static char __ipipe_printk_buf[__LOG_BUF_LEN];
+
+void __ipipe_flush_printk (unsigned virq, void *cookie)
+{
+	char *p = __ipipe_printk_buf;
+	int len, lmax, out = 0;
+	unsigned long flags;
+
+	goto start;
+
+	do {
+		spin_unlock_irqrestore(&__ipipe_printk_lock, flags);
+ start:
+		lmax = __ipipe_printk_fill;
+		while (out < lmax) {
+			len = strlen(p) + 1;
+			printk("%s",p);
+			p += len;
+			out += len;
+		}
+		spin_lock_irqsave(&__ipipe_printk_lock, flags);
+	}
+	while (__ipipe_printk_fill != lmax);
+
+	__ipipe_printk_fill = 0;
+
+	spin_unlock_irqrestore(&__ipipe_printk_lock, flags);
+}
+
 /**
  * printk - print a kernel message
  * @fmt: format string
@@ -588,6 +623,65 @@
 
 asmlinkage int printk(const char *fmt, ...)
 {
+	int r, fbytes, oldcount;
+	unsigned long flags;
+	int sprintk = 1;
+	int cs = -1;
+	va_list args;
+
+	va_start(args, fmt);
+
+	local_irq_save_hw(flags);
+
+	if (test_bit(IPIPE_SPRINTK_FLAG, &__ipipe_current_domain->flags) ||
+	    oops_in_progress)
+		cs = ipipe_disable_context_check(ipipe_processor_id());
+	else if (__ipipe_current_domain == ipipe_root_domain) {
+		struct ipipe_domain *dom;
+
+		list_for_each_entry(dom, &__ipipe_pipeline, p_link) {
+			if (dom == ipipe_root_domain)
+				break;
+			if (test_bit(IPIPE_STALL_FLAG,
+				     &ipipe_cpudom_var(dom, status)))
+				sprintk = 0;
+		}
+	} else
+		sprintk = 0;
+
+	local_irq_restore_hw(flags);
+
+	if (sprintk) {
+		r = vprintk(fmt, args);
+		if (cs != -1)
+			ipipe_restore_context_check(ipipe_processor_id(), cs);
+		goto out;
+	}
+
+	spin_lock_irqsave(&__ipipe_printk_lock, flags);
+
+	oldcount = __ipipe_printk_fill;
+	fbytes = __LOG_BUF_LEN - oldcount;
+
+	if (fbytes > 1)	{
+		r = vscnprintf(__ipipe_printk_buf + __ipipe_printk_fill,
+			       fbytes, fmt, args) + 1; /* account for the null byte */
+		__ipipe_printk_fill += r;
+	} else
+		r = 0;
+
+	spin_unlock_irqrestore(&__ipipe_printk_lock, flags);
+
+	if (oldcount == 0)
+		ipipe_trigger_irq(__ipipe_printk_virq);
+out:
+	va_end(args);
+
+	return r;
+}
+#else /* !CONFIG_IPIPE */
+asmlinkage int printk(const char *fmt, ...)
+{
 	va_list args;
 	int r;
 
@@ -597,6 +691,7 @@
 
 	return r;
 }
+#endif /* CONFIG_IPIPE */
 
 /* cpu currently holding logbuf_lock */
 static volatile unsigned int printk_cpu = UINT_MAX;
diff -urN source_powerpc_none/kernel/sched.c source_powerpc_none.ipipe/kernel/sched.c
--- source_powerpc_none/kernel/sched.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/sched.c	2009-12-22 12:44:08.000000000 -0500
@@ -2351,7 +2351,7 @@
 	smp_wmb();
 	rq = orig_rq = task_rq_lock(p, &flags);
 	update_rq_clock(rq);
-	if (!(p->state & state))
+	if (!(p->state & state) || (p->state & (TASK_NOWAKEUP|TASK_ATOMICSWITCH)))
 		goto out;
 
 	if (p->se.on_rq)
@@ -2825,13 +2825,15 @@
 #endif
 	if (current->set_child_tid)
 		put_user(task_pid_vnr(current), current->set_child_tid);
+
+ 	ipipe_init_notify(current);
 }
 
 /*
  * context_switch - switch to the new MM and the new
  * thread's register state.
  */
-static inline void
+static inline int
 context_switch(struct rq *rq, struct task_struct *prev,
 	       struct task_struct *next)
 {
@@ -2873,12 +2875,23 @@
 	switch_to(prev, next, prev);
 
 	barrier();
+
+#ifdef CONFIG_IPIPE_DELAYED_ATOMICSW
+	current->state &= ~TASK_ATOMICSWITCH;
+#else
+	prev->state &= ~TASK_ATOMICSWITCH;
+#endif
+	if (task_hijacked(prev))
+		return 1;
+
 	/*
 	 * this_rq must be evaluated again because prev may have moved
 	 * CPUs since it called schedule(), thus the 'rq' on its stack
 	 * frame will be invalid.
 	 */
 	finish_task_switch(this_rq(), prev);
+
+	return 0;
 }
 
 /*
@@ -5261,6 +5274,7 @@
 
 void __kprobes add_preempt_count(int val)
 {
+ 	ipipe_check_context(ipipe_root_domain);
 #ifdef CONFIG_DEBUG_PREEMPT
 	/*
 	 * Underflow?
@@ -5283,6 +5297,7 @@
 
 void __kprobes sub_preempt_count(int val)
 {
+ 	ipipe_check_context(ipipe_root_domain);
 #ifdef CONFIG_DEBUG_PREEMPT
 	/*
 	 * Underflow?
@@ -5331,6 +5346,7 @@
  */
 static inline void schedule_debug(struct task_struct *prev)
 {
+	ipipe_check_context(ipipe_root_domain);
 	/*
 	 * Test if we are atomic. Since do_exit() needs to call into
 	 * schedule() atomically, we ignore that path for now.
@@ -5423,6 +5439,9 @@
 	rcu_sched_qs(cpu);
 	prev = rq->curr;
 	switch_count = &prev->nivcsw;
+ 	if (unlikely(prev->state & TASK_ATOMICSWITCH))
+		/* Pop one disable level -- one still remains. */
+		preempt_enable();
 
 	release_kernel_lock(prev);
 need_resched_nonpreemptible:
@@ -5460,15 +5479,18 @@
 		rq->curr = next;
 		++*switch_count;
 
-		context_switch(rq, prev, next); /* unlocks the rq */
+		if (context_switch(rq, prev, next))
+			return; /* task hijacked by higher domain */
 		/*
 		 * the context switch might have flipped the stack from under
 		 * us, hence refresh the local variables.
 		 */
 		cpu = smp_processor_id();
 		rq = cpu_rq(cpu);
-	} else
+	} else {
+		prev->state &= ~TASK_ATOMICSWITCH;
 		spin_unlock_irq(&rq->lock);
+	}
 
 	post_schedule(rq);
 
@@ -6330,6 +6352,7 @@
 
 	oldprio = p->prio;
 	__setscheduler(rq, p, policy, param->sched_priority);
+  	ipipe_setsched_notify(p);
 
 	if (running)
 		p->sched_class->set_curr_task(rq);
@@ -6978,6 +7001,7 @@
 #else
 	task_thread_info(idle)->preempt_count = 0;
 #endif
+	ipipe_check_context(ipipe_root_domain);
 	/*
 	 * The idle tasks have their own, simple scheduling class:
 	 */
@@ -10800,6 +10824,63 @@
 };
 #endif	/* CONFIG_CGROUP_CPUACCT */
 
+#ifdef CONFIG_IPIPE
+
+int ipipe_setscheduler_root (struct task_struct *p, int policy, int prio)
+{
+	const struct sched_class *prev_class = p->sched_class;
+	int oldprio, on_rq, running;
+	unsigned long flags;
+	struct rq *rq;
+
+	spin_lock_irqsave(&p->pi_lock, flags);
+	rq = __task_rq_lock(p);
+	update_rq_clock(rq);
+	on_rq = p->se.on_rq;
+	running = task_running(rq, p);
+
+	if (on_rq)
+		deactivate_task(rq, p, 0);
+	if (running)
+		p->sched_class->put_prev_task(rq, p);
+
+	oldprio = p->prio;
+	__setscheduler(rq, p, policy, prio);
+	ipipe_setsched_notify(p);
+
+	if (running)
+		p->sched_class->set_curr_task(rq);
+	if (on_rq) {
+		activate_task(rq, p, 0);
+		check_class_changed(rq, p, prev_class, oldprio, running);
+	}
+	__task_rq_unlock(rq);
+	spin_unlock_irqrestore(&p->pi_lock, flags);
+
+	rt_mutex_adjust_pi(p);
+
+	return 0;
+}
+
+EXPORT_SYMBOL(ipipe_setscheduler_root);
+
+int ipipe_reenter_root (struct task_struct *prev, int policy, int prio)
+{
+	finish_task_switch(this_rq(), prev);
+
+	(void)reacquire_kernel_lock(current);
+	preempt_enable_no_resched();
+
+	if (current->policy != policy || current->rt_priority != prio)
+		return ipipe_setscheduler_root(current, policy, prio);
+
+	return 0;
+}
+
+EXPORT_SYMBOL(ipipe_reenter_root);
+
+#endif /* CONFIG_IPIPE */
+
 #ifndef CONFIG_SMP
 
 int rcu_expedited_torture_stats(char *page)
diff -urN source_powerpc_none/kernel/signal.c source_powerpc_none.ipipe/kernel/signal.c
--- source_powerpc_none/kernel/signal.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/signal.c	2009-12-22 12:44:08.000000000 -0500
@@ -518,6 +518,7 @@
 	unsigned int mask;
 
 	set_tsk_thread_flag(t, TIF_SIGPENDING);
+	ipipe_sigwake_notify(t); /* TIF_SIGPENDING must be set first. */
 
 	/*
 	 * For SIGKILL, we want to wake it up in the stopped/traced/killable
diff -urN source_powerpc_none/kernel/time/tick-common.c source_powerpc_none.ipipe/kernel/time/tick-common.c
--- source_powerpc_none/kernel/time/tick-common.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/time/tick-common.c	2009-12-22 12:44:08.000000000 -0500
@@ -69,7 +69,7 @@
 		write_sequnlock(&xtime_lock);
 	}
 
-	update_process_times(user_mode(get_irq_regs()));
+	update_root_process_times(get_irq_regs());
 	profile_tick(CPU_PROFILING);
 }
 
@@ -177,6 +177,10 @@
 
 	td->evtdev = newdev;
 
+	/* I-pipe: derive global tick IRQ from CPU 0 */
+	if (cpu == 0)
+		ipipe_update_tick_evtdev(newdev);
+
 	/*
 	 * When the device is not per cpu, pin the interrupt to the
 	 * current cpu:
diff -urN source_powerpc_none/kernel/time/tick-sched.c source_powerpc_none.ipipe/kernel/time/tick-sched.c
--- source_powerpc_none/kernel/time/tick-sched.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/time/tick-sched.c	2009-12-22 12:44:08.000000000 -0500
@@ -525,7 +525,7 @@
 		ts->idle_jiffies++;
 	}
 
-	update_process_times(user_mode(regs));
+	update_root_process_times(regs);
 	profile_tick(CPU_PROFILING);
 
 	while (tick_nohz_reprogram(ts, now)) {
@@ -676,7 +676,7 @@
 			touch_softlockup_watchdog();
 			ts->idle_jiffies++;
 		}
-		update_process_times(user_mode(regs));
+		update_root_process_times(regs);
 		profile_tick(CPU_PROFILING);
 	}
 
diff -urN source_powerpc_none/kernel/timer.c source_powerpc_none.ipipe/kernel/timer.c
--- source_powerpc_none/kernel/timer.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/timer.c	2009-12-22 12:44:08.000000000 -0500
@@ -1204,6 +1204,25 @@
 	run_posix_cpu_timers(p);
 }
 
+#ifdef CONFIG_IPIPE
+
+void update_root_process_times(struct pt_regs *regs)
+{
+	int cpu, user_tick = user_mode(regs);
+
+	if (__ipipe_root_tick_p(regs)) {
+		update_process_times(user_tick);
+		return;
+	}
+
+	run_local_timers();
+	cpu = smp_processor_id();
+	rcu_check_callbacks(cpu, user_tick);
+	run_posix_cpu_timers(current);
+}
+
+#endif
+
 /*
  * This function runs timers and the timer-tq in bottom half context.
  */
diff -urN source_powerpc_none/kernel/trace/ftrace.c source_powerpc_none.ipipe/kernel/trace/ftrace.c
--- source_powerpc_none/kernel/trace/ftrace.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/kernel/trace/ftrace.c	2009-12-22 12:44:08.000000000 -0500
@@ -28,6 +28,7 @@
 #include <linux/ctype.h>
 #include <linux/list.h>
 #include <linux/hash.h>
+#include <linux/ipipe.h>
 
 #include <trace/events/sched.h>
 
@@ -1142,6 +1143,9 @@
 
 static void ftrace_run_update_code(int command)
 {
+#ifdef CONFIG_IPIPE
+	unsigned long flags;
+#endif /* CONFIG_IPIPE */
 	int ret;
 
 	ret = ftrace_arch_code_modify_prepare();
@@ -1149,7 +1153,13 @@
 	if (ret)
 		return;
 
+#ifdef CONFIG_IPIPE
+	flags = ipipe_critical_enter(NULL);
+	__ftrace_modify_code(&command);
+	ipipe_critical_exit(flags);
+#else  /* !CONFIG_IPIPE */
 	stop_machine(__ftrace_modify_code, &command, NULL);
+#endif /* !CONFIG_IPIPE */
 
 	ret = ftrace_arch_code_modify_post_process();
 	FTRACE_WARN_ON(ret);
@@ -2648,9 +2658,9 @@
 	}
 
 	/* disable interrupts to prevent kstop machine */
-	local_irq_save(flags);
+	local_irq_save_hw_notrace(flags);
 	ftrace_update_code(mod);
-	local_irq_restore(flags);
+	local_irq_restore_hw_notrace(flags);
 	mutex_unlock(&ftrace_lock);
 
 	return 0;
@@ -2729,9 +2739,9 @@
 	/* Keep the ftrace pointer to the stub */
 	addr = (unsigned long)ftrace_stub;
 
-	local_irq_save(flags);
+	local_irq_save_hw_notrace(flags);
 	ftrace_dyn_arch_init(&addr);
-	local_irq_restore(flags);
+	local_irq_restore_hw_notrace(flags);
 
 	/* ftrace_dyn_arch_init places the return code in addr */
 	if (addr)
diff -urN source_powerpc_none/lib/Kconfig.debug source_powerpc_none.ipipe/lib/Kconfig.debug
--- source_powerpc_none/lib/Kconfig.debug	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/lib/Kconfig.debug	2009-12-22 12:44:08.000000000 -0500
@@ -136,6 +136,8 @@
 	  - Enable verbose reporting from modpost to help solving
 	    the section mismatches reported.
 
+source "kernel/ipipe/Kconfig.debug"
+
 config DEBUG_KERNEL
 	bool "Kernel debugging"
 	help
diff -urN source_powerpc_none/lib/bust_spinlocks.c source_powerpc_none.ipipe/lib/bust_spinlocks.c
--- source_powerpc_none/lib/bust_spinlocks.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/lib/bust_spinlocks.c	2009-12-22 12:44:08.000000000 -0500
@@ -13,6 +13,7 @@
 #include <linux/wait.h>
 #include <linux/vt_kern.h>
 #include <linux/console.h>
+#include <linux/ipipe_trace.h>
 
 
 void __attribute__((weak)) bust_spinlocks(int yes)
@@ -24,6 +25,7 @@
 		unblank_screen();
 #endif
 		console_unblank();
+  		ipipe_trace_panic_dump();
 		if (--oops_in_progress == 0)
 			wake_up_klogd();
 	}
diff -urN source_powerpc_none/lib/ioremap.c source_powerpc_none.ipipe/lib/ioremap.c
--- source_powerpc_none/lib/ioremap.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/lib/ioremap.c	2009-12-22 12:44:08.000000000 -0500
@@ -85,8 +85,8 @@
 		if (err)
 			break;
 	} while (pgd++, addr = next, addr != end);
-
-	flush_cache_vmap(start, end);
+	__ipipe_pin_range_globally(start, end);
+ 	flush_cache_vmap(start, end);
 
 	return err;
 }
diff -urN source_powerpc_none/lib/smp_processor_id.c source_powerpc_none.ipipe/lib/smp_processor_id.c
--- source_powerpc_none/lib/smp_processor_id.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/lib/smp_processor_id.c	2009-12-22 12:44:08.000000000 -0500
@@ -12,10 +12,13 @@
 	unsigned long preempt_count = preempt_count();
 	int this_cpu = raw_smp_processor_id();
 
+	if (!ipipe_root_domain_p)
+		goto out;
+
 	if (likely(preempt_count))
 		goto out;
 
-	if (irqs_disabled())
+	if (irqs_disabled() || irqs_disabled_hw())
 		goto out;
 
 	/*
diff -urN source_powerpc_none/lib/spinlock_debug.c source_powerpc_none.ipipe/lib/spinlock_debug.c
--- source_powerpc_none/lib/spinlock_debug.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/lib/spinlock_debug.c	2009-12-22 12:44:08.000000000 -0500
@@ -133,6 +133,8 @@
 	debug_spin_lock_after(lock);
 }
 
+EXPORT_SYMBOL(_raw_spin_lock);
+
 int _raw_spin_trylock(spinlock_t *lock)
 {
 	int ret = __raw_spin_trylock(&lock->raw_lock);
@@ -148,12 +150,16 @@
 	return ret;
 }
 
+EXPORT_SYMBOL(_raw_spin_trylock);
+
 void _raw_spin_unlock(spinlock_t *lock)
 {
 	debug_spin_unlock(lock);
 	__raw_spin_unlock(&lock->raw_lock);
 }
 
+EXPORT_SYMBOL(_raw_spin_unlock);
+
 static void rwlock_bug(rwlock_t *lock, const char *msg)
 {
 	if (!debug_locks_off())
@@ -199,6 +205,8 @@
 	__raw_read_lock(&lock->raw_lock);
 }
 
+EXPORT_SYMBOL(_raw_read_lock);
+
 int _raw_read_trylock(rwlock_t *lock)
 {
 	int ret = __raw_read_trylock(&lock->raw_lock);
@@ -212,12 +220,16 @@
 	return ret;
 }
 
+EXPORT_SYMBOL(_raw_read_trylock);
+
 void _raw_read_unlock(rwlock_t *lock)
 {
 	RWLOCK_BUG_ON(lock->magic != RWLOCK_MAGIC, lock, "bad magic");
 	__raw_read_unlock(&lock->raw_lock);
 }
 
+EXPORT_SYMBOL(_raw_read_unlock);
+
 static inline void debug_write_lock_before(rwlock_t *lock)
 {
 	RWLOCK_BUG_ON(lock->magic != RWLOCK_MAGIC, lock, "bad magic");
@@ -275,6 +287,8 @@
 	debug_write_lock_after(lock);
 }
 
+EXPORT_SYMBOL(_raw_write_lock);
+
 int _raw_write_trylock(rwlock_t *lock)
 {
 	int ret = __raw_write_trylock(&lock->raw_lock);
@@ -290,8 +304,12 @@
 	return ret;
 }
 
+EXPORT_SYMBOL(_raw_write_trylock);
+
 void _raw_write_unlock(rwlock_t *lock)
 {
 	debug_write_unlock(lock);
 	__raw_write_unlock(&lock->raw_lock);
 }
+
+EXPORT_SYMBOL(_raw_write_unlock);
diff -urN source_powerpc_none/mm/memory.c source_powerpc_none.ipipe/mm/memory.c
--- source_powerpc_none/mm/memory.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/mm/memory.c	2009-12-22 12:44:08.000000000 -0500
@@ -56,6 +56,7 @@
 #include <linux/kallsyms.h>
 #include <linux/swapops.h>
 #include <linux/elf.h>
+#include <linux/vmalloc.h>
 
 #include <asm/io.h>
 #include <asm/pgalloc.h>
@@ -566,6 +567,32 @@
 	return pfn_to_page(pfn);
 }
 
+static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma)
+{
+	/*
+	 * If the source page was a PFN mapping, we don't have
+	 * a "struct page" for it. We do a best-effort copy by
+	 * just copying from the original user address. If that
+	 * fails, we just zero-fill it. Live with it.
+	 */
+	if (unlikely(!src)) {
+		void *kaddr = kmap_atomic(dst, KM_USER0);
+		void __user *uaddr = (void __user *)(va & PAGE_MASK);
+
+		/*
+		 * This really shouldn't fail, because the page is there
+		 * in the page tables. But it might just be unreadable,
+		 * in which case we just give up and fill the result with
+		 * zeroes.
+		 */
+		if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE))
+			memset(kaddr, 0, PAGE_SIZE);
+		kunmap_atomic(kaddr, KM_USER0);
+		flush_dcache_page(dst);
+	} else
+		copy_user_highpage(dst, src, va, vma);
+}
+
 /*
  * copy one vm_area from one task to the other. Assumes the page tables
  * already present in the new task to be cleared in the whole range
@@ -574,8 +601,8 @@
 
 static inline void
 copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
-		pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma,
-		unsigned long addr, int *rss)
+	     pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma,
+	     unsigned long addr, int *rss, struct page *uncow_page)
 {
 	unsigned long vm_flags = vma->vm_flags;
 	pte_t pte = *src_pte;
@@ -614,6 +641,21 @@
 	 * in the parent and the child
 	 */
 	if (is_cow_mapping(vm_flags)) {
+#ifdef CONFIG_IPIPE
+		if (uncow_page) {
+			struct page *old_page = vm_normal_page(vma, addr, pte);
+			cow_user_page(uncow_page, old_page, addr, vma);
+			pte = mk_pte(uncow_page, vma->vm_page_prot);
+
+			if (vm_flags & VM_SHARED)
+				pte = pte_mkclean(pte);
+			pte = pte_mkold(pte);
+
+			page_add_new_anon_rmap(uncow_page, vma, addr);
+			rss[!!PageAnon(uncow_page)]++;
+			goto out_set_pte;
+		}
+#endif /* CONFIG_IPIPE */
 		ptep_set_wrprotect(src_mm, addr, src_pte);
 		pte = pte_wrprotect(pte);
 	}
@@ -645,13 +687,27 @@
 	pte_t *src_pte, *dst_pte;
 	spinlock_t *src_ptl, *dst_ptl;
 	int progress = 0;
+	struct page *uncow_page = NULL;
 	int rss[2];
-
+#ifdef CONFIG_IPIPE
+	int do_cow_break = 0;
+again:
+	if (do_cow_break) {
+		uncow_page = alloc_page_vma(GFP_HIGHUSER, vma, addr);
+		if (!uncow_page)
+			return -ENOMEM;
+		do_cow_break = 0;
+	}
+#else
 again:
+#endif
 	rss[1] = rss[0] = 0;
 	dst_pte = pte_alloc_map_lock(dst_mm, dst_pmd, addr, &dst_ptl);
-	if (!dst_pte)
+	if (!dst_pte) {
+		if (uncow_page)
+			page_cache_release(uncow_page);
 		return -ENOMEM;
+	}
 	src_pte = pte_offset_map_nested(src_pmd, addr);
 	src_ptl = pte_lockptr(src_mm, src_pmd);
 	spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
@@ -674,7 +730,26 @@
 			progress++;
 			continue;
 		}
-		copy_one_pte(dst_mm, src_mm, dst_pte, src_pte, vma, addr, rss);
+#ifdef CONFIG_IPIPE
+		if (likely(uncow_page == NULL) && likely(pte_present(*src_pte))) {
+			if (is_cow_mapping(vma->vm_flags)) {
+				if (((vma->vm_flags|src_mm->def_flags) & (VM_LOCKED|VM_PINNED))
+				    == (VM_LOCKED|VM_PINNED)) {
+				    	arch_leave_lazy_mmu_mode();
+				    	spin_unlock(src_ptl);
+				    	pte_unmap_nested(src_pte);
+				    	add_mm_rss(dst_mm, rss[0], rss[1]);
+				    	pte_unmap_unlock(dst_pte, dst_ptl);
+				    	cond_resched();
+					do_cow_break = 1;
+					goto again;
+				}
+			}
+		}
+#endif
+		copy_one_pte(dst_mm, src_mm, dst_pte,
+			     src_pte, vma, addr, rss, uncow_page);
+		uncow_page = NULL;
 		progress += 8;
 	} while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end);
 
@@ -1941,32 +2016,6 @@
 	return pte;
 }
 
-static inline void cow_user_page(struct page *dst, struct page *src, unsigned long va, struct vm_area_struct *vma)
-{
-	/*
-	 * If the source page was a PFN mapping, we don't have
-	 * a "struct page" for it. We do a best-effort copy by
-	 * just copying from the original user address. If that
-	 * fails, we just zero-fill it. Live with it.
-	 */
-	if (unlikely(!src)) {
-		void *kaddr = kmap_atomic(dst, KM_USER0);
-		void __user *uaddr = (void __user *)(va & PAGE_MASK);
-
-		/*
-		 * This really shouldn't fail, because the page is there
-		 * in the page tables. But it might just be unreadable,
-		 * in which case we just give up and fill the result with
-		 * zeroes.
-		 */
-		if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE))
-			memset(kaddr, 0, PAGE_SIZE);
-		kunmap_atomic(kaddr, KM_USER0);
-		flush_dcache_page(dst);
-	} else
-		copy_user_highpage(dst, src, va, vma);
-}
-
 /*
  * This routine handles present pages, when users try to write
  * to a shared page. It is done by copying the page to a new address
@@ -3377,3 +3426,111 @@
 }
 EXPORT_SYMBOL(might_fault);
 #endif
+
+#ifdef CONFIG_IPIPE
+
+static inline int ipipe_pin_pte_range(struct mm_struct *mm, pmd_t *pmd,
+				      struct vm_area_struct *vma,
+				      unsigned long addr, unsigned long end)
+{
+	spinlock_t *ptl;
+	pte_t *pte;
+
+	do {
+		pte = pte_offset_map_lock(mm, pmd, addr, &ptl);
+		if (!pte)
+			continue;
+
+		if (!pte_present(*pte) || pte_write(*pte)) {
+			pte_unmap_unlock(pte, ptl);
+			continue;
+		}
+
+		if (do_wp_page(mm, vma, addr, pte, pmd, ptl, *pte) == VM_FAULT_OOM)
+			return -ENOMEM;
+	} while (addr += PAGE_SIZE, addr != end);
+	return 0;
+}
+
+static inline int ipipe_pin_pmd_range(struct mm_struct *mm, pud_t *pud,
+				      struct vm_area_struct *vma,
+				      unsigned long addr, unsigned long end)
+{
+	unsigned long next;
+	pmd_t *pmd;
+
+	pmd = pmd_offset(pud, addr);
+	do {
+		next = pmd_addr_end(addr, end);
+		if (pmd_none_or_clear_bad(pmd))
+			continue;
+		if (ipipe_pin_pte_range(mm, pmd, vma, addr, next))
+			return -ENOMEM;
+	} while (pmd++, addr = next, addr != end);
+	return 0;
+}
+
+static inline int ipipe_pin_pud_range(struct mm_struct *mm, pgd_t *pgd,
+				      struct vm_area_struct *vma,
+				      unsigned long addr, unsigned long end)
+{
+	unsigned long next;
+	pud_t *pud;
+
+	pud = pud_offset(pgd, addr);
+	do {
+		next = pud_addr_end(addr, end);
+		if (pud_none_or_clear_bad(pud))
+			continue;
+		if (ipipe_pin_pmd_range(mm, pud, vma, addr, next))
+			return -ENOMEM;
+	} while (pud++, addr = next, addr != end);
+	return 0;
+}
+
+int ipipe_disable_ondemand_mappings(struct task_struct *tsk)
+{
+	unsigned long addr, next, end;
+	struct vm_area_struct *vma;
+	struct mm_struct *mm;
+	int result = 0;
+	pgd_t *pgd;
+
+	mm = get_task_mm(tsk);
+	if (!mm)
+		return -EPERM;
+
+	down_write(&mm->mmap_sem);
+	if (mm->def_flags & VM_PINNED)
+		goto done_mm;
+
+	for (vma = mm->mmap; vma; vma = vma->vm_next) {
+		if (!is_cow_mapping(vma->vm_flags)
+		    || !(vma->vm_flags & VM_WRITE))
+			continue;
+
+		addr = vma->vm_start;
+		end = vma->vm_end;
+
+		pgd = pgd_offset(mm, addr);
+		do {
+			next = pgd_addr_end(addr, end);
+			if (pgd_none_or_clear_bad(pgd))
+				continue;
+			if (ipipe_pin_pud_range(mm, pgd, vma, addr, next)) {
+				result = -ENOMEM;
+				goto done_mm;
+			}
+		} while (pgd++, addr = next, addr != end);
+	}
+	mm->def_flags |= VM_PINNED;
+
+  done_mm:
+	up_write(&mm->mmap_sem);
+	mmput(mm);
+	return result;
+}
+
+EXPORT_SYMBOL(ipipe_disable_ondemand_mappings);
+
+#endif
diff -urN source_powerpc_none/mm/mlock.c source_powerpc_none.ipipe/mm/mlock.c
--- source_powerpc_none/mm/mlock.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/mm/mlock.c	2009-12-22 12:44:08.000000000 -0500
@@ -515,10 +515,10 @@
 static int do_mlockall(int flags)
 {
 	struct vm_area_struct * vma, * prev = NULL;
-	unsigned int def_flags = 0;
+	unsigned int def_flags = current->mm->def_flags & VM_PINNED;
 
 	if (flags & MCL_FUTURE)
-		def_flags = VM_LOCKED;
+		def_flags |= VM_LOCKED;
 	current->mm->def_flags = def_flags;
 	if (flags == MCL_FUTURE)
 		goto out;
diff -urN source_powerpc_none/mm/mmu_context.c source_powerpc_none.ipipe/mm/mmu_context.c
--- source_powerpc_none/mm/mmu_context.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/mm/mmu_context.c	2009-12-22 12:44:08.000000000 -0500
@@ -23,6 +23,7 @@
 {
 	struct mm_struct *active_mm;
 	struct task_struct *tsk = current;
+	unsigned long flags;
 
 	task_lock(tsk);
 	active_mm = tsk->active_mm;
@@ -31,7 +32,9 @@
 		tsk->active_mm = mm;
 	}
 	tsk->mm = mm;
-	switch_mm(active_mm, mm, tsk);
+	ipipe_mm_switch_protect(flags);
+	__switch_mm(active_mm, mm, tsk);
+	ipipe_mm_switch_unprotect(flags);
 	task_unlock(tsk);
 
 	if (active_mm != mm)
diff -urN source_powerpc_none/mm/vmalloc.c source_powerpc_none.ipipe/mm/vmalloc.c
--- source_powerpc_none/mm/vmalloc.c	2009-12-02 22:51:21.000000000 -0500
+++ source_powerpc_none.ipipe/mm/vmalloc.c	2009-12-22 12:44:08.000000000 -0500
@@ -172,6 +172,8 @@
 			return err;
 	} while (pgd++, addr = next, addr != end);
 
+ 	__ipipe_pin_range_globally(start, end);
+
 	return nr;
 }
 

-- 
Len Sorensen

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Thu, 2010-01-14 at 08:53 +0100, Richard Cochran wrote: 
> On Tue, Jan 12, 2010 at 05:18:02PM -0500, Lennart Sorensen wrote:
> > Well I have not been able to find the magic invocation that lets me take
> > the DENX tree (which I have had around for a long time just to look at
> > occationally, whenever I was trying to get an ipipe patch to apply),
> > apply the ipipe patch, revert the DENX changes to get back to a release
> > kernel, and generate a diff of the ipipe changes.  It has never worked
> > when I tried.
> 
> Lennart,
> 
> It is not so hard (using git) to remove the Denx patches from the
> ipipe tree. I did this myself for 2.6.30 in about a half an hour. If
> you don't know how to use git, then you would have to consider the
> additional time you need to get to understand it. (For me, it was only
> a year or so ;)
> 
> In the ipipe tree, the Denx commits have been "squashed" together into
> one or two really large commits. So, you can just cherry pick the
> adeos commits into a new branch, with a few minor fixups.
> 
> Philippe,
> 
> I actually agree with Lennart that the Denx stuff is an
> annoyance. When considering my "no-denx" branch that I made, I could
> not see any significant Denx change that adeos builds upon. There were
> a few Denx fixes for one specific board that were close to the adeos
> changes, but these were only a few, and easy to fix. So, I could not
> understand why Denx is a prerequisite for adeos.
> 

As I explained earlier in my reply to Lennart, the DENX tree is not a
pre-requisite for having the pipeline run on each and every hw platform,
but this is still the case for some, because they are not
stable/complete/good enough in mainline yet. The reason to stick with it
stems from this fact.

What has to be reassessed, is the number of platforms Xenomai supports
that still need DENX bits today; if only a few of them remain in this
category, then it's probably sound to start maintaining the pipeline
support for them in a separate tree, rebasing I-pipe mainline over Linux
mainline for ppc as well. I have no issue with that.

> I understand that Denx sponsored the original PowerPC Xenomai port. Is
> the reason that ipipe is based on Denx simply to honor that fact? If
> so, I would not think it a bad reason at all.
> 

No, it's not, because if I had chosen to base the pipeline code for ppc
over mainline at that time, I would have maintained a DENX-based version
for the very reason you mentioned, until it proves useless.

> However, I would still prefer the following ordering for the changes:
> 
> 1. stable linux (2.6.xx.y)
> 2. adeos arch indepedendent
> 3. adeos powerpc
> 4. denx
> 5. adeos for denx (minimal changes, I expect)
> 

What we may be aiming at, if workable, is something like:

ipipe-*-mainline
ipipe-*-amcc
ipipe-*-512x

Maybe one for the PA6T as well, if we want to keep supporting the old A2
board rev. I'm unsure right now, since B0 is fine in mainline already.

A mainline pipeline branch for everything that directly works over
mainline, and platform-specific branches for those that do not. Those
special branches would then disappear as soon as mainline is fine for
the platforms they host as well.

> Richard
> 
> _______________________________________________
> Adeos-main mailing list
> Adeos-main@domain.hid
> https://mail.gna.org/listinfo/adeos-main


-- 
Philippe.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Wolfgang Denk]

Dear Philippe,

in message <1263765606.2428.908.camel@domain.hid> you wrote:
>
> What we may be aiming at, if workable, is something like:
> 
> ipipe-*-mainline
> ipipe-*-amcc
> ipipe-*-512x

This sounds fine to me.

I guess we don't even need a 'ipipe-*-amcc' branch - the only things
that should be in our tree but not in mainline are the Synopsis USB
and S-ATA drivers (which are in such a state that they have no chance
of being accepted for mainline), and these additional features are
probably not of relevance for use with Xenomai.

ACK for 'ipipe-*-512x', but this is a work-in-progress tree, with the
clear intention to push this stuff into mainline ASAP.

> Maybe one for the PA6T as well, if we want to keep supporting the old A2
> board rev. I'm unsure right now, since B0 is fine in mainline already.

Are there any significant differences between A2 and Bx as far as
Xenomai in concerned? I mean, would a mainline patch (for Bx) be
missing anything so it doesn't run on A2?

> A mainline pipeline branch for everything that directly works over
> mainline, and platform-specific branches for those that do not. Those
> special branches would then disappear as soon as mainline is fine for
> the platforms they host as well.

I like this approach :-)

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@domain.hid
"We don't care.  We don't have to.  We're the Phone Company."

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Sun, 2010-01-17 at 23:51 +0100, Wolfgang Denk wrote:
> Dear Philippe,
> 
> in message <1263765606.2428.908.camel@domain.hid> you wrote:
> >
> > What we may be aiming at, if workable, is something like:
> > 
> > ipipe-*-mainline
> > ipipe-*-amcc
> > ipipe-*-512x
> 
> This sounds fine to me.
> 
> I guess we don't even need a 'ipipe-*-amcc' branch - the only things
> that should be in our tree but not in mainline are the Synopsis USB
> and S-ATA drivers (which are in such a state that they have no chance
> of being accepted for mainline), and these additional features are
> probably not of relevance for use with Xenomai.
> 

The 8250 serial support put aside, I don't think we have any hunks in
non-core Linux code, so we likely don't need this branch indeed.

> ACK for 'ipipe-*-512x', but this is a work-in-progress tree, with the
> clear intention to push this stuff into mainline ASAP.
> 

Ok.

> > Maybe one for the PA6T as well, if we want to keep supporting the old A2
> > board rev. I'm unsure right now, since B0 is fine in mainline already.
> 
> Are there any significant differences between A2 and Bx as far as
> Xenomai in concerned? I mean, would a mainline patch (for Bx) be
> missing anything so it doesn't run on A2?

Basically the work-arounds to fix the A2 errata; some of them have to be
adapted to run in any context from the pipeline code (some bits in the
process switching and tlbie handling code come to mind). This said, this
could be solved by a supplemental patch moving those work-arounds on top
of a mainline-based support.

> 
> > A mainline pipeline branch for everything that directly works over
> > mainline, and platform-specific branches for those that do not. Those
> > special branches would then disappear as soon as mainline is fine for
> > the platforms they host as well.
> 
> I like this approach :-)
> 

Ok, thanks. ETA is in the 2.6.33 time frame.

> Best regards,
> 
> Wolfgang Denk
> 


-- 
Philippe.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Mon, 2010-01-18 at 12:51 -0500, Lennart Sorensen wrote:
> On Sun, Jan 17, 2010 at 11:00:06PM +0100, Philippe Gerum wrote:
> > As I explained earlier in my reply to Lennart, the DENX tree is not a
> > pre-requisite for having the pipeline run on each and every hw platform,
> > but this is still the case for some, because they are not
> > stable/complete/good enough in mainline yet. The reason to stick with it
> > stems from this fact.
> > 
> > What has to be reassessed, is the number of platforms Xenomai supports
> > that still need DENX bits today; if only a few of them remain in this
> > category, then it's probably sound to start maintaining the pipeline
> > support for them in a separate tree, rebasing I-pipe mainline over Linux
> > mainline for ppc as well. I have no issue with that.
> 
> Sounds great to me.
> 
> > What we may be aiming at, if workable, is something like:
> > 
> > ipipe-*-mainline
> > ipipe-*-amcc
> > ipipe-*-512x
> > 
> > Maybe one for the PA6T as well, if we want to keep supporting the old A2
> > board rev. I'm unsure right now, since B0 is fine in mainline already.
> > 
> > A mainline pipeline branch for everything that directly works over
> > mainline, and platform-specific branches for those that do not. Those
> > special branches would then disappear as soon as mainline is fine for
> > the platforms they host as well.
> 
> I also noticed on Friday that applying both x86 and powerpc ipipe
> patches to one kernel tree is a bit problematic.  Obviously both
> include the noarch section, but dealing with that is simple.
> The only actual problem that had to be solved is that both xenomai
> architectures try to include arch/<archname>/xenomai/Kconfig which
> conflict with each other.  Merging the x86 and powerpc Kconfig bits
> together in one file isn't hard and allows both to work with a single
> included file.
> 
> The reason I want this is that I very much intend to have a single 2.6.32
> kernel source build with the same patches on both our x86 based embedded
> systems and our powerpc based embedded systems.
> 
> Is there any interest in getting such cleanup done to make the code
> just apply all together?  If so I will try to finish adding the other
> architectures together too.
> 

If you can make this work cleanly, while keeping the ability to separate
arch-dep sections in different patches, I'm a taker. Even if having
separate per-architecture patches is better for decoupling their release
cycle (issuing a jumbo patch each time minor updates appear only in a
single arch-dep section is not suitable), we could also provide an
integrated patch by combining all per-arch bits and a single no-arch
section.


-- 
Philippe.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Richard Cochran]

Philippe,

After updating my ipipe git repo, I notice that you have done
something new with ipipe-2.6.32-powerpc, back in January.  Previously,
the Denx bits appeared as "squashed" commits. Now, the Denx commits
appear individually.

I am curious, what's up with that?

Thanks,
Richard

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Tue, 2010-02-23 at 08:53 +0100, Richard Cochran wrote:
> Philippe,
> 
> After updating my ipipe git repo, I notice that you have done
> something new with ipipe-2.6.32-powerpc, back in January.  Previously,
> the Denx bits appeared as "squashed" commits. Now, the Denx commits
> appear individually.
> 
> I am curious, what's up with that?
> 

I wanted to rebase on the official DENX stable branch, since my work was
previously based on a dev snapshot, and I did this by fetching+merging
that branch into mine, based on v2.6.32.2; in that case, the changes
appear as individual commits I guess. I did not want to mess with jumbo
patches and the error-prone manual operations they entail.

With ipipe-2.6.33 solely tracking mainline, only the few temporary trees
with not-mainlined-yet features/support will pull from the DENX tree to
get them.

> Thanks,
> Richard
> 
> _______________________________________________
> Adeos-main mailing list
> Adeos-main@domain.hid
> https://mail.gna.org/listinfo/adeos-main


-- 
Philippe.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Richard Cochran]

On Tue, Feb 23, 2010 at 11:07:57AM +0100, Philippe Gerum wrote:
> With ipipe-2.6.33 solely tracking mainline, only the few temporary trees
> with not-mainlined-yet features/support will pull from the DENX tree to
> get them.

Okay, its now clear to me. Thanks for the explanation.

Richard

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Philippe Gerum]

On Tue, 2010-02-23 at 11:57 +0100, Richard Cochran wrote:
> On Tue, Feb 23, 2010 at 11:07:57AM +0100, Philippe Gerum wrote:
> > With ipipe-2.6.33 solely tracking mainline, only the few temporary trees
> > with not-mainlined-yet features/support will pull from the DENX tree to
> > get them.
> 
> Okay, its now clear to me. Thanks for the explanation.

You are welcome. Out of curiosity, what was unclear exactly?

> 
> Richard
> 
> _______________________________________________
> Adeos-main mailing list
> Adeos-main@domain.hid
> https://mail.gna.org/listinfo/adeos-main


-- 
Philippe.

Re: [Xenomai-core] [Adeos-main] I-pipe for 2.6.32 PPC

[Richard Cochran]

On Thu, Feb 25, 2010 at 05:48:08PM +0100, Philippe Gerum wrote:
> 
> You are welcome. Out of curiosity, what was unclear exactly?

We just had a nice discussion regarding powerpc support in
ipipe/mainline/denx. Now it is clear to me, what your plan is!

Thanks for your efforts,

Richard