* Re: [PATCH 1/4] of/device: Replace of_device with platform_device in includes and core code
From: David Miller @ 2010-07-23 5:50 UTC (permalink / raw)
To: grant.likely
Cc: sfr, monstr, microblaze-uclinux, devicetree-discuss, linux-kernel,
linuxppc-dev, sparclinux
In-Reply-To: <20100722223006.21557.84275.stgit@angua>
From: Grant Likely <grant.likely@secretlab.ca>
Date: Thu, 22 Jul 2010 16:30:06 -0600
> of_device is currently just an #define alias to platform_device until it
> gets removed entirely. This patch removes references to it from the
> include directories and the core drivers/of code.
>
> Signed-off-by: Grant Likely <grant.likely@secretlab.ca>
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply
* Re: [PATCH][RFC] preempt_count corruption across H_CEDE call with CONFIG_PREEMPT on pseries
From: Benjamin Herrenschmidt @ 2010-07-23 5:11 UTC (permalink / raw)
To: svaidy
Cc: Stephen Rothwell, Darren Hart, Gautham R Shenoy, Steven Rostedt,
linuxppc-dev, Will Schmidt, Paul Mackerras, Thomas Gleixner
In-Reply-To: <20100723050814.GA26162@dirshya.in.ibm.com>
On Fri, 2010-07-23 at 10:38 +0530, Vaidyanathan Srinivasan wrote:
> Yes. extended_cede_processor() will return with interrupts enabled in
> the cpu. (This is done by the hypervisor). Under normal cases we
> cannot be interrupted because no IO interrupts are routed to us after
> xics_teardown_cpu() and since the CPU is out of the map, nobody will
> send us IPIs.
What about decrementer ?
> Though H_CEDE will return with interrupts enabled, it is unlikely that
> an interrupt can be delivered in this context.
Well, if interrupts are soft-disabled, even if one occurs, we will just
mask and return, so that at least should be ok.
Cheers,
Ben.
^ permalink raw reply
* Re: [PATCH][RFC] preempt_count corruption across H_CEDE call with CONFIG_PREEMPT on pseries
From: Benjamin Herrenschmidt @ 2010-07-23 5:09 UTC (permalink / raw)
To: Darren Hart
Cc: Stephen Rothwell, Gautham R Shenoy, Steven Rostedt, linuxppc-dev,
Will Schmidt, Paul Mackerras, Thomas Gleixner
In-Reply-To: <4C491E14.9010100@us.ibm.com>
On Thu, 2010-07-22 at 21:44 -0700, Darren Hart wrote:
> suggestion I updated the instrumentation to display the
> local_save_flags and irqs_disabled_flags:
> Jul 22 23:36:58 igoort1 kernel: local flags: 0, irqs disabled: 1
> Jul 22 23:36:58 igoort1 kernel: before H_CEDE current->stack: c00000010e9e3ce0, pcnt: 1
> Jul 22 23:36:58 igoort1 kernel: after H_CEDE current->stack: c00000010e9e3ce0, pcnt: 1
>
> I'm not sure if I'm reading that right, but I believe interrupts are
> intended to be disabled here. If accomplished via the
> spin_lock_irqsave() this would behave differently on RT. However, this
> path disables the interrupts handled by xics, all but the IPIs anyway.
> On RT I disabled the decrementer as well.
>
> Is it possible for RT to be receiving other interrupts here?
Also you may want to call hard_irq_disable() to -really- disable
interrupts ... since we do lazy-disable on powerpc
Cheers,
Ben.
^ permalink raw reply
* Re: [PATCH][RFC] preempt_count corruption across H_CEDE call with CONFIG_PREEMPT on pseries
From: Vaidyanathan Srinivasan @ 2010-07-23 5:08 UTC (permalink / raw)
To: Darren Hart
Cc: Stephen Rothwell, Gautham R Shenoy, Steven Rostedt, linuxppc-dev,
Will Schmidt, Paul Mackerras, Thomas Gleixner
In-Reply-To: <4C491E14.9010100@us.ibm.com>
* Darren Hart <dvhltc@us.ibm.com> [2010-07-22 21:44:04]:
> On 07/22/2010 04:57 PM, Darren Hart wrote:
> > On 07/22/2010 03:25 PM, Benjamin Herrenschmidt wrote:
> >> On Thu, 2010-07-22 at 11:24 -0700, Darren Hart wrote:
> >>>
> >>> 1) How can the preempt_count() get mangled across the H_CEDE hcall?
> >>> 2) Should we call preempt_enable() in cpu_idle() prior to cpu_die() ?
> >>
> >> The preempt count is on the thread info at the bottom of the stack.
> >>
> >> Can you check the stack pointers ?
> >
> > Hi Ben, thanks for looking.
> >
> > I instrumented the area around extended_cede_processor() as follows
> > (please confirm I'm getting the stack pointer correctly).
> >
> > while (get_preferred_offline_state(cpu) == CPU_STATE_INACTIVE) {
> > asm("mr %0,1" : "=r" (sp));
> > printk("before H_CEDE current->stack: %lx, pcnt: %x\n", sp, preempt_count());
> > extended_cede_processor(cede_latency_hint);
> > asm("mr %0,1" : "=r" (sp));
> > printk("after H_CEDE current->stack: %lx, pcnt: %x\n", sp, preempt_count());
> > }
> >
> >
> > On Mainline (2.6.33.6, CONFIG_PREEMPT=y) I see this:
> > Jul 22 18:37:08 igoort1 kernel: before H_CEDE current->stack: c00000010e9e3ce0, pcnt: 1
> > Jul 22 18:37:08 igoort1 kernel: after H_CEDE current->stack: c00000010e9e3ce0, pcnt: 1
> >
> > This surprised me as preempt_count is 1 before and after, so no
> > corruption appears to occur on mainline. This makes the pcnt of 65 I see
> > without the preempt_count()=0 hack very strange. I ran several hundred
> > off/on cycles. The issue of preempt_count being 1 is still addressed by
> > this patch however.
> >
> > On PREEMPT_RT (2.6.33.5-rt23 - tglx, sorry, rt/2.6.33 next time, promise):
> > Jul 22 18:51:11 igoort1 kernel: before H_CEDE current->stack: c000000089bcfcf0, pcnt: 1
> > Jul 22 18:51:11 igoort1 kernel: after H_CEDE current->stack: c000000089bcfcf0, pcnt: ffffffff
> >
> > In both cases the stack pointer appears unchanged.
> >
> > Note: there is a BUG triggered in between these statements as the
> > preempt_count causes the printk to trigger:
> > Badness at kernel/sched.c:5572
>
> At Steven's suggestion I updated the instrumentation to display the
> local_save_flags and irqs_disabled_flags:
>
> while (get_preferred_offline_state(cpu) == CPU_STATE_INACTIVE) {
> local_save_flags(flags);
> printk("local flags: %lx, irqs disabled: %d\n", flags, irqs_disabled_flags(flags));
> asm("mr %0,1" : "=r" (sp));
> printk("before H_CEDE current->stack: %lx, pcnt: %x\n", sp, preempt_count());
> extended_cede_processor(cede_latency_hint);
> asm("mr %0,1" : "=r" (sp));
> printk("after H_CEDE current->stack: %lx, pcnt: %x\n", sp, preempt_count());
> }
>
>
> Jul 22 23:36:58 igoort1 kernel: local flags: 0, irqs disabled: 1
> Jul 22 23:36:58 igoort1 kernel: before H_CEDE current->stack: c00000010e9e3ce0, pcnt: 1
> Jul 22 23:36:58 igoort1 kernel: after H_CEDE current->stack: c00000010e9e3ce0, pcnt: 1
>
> I'm not sure if I'm reading that right, but I believe interrupts are
> intended to be disabled here. If accomplished via the
> spin_lock_irqsave() this would behave differently on RT. However, this
> path disables the interrupts handled by xics, all but the IPIs anyway.
> On RT I disabled the decrementer as well.
>
> Is it possible for RT to be receiving other interrupts here?
Yes. extended_cede_processor() will return with interrupts enabled in
the cpu. (This is done by the hypervisor). Under normal cases we
cannot be interrupted because no IO interrupts are routed to us after
xics_teardown_cpu() and since the CPU is out of the map, nobody will
send us IPIs.
Though H_CEDE will return with interrupts enabled, it is unlikely that
an interrupt can be delivered in this context.
--Vaidy
^ permalink raw reply
* Re: [PATCH][RFC] preempt_count corruption across H_CEDE call with CONFIG_PREEMPT on pseries
From: Darren Hart @ 2010-07-23 4:44 UTC (permalink / raw)
To: Benjamin Herrenschmidt
Cc: Stephen Rothwell, Gautham R Shenoy, Steven Rostedt, linuxppc-dev,
Will Schmidt, Paul Mackerras, Thomas Gleixner
In-Reply-To: <4C48DADE.1050409@us.ibm.com>
On 07/22/2010 04:57 PM, Darren Hart wrote:
> On 07/22/2010 03:25 PM, Benjamin Herrenschmidt wrote:
>> On Thu, 2010-07-22 at 11:24 -0700, Darren Hart wrote:
>>>
>>> 1) How can the preempt_count() get mangled across the H_CEDE hcall?
>>> 2) Should we call preempt_enable() in cpu_idle() prior to cpu_die() ?
>>
>> The preempt count is on the thread info at the bottom of the stack.
>>
>> Can you check the stack pointers ?
>
> Hi Ben, thanks for looking.
>
> I instrumented the area around extended_cede_processor() as follows
> (please confirm I'm getting the stack pointer correctly).
>
> while (get_preferred_offline_state(cpu) == CPU_STATE_INACTIVE) {
> asm("mr %0,1" : "=r" (sp));
> printk("before H_CEDE current->stack: %lx, pcnt: %x\n", sp, preempt_count());
> extended_cede_processor(cede_latency_hint);
> asm("mr %0,1" : "=r" (sp));
> printk("after H_CEDE current->stack: %lx, pcnt: %x\n", sp, preempt_count());
> }
>
>
> On Mainline (2.6.33.6, CONFIG_PREEMPT=y) I see this:
> Jul 22 18:37:08 igoort1 kernel: before H_CEDE current->stack: c00000010e9e3ce0, pcnt: 1
> Jul 22 18:37:08 igoort1 kernel: after H_CEDE current->stack: c00000010e9e3ce0, pcnt: 1
>
> This surprised me as preempt_count is 1 before and after, so no
> corruption appears to occur on mainline. This makes the pcnt of 65 I see
> without the preempt_count()=0 hack very strange. I ran several hundred
> off/on cycles. The issue of preempt_count being 1 is still addressed by
> this patch however.
>
> On PREEMPT_RT (2.6.33.5-rt23 - tglx, sorry, rt/2.6.33 next time, promise):
> Jul 22 18:51:11 igoort1 kernel: before H_CEDE current->stack: c000000089bcfcf0, pcnt: 1
> Jul 22 18:51:11 igoort1 kernel: after H_CEDE current->stack: c000000089bcfcf0, pcnt: ffffffff
>
> In both cases the stack pointer appears unchanged.
>
> Note: there is a BUG triggered in between these statements as the
> preempt_count causes the printk to trigger:
> Badness at kernel/sched.c:5572
At Steven's suggestion I updated the instrumentation to display the
local_save_flags and irqs_disabled_flags:
while (get_preferred_offline_state(cpu) == CPU_STATE_INACTIVE) {
local_save_flags(flags);
printk("local flags: %lx, irqs disabled: %d\n", flags, irqs_disabled_flags(flags));
asm("mr %0,1" : "=r" (sp));
printk("before H_CEDE current->stack: %lx, pcnt: %x\n", sp, preempt_count());
extended_cede_processor(cede_latency_hint);
asm("mr %0,1" : "=r" (sp));
printk("after H_CEDE current->stack: %lx, pcnt: %x\n", sp, preempt_count());
}
Jul 22 23:36:58 igoort1 kernel: local flags: 0, irqs disabled: 1
Jul 22 23:36:58 igoort1 kernel: before H_CEDE current->stack: c00000010e9e3ce0, pcnt: 1
Jul 22 23:36:58 igoort1 kernel: after H_CEDE current->stack: c00000010e9e3ce0, pcnt: 1
I'm not sure if I'm reading that right, but I believe interrupts are
intended to be disabled here. If accomplished via the
spin_lock_irqsave() this would behave differently on RT. However, this
path disables the interrupts handled by xics, all but the IPIs anyway.
On RT I disabled the decrementer as well.
Is it possible for RT to be receiving other interrupts here?
--
Darren Hart
IBM Linux Technology Center
Real-Time Linux Team
^ permalink raw reply
* RE: [PATCH] RapidIO,powerpc/85xx: remove MCSR_MASK in fsl_rio
From: Li Yang-R58472 @ 2010-07-23 3:59 UTC (permalink / raw)
To: Kumar Gala, Alexandre Bounine, akpm, linux-kernel, linuxppc-dev
In-Reply-To: <1279730810-24926-1-git-send-email-alexandre.bounine@idt.com>
>Subject: [PATCH] RapidIO,powerpc/85xx: remove MCSR_MASK in fsl_rio
>
>Fixes compile problem caused by MCSR_MASK removal from book-E =
definitions.
Hi Alex,
Only with your patch, there will still be problem on SRIO platforms =
other than MPC85xx.
I have posted a patch series to fix this together with several =
compatibility issues a month before.
http://patchwork.ozlabs.org/patch/56135/
http://patchwork.ozlabs.org/patch/56136/
http://patchwork.ozlabs.org/patch/56138/
http://patchwork.ozlabs.org/patch/56137/
Can anyone pick the patch series quickly as currently there is a compile =
error when SRIO is enabled.
- Leo
>
>Signed-off-by: Alexandre Bounine <alexandre.bounine@idt.com>
>Cc: Kumar Gala <galak@kernel.crashing.org>
>Cc: Grant Likely <grant.likely@secretlab.ca>
>Cc: Matt Porter <mporter@kernel.crashing.org>
>---
> arch/powerpc/sysdev/fsl_rio.c | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
>diff --git a/arch/powerpc/sysdev/fsl_rio.c =
b/arch/powerpc/sysdev/fsl_rio.c
>index 30e1626..c58df58 100644
>--- a/arch/powerpc/sysdev/fsl_rio.c
>+++ b/arch/powerpc/sysdev/fsl_rio.c
>@@ -245,7 +245,7 @@ static int (*saved_mcheck_exception)(struct pt_regs
>*regs); static int fsl_rio_mcheck_exception(struct pt_regs *regs) {
> const struct exception_table_entry *entry =3D NULL;
>- unsigned long reason =3D (mfspr(SPRN_MCSR) & MCSR_MASK);
>+ unsigned long reason =3D mfspr(SPRN_MCSR);
>
> if (reason & MCSR_BUS_RBERR) {
> reason =3D in_be32((u32 *)(rio_regs_win + RIO_LTLEDCSR));
^ permalink raw reply
* [git pull] Please pull powerpc.git merge branch
From: Benjamin Herrenschmidt @ 2010-07-23 4:14 UTC (permalink / raw)
To: Linus Torvalds
Cc: linuxppc-dev list, Andrew Morton, Russell King, Linux Kernel list
Hi Linus !
Here's a few things for 2.6.35.
One's a fallover from the lmb->memblock rename that I missed, which causes
a bunch of pseries machine to boot with 128M of memory and that's it :-)
There's a few patches fixing a nasty race we found in our hugepage code,
a kexec fix from Kumar, and an .lds fix from Sam for a bug that as far
as we know only affects powerpc.
Cheers,
Ben.
The following changes since commit b37fa16e78d6f9790462b3181602a26b5af36260:
Linus Torvalds (1):
Linux 2.6.35-rc6
are available in the git repository at:
git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc.git merge
Anton Blanchard (1):
powerpc/mm: Handle hypervisor pte insert failure in __hash_page_huge
Benjamin Herrenschmidt (5):
powerpc/mm: Move around testing of _PAGE_PRESENT in hash code
powerpc/mm: Fix bugs in huge page hashing
powerpc/mm: Add some debug output when hash insertion fails
powerpc: Fix erroneous lmb->memblock conversions
Merge commit 'kumar/merge' into merge
Kumar Gala (1):
powerpc/kexec: Fix boundary case for book-e kexec memory limits
Sam Ravnborg (1):
vmlinux.lds: fix .data..init_task output section (fix popwerpc boot)
arch/powerpc/include/asm/kexec.h | 6 +-
arch/powerpc/include/asm/mmu-hash64.h | 4 +-
arch/powerpc/kernel/prom.c | 2 +-
arch/powerpc/mm/hash_low_64.S | 9 ----
arch/powerpc/mm/hash_utils_64.c | 53 +++++++++++++++++------
arch/powerpc/mm/hugetlbpage-hash64.c | 40 +++++++++--------
arch/powerpc/mm/numa.c | 24 +++++-----
arch/powerpc/platforms/pseries/hotplug-memory.c | 22 +++++-----
include/asm-generic/vmlinux.lds.h | 2 +-
9 files changed, 93 insertions(+), 69 deletions(-)
^ permalink raw reply
* Re: [PATCH] powerpc: ONLINE to OFFLINE CPU state transition during removal
From: Vaidyanathan Srinivasan @ 2010-07-23 4:13 UTC (permalink / raw)
To: Robert Jennings, Nathan Fontenot, Benjamin Herrenschmidt,
Paul Mackerras, Gautham R Shenoy, Julia Lawall, linuxppc-dev
In-Reply-To: <20100723024343.GC32534@linux.vnet.ibm.com>
* Robert Jennings <rcj@linux.vnet.ibm.com> [2010-07-22 21:43:44]:
> If a CPU remove is attempted using the 'release' interface on hardware
> which supports extended cede, the CPU will be put in the INACTIVE state
> rather than the OFFLINE state due to the default preferred_offline_state
> in that situation. In the INACTIVE state it will fail to be removed.
>
> This patch changes the preferred offline state to OFFLINE when an CPU is
> in the ONLINE state. After cpu_down() is called in dlpar_offline_cpu()
> the CPU will be OFFLINE and CPU removal can continue.
Hi Robert,
Thanks for the patch. In dlpar operation, we would offline the CPU
first using the sysfs online file and then write to the sysfs release
file to complete the sequence right? The current code in
dlpar_offline_cpu() would work as long as the cpu is in either
inactive state or offline state (in case of unsupported platform).
Is the dlpar tools being changed to complete the operation with one
sysfs write to release file?
> Signed-off-by: Robert Jennings <rcj@linux.vnet.ibm.com>
>
> ---
>
> diff --git a/arch/powerpc/platforms/pseries/dlpar.c b/arch/powerpc/platforms/pseries/dlpar.c
> index d71e585..227c1c3 100644
> --- a/arch/powerpc/platforms/pseries/dlpar.c
> +++ b/arch/powerpc/platforms/pseries/dlpar.c
> @@ -463,6 +463,7 @@ static int dlpar_offline_cpu(struct device_node *dn)
> break;
>
> if (get_cpu_current_state(cpu) == CPU_STATE_ONLINE) {
> + set_preferred_offline_state(cpu, CPU_STATE_OFFLINE);
> cpu_maps_update_done();
> rc = cpu_down(cpu);
> if (rc)
The patch looks good. Will need to test out the various scenarios so
that the preferred_offline_state do not get flipped before cpu_down()
is called. This is unlikely, but still we need to validate
a concurrent sysfs online file write and sysfs release file write.
--Vaidy
^ permalink raw reply
* [PATCH 3/5 v2] powerpc/mm: Fix bugs in huge page hashing
From: Benjamin Herrenschmidt @ 2010-07-23 3:04 UTC (permalink / raw)
To: linuxppc-dev
There's a couple of nasty bugs lurking in our huge page hashing code.
First, we don't check the access permission atomically with setting
the _PAGE_BUSY bit, which means that the PTE value we end up using
for the hashing might be different than the one we have checked
the access permissions for.
We've seen cases where that leads us to try to use an invalidated
PTE for hashing, causing all sort of "interesting" issues.
Then, we also failed to set _PAGE_DIRTY on a write access.
Finally, a minor tweak but we should return 0 when we find the
PTE busy, in order to just re-execute the access, rather than 1
which means going to do_page_fault().
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/mm/hugetlbpage-hash64.c | 31 +++++++++++++------------------
1 files changed, 13 insertions(+), 18 deletions(-)
diff --git a/arch/powerpc/mm/hugetlbpage-hash64.c b/arch/powerpc/mm/hugetlbpage-hash64.c
index c9acd79..faae9ec 100644
--- a/arch/powerpc/mm/hugetlbpage-hash64.c
+++ b/arch/powerpc/mm/hugetlbpage-hash64.c
@@ -21,21 +21,13 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid,
unsigned long old_pte, new_pte;
unsigned long va, rflags, pa, sz;
long slot;
- int err = 1;
BUG_ON(shift != mmu_psize_defs[mmu_psize].shift);
/* Search the Linux page table for a match with va */
va = hpt_va(ea, vsid, ssize);
- /*
- * Check the user's access rights to the page. If access should be
- * prevented then send the problem up to do_page_fault.
- */
- if (unlikely(access & ~pte_val(*ptep)))
- goto out;
- /*
- * At this point, we have a pte (old_pte) which can be used to build
+ /* At this point, we have a pte (old_pte) which can be used to build
* or update an HPTE. There are 2 cases:
*
* 1. There is a valid (present) pte with no associated HPTE (this is
@@ -49,9 +41,17 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid,
do {
old_pte = pte_val(*ptep);
- if (old_pte & _PAGE_BUSY)
- goto out;
+ /* If PTE busy, retry the access */
+ if (unlikely(old_pte & _PAGE_BUSY))
+ return 0;
+ /* If PTE permissions don't match, take page fault */
+ if (unlikely(access & ~old_pte))
+ return 1;
+ /* Try to lock the PTE, add ACCESSED and DIRTY if it was
+ * a write access */
new_pte = old_pte | _PAGE_BUSY | _PAGE_ACCESSED;
+ if (access & _PAGE_RW)
+ new_pte |= _PAGE_DIRTY;
} while(old_pte != __cmpxchg_u64((unsigned long *)ptep,
old_pte, new_pte));
@@ -127,8 +127,7 @@ repeat:
*/
if (unlikely(slot == -2)) {
*ptep = __pte(old_pte);
- err = -1;
- goto out;
+ return -1;
}
new_pte |= (slot << 12) & (_PAGE_F_SECOND | _PAGE_F_GIX);
@@ -138,9 +137,5 @@ repeat:
* No need to use ldarx/stdcx here
*/
*ptep = __pte(new_pte & ~_PAGE_BUSY);
-
- err = 0;
-
- out:
- return err;
+ return 0;
}
^ permalink raw reply related
* Re: [PATCH 3/5] powerpc/mm: Fix bugs in huge page hashing
From: Benjamin Herrenschmidt @ 2010-07-23 3:03 UTC (permalink / raw)
To: linuxppc-dev
In-Reply-To: <1279845704-14848-3-git-send-email-benh@kernel.crashing.org>
On Fri, 2010-07-23 at 10:41 +1000, Benjamin Herrenschmidt wrote:
> There's a couple of nasty bugs lurking in our huge page hashing code.
>
> First, we don't check the access permission atomically with setting
> the _PAGE_BUSY bit, which means that the PTE value we end up using
> for the hashing might be different than the one we have checked
> the access permissions for.
>
> We've seen cases where that leads us to try to use an invalidated
> PTE for hashing, causing all sort of "interesting" issues.
>
> Then, we also failed to set _PAGE_DIRTY on a write access.
>
> This fixes both, while also simplifying the code a bit.
The changeset lacks a comment about a change to the return value
when hitting _PAGE_BUSY and ...
> do {
> old_pte = pte_val(*ptep);
> - if (old_pte & _PAGE_BUSY)
> - goto out;
> + /* If PTE busy, retry the access */
> + if (unlikely(old_pte & _PAGE_BUSY))
> + return 0;
> + /* If PTE permissions don't match, take page fault */
> + if (unlikely(access & ~pte_val(*ptep)))
> + return 1;
old_pte will do just fine instead of reloading it
Thanks Michael !
I'll respin that one.
Cheers,
Ben.
> + /* Try to lock the PTE, add ACCESSED and DIRTY if it was
> + * a write access */
> new_pte = old_pte | _PAGE_BUSY | _PAGE_ACCESSED;
> + if (access & _PAGE_RW)
> + new_pte |= _PAGE_DIRTY;
> } while(old_pte != __cmpxchg_u64((unsigned long *)ptep,
> old_pte, new_pte));
>
> @@ -127,8 +127,7 @@ repeat:
> */
> if (unlikely(slot == -2)) {
> *ptep = __pte(old_pte);
> - err = -1;
> - goto out;
> + return -1;
> }
>
> new_pte |= (slot << 12) & (_PAGE_F_SECOND | _PAGE_F_GIX);
> @@ -138,9 +137,5 @@ repeat:
> * No need to use ldarx/stdcx here
> */
> *ptep = __pte(new_pte & ~_PAGE_BUSY);
> -
> - err = 0;
> -
> - out:
> - return err;
> + return 0;
> }
^ permalink raw reply
* Re: [PATCH] fix of_flat_dt_is_compatible to match the full compatible string
From: Benjamin Herrenschmidt @ 2010-07-23 2:45 UTC (permalink / raw)
To: Stuart Yoder; +Cc: linuxppc-dev
In-Reply-To: <1279841330-28695-1-git-send-email-stuart.yoder@freescale.com>
On Thu, 2010-07-22 at 18:28 -0500, Stuart Yoder wrote:
> From: Stuart Yoder <stuart.yoder@freescale.com>
>
> With the previous string comparison, a device tree
> compatible of "foo-bar" would match as compatible
> with a driver looking for "foo".
>
> Signed-off-by: Stuart Yoder <stuart.yoder@freescale.com>
Beware you are doing two changes in one here and only documenting one...
You also removed the case insenstivity.
Now, those things are supposed to be case sensitive afaik, but we have
enough legacy HW with more/less crap DTs and I'd be careful with
changing that without a good reason (ie, it breaks not to do it ?).
No objection with fixing the partial match tho.
Cheers,
Ben.
> ---
> drivers/of/fdt.c | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
> index dee4fb5..f5239c0 100644
> --- a/drivers/of/fdt.c
> +++ b/drivers/of/fdt.c
> @@ -169,7 +169,7 @@ int __init of_flat_dt_is_compatible(unsigned long node, const char *compat)
> if (cp == NULL)
> return 0;
> while (cplen > 0) {
> - if (strncasecmp(cp, compat, strlen(compat)) == 0)
> + if (!strcmp(cp, compat))
> return 1;
> l = strlen(cp) + 1;
> cp += l;
^ permalink raw reply
* [PATCH] powerpc: ONLINE to OFFLINE CPU state transition during removal
From: Robert Jennings @ 2010-07-23 2:43 UTC (permalink / raw)
To: Nathan Fontenot, Benjamin Herrenschmidt, Paul Mackerras,
Gautham R Shenoy, Julia Lawall, Vaidyanathan Srinivasan,
linuxppc-dev
If a CPU remove is attempted using the 'release' interface on hardware
which supports extended cede, the CPU will be put in the INACTIVE state
rather than the OFFLINE state due to the default preferred_offline_state
in that situation. In the INACTIVE state it will fail to be removed.
This patch changes the preferred offline state to OFFLINE when an CPU is
in the ONLINE state. After cpu_down() is called in dlpar_offline_cpu()
the CPU will be OFFLINE and CPU removal can continue.
Signed-off-by: Robert Jennings <rcj@linux.vnet.ibm.com>
---
diff --git a/arch/powerpc/platforms/pseries/dlpar.c b/arch/powerpc/platforms/pseries/dlpar.c
index d71e585..227c1c3 100644
--- a/arch/powerpc/platforms/pseries/dlpar.c
+++ b/arch/powerpc/platforms/pseries/dlpar.c
@@ -463,6 +463,7 @@ static int dlpar_offline_cpu(struct device_node *dn)
break;
if (get_cpu_current_state(cpu) == CPU_STATE_ONLINE) {
+ set_preferred_offline_state(cpu, CPU_STATE_OFFLINE);
cpu_maps_update_done();
rc = cpu_down(cpu);
if (rc)
^ permalink raw reply related
* [PATCH 1/1] Added PowerMac10,2 device descriptor
From: Mark Crichton @ 2010-07-23 2:40 UTC (permalink / raw)
To: linuxppc-dev, Benjamin Herrenschmidt
Small patch to recognize the Mac Mini's that were quietly rolled out
~2005. Work needs to
be done on the WOL (Radeon and network card).
Signed-off-by: Mark Crichton <crichton@gmail.com>
---
arch/powerpc/platforms/powermac/feature.c | 4 ++++
1 files changed, 4 insertions(+), 0 deletions(-)
diff --git a/arch/powerpc/platforms/powermac/feature.c
b/arch/powerpc/platforms/powermac/feature.c
index 9e1b9fd..fa90384 100644
--- a/arch/powerpc/platforms/powermac/feature.c
+++ b/arch/powerpc/platforms/powermac/feature.c
@@ -2191,6 +2191,10 @@ static struct pmac_mb_def pmac_mb_defs[] = {
PMAC_TYPE_UNKNOWN_INTREPID, intrepid_features,
PMAC_MB_MAY_SLEEP,
},
+ { "PowerMac10,2", "Mac mini (Late 2005)",
+ PMAC_TYPE_UNKNOWN_INTREPID, intrepid_features,
+ PMAC_MB_MAY_SLEEP,
+ },
{ "iMac,1", "iMac (first generation)",
PMAC_TYPE_ORIG_IMAC, paddington_features,
0
--
1.7.1
^ permalink raw reply related
* Re: linux-next: OOPS at bot time
From: Michael Ellerman @ 2010-07-23 2:05 UTC (permalink / raw)
To: Ben Hutchings; +Cc: Stephen Rothwell, ppc-dev, LKML, Jesse Barnes
In-Reply-To: <1279847985.4883.391.camel@localhost>
[-- Attachment #1: Type: text/plain, Size: 2133 bytes --]
On Fri, 2010-07-23 at 02:19 +0100, Ben Hutchings wrote:
> On Fri, 2010-07-23 at 10:22 +1000, Stephen Rothwell wrote:
> > Hi all,
> >
> > My Power7 boot test paniced like this: (next-20100722)
> >
> > %GQLogic Fibre Channel HBA Driver: 8.03.03-k0
> > qla2xxx 0002:01:00.2: enabling device (0144 -> 0146)
> > qla2xxx 0002:01:00.2: Found an ISP8001, irq 35, iobase 0xd000080080014000
> > ------------[ cut here ]------------
> > kernel BUG at drivers/pci/msi.c:205!
> [...]
> > Call Trace:
> > [c00000000278b270] [c000000000048d9c] .rtas_setup_msi_irqs+0x1d8/0x254 (unreliable)
> > [c00000000278b360] [c00000000002a9cc] .arch_setup_msi_irqs+0x34/0x4c
> > [c00000000278b3e0] [c0000000002fd3fc] .pci_enable_msix+0x49c/0x4ac
> [...]
> > That line number is this:
> >
> > BUG_ON(!(entry->msg.address_hi | entry->msg.address_lo |
> > entry->msg.data));
> >
> > in read_msi_msg_desc(). That BUG_ON was added by commit
> > 2ca1af9aa3285c6a5f103ed31ad09f7399fc65d7 ("PCI: MSI: Remove unsafe and
> > unnecessary hardware access") from the pci tree.
>
> I wanted to assert that read_msi_msg_desc() is only used to update
> MSI/MSI-X descriptors that have already been generated by Linux. It
> looks like you found an exception.
>
> We could make read_msi_msg() fall back to reading from the hardware, but
> I think that what the pSeries code is trying to do - save an MSI message
> generated by firmware - is different from what the other callers want.
> Instead we could add:
>
> void save_msi_msg(unsigned int irq)
> {
> struct irq_desc *desc = irq_to_desc(irq);
> struct msi_desc *entry = get_irq_desc_msi(desc);
> struct msi_msg *msg = &entry->msg;
>
> /* ...followed by the old implementation of read_msi_msg_desc() */
> }
>
> Possibly conditional on something like CONFIG_ARCH_NEEDS_SAVE_MSI_MSG.
Maybe.
But then you end up with read_msi_msg(), which doesn't actually read
anything, which I think is confusing. I'd rather read_msi_msg() read the
message, from the device, and we have another routine which returns the
previously saved msg from the msi_desc.
cheers
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply
* Re: [PATCH] powerpc: print cores passed to firmware in decimal
From: Michael Neuling @ 2010-07-23 1:44 UTC (permalink / raw)
To: Jesse Larrew, benh; +Cc: linuxppc-dev
In-Reply-To: <4C48B36E.6040505@linux.vnet.ibm.com>
Currently we look pretty stupid when printing out the number of cores
passed to FW
Max number of cores passed to firmware: 0x0000000000000080
So I've change this to print in decimal:
Max number of cores passed to firmware: 128 (NR_CPUS = 256)
This required adding a prom_print_dec() function.
Signed-off-by: Michael Neuling <mikey@neuling.org>
---
> > cores = (u32 *)PTRRELOC(&ibm_architecture_vec[IBM_ARCH_VEC_NRCO
RES_OFFSET]);
> > if (*cores != NR_CPUS) {
> > prom_printf("WARNING ! "
> > - "ibm_architecture_vec structure inconsisten
t: 0x%x !\n",
> > + "ibm_architecture_vec structure inconsisten
t: 0x%i !\n",
> > *cores);
> >
>
> Since we're changing from hex to decimal, we shouldn't print the "0x".
Oops yes. Thanks!
BTW please CC me directly rather than sending only to the list.
arch/powerpc/kernel/prom_init.c | 35 ++++++++++++++++++++++++++++++++---
1 files changed, 32 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/kernel/prom_init.c b/arch/powerpc/kernel/prom_init.c
index 3b6f8ae..28b9116 100644
--- a/arch/powerpc/kernel/prom_init.c
+++ b/arch/powerpc/kernel/prom_init.c
@@ -311,6 +311,27 @@ static void __init prom_print_hex(unsigned long val)
call_prom("write", 3, 1, _prom->stdout, buf, nibbles);
}
+/* max number of decimal digits in an unsigned long */
+#define UL_DIGITS 21
+static void __init prom_print_dec(unsigned long val)
+{
+ int i, size;
+ char buf[UL_DIGITS+1];
+ struct prom_t *_prom = &RELOC(prom);
+
+ for (i = UL_DIGITS-1; i >= 0; i--) {
+ buf[i] = (val % 10) + '0';
+ val = val/10;
+ if (val == 0)
+ break;
+ }
+ /* shift stuff down */
+ size = UL_DIGITS - i;
+ for (i = 0 ; i < size ; i++)
+ buf[i] = buf[i + UL_DIGITS - size];
+ buf[size+1] = '\0';
+ call_prom("write", 3, 1, _prom->stdout, buf, size);
+}
static void __init prom_printf(const char *format, ...)
{
@@ -350,6 +371,14 @@ static void __init prom_printf(const char *format, ...)
v = va_arg(args, unsigned long);
prom_print_hex(v);
break;
+ case 'l':
+ ++q;
+ if (*q == 'u') { /* '%lu' */
+ ++q;
+ v = va_arg(args, unsigned long);
+ prom_print_dec(v);
+ }
+ break;
}
}
}
@@ -869,12 +898,12 @@ static void __init prom_send_capabilities(void)
cores = (u32 *)PTRRELOC(&ibm_architecture_vec[IBM_ARCH_VEC_NRCORES_OFFSET]);
if (*cores != NR_CPUS) {
prom_printf("WARNING ! "
- "ibm_architecture_vec structure inconsistent: 0x%x !\n",
+ "ibm_architecture_vec structure inconsistent: %lu !\n",
*cores);
} else {
*cores = DIV_ROUND_UP(NR_CPUS, prom_count_smt_threads());
- prom_printf("Max number of cores passed to firmware: 0x%x\n",
- (unsigned long)*cores);
+ prom_printf("Max number of cores passed to firmware: %lu (NR_CPUS = %lu)\n",
+ *cores, NR_CPUS);
}
/* try calling the ibm,client-architecture-support method */
^ permalink raw reply related
* Re: linux-next: OOPS at bot time
From: Ben Hutchings @ 2010-07-23 1:19 UTC (permalink / raw)
To: Stephen Rothwell; +Cc: ppc-dev, LKML, Jesse Barnes
In-Reply-To: <20100723102202.871a3131.sfr@canb.auug.org.au>
On Fri, 2010-07-23 at 10:22 +1000, Stephen Rothwell wrote:
> Hi all,
>
> My Power7 boot test paniced like this: (next-20100722)
>
> %GQLogic Fibre Channel HBA Driver: 8.03.03-k0
> qla2xxx 0002:01:00.2: enabling device (0144 -> 0146)
> qla2xxx 0002:01:00.2: Found an ISP8001, irq 35, iobase 0xd000080080014000
> ------------[ cut here ]------------
> kernel BUG at drivers/pci/msi.c:205!
[...]
> Call Trace:
> [c00000000278b270] [c000000000048d9c] .rtas_setup_msi_irqs+0x1d8/0x254 (unreliable)
> [c00000000278b360] [c00000000002a9cc] .arch_setup_msi_irqs+0x34/0x4c
> [c00000000278b3e0] [c0000000002fd3fc] .pci_enable_msix+0x49c/0x4ac
[...]
> That line number is this:
>
> BUG_ON(!(entry->msg.address_hi | entry->msg.address_lo |
> entry->msg.data));
>
> in read_msi_msg_desc(). That BUG_ON was added by commit
> 2ca1af9aa3285c6a5f103ed31ad09f7399fc65d7 ("PCI: MSI: Remove unsafe and
> unnecessary hardware access") from the pci tree.
I wanted to assert that read_msi_msg_desc() is only used to update
MSI/MSI-X descriptors that have already been generated by Linux. It
looks like you found an exception.
We could make read_msi_msg() fall back to reading from the hardware, but
I think that what the pSeries code is trying to do - save an MSI message
generated by firmware - is different from what the other callers want.
Instead we could add:
void save_msi_msg(unsigned int irq)
{
struct irq_desc *desc = irq_to_desc(irq);
struct msi_desc *entry = get_irq_desc_msi(desc);
struct msi_msg *msg = &entry->msg;
/* ...followed by the old implementation of read_msi_msg_desc() */
}
Possibly conditional on something like CONFIG_ARCH_NEEDS_SAVE_MSI_MSG.
Ben.
--
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
^ permalink raw reply
* [PATCH] Adding ADMA support for PPC460EX DMA engine.
From: tmarri @ 2010-07-23 0:57 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-raid, tmarri, dan.j.williams, linuxppc-dev
From: Tirumala Marri <tmarri@amcc.com>
This patch will add ADMA support for DMA engine and HW offload for
XOR/ADG (RAID-5/6) functionalities.
1. It supports memcpy, xor, GF(2) based RAID-6.
2. It supports interrupt based DMA completions.
3. Also supports memcpy in RAID-1 case.
Kernel version: 2.6.35-rc5
Testing:
Created RAID-5/6 arrays usign mdadm.
And ran raw IO and filesystem IO to the RAID array.
Chunk size 4k,64k was tested.
RAID rebuild , disk fail, resync tested.
File names:
This code is similar to ppc440spe . So I named the files as
drivers/dma/ppc4xx/adma1.c and drivers/dma/ppc4xx/adma1.h
Signed-off-by: Tirumala R Marri <tmarri@apm.com>
---
arch/powerpc/boot/dts/canyonlands.dts | 23 +-
drivers/dma/Kconfig | 10 +
drivers/dma/Makefile | 1 +
drivers/dma/ppc4xx/Makefile | 1 +
drivers/dma/ppc4xx/adma1.c | 4119 +++++++++++++++++++++++++++++++++
drivers/dma/ppc4xx/adma1.h | 192 ++
drivers/dma/ppc4xx/dma.h | 20 +-
7 files changed, 4362 insertions(+), 4 deletions(-)
create mode 100644 drivers/dma/ppc4xx/adma1.c
create mode 100644 drivers/dma/ppc4xx/adma1.h
diff --git a/arch/powerpc/boot/dts/canyonlands.dts b/arch/powerpc/boot/dts/canyonlands.dts
index cd56bb5..eb3ca8c 100644
--- a/arch/powerpc/boot/dts/canyonlands.dts
+++ b/arch/powerpc/boot/dts/canyonlands.dts
@@ -114,7 +114,10 @@
interrupt-parent = <&UIC1>;
interrupts = <11 1>;
};
-
+ MQ0: mq {
+ compatible = "ibm,mq-460ex";
+ dcr-reg = <0x040 0x020>;
+ };
plb {
compatible = "ibm,plb-460ex", "ibm,plb4";
#address-cells = <2>;
@@ -162,6 +165,24 @@
interrupt-parent = <&UIC2>;
interrupts = <0x1e 4>;
};
+ I2O: i2o@400100000 {
+ compatible = "ibm,i2o-460ex";
+ reg = <0x00000004 0x00100000 0x100>;
+ dcr-reg = <0x060 0x020>;
+ };
+ ADMA: adma {
+ compatible = "amcc,dma-460ex";
+ device_type = "dma";
+ reg = <0x00000004 0x00100200 0x100>;
+ interrupt-parent = <&ADMA>;
+ interrupts =<0 1 2>;
+ #interrupt-cells = <1>;
+ #address-cells = <0>;
+ #size-cells = <0>;
+ interrupt-map = </*FIFO need service */ 0 &UIC0 0x16 4
+ /*FIFO FULL */ 1 &UIC0 0x15 4
+ /*FIFO HSDMA err */ 2 &UIC1 0x16 4>;
+ };
POB0: opb {
compatible = "ibm,opb-460ex", "ibm,opb";
diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
index 9e01e96..77f9ea0 100644
--- a/drivers/dma/Kconfig
+++ b/drivers/dma/Kconfig
@@ -163,6 +163,16 @@ config TIMB_DMA
help
Enable support for the Timberdale FPGA DMA engine.
+config AMCC_PPC460EX_ADMA
+ tristate "AMCC PPC460Ex ADMA support"
+ depends on 460EX
+ select DMA_ENGINE
+ select ARCH_HAS_ASYNC_TX_FIND_CHANNEL
+ help
+ Enable support for the AMCC PPC460Ex RAID engines.
+ Also adds HW acceleration for memset and memcpy.
+ Enabling RAID-5/6 would also need HW key.
+
config ARCH_HAS_ASYNC_TX_FIND_CHANNEL
bool
diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile
index 0fe5ebb..1d0ccfc 100644
--- a/drivers/dma/Makefile
+++ b/drivers/dma/Makefile
@@ -20,6 +20,7 @@ obj-$(CONFIG_TXX9_DMAC) += txx9dmac.o
obj-$(CONFIG_SH_DMAE) += shdma.o
obj-$(CONFIG_COH901318) += coh901318.o coh901318_lli.o
obj-$(CONFIG_AMCC_PPC440SPE_ADMA) += ppc4xx/
+obj-$(CONFIG_AMCC_PPC460EX_ADMA) += ppc4xx/
obj-$(CONFIG_TIMB_DMA) += timb_dma.o
obj-$(CONFIG_STE_DMA40) += ste_dma40.o ste_dma40_ll.o
obj-$(CONFIG_PL330_DMA) += pl330.o
diff --git a/drivers/dma/ppc4xx/Makefile b/drivers/dma/ppc4xx/Makefile
index b3d259b..435a086 100644
--- a/drivers/dma/ppc4xx/Makefile
+++ b/drivers/dma/ppc4xx/Makefile
@@ -1 +1,2 @@
obj-$(CONFIG_AMCC_PPC440SPE_ADMA) += adma.o
+obj-$(CONFIG_AMCC_PPC460EX_ADMA) += adma1.o
diff --git a/drivers/dma/ppc4xx/adma1.c b/drivers/dma/ppc4xx/adma1.c
new file mode 100644
index 0000000..30c2229
--- /dev/null
+++ b/drivers/dma/ppc4xx/adma1.c
@@ -0,0 +1,4119 @@
+/*
+ * Copyright(c) 2010 Applied Micro (APM). All rights reserved.
+ *
+ * Author: Tirumala Reddy Marri tmarri@apm.com
+ *
+ * This driver follows Dan Williams and Yuri Tikhonovs implementations.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; if not, write to the Free Software Foundation, Inc., 59
+ * Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * The full GNU General Public License is included in this distribution in the
+ * file called COPYING.
+ *
+ */
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/async_tx.h>
+#include <linux/delay.h>
+#include <linux/dma-mapping.h>
+#include <linux/spinlock.h>
+#include <linux/interrupt.h>
+#include <linux/platform_device.h>
+#include <linux/uaccess.h>
+#include <linux/of_platform.h>
+#include <linux/proc_fs.h>
+#include <asm/dcr.h>
+#include <asm/dcr-regs.h>
+#include "dma.h"
+#include "adma1.h"
+
+enum ppc_adma_init_code {
+ PPC_ADMA_INIT_OK = 0,
+ PPC_ADMA_INIT_MEMRES,
+ PPC_ADMA_INIT_MEMREG,
+ PPC_ADMA_INIT_ALLOC,
+ PPC_ADMA_INIT_COHERENT,
+ PPC_ADMA_INIT_CHANNEL,
+ PPC_ADMA_INIT_IRQ1,
+ PPC_ADMA_INIT_IRQ2,
+ PPC_ADMA_INIT_REGISTER
+};
+
+static char *ppc_adma_errors[] = {
+ [PPC_ADMA_INIT_OK] = "ok",
+ [PPC_ADMA_INIT_MEMRES] = "failed to get memory resource",
+ [PPC_ADMA_INIT_MEMREG] = "failed to request memory region",
+ [PPC_ADMA_INIT_ALLOC] = "failed to allocate memory for adev "
+ "structure",
+ [PPC_ADMA_INIT_COHERENT] = "failed to allocate coherent memory for "
+ "hardware descriptors",
+ [PPC_ADMA_INIT_CHANNEL] = "failed to allocate memory for channel",
+ [PPC_ADMA_INIT_IRQ1] = "failed to request first irq",
+ [PPC_ADMA_INIT_IRQ2] = "failed to request second irq",
+ [PPC_ADMA_INIT_REGISTER] = "failed to register dma async device",
+};
+
+static enum ppc_adma_init_code
+ppc460ex_adma_devices[PPC460EX_ADMA_ENGINES_NUM];
+
+struct ppc_dma_chan_ref {
+ struct dma_chan *chan;
+ struct list_head node;
+};
+
+/*
+ * The list of channels exported by ppc460ex ADMA
+ */
+struct list_head
+ppc460ex_adma_chan_list = LIST_HEAD_INIT(ppc460ex_adma_chan_list);
+
+/*
+ * Pointer to DMA0 CP/CS FIFO
+ */
+static void *ppc460ex_dma_fifo_buf;
+
+/*
+ * Pointers to last submitted to DMA0
+ */
+static struct ppc460ex_adma_desc_slot *chan_last_sub[1];
+static struct ppc460ex_adma_desc_slot *chan_first_cdb[1];
+
+/*
+ * Since RXOR operations use the common register (MQ0_CF2H) for setting-up
+ * the block size in transactions, then we do not allow to activate more than
+ * only one RXOR transactions simultaneously. So use this var to store
+ * the information about is RXOR currently active (PPC460EX_RXOR_RUN bit is
+ * set) or not (PPC460EX_RXOR_RUN is clear).
+ */
+static unsigned long ppc460ex_rxor_state;
+/*
+ * This array is used in data-check operations for storing a pattern
+ */
+static char ppc460ex_qword[16];
+
+
+/*
+ * These are used in enable & check routines
+ */
+static u32 ppc460ex_r6_enabled;
+static u32 ppc460ex_r5_enabled;
+
+static struct pc460ex_adma_chan *ppc460ex_r6_tchan;
+static struct pc460ex_adma_chan *ppc460ex_r5_tchan;
+static struct completion ppc460ex_r6_test_comp;
+static struct completion ppc460ex_r5_test_comp;
+static atomic_t ppc460ex_adma_err_irq_ref;
+static dcr_host_t ppc460ex_mq_dcr_host;
+static unsigned int ppc460ex_mq_dcr_len;
+
+static int ppc460ex_adma_alloc_chan_resources(struct dma_chan *chan);
+static struct ppc460ex_adma_desc_slot *ppc460ex_adma_alloc_slots(
+ struct ppc460ex_adma_chan *chan, int num_slots,
+ int slots_per_op);
+
+/******************************************************************************
+ * Command (Descriptor) Blocks low-level routines
+ ******************************************************************************/
+/*
+ * ppc460ex_desc_init_interrupt - initialize the descriptor for INTERRUPT
+ * pseudo operation
+ */
+static inline void ppc460ex_desc_init_interrupt(
+ struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan)
+{
+ memset(desc->hw_desc, 0, sizeof(struct dma_cdb));
+ /*
+ * NOP with interrupt
+ */
+ set_bit(PPC460EX_DESC_INT, &desc->flags);
+}
+/*
+ * ppc460ex_desc_init_pqzero_sum - initialize the descriptor
+ * for PQ_ZERO_SUM operation
+ */
+static void ppc460ex_desc_init_pqzero_sum(
+ struct ppc460ex_adma_desc_slot *desc,
+ int dst_cnt, int src_cnt)
+{
+ struct dma_cdb *hw_desc;
+ struct ppc460ex_adma_desc_slot *iter;
+ int i = 0;
+ u8 dopc = (dst_cnt == 2) ? DMA_CDB_OPC_MULTICAST :
+ DMA_CDB_OPC_MV_SG1_SG2;
+ /*
+ * Initialize starting from 2nd or 3rd descriptor dependent
+ * on dst_cnt. First one or two slots are for cloning P
+ * and/or Q to chan->pdest and/or chan->qdest as we have
+ * to preserve original P/Q.
+ */
+ iter = list_first_entry(&desc->group_list,
+ struct ppc460ex_adma_desc_slot, chain_node);
+ iter = list_entry(iter->chain_node.next,
+ struct ppc460ex_adma_desc_slot, chain_node);
+
+ if (dst_cnt > 1) {
+ iter = list_entry(iter->chain_node.next,
+ struct ppc460ex_adma_desc_slot, chain_node);
+ }
+ /*
+ * initialize each source descriptor in chain
+ */
+ list_for_each_entry_from(iter, &desc->group_list, chain_node) {
+ hw_desc = iter->hw_desc;
+ memset(iter->hw_desc, 0, sizeof(struct dma_cdb));
+ iter->src_cnt = 0;
+ iter->dst_cnt = 0;
+
+ /* This is a ZERO_SUM operation:
+ * - <src_cnt> descriptors starting from 2nd or 3rd
+ * descriptor are for GF-XOR operations;
+ * - remaining <dst_cnt> descriptors are for checking the result
+ */
+ if (i++ < src_cnt)
+ /* MV_SG1_SG2 if only Q is being verified
+ * MULTICAST if both P and Q are being verified
+ */
+ hw_desc->opc = dopc;
+ else
+ /*
+ * DMA_CDB_OPC_DCHECK128 operation
+ */
+ hw_desc->opc = DMA_CDB_OPC_DCHECK128;
+
+ if (likely(!list_is_last(&iter->chain_node,
+ &desc->group_list))) {
+ /*
+ * set 'next' pointer
+ */
+ iter->hw_next = list_entry(iter->chain_node.next,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ } else {
+ /* this is the last descriptor.
+ * this slot will be pasted from ADMA level
+ * each time it wants to configure parameters
+ * of the transaction (src, dst, ...)
+ */
+ iter->hw_next = NULL;
+ /* always enable interrupt generation since we get
+ * the status of pqzero from the handler
+ */
+ set_bit(PPC460EX_DESC_INT, &iter->flags);
+ }
+ }
+ desc->src_cnt = src_cnt;
+ desc->dst_cnt = dst_cnt;
+}
+/*
+ * ppc460ex_desc_init_memcpy - initialize the descriptor for MEMCPY operation
+ */
+static inline void ppc460ex_desc_init_memcpy(
+ struct ppc460ex_adma_desc_slot *desc,
+ unsigned long flags)
+{
+ struct dma_cdb *hw_desc = desc->hw_desc;
+
+ memset(desc->hw_desc, 0, sizeof(struct dma_cdb));
+ desc->hw_next = NULL;
+ desc->src_cnt = 1;
+ desc->dst_cnt = 1;
+
+ if (flags & DMA_PREP_INTERRUPT)
+ set_bit(PPC460EX_DESC_INT, &desc->flags);
+ else
+ clear_bit(PPC460EX_DESC_INT, &desc->flags);
+
+ hw_desc->opc = DMA_CDB_OPC_MV_SG1_SG2;
+}
+
+/*
+ * ppc460ex_desc_init_memset - initialize the descriptor for MEMSET operation
+ */
+static inline void ppc460ex_desc_init_memset(
+ struct ppc460ex_adma_desc_slot *desc, int value,
+ unsigned long flags)
+{
+ struct dma_cdb *hw_desc = desc->hw_desc;
+
+ memset(desc->hw_desc, 0, sizeof(struct dma_cdb));
+ desc->hw_next = NULL;
+ desc->src_cnt = 1;
+ desc->dst_cnt = 1;
+
+ if (flags & DMA_PREP_INTERRUPT)
+ set_bit(PPC460EX_DESC_INT, &desc->flags);
+ else
+ clear_bit(PPC460EX_DESC_INT, &desc->flags);
+
+ hw_desc->sg1u = hw_desc->sg1l = cpu_to_le32((u32)value);
+ hw_desc->sg3u = hw_desc->sg3l = cpu_to_le32((u32)value);
+ hw_desc->opc = DMA_CDB_OPC_DFILL128;
+}
+/*
+ * ppc460ex_desc_assign_cookie - assign a cookie
+ */
+static dma_cookie_t ppc460ex_desc_assign_cookie(struct ppc460ex_adma_chan *chan,
+ struct ppc460ex_adma_desc_slot *desc)
+{
+ dma_cookie_t cookie = chan->common.cookie;
+ cookie++;
+ if (cookie < 0)
+ cookie = 1;
+ chan->common.cookie = desc->async_tx.cookie = cookie;
+ return cookie;
+}
+/*
+ * ppc460ex_desc_set_src_addr - set source address into the descriptor
+ */
+static inline void ppc460ex_desc_set_src_addr(
+ struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan, int src_idx,
+ dma_addr_t addrh, dma_addr_t addrl)
+{
+ struct dma_cdb *dma_hw_desc;
+ phys_addr_t addr64, tmplow, tmphi;
+
+ if (!addrh) {
+ addr64 = addrl;
+ tmphi = (addr64 >> 32);
+ tmplow = (addr64 & 0xFFFFFFFF);
+ } else {
+ tmphi = addrh;
+ tmplow = addrl;
+ }
+ dma_hw_desc = desc->hw_desc;
+ dma_hw_desc->sg1l = cpu_to_le32((u32)tmplow);
+ dma_hw_desc->sg1u = cpu_to_le32((u32)tmphi);
+}
+
+/*
+ * ppc460ex_desc_set_src_mult - set source address mult into the descriptor
+ */
+static inline void ppc460ex_desc_set_src_mult(
+ struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan, u32 mult_index, int sg_index,
+ unsigned char mult_value)
+{
+ struct dma_cdb *dma_hw_desc;
+ u32 *psgu;
+
+ dma_hw_desc = desc->hw_desc;
+
+ switch (sg_index) {
+ /*
+ * for RXOR operations set multiplier
+ * into source cued address
+ */
+ case DMA_CDB_SG_SRC:
+ psgu = &dma_hw_desc->sg1u;
+ break;
+ /*
+ * for WXOR operations set multiplier
+ * into destination cued address(es)
+ */
+ case DMA_CDB_SG_DST1:
+ psgu = &dma_hw_desc->sg2u;
+ break;
+ case DMA_CDB_SG_DST2:
+ psgu = &dma_hw_desc->sg3u;
+ break;
+ default:
+ BUG();
+ }
+
+ *psgu |= cpu_to_le32(mult_value << mult_index);
+}
+
+/*
+ * ppc460ex_desc_set_dest_addr - set destination address into the descriptor
+ */
+static inline void ppc460ex_desc_set_dest_addr(
+ struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan,
+ dma_addr_t addrh, dma_addr_t addrl,
+ u32 dst_idx)
+{
+ struct dma_cdb *dma_hw_desc;
+ phys_addr_t addr64, tmphi, tmplow;
+ u32 *psgu, *psgl;
+
+ if (!addrh) {
+ addr64 = addrl;
+ tmphi = (addr64 >> 32);
+ tmplow = (addr64 & 0xFFFFFFFF);
+ } else {
+ tmphi = addrh;
+ tmplow = addrl;
+ }
+ dma_hw_desc = desc->hw_desc;
+
+ psgu = dst_idx ? &dma_hw_desc->sg3u : &dma_hw_desc->sg2u;
+ psgl = dst_idx ? &dma_hw_desc->sg3l : &dma_hw_desc->sg2l;
+
+ *psgl = cpu_to_le32((u32)tmplow);
+ *psgu |= cpu_to_le32((u32)tmphi);
+}
+
+/*
+ * ppc460ex_desc_set_byte_count - set number of data bytes involved
+ * into the operation
+ */
+static inline void ppc460ex_desc_set_byte_count(
+ struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan, u32 byte_count)
+{
+ struct dma_cdb *dma_hw_desc;
+
+ dma_hw_desc = desc->hw_desc;
+ dma_hw_desc->cnt = cpu_to_le32(byte_count);
+}
+/*
+ * ppc460ex_desc_set_rxor_block_size - set RXOR block size
+ */
+static inline void ppc460ex_desc_set_rxor_block_size(u32 byte_count)
+{
+ /*
+ * assume that byte_count is aligned on the 512-boundary;
+ * thus write it directly to the register (bits 23:31 are
+ * reserved there).
+ */
+ mtdcr(DCRN_MQ0_CF2H, byte_count);
+}
+/*
+ * ppc460ex_desc_set_dcheck - set CHECK pattern
+ */
+static inline void ppc460ex_desc_set_dcheck(
+ struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan, u8 *qword)
+{
+ struct dma_cdb *dma_hw_desc;
+
+ dma_hw_desc = desc->hw_desc;
+ out_le32(&dma_hw_desc->sg3l, qword[0]);
+ out_le32(&dma_hw_desc->sg3u, qword[4]);
+ out_le32(&dma_hw_desc->sg2l, qword[8]);
+ out_le32(&dma_hw_desc->sg2u, qword[12]);
+}
+/*
+ * ppc460ex_desc_get_src_num - extract the number of source addresses from
+ * the descriptor
+ */
+static inline u32 ppc460ex_desc_get_src_num(
+ struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan)
+{
+ struct dma_cdb *dma_hw_desc;
+
+ dma_hw_desc = desc->hw_desc;
+
+ switch (dma_hw_desc->opc) {
+ case DMA_CDB_OPC_NO_OP:
+ case DMA_CDB_OPC_DFILL128:
+ return 0;
+ case DMA_CDB_OPC_DCHECK128:
+ return 1;
+ case DMA_CDB_OPC_MV_SG1_SG2:
+ case DMA_CDB_OPC_MULTICAST:
+ /*
+ * Only for RXOR operations we have more than
+ * one source
+ */
+ if (le32_to_cpu(dma_hw_desc->sg1u) &
+ DMA_CUED_XOR_WIN_MSK) {
+ /* RXOR op, there are 2 or 3 sources */
+ if (((le32_to_cpu(dma_hw_desc->sg1u) >>
+ DMA_CUED_REGION_OFF) &
+ DMA_CUED_REGION_MSK) == DMA_RXOR12) {
+ /* RXOR 1-2 */
+ return 2;
+ } else {
+ /* RXOR 1-2-3/1-2-4/1-2-5 */
+ return 3;
+ }
+ }
+ return 1;
+ default:
+ dev_dbg(chan->device->common.dev, "%s: unknown OPC 0x%02x\n",
+ __func__, dma_hw_desc->opc);
+ BUG();
+ }
+
+ return 0;
+}
+
+/*
+ * ppc460ex_desc_get_dst_num - get the number of destination addresses in
+ * this descriptor
+ */
+static inline u32 ppc460ex_desc_get_dst_num(
+ struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan)
+{
+ struct dma_cdb *dma_hw_desc;
+
+ /*
+ * May be 1 or 2 destinations
+ */
+ dma_hw_desc = desc->hw_desc;
+ switch (dma_hw_desc->opc) {
+ case DMA_CDB_OPC_NO_OP:
+ case DMA_CDB_OPC_DCHECK128:
+ return 0;
+ case DMA_CDB_OPC_MV_SG1_SG2:
+ case DMA_CDB_OPC_DFILL128:
+ return 1;
+ case DMA_CDB_OPC_MULTICAST:
+ return 2;
+ default:
+ dev_dbg(chan->device->common.dev, "%s: unknown OPC 0x%02x\n",
+ __func__, dma_hw_desc->opc);
+ BUG();
+ }
+ return 0;
+}
+/*
+ * ppc460ex_desc_get_src_addr - extract the source address from the descriptor
+ */
+static inline u32 ppc460ex_desc_get_src_addr(
+ struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan, int src_idx)
+{
+ struct dma_cdb *dma_hw_desc;
+ u32 sg11;
+
+ dma_hw_desc = desc->hw_desc;
+ /*
+ * May have 0, 1, 2, or 3 sources
+ */
+ switch (dma_hw_desc->opc) {
+ case DMA_CDB_OPC_NO_OP:
+ case DMA_CDB_OPC_DFILL128:
+ return 0;
+ case DMA_CDB_OPC_DCHECK128:
+ if (unlikely(src_idx)) {
+ dev_dbg(chan->device->common.dev,
+ "%s: try to get %d source for"
+ " DCHECK128\n", __func__, src_idx);
+ BUG();
+ }
+ return le32_to_cpu(dma_hw_desc->sg1l);
+ case DMA_CDB_OPC_MULTICAST:
+ case DMA_CDB_OPC_MV_SG1_SG2:
+ if (unlikely(src_idx > 2)) {
+ dev_dbg(chan->device->common.dev,
+ "%s: try to get %d source from"
+ " DMA descr\n", __func__, src_idx);
+ BUG();
+ }
+ if (src_idx) {
+ if (le32_to_cpu(dma_hw_desc->sg1u) &
+ DMA_CUED_XOR_WIN_MSK) {
+ u8 region;
+
+ if (src_idx == 1)
+ return le32_to_cpu(
+ dma_hw_desc->sg1l) +
+ desc->unmap_len;
+
+ region = (le32_to_cpu(
+ dma_hw_desc->sg1u)) >>
+ DMA_CUED_REGION_OFF;
+
+ region &= DMA_CUED_REGION_MSK;
+ switch (region) {
+ case DMA_RXOR123:
+ return le32_to_cpu(
+ dma_hw_desc->sg1l) +
+ (desc->unmap_len << 1);
+ case DMA_RXOR124:
+ return le32_to_cpu(
+ dma_hw_desc->sg1l) +
+ (desc->unmap_len * 3);
+ case DMA_RXOR125:
+ return le32_to_cpu(
+ dma_hw_desc->sg1l) +
+ (desc->unmap_len << 2);
+ default:
+ dev_dbg(chan->device->common.dev,
+ "%s: try to"
+ " get src3 for region %02x"
+ "PPC460EX_DESC_RXOR12?\n",
+ __func__, region);
+ BUG();
+ }
+ } else {
+ dev_dbg(chan->device->common.dev,
+ "%s: try to get %d"
+ " source for non-cued descr\n",
+ __func__, src_idx);
+ BUG();
+ }
+ }
+ return le32_to_cpu(dma_hw_desc->sg1l);
+ default:
+ dev_dbg(chan->device->common.dev, "%s: unknown OPC 0x%02x\n",
+ __func__, dma_hw_desc->opc);
+ BUG();
+ }
+ sg11 = le32_to_cpu(dma_hw_desc->sg1l);
+ return sg11;
+}
+
+/*
+ * ppc460ex_desc_get_dest_addr - extract the destination address from the
+ * descriptor
+ */
+static inline u32 ppc460ex_desc_get_dest_addr(
+ struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan, int idx)
+{
+ struct dma_cdb *dma_hw_desc;
+
+ dma_hw_desc = desc->hw_desc;
+
+ if (likely(!idx))
+ return le32_to_cpu(dma_hw_desc->sg2l);
+ return le32_to_cpu(dma_hw_desc->sg3l);
+}
+/*
+ * ppc460ex_desc_get_link - get the address of the descriptor that
+ * follows this one
+ */
+static inline u32 ppc460ex_desc_get_link(struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan)
+{
+ if (!desc->hw_next)
+ return 0;
+
+ return desc->hw_next->phys;
+}
+/******************************************************************************
+ * ADMA channel low-level routines
+ ******************************************************************************/
+static inline u32 ppc460ex_chan_get_current_descriptor(
+ struct ppc460ex_adma_chan *chan);
+static inline void ppc460ex_chan_append(struct ppc460ex_adma_chan *chan);
+static void ppc460ex_dma_put_desc(struct ppc460ex_adma_chan *chan,
+ struct ppc460ex_adma_desc_slot *desc);
+
+/*
+ * ppc460ex_adma_device_clear_eot_status - interrupt ack to XOR or DMA engine
+ */
+static inline void ppc460ex_adma_device_clear_eot_status(
+ struct ppc460ex_adma_chan *chan)
+{
+ struct dma_regs *dma_reg;
+ u8 *p = chan->device->dma_desc_pool_virt;
+ struct dma_cdb *cdb;
+ u32 rv, i;
+
+ /*
+ * read FIFO to ack
+ */
+ dma_reg = (struct dma_regs *)chan->device->dma_reg;
+ while ((rv = in_le32(&dma_reg->csfpl))) {
+ i = rv & DMA_CDB_ADDR_MSK;
+ cdb = (struct dma_cdb *)&p[i -
+ (u32)chan->device->dma_desc_pool];
+
+ /* Clear opcode to ack. This is necessary for
+ * ZeroSum operations only
+ */
+ cdb->opc = 0;
+
+ if (test_bit(PPC460EX_RXOR_RUN,
+ &ppc460ex_rxor_state)) {
+ /* probably this is a completed RXOR op,
+ * get pointer to CDB using the fact that
+ * physical and virtual addresses of CDB
+ * in pools have the same offsets
+ */
+ if (le32_to_cpu(cdb->sg1u) &
+ DMA_CUED_XOR_BASE) {
+ /* this is a RXOR */
+ clear_bit(PPC460EX_RXOR_RUN,
+ &ppc460ex_rxor_state);
+ }
+ }
+
+ if (rv & DMA_CDB_STATUS_MSK) {
+ /*
+ * ZeroSum check failed
+ */
+ struct ppc460ex_adma_desc_slot *iter;
+ dma_addr_t phys = rv & ~DMA_CDB_MSK;
+
+ /*
+ * Update the status of corresponding
+ * descriptor.
+ */
+ list_for_each_entry(iter, &chan->chain,
+ chain_node) {
+ if (iter->phys == phys)
+ break;
+ }
+ /*
+ * if cannot find the corresponding
+ * slot it's a bug
+ */
+ BUG_ON(&iter->chain_node == &chan->chain);
+
+ if (iter->xor_check_result)
+ *iter->xor_check_result |=
+ rv & DMA_CDB_STATUS_MSK;
+ }
+ }
+
+ rv = in_le32(&dma_reg->dsts);
+ if (rv) {
+ dev_dbg(chan->device->common.dev,
+ "DMA%d err status: 0x%x\n", chan->device->id,
+ rv);
+ /*
+ * write back to clear
+ */
+ out_le32(&dma_reg->dsts, rv);
+ }
+}
+/*
+ * ppc460ex_chan_is_busy - get the channel status
+ */
+static inline int ppc460ex_chan_is_busy(struct ppc460ex_adma_chan *chan)
+{
+ int busy = 0;
+ struct dma_regs *dma_reg;
+
+ dma_reg = (struct dma_regs *)chan->device->dma_reg;
+ /*
+ * if command FIFO's head and tail pointers are equal and
+ * status tail is the same as command, then channel is free
+ */
+ if (dma_reg->cpfhp != dma_reg->cpftp ||
+ dma_reg->cpftp != dma_reg->csftp)
+ busy = 1;
+
+ return busy;
+}
+/*
+ * ppc460ex_chan_append - update the h/w chain in the channel
+ */
+void ppc460ex_chan_append(struct ppc460ex_adma_chan *chan)
+{
+ struct dma_regs *dma_reg;
+ struct ppc460ex_adma_desc_slot *iter;
+ u32 cur_desc;
+ unsigned long flags;
+
+ local_irq_save(flags);
+ dma_reg = (struct dma_regs *)chan->device->dma_reg;
+ cur_desc = ppc460ex_chan_get_current_descriptor(chan);
+
+ if (likely(cur_desc)) {
+ iter = chan_last_sub[chan->device->id];
+ BUG_ON(!iter);
+ } else {
+ /*
+ * first peer
+ */
+ iter = chan_first_cdb[chan->device->id];
+ BUG_ON(!iter);
+ ppc460ex_dma_put_desc(chan, iter);
+ chan->hw_chain_inited = 1;
+ }
+ /*
+ * is there something new to append
+ */
+ if (!iter->hw_next)
+ goto out;
+ /*
+ * flush descriptors from the s/w queue to fifo
+ */
+ list_for_each_entry_continue(iter, &chan->chain, chain_node) {
+ ppc460ex_dma_put_desc(chan, iter);
+ if (!iter->hw_next)
+ break;
+ }
+out:
+ local_irq_restore(flags);
+}
+
+/*
+ * ppc460ex_chan_get_current_descriptor - get the currently executed descriptor
+ */
+u32 ppc460ex_chan_get_current_descriptor(struct ppc460ex_adma_chan *chan)
+{
+ struct dma_regs *dma_reg;
+
+ if (unlikely(!chan->hw_chain_inited))
+ /*
+ * h/w descriptor chain is not initialized yet
+ */
+ return 0;
+
+ dma_reg = (struct dma_regs *)chan->device->dma_reg;
+ return (le32_to_cpu(dma_reg->acpl)) & (~DMA_CDB_MSK);
+}
+/*
+ * ppc460ex_dma_put_desc - put DMA0,1 descriptor to FIFO
+ */
+static void ppc460ex_dma_put_desc(struct ppc460ex_adma_chan *chan,
+ struct ppc460ex_adma_desc_slot *desc)
+{
+ u32 pcdb;
+ struct dma_regs *dma_reg =
+ dma_reg = (struct dma_regs *)chan->device->dma_reg;
+
+ pcdb = desc->phys;
+ if (!test_bit(PPC460EX_DESC_INT, &desc->flags))
+ pcdb |= DMA_CDB_NO_INT;
+
+ chan_last_sub[chan->device->id] = desc;
+
+ out_le32(&dma_reg->cpfpl, pcdb);
+
+}
+/******************************************************************************
+ * ADMA device level
+ *****************************************************************************/
+
+static dma_cookie_t ppc460ex_adma_tx_submit(
+ struct dma_async_tx_descriptor *tx);
+static inline void ppc460ex_adma_set_dest(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ dma_addr_t addr, int index);
+
+static void ppc460ex_adma_pqzero_sum_set_src_mult(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ unsigned char mult, int index, int dst_pos);
+static void ppc460ex_adma_pq_set_src_mult(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ unsigned char mult, int index, int dst_pos);
+static struct dma_async_tx_descriptor *ppc460ex_adma_prep_dma_mq_xor(
+ struct dma_chan *chan, dma_addr_t dst,
+ dma_addr_t *src, unsigned int src_cnt,
+ size_t len, unsigned long flags);
+static struct dma_async_tx_descriptor *ppc460ex_adma_prep_dma_p(
+ struct dma_chan *chan, dma_addr_t *dst, dma_addr_t *src,
+ unsigned int src_cnt, unsigned char *scf,
+ size_t len, unsigned long flags);
+static struct ppc460ex_adma_desc_slot *ppc460ex_dma01_prep_mult(
+ struct ppc460ex_adma_chan *ppc460ex_chan,
+ dma_addr_t *dst, int dst_cnt, dma_addr_t *src, int src_cnt,
+ const unsigned char *scf, size_t len, unsigned long flags);
+static struct ppc460ex_adma_desc_slot *ppc460ex_dma_prep_pq(
+ struct ppc460ex_adma_chan *ppc460ex_chan,
+ dma_addr_t *dst, unsigned int dst_cnt,
+ dma_addr_t *src, unsigned int src_cnt, unsigned char *scf,
+ size_t len, unsigned long flags);
+
+static void ppc460ex_adma_pqxor_set_src_mult(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ unsigned char mult, int index, int dst_pos);
+static void ppc460ex_adma_pqxor_set_src(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ dma_addr_t addr, int index);
+
+static void ppc460ex_adma_pqxor_set_dest(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ dma_addr_t *addrs, unsigned long flags);
+static void ppc460ex_adma_pqzero_sum_set_dest(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ dma_addr_t paddr, dma_addr_t qaddr);
+static void ppc460ex_desc_init_pq(struct ppc460ex_adma_desc_slot *desc,
+ int dst_cnt, int src_cnt, unsigned long flags,
+ unsigned long op);
+static inline void ppc460ex_adma_memcpy_xor_set_src(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ dma_addr_t addr, int index);
+#define DMA_CTRL_FLAGS_LAST DMA_PREP_FENCE
+#define DMA_PREP_ZERO_P (DMA_CTRL_FLAGS_LAST << 1)
+#define DMA_PREP_ZERO_Q (DMA_PREP_ZERO_P << 1)
+/*
+ * ppc460ex_can_rxor - check if the operands may be processed with RXOR
+ */
+int ppc460ex_can_rxor(struct page **srcs, int src_cnt, size_t len)
+{
+ int i, order = 0, state = 0;
+
+ if (unlikely(!(src_cnt > 1)))
+ return 0;
+
+ for (i = 1; i < src_cnt; i++) {
+ char *cur_addr = page_address(srcs[i]);
+ char *old_addr = page_address(srcs[i-1]);
+ switch (state) {
+ case 0:
+ if (cur_addr == old_addr + len) {
+ /* direct RXOR */
+ order = 1;
+ state = 1;
+ } else
+ if (old_addr == cur_addr + len) {
+ /* reverse RXOR */
+ order = -1;
+ state = 1;
+ } else
+ goto out;
+ break;
+ case 1:
+ if ((i == src_cnt-2) ||
+ (order == -1 && cur_addr != old_addr - len)) {
+ order = 0;
+ state = 0;
+ } else
+ if ((cur_addr == old_addr + len*order) ||
+ (cur_addr == old_addr + 2*len) ||
+ (cur_addr == old_addr + 3*len)) {
+ state = 2;
+ } else {
+ order = 0;
+ state = 0;
+ }
+ break;
+ case 2:
+ order = 0;
+ state = 0;
+ break;
+ }
+ }
+
+out:
+ if (state == 1 || state == 2)
+ return 1;
+
+ return 0;
+}
+/*
+ * ppc460ex_adma_device_estimate - estimate the efficiency of processing
+ * the operation given on this channel. It's assumed that 'chan' is
+ * capable to process 'cap' type of operation.
+ * @chan: channel to use
+ * @cap: type of transaction
+ * @dst_lst: array of destination pointers
+ * @src_lst: array of source pointers
+ * @src_cnt: number of source operands
+ * @src_sz: size of each source operand
+ */
+int ppc460ex_adma_estimate(struct dma_chan *chan,
+ enum dma_transaction_type cap, struct page **dst_lst, int dst_cnt,
+ struct page **src_lst, int src_cnt, size_t src_sz)
+{
+ int ef = 1;
+
+ if (cap == DMA_PQ || cap == DMA_PQ_VAL) {
+ /*
+ * If RAID-6 capabilities were not activated don't try
+ * to use them
+ */
+ if (unlikely(!ppc460ex_r6_enabled))
+ return -1;
+ }
+ if (cap == DMA_XOR) {
+ if (unlikely(!ppc460ex_r5_enabled))
+ return -1;
+ }
+ /* in the current implementation of ppc460ex ADMA driver it
+ * makes sense to pick out only pqxor case, because it may be
+ * processed:
+ * (1) either using Biskup method on DMA2;
+ * (2) or on DMA0/1.
+ * Thus we give a favour to (1) if the sources are suitable;
+ * else let it be processed on one of the DMA0/1 engines.
+ */
+ if (cap == DMA_PQ) {
+ if (ppc460ex_can_rxor(src_lst, src_cnt, src_sz))
+ ef = 3; /* override (dma0 + idle) */
+ else
+ ef = 0; /* can't process on DMA2 if !rxor */
+ }
+
+ /*
+ * channel idleness increases the priority
+ */
+ if (likely(ef) &&
+ !ppc460ex_chan_is_busy(to_ppc460ex_adma_chan(chan)))
+ ef++;
+
+ return ef;
+}
+struct dma_chan *
+ppc460ex_async_tx_find_best_channel(enum dma_transaction_type cap,
+ struct page **dst_lst, int dst_cnt, struct page **src_lst,
+ int src_cnt, size_t src_sz)
+{
+ struct dma_chan *best_chan = NULL;
+ struct ppc_dma_chan_ref *ref;
+ int best_rank = -1;
+
+ if (unlikely(!src_sz))
+ return NULL;
+ if (src_sz > PAGE_SIZE) {
+ switch (cap) {
+ case DMA_PQ:
+ if (src_cnt == 1 && dst_lst[1] == src_lst[0])
+ return NULL;
+ if (src_cnt == 2 && dst_lst[1] == src_lst[1])
+ return NULL;
+ break;
+ default:
+ break;
+ }
+ }
+ list_for_each_entry(ref, &ppc460ex_adma_chan_list, node) {
+ if (dma_has_cap(cap, ref->chan->device->cap_mask)) {
+ int rank;
+
+ rank = ppc460ex_adma_estimate(ref->chan, cap, dst_lst,
+ dst_cnt, src_lst, src_cnt, src_sz);
+ if (rank > best_rank) {
+ best_rank = rank;
+ best_chan = ref->chan;
+ }
+ }
+ }
+ return best_chan;
+}
+/*
+ * ppc460ex_get_group_entry - get group entry with index idx
+ * @tdesc: is the last allocated slot in the group.
+ */
+struct ppc460ex_adma_desc_slot *
+ppc460ex_get_group_entry(struct ppc460ex_adma_desc_slot *tdesc,
+ u32 entry_idx)
+{
+ struct ppc460ex_adma_desc_slot *iter = tdesc->group_head;
+ int i = 0;
+
+ if (entry_idx < 0 || entry_idx >= (tdesc->src_cnt + tdesc->dst_cnt)) {
+ pr_debug("%s: entry_idx %d, src_cnt %d, dst_cnt %d\n",
+ __func__, entry_idx, tdesc->src_cnt,
+ tdesc->dst_cnt);
+ BUG();
+ }
+ list_for_each_entry(iter, &tdesc->group_list, chain_node) {
+ if (i++ == entry_idx)
+ break;
+ }
+ return iter;
+}
+/*
+ * ppc460ex_adma_free_slots - flags descriptor slots for reuse
+ * @slot: Slot to free
+ * Caller must hold &ppc460ex_chan->lock while calling this function
+ */
+void ppc460ex_adma_free_slots(struct ppc460ex_adma_desc_slot *slot,
+ struct ppc460ex_adma_chan *chan)
+{
+ int stride = slot->slots_per_op;
+
+ while (stride--) {
+ slot->slots_per_op = 0;
+ slot = list_entry(slot->slot_node.next,
+ struct ppc460ex_adma_desc_slot,
+ slot_node);
+ }
+}
+
+void ppc460ex_adma_unmap(struct ppc460ex_adma_chan *chan,
+ struct ppc460ex_adma_desc_slot *desc)
+{
+ u32 src_cnt, dst_cnt;
+ dma_addr_t addr;
+ /*
+ * get the number of sources & destination
+ * included in this descriptor and unmap
+ * them all
+ */
+ src_cnt = ppc460ex_desc_get_src_num(desc, chan);
+ dst_cnt = ppc460ex_desc_get_dst_num(desc, chan);
+
+ /*
+ * unmap destinations
+ */
+ if (!(desc->async_tx.flags & DMA_COMPL_SKIP_DEST_UNMAP)) {
+ while (dst_cnt--) {
+ addr = ppc460ex_desc_get_dest_addr(
+ desc, chan, dst_cnt);
+ dma_unmap_page(chan->device->dev,
+ addr, desc->unmap_len,
+ DMA_FROM_DEVICE);
+ }
+ }
+
+ /*
+ * unmap sources
+ */
+ if (!(desc->async_tx.flags & DMA_COMPL_SKIP_SRC_UNMAP)) {
+ while (src_cnt--) {
+ addr = ppc460ex_desc_get_src_addr(
+ desc, chan, src_cnt);
+ dma_unmap_page(chan->device->dev,
+ addr, desc->unmap_len,
+ DMA_TO_DEVICE);
+ }
+ }
+
+}
+/*
+ * ppc460ex_adma_run_tx_complete_actions - call functions to be called
+ * upon complete
+ */
+dma_cookie_t ppc460ex_adma_run_tx_complete_actions(
+ struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan,
+ dma_cookie_t cookie)
+{
+ int i;
+
+ BUG_ON(desc->async_tx.cookie < 0);
+ if (desc->async_tx.cookie > 0) {
+ cookie = desc->async_tx.cookie;
+ desc->async_tx.cookie = 0;
+
+ /*
+ * call the callback (must not sleep or submit new
+ * operations to this channel)
+ */
+ if (desc->async_tx.callback)
+ desc->async_tx.callback(
+ desc->async_tx.callback_param);
+
+ /* unmap dma addresses
+ * (unmap_single vs unmap_page?)
+ *
+ * actually, ppc's dma_unmap_page() functions are empty, so
+ * the following code is just for the sake of completeness
+ */
+ if (chan && chan->needs_unmap && desc->group_head &&
+ desc->unmap_len) {
+ struct ppc460ex_adma_desc_slot *unmap =
+ desc->group_head;
+ /*
+ * assume 1 slot per op always
+ */
+ u32 slot_count = unmap->slot_cnt;
+
+ /*
+ * Run through the group list and unmap addresses
+ */
+ for (i = 0; i < slot_count; i++) {
+ BUG_ON(!unmap);
+ ppc460ex_adma_unmap(chan, unmap);
+ unmap = unmap->hw_next;
+ }
+ }
+ }
+
+ /*
+ * run dependent operations
+ */
+ dma_run_dependencies(&desc->async_tx);
+
+ return cookie;
+}
+/*
+ * ppc460ex_adma_clean_slot - clean up CDB slot (if ack is set)
+ */
+int ppc460ex_adma_clean_slot(struct ppc460ex_adma_desc_slot *desc,
+ struct ppc460ex_adma_chan *chan)
+{
+ struct dma_cdb *cdb ;
+ /*
+ * the client is allowed to attach dependent operations
+ * until 'ack' is set
+ */
+ if (!async_tx_test_ack(&desc->async_tx))
+ return 0;
+
+ /*
+ * leave the last descriptor in the chain
+ * so we can append to it
+ */
+ if (list_is_last(&desc->chain_node, &chan->chain) ||
+ desc->phys == ppc460ex_chan_get_current_descriptor(chan))
+ return 1;
+
+ /* our DMA interrupt handler clears opc field of
+ * each processed descriptor. For all types of
+ * operations except for ZeroSum we do not actually
+ * need ack from the interrupt handler. ZeroSum is a
+ * gtcial case since the result of this operation
+ * is available from the handler only, so if we see
+ * such type of descriptor (which is unprocessed yet)
+ * then leave it in chain.
+ */
+ cdb = desc->hw_desc;
+ if (cdb->opc == DMA_CDB_OPC_DCHECK128)
+ return 1;
+
+ dev_dbg(chan->device->common.dev, "\tfree slot %lx: %d stride: %d\n",
+ (ulong)desc->phys, desc->idx, desc->slots_per_op);
+
+ list_del(&desc->chain_node);
+ ppc460ex_adma_free_slots(desc, chan);
+ return 0;
+}
+/*
+ *__ppc460ex_adma_slot_cleanup - this is the common clean-up routine
+ *which runs through the channel CDBs list until reach the descriptor
+ *currently processed. When routine determines that all CDBs of group
+ *are completed then corresponding callbacks (if any) are called and slots
+ *are freed.
+ */
+void __ppc460ex_adma_slot_cleanup(struct ppc460ex_adma_chan *chan)
+{
+ struct ppc460ex_adma_desc_slot *iter, *_iter, *group_start = NULL;
+ dma_cookie_t cookie = 0;
+ u32 current_desc = ppc460ex_chan_get_current_descriptor(chan);
+ int busy = ppc460ex_chan_is_busy(chan);
+ int seen_current = 0, slot_cnt = 0, slots_per_op = 0;
+
+ dev_dbg(chan->device->common.dev, "ppc460ex adma%d: %s\n",
+ chan->device->id, __func__);
+
+ if (!current_desc) {
+ /*
+ * There were no transactions yet, so
+ * nothing to clean
+ */
+ return;
+ }
+
+ /*
+ * free completed slots from the chain starting with
+ * the oldest descriptor
+ */
+ list_for_each_entry_safe(iter, _iter, &chan->chain,
+ chain_node) {
+ dev_dbg(chan->device->common.dev, "\tcookie: %d slot: %d "
+ "busy: %d this_desc: %#x next_desc: %#x cur: %#x ack: %d\n",
+ iter->async_tx.cookie, iter->idx, busy, (u32)iter->phys,
+ ppc460ex_desc_get_link(iter, chan), current_desc,
+ async_tx_test_ack(&iter->async_tx));
+ prefetch(_iter);
+ prefetch(&_iter->async_tx);
+
+ /*
+ * do not advance past the current descriptor loaded into the
+ * hardware channel,subsequent descriptors are either in process
+ * or have not been submitted
+ */
+ if (seen_current)
+ break;
+
+ /*
+ * stop the search if we reach the current descriptor and the
+ * channel is busy, or if it appears that the current descriptor
+ * needs to be re-read (i.e. has been appended to)
+ */
+ if (iter->phys == current_desc) {
+ BUG_ON(seen_current++);
+ if (busy || ppc460ex_desc_get_link(iter, chan)) {
+ /* not all descriptors of the group have
+ * been completed; exit.
+ */
+ break;
+ }
+ }
+
+ /*
+ * detect the start of a group transaction
+ */
+ if (!slot_cnt && !slots_per_op) {
+ slot_cnt = iter->slot_cnt;
+ slots_per_op = iter->slots_per_op;
+ if (slot_cnt <= slots_per_op) {
+ slot_cnt = 0;
+ slots_per_op = 0;
+ }
+ }
+
+ if (slot_cnt) {
+ if (!group_start)
+ group_start = iter;
+ slot_cnt -= slots_per_op;
+ }
+
+ /*
+ * all the members of a group are complete
+ */
+ if (slots_per_op != 0 && slot_cnt == 0) {
+ struct ppc460ex_adma_desc_slot *grp_iter, *_grp_iter;
+ int end_of_chain = 0;
+
+ /*
+ * clean up the group
+ */
+ slot_cnt = group_start->slot_cnt;
+ grp_iter = group_start;
+ list_for_each_entry_safe_from(grp_iter, _grp_iter,
+ &chan->chain, chain_node) {
+
+ cookie = ppc460ex_adma_run_tx_complete_actions(
+ grp_iter, chan, cookie);
+
+ slot_cnt -= slots_per_op;
+ end_of_chain = ppc460ex_adma_clean_slot(
+ grp_iter, chan);
+ if (end_of_chain && slot_cnt) {
+ /*
+ * Should wait for ZeroSum complete
+ */
+ if (cookie > 0)
+ chan->completed_cookie = cookie;
+ return;
+ }
+
+ if (slot_cnt == 0 || end_of_chain)
+ break;
+ }
+
+ /*
+ * the group should be complete at this point
+ */
+ BUG_ON(slot_cnt);
+
+ slots_per_op = 0;
+ group_start = NULL;
+ if (end_of_chain)
+ break;
+ else
+ continue;
+ /*
+ * wait for group completion
+ */
+ } else if (slots_per_op)
+ continue;
+
+ cookie = ppc460ex_adma_run_tx_complete_actions(iter, chan,
+ cookie);
+
+ if (ppc460ex_adma_clean_slot(iter, chan))
+ break;
+ }
+
+ BUG_ON(!seen_current);
+}
+/*
+ * ppc460ex_adma_tasklet - clean up watch-dog initiator
+ */
+void ppc460ex_adma_tasklet(unsigned long data)
+{
+ struct ppc460ex_adma_chan *chan = (struct ppc460ex_adma_chan *) data;
+ spin_lock_nested(&chan->lock, SINGLE_DEPTH_NESTING);
+ __ppc460ex_adma_slot_cleanup(chan);
+ spin_unlock(&chan->lock);
+}
+
+/*
+ * ppc460ex_adma_slot_cleanup - clean up scheduled initiator
+ */
+void ppc460ex_adma_slot_cleanup(struct ppc460ex_adma_chan *chan)
+{
+ spin_lock_bh(&chan->lock);
+ __ppc460ex_adma_slot_cleanup(chan);
+ spin_unlock_bh(&chan->lock);
+}
+/*
+ * ppc460ex_adma_alloc_slots - allocate free slots (if any)
+ */
+struct ppc460ex_adma_desc_slot *ppc460ex_adma_alloc_slots(
+ struct ppc460ex_adma_chan *chan, int num_slots,
+ int slots_per_op)
+{
+ struct ppc460ex_adma_desc_slot *iter = NULL, *_iter;
+ struct ppc460ex_adma_desc_slot *alloc_start = NULL;
+ struct list_head chain = LIST_HEAD_INIT(chain);
+ int slots_found, retry = 0;
+
+ BUG_ON(!num_slots || !slots_per_op);
+ /* start search from the last allocated descrtiptor
+ * if a contiguous allocation can not be found start searching
+ * from the beginning of the list
+ */
+retry:
+ slots_found = 0;
+ if (retry == 0)
+ iter = chan->last_used;
+ else
+ iter = list_entry(&chan->all_slots,
+ struct ppc460ex_adma_desc_slot,
+ slot_node);
+ list_for_each_entry_safe_continue(iter, _iter, &chan->all_slots,
+ slot_node) {
+ prefetch(_iter);
+ prefetch(&_iter->async_tx);
+ if (iter->slots_per_op) {
+ slots_found = 0;
+ continue;
+ }
+
+ /*
+ * start the allocation if the slot is correctly aligned
+ */
+ if (!slots_found++)
+ alloc_start = iter;
+ if (slots_found == num_slots) {
+ struct ppc460ex_adma_desc_slot *alloc_tail = NULL;
+ struct ppc460ex_adma_desc_slot *last_used = NULL;
+
+ iter = alloc_start;
+ while (num_slots) {
+ int i;
+ /*
+ * pre-ack all but the last descriptor
+ */
+ if (num_slots != slots_per_op)
+ async_tx_ack(&iter->async_tx);
+
+ list_add_tail(&iter->chain_node, &chain);
+ alloc_tail = iter;
+ iter->async_tx.cookie = 0;
+ iter->hw_next = NULL;
+ iter->flags = 0;
+ iter->slot_cnt = num_slots;
+ iter->xor_check_result = NULL;
+ for (i = 0; i < slots_per_op; i++) {
+ iter->slots_per_op = slots_per_op - i;
+ last_used = iter;
+ iter = list_entry(iter->slot_node.next,
+ struct ppc460ex_adma_desc_slot,
+ slot_node);
+ }
+ num_slots -= slots_per_op;
+ }
+ alloc_tail->group_head = alloc_start;
+ alloc_tail->async_tx.cookie = -EBUSY;
+ list_splice(&chain, &alloc_tail->group_list);
+ chan->last_used = last_used;
+ return alloc_tail;
+ }
+ }
+ if (!retry++)
+ goto retry;
+
+ /*
+ * try to free some slots if the allocation fails
+ */
+ tasklet_schedule(&chan->irq_tasklet);
+ return NULL;
+}
+/*
+ * ppc460ex_adma_alloc_chan_resources - allocate pools for CDB slots
+ */
+int ppc460ex_adma_alloc_chan_resources(struct dma_chan *chan)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan;
+ struct ppc460ex_adma_desc_slot *slot = NULL;
+ char *hw_desc;
+ int i, db_sz;
+ int init;
+
+ ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ init = ppc460ex_chan->slots_allocated ? 0 : 1;
+ chan->chan_id = ppc460ex_chan->device->id;
+
+ /*
+ * Allocate descriptor slots
+ */
+ i = ppc460ex_chan->slots_allocated;
+ db_sz = sizeof(struct dma_cdb);
+
+ for (; i < (ppc460ex_chan->device->pool_size / db_sz); i++) {
+ slot = kzalloc(sizeof(struct ppc460ex_adma_desc_slot),
+ GFP_KERNEL);
+ if (!slot) {
+ dev_dbg(ppc460ex_chan->device->common.dev,
+ "EX/GT ADMA Channel only initialized"
+ " %d descriptor slots", i--);
+ break;
+ }
+ hw_desc = (char *) ppc460ex_chan->device->dma_desc_pool_virt;
+ slot->hw_desc = (void *) &hw_desc[i * db_sz];
+ dma_async_tx_descriptor_init(&slot->async_tx, chan);
+ slot->async_tx.tx_submit = ppc460ex_adma_tx_submit;
+ INIT_LIST_HEAD(&slot->chain_node);
+ INIT_LIST_HEAD(&slot->slot_node);
+ INIT_LIST_HEAD(&slot->group_list);
+ slot->phys = ppc460ex_chan->device->dma_desc_pool + i * db_sz;
+ slot->idx = i;
+
+ spin_lock_bh(&ppc460ex_chan->lock);
+ ppc460ex_chan->slots_allocated++;
+ list_add_tail(&slot->slot_node, &ppc460ex_chan->all_slots);
+ spin_unlock_bh(&ppc460ex_chan->lock);
+ }
+ if (i && !ppc460ex_chan->last_used) {
+ ppc460ex_chan->last_used =
+ list_entry(ppc460ex_chan->all_slots.next,
+ struct ppc460ex_adma_desc_slot,
+ slot_node);
+ }
+
+ dev_dbg(ppc460ex_chan->device->common.dev,
+ "ppc460ex adma%d: allocated %d descriptor slots\n",
+ ppc460ex_chan->device->id, i);
+
+ /*
+ * initialize the channel and the chain with a null operation
+ */
+ if (init) {
+ ppc460ex_chan->hw_chain_inited = 0;
+ /*
+ * Use WXOR for self-testing
+ */
+ if (!ppc460ex_r6_tchan)
+ ppc460ex_r6_tchan = ppc460ex_chan;
+ ppc460ex_chan->needs_unmap = 1;
+ }
+ return (i > 0) ? i : -ENOMEM;
+}
+/*
+ * ppc460ex_desc_set_link - set the address of descriptor following this
+ * descriptor in chain
+ */
+static inline void ppc460ex_desc_set_link(struct ppc460ex_adma_chan *chan,
+ struct ppc460ex_adma_desc_slot *prev_desc,
+ struct ppc460ex_adma_desc_slot *next_desc)
+{
+ unsigned long flags;
+
+ if (unlikely(!prev_desc || !next_desc ||
+ (prev_desc->hw_next && prev_desc->hw_next != next_desc))) {
+ /* If previous next is overwritten something is wrong.
+ * though we may refetch from append to initiate list
+ * processing; in this case - it's ok.
+ */
+ dev_dbg(chan->device->common.dev,
+ "%s: prev_desc=0x%p; next_desc=0x%p; "
+ "prev->hw_next=0x%p\n", __func__, prev_desc,
+ next_desc, prev_desc ? prev_desc->hw_next : 0);
+ BUG();
+ }
+
+ local_irq_save(flags);
+
+ /*
+ * do s/w chaining both for DMA and XOR descriptors
+ */
+ prev_desc->hw_next = next_desc;
+ local_irq_restore(flags);
+}
+/*
+ * ppc460ex_adma_check_threshold - append CDBs to h/w chain if threshold
+ * has been achieved
+ */
+static void ppc460ex_adma_check_threshold(struct ppc460ex_adma_chan *chan)
+{
+ dev_dbg(chan->device->common.dev, "ppc460ex adma%d: pending: %d\n",
+ chan->device->id, chan->pending);
+
+ if (chan->pending >= PPC460EX_ADMA_THRESHOLD) {
+ chan->pending = 0;
+ ppc460ex_chan_append(chan);
+ }
+}
+/*
+ * ppc460ex_adma_tx_submit - submit new descriptor group to the channel
+ * (it's not necessary that descriptors will be submitted to the h/w
+ * chains too right now)
+ */
+static dma_cookie_t ppc460ex_adma_tx_submit(struct dma_async_tx_descriptor *tx)
+{
+ struct ppc460ex_adma_desc_slot *sw_desc = tx_to_ppc460ex_adma_slot(tx);
+ struct ppc460ex_adma_chan *chan = to_ppc460ex_adma_chan(tx->chan);
+ struct ppc460ex_adma_desc_slot *group_start, *old_chain_tail;
+ int slot_cnt;
+ int slots_per_op;
+ dma_cookie_t cookie;
+
+ group_start = sw_desc->group_head;
+ slot_cnt = group_start->slot_cnt;
+ slots_per_op = group_start->slots_per_op;
+
+ spin_lock_bh(&chan->lock);
+
+ cookie = ppc460ex_desc_assign_cookie(chan, sw_desc);
+
+ if (unlikely(list_empty(&chan->chain))) {
+ /*
+ *first peer
+ */
+ list_splice_init(&sw_desc->group_list, &chan->chain);
+ chan_first_cdb[chan->device->id] = group_start;
+ } else {
+ /*
+ * isn't first peer, bind CDBs to chain
+ */
+ old_chain_tail = list_entry(chan->chain.prev,
+ struct ppc460ex_adma_desc_slot, chain_node);
+ list_splice_init(&sw_desc->group_list,
+ &old_chain_tail->chain_node);
+ /*
+ * fix up the hardware chain
+ */
+ ppc460ex_desc_set_link(chan, old_chain_tail, group_start);
+ }
+
+ /*
+ * increment the pending count by the number of operations
+ */
+ chan->pending += slot_cnt / slots_per_op;
+ ppc460ex_adma_check_threshold(chan);
+ spin_unlock_bh(&chan->lock);
+
+ dev_dbg(chan->device->common.dev,
+ "ppc460ex adma%d: %s cookie: %d slot: %d tx %p\n",
+ chan->device->id, __func__,
+ sw_desc->async_tx.cookie, sw_desc->idx, sw_desc);
+ return cookie;
+}
+/*
+ * ppc460ex_adma_prep_dma_interrupt - prepare CDB for a pseudo DMA operation
+ */
+struct dma_async_tx_descriptor *ppc460ex_adma_prep_dma_interrupt(
+ struct dma_chan *chan, unsigned long flags)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ struct ppc460ex_adma_desc_slot *sw_desc, *group_start;
+ int slot_cnt, slots_per_op;
+
+ dev_dbg(ppc460ex_chan->device->common.dev,
+ "ppc460ex adma%d: %s\n", ppc460ex_chan->device->id,
+ __func__);
+
+ spin_lock_bh(&ppc460ex_chan->lock);
+ slot_cnt = slots_per_op = 1;
+ sw_desc = ppc460ex_adma_alloc_slots(ppc460ex_chan, slot_cnt,
+ slots_per_op);
+ if (sw_desc) {
+ group_start = sw_desc->group_head;
+ ppc460ex_desc_init_interrupt(group_start, ppc460ex_chan);
+ group_start->unmap_len = 0;
+ sw_desc->async_tx.flags = flags;
+ }
+ spin_unlock_bh(&ppc460ex_chan->lock);
+
+ return sw_desc ? &sw_desc->async_tx : NULL;
+}
+/*
+ * ppc460ex_adma_prep_dma_pqzero_sum - prepare CDB group for
+ * a PQ_ZERO_SUM operation
+ */
+struct dma_async_tx_descriptor *ppc460ex_adma_prep_dma_pqzero_sum(
+ struct dma_chan *chan, dma_addr_t *pq, dma_addr_t *src,
+ unsigned int src_cnt, const unsigned char *scf, size_t len,
+ enum sum_check_flags *pqres, unsigned long flags)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan;
+ struct ppc460ex_adma_desc_slot *sw_desc, *iter;
+ dma_addr_t pdest, qdest;
+ int slot_cnt, slots_per_op, idst, dst_cnt;
+
+ ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+
+ if (flags & DMA_PREP_PQ_DISABLE_P)
+ pdest = 0;
+ else
+ pdest = pq[0];
+
+ if (flags & DMA_PREP_PQ_DISABLE_Q)
+ qdest = 0;
+ else
+ qdest = pq[1];
+
+
+ /* Always use WXOR for P/Q calculations (two destinations).
+ * Need 1 or 2 extra slots to verify results are zero.
+ */
+ idst = dst_cnt = (pdest && qdest) ? 2 : 1;
+
+ /* One additional slot per destination to clone P/Q
+ * before calculation (we have to preserve destinations).
+ */
+ slot_cnt = src_cnt + dst_cnt * 2;
+ slots_per_op = 1;
+
+ spin_lock_bh(&ppc460ex_chan->lock);
+ sw_desc = ppc460ex_adma_alloc_slots(ppc460ex_chan, slot_cnt,
+ slots_per_op);
+ if (sw_desc) {
+ ppc460ex_desc_init_pqzero_sum(sw_desc, dst_cnt, src_cnt);
+
+ /*
+ * Setup byte count for each slot just allocated
+ */
+ sw_desc->async_tx.flags = flags;
+ list_for_each_entry(iter, &sw_desc->group_list, chain_node) {
+ ppc460ex_desc_set_byte_count(iter, ppc460ex_chan,
+ len);
+ iter->unmap_len = len;
+ }
+
+ if (pdest) {
+ struct dma_cdb *hw_desc;
+ struct ppc460ex_adma_chan *chan;
+
+ iter = sw_desc->group_head;
+ chan = to_ppc460ex_adma_chan(iter->async_tx.chan);
+ memset(iter->hw_desc, 0, sizeof(struct dma_cdb));
+ iter->hw_next = list_entry(iter->chain_node.next,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = DMA_CDB_OPC_MV_SG1_SG2;
+ iter->src_cnt = 0;
+ iter->dst_cnt = 0;
+ ppc460ex_desc_set_dest_addr(iter, chan, 0,
+ ppc460ex_chan->pdest, 0);
+ ppc460ex_desc_set_src_addr(iter, chan, 0, 0, pdest);
+ ppc460ex_desc_set_byte_count(iter, ppc460ex_chan,
+ len);
+ iter->unmap_len = 0;
+ /*
+ * override pdest to preserve original P
+ */
+ pdest = ppc460ex_chan->pdest;
+ }
+ if (qdest) {
+ struct dma_cdb *hw_desc;
+ struct ppc460ex_adma_chan *chan;
+
+ iter = list_first_entry(&sw_desc->group_list,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ chan = to_ppc460ex_adma_chan(iter->async_tx.chan);
+
+ if (pdest) {
+ iter = list_entry(iter->chain_node.next,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ }
+
+ memset(iter->hw_desc, 0, sizeof(struct dma_cdb));
+ iter->hw_next = list_entry(iter->chain_node.next,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = DMA_CDB_OPC_MV_SG1_SG2;
+ iter->src_cnt = 0;
+ iter->dst_cnt = 0;
+ ppc460ex_desc_set_dest_addr(iter, chan, 0,
+ ppc460ex_chan->qdest, 0);
+ ppc460ex_desc_set_src_addr(iter, chan, 0, 0, qdest);
+ ppc460ex_desc_set_byte_count(iter, ppc460ex_chan,
+ len);
+ iter->unmap_len = 0;
+ /*
+ * override qdest to preserve original Q
+ */
+ qdest = ppc460ex_chan->qdest;
+ }
+
+ /*
+ * Setup destinations for P/Q ops
+ */
+ ppc460ex_adma_pqzero_sum_set_dest(sw_desc, pdest, qdest);
+
+ /*
+ * Setup zero QWORDs into DCHECK CDBs
+ */
+ idst = dst_cnt;
+ list_for_each_entry_reverse(iter, &sw_desc->group_list,
+ chain_node) {
+ /*
+ * The last CDB corresponds to Q-parity check,
+ * the one before last CDB corresponds
+ * P-parity check
+ */
+ if (idst == DMA_DEST_MAX_NUM) {
+ if (idst == dst_cnt) {
+ set_bit(PPC460EX_DESC_QCHECK,
+ &iter->flags);
+ } else {
+ set_bit(PPC460EX_DESC_PCHECK,
+ &iter->flags);
+ }
+ } else {
+ if (qdest) {
+ set_bit(PPC460EX_DESC_QCHECK,
+ &iter->flags);
+ } else {
+ set_bit(PPC460EX_DESC_PCHECK,
+ &iter->flags);
+ }
+ }
+ iter->xor_check_result = pqres;
+
+ /*
+ * set it to zero, if check fail then result will
+ * be updated
+ */
+ *iter->xor_check_result = 0;
+ ppc460ex_desc_set_dcheck(iter, ppc460ex_chan,
+ ppc460ex_qword);
+
+ if (!(--dst_cnt))
+ break;
+ }
+
+ /*
+ * Setup sources and mults for P/Q ops
+ */
+ list_for_each_entry_continue_reverse(iter, &sw_desc->group_list,
+ chain_node) {
+ struct ppc460ex_adma_chan *chan;
+ u32 mult_dst;
+
+ chan = to_ppc460ex_adma_chan(iter->async_tx.chan);
+ ppc460ex_desc_set_src_addr(iter, chan, 0,
+ DMA_CUED_XOR_HB,
+ src[src_cnt - 1]);
+ if (qdest) {
+ mult_dst = (dst_cnt - 1) ? DMA_CDB_SG_DST2 :
+ DMA_CDB_SG_DST1;
+ ppc460ex_desc_set_src_mult(iter, chan,
+ DMA_CUED_MULT1_OFF,
+ mult_dst,
+ scf[src_cnt - 1]);
+ }
+ if (!(--src_cnt))
+ break;
+ }
+ }
+ spin_unlock_bh(&ppc460ex_chan->lock);
+ return sw_desc ? &sw_desc->async_tx : NULL;
+}
+/*
+ * ppc460ex_adma_prep_dma_xor_zero_sum - prepare CDB group for
+ * XOR ZERO_SUM operation
+ */
+static struct dma_async_tx_descriptor *ppc460ex_adma_prep_dma_xor_zero_sum(
+ struct dma_chan *chan, dma_addr_t *src, unsigned int src_cnt,
+ size_t len, enum sum_check_flags *result, unsigned long flags)
+{
+ struct dma_async_tx_descriptor *tx;
+ dma_addr_t pq[2];
+
+ /*
+ * validate P, disable Q
+ */
+ pq[0] = src[0];
+ pq[1] = 0;
+ flags |= DMA_PREP_PQ_DISABLE_Q;
+
+ tx = ppc460ex_adma_prep_dma_pqzero_sum(chan, pq, &src[1],
+ src_cnt - 1, 0, len,
+ result, flags);
+ return tx;
+}
+/*
+ * ppc460ex_desc_init_pq - initialize the descriptor for PQ_XOR operation
+ */
+static inline void ppc460ex_desc_init_pq(struct ppc460ex_adma_desc_slot *desc,
+ int dst_cnt, int src_cnt, unsigned long flags,
+ unsigned long op)
+{
+ struct dma_cdb *hw_desc;
+ struct ppc460ex_adma_desc_slot *iter;
+ u8 dopc;
+
+
+ /*
+ * Common initialization of a PQ descriptors chain
+ */
+
+ set_bits(op, &desc->flags);
+ desc->src_cnt = src_cnt;
+ desc->dst_cnt = dst_cnt;
+
+ dopc = (desc->dst_cnt == DMA_DEST_MAX_NUM) ?
+ DMA_CDB_OPC_MULTICAST : DMA_CDB_OPC_MV_SG1_SG2;
+
+ list_for_each_entry(iter, &desc->group_list, chain_node) {
+ hw_desc = iter->hw_desc;
+ memset(iter->hw_desc, 0, sizeof(struct dma_cdb));
+
+ if (likely(!list_is_last(&iter->chain_node,
+ &desc->group_list))) {
+ /*
+ * set 'next' pointer
+ */
+ iter->hw_next = list_entry(iter->chain_node.next,
+ struct ppc460ex_adma_desc_slot, chain_node);
+ clear_bit(PPC460EX_DESC_INT, &iter->flags);
+ } else {
+ /* this is the last descriptor.
+ * this slot will be pasted from ADMA level
+ * each time it wants to configure parameters
+ * of the transaction (src, dst, ...)
+ */
+ iter->hw_next = NULL;
+ if (flags & DMA_PREP_INTERRUPT)
+ set_bit(PPC460EX_DESC_INT, &iter->flags);
+ else
+ clear_bit(PPC460EX_DESC_INT, &iter->flags);
+ }
+ }
+
+ /* Set OPS depending on WXOR/RXOR type of operation */
+ if (!test_bit(PPC460EX_DESC_RXOR, &desc->flags)) {
+ /* This is a WXOR only chain:
+ * - first descriptors are for zeroing destinations
+ * if PPC460EX_ZERO_P/Q set;
+ * - descriptors remained are for GF-XOR operations.
+ */
+ iter = list_first_entry(&desc->group_list,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+
+ if (test_bit(PPC460EX_ZERO_P, &desc->flags)) {
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = DMA_CDB_OPC_MV_SG1_SG2;
+ iter = list_first_entry(&iter->chain_node,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ }
+
+ if (test_bit(PPC460EX_ZERO_Q, &desc->flags)) {
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = DMA_CDB_OPC_MV_SG1_SG2;
+ iter = list_first_entry(&iter->chain_node,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ }
+
+ list_for_each_entry_from(iter, &desc->group_list, chain_node) {
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = dopc;
+ }
+ } else {
+ /* This is either RXOR-only or mixed RXOR/WXOR
+ * The first 1 or 2 slots in chain are always RXOR,
+ * if need to calculate P & Q, then there are two
+ * RXOR slots; if only P or only Q, then there is one
+ */
+ iter = list_first_entry(&desc->group_list,
+ struct ppc460ex_adma_desc_slot, chain_node);
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = DMA_CDB_OPC_MV_SG1_SG2;
+
+ if (desc->dst_cnt == DMA_DEST_MAX_NUM) {
+ iter = list_first_entry(&iter->chain_node,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = DMA_CDB_OPC_MV_SG1_SG2;
+ }
+
+ /*
+ * The remain descs (if any) are WXORs
+ */
+ if (test_bit(PPC460EX_DESC_WXOR, &desc->flags)) {
+ iter = list_first_entry(&iter->chain_node,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ list_for_each_entry_from(iter, &desc->group_list,
+ chain_node) {
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = dopc;
+ }
+ }
+ }
+}
+
+/*
+ * ppc460ex_adma_prep_dma_memcpy - prepare CDB for a MEMCPY operation
+ */
+static struct dma_async_tx_descriptor *ppc460ex_adma_prep_dma_memcpy(
+ struct dma_chan *chan, dma_addr_t dma_dest,
+ dma_addr_t dma_src, size_t len, unsigned long flags)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ struct ppc460ex_adma_desc_slot *sw_desc, *group_start;
+ int slot_cnt, slots_per_op;
+ if (unlikely(!len))
+ return NULL;
+ BUG_ON(unlikely(len > PPC460EX_ADMA_DMA_MAX_BYTE_COUNT));
+ spin_lock_bh(&ppc460ex_chan->lock);
+ dev_dbg(ppc460ex_chan->device->common.dev,
+ "ppc460ex adma%d: %s len: %u int_en %d\n",
+ ppc460ex_chan->device->id, __func__,
+ len, flags & DMA_PREP_INTERRUPT ? 1 : 0);
+
+ slot_cnt = slots_per_op = 1;
+ sw_desc = ppc460ex_adma_alloc_slots(ppc460ex_chan, slot_cnt,
+ slots_per_op);
+ if (sw_desc) {
+ group_start = sw_desc->group_head;
+ ppc460ex_desc_init_memcpy(group_start, flags);
+ ppc460ex_adma_set_dest(group_start, dma_dest, 0);
+ ppc460ex_adma_memcpy_xor_set_src(group_start, dma_src, 0);
+ ppc460ex_desc_set_byte_count(group_start, ppc460ex_chan, len);
+ sw_desc->unmap_len = len;
+ sw_desc->async_tx.flags = flags;
+ }
+ spin_unlock_bh(&ppc460ex_chan->lock);
+ return sw_desc ? &sw_desc->async_tx : NULL;
+}
+
+/*
+ * ppc460ex_adma_prep_dma_memset - prepare CDB for a MEMSET operation
+ */
+static struct dma_async_tx_descriptor *ppc460ex_adma_prep_dma_memset(
+ struct dma_chan *chan, dma_addr_t dma_dest, int value,
+ size_t len, unsigned long flags)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ struct ppc460ex_adma_desc_slot *sw_desc, *group_start;
+ int slot_cnt, slots_per_op;
+ if (unlikely(!len))
+ return NULL;
+ BUG_ON(unlikely(len > PPC460EX_ADMA_DMA_MAX_BYTE_COUNT));
+
+ spin_lock_bh(&ppc460ex_chan->lock);
+
+ dev_dbg(ppc460ex_chan->device->common.dev,
+ "ppc460ex adma%d: %s cal: %u len: %u int_en %d\n",
+ ppc460ex_chan->device->id, __func__, value, len,
+ flags & DMA_PREP_INTERRUPT ? 1 : 0);
+
+ slot_cnt = slots_per_op = 1;
+ sw_desc = ppc460ex_adma_alloc_slots(ppc460ex_chan, slot_cnt,
+ slots_per_op);
+ if (sw_desc) {
+ group_start = sw_desc->group_head;
+ ppc460ex_desc_init_memset(group_start, value, flags);
+ ppc460ex_adma_set_dest(group_start, dma_dest, 0);
+ ppc460ex_desc_set_byte_count(group_start, ppc460ex_chan, len);
+ sw_desc->unmap_len = len;
+ sw_desc->async_tx.flags = flags;
+ }
+ spin_unlock_bh(&ppc460ex_chan->lock);
+
+ return sw_desc ? &sw_desc->async_tx : NULL;
+}
+static void ppc460ex_adma_pq_zero_op(struct ppc460ex_adma_desc_slot *iter,
+ struct ppc460ex_adma_chan *chan, dma_addr_t addr)
+{
+ /*
+ * To clear destinations update the descriptor
+ * (P or Q depending on index) as follows:
+ * addr is destination (0 corresponds to SG2):
+ */
+ ppc460ex_desc_set_dest_addr(iter, chan, DMA_CUED_XOR_BASE, addr, 0);
+
+ /* ... and the addr is source: */
+ ppc460ex_desc_set_src_addr(iter, chan, 0, DMA_CUED_XOR_HB, addr);
+
+}
+/*
+ * ppc460ex_adma_set_dest - set destination address into descriptor
+ */
+static inline void ppc460ex_adma_set_dest(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ dma_addr_t addr, int index)
+{
+ struct ppc460ex_adma_chan *chan;
+
+ chan = to_ppc460ex_adma_chan(sw_desc->async_tx.chan);
+ BUG_ON(index >= sw_desc->dst_cnt);
+
+ /*
+ * to do: support transfers lengths
+ * PPC460EX_ADMA_DMA/XOR_MAX_BYTE_COUNT
+ */
+ ppc460ex_desc_set_dest_addr(sw_desc->group_head,
+ chan, 0, addr, index);
+}
+/*
+ * ppc460ex_adma_pq_xor_set_dest - set destination address into descriptor
+ * for the PQXOR operation
+ */
+static void ppc460ex_adma_pqxor_set_dest(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ dma_addr_t *addrs, unsigned long flags)
+{
+ struct ppc460ex_adma_desc_slot *iter;
+ struct ppc460ex_adma_chan *chan;
+ dma_addr_t paddr, qaddr;
+ dma_addr_t addr = 0, ppath, qpath;
+ int index = 0;
+
+ chan = to_ppc460ex_adma_chan(sw_desc->async_tx.chan);
+ if (flags & DMA_PREP_PQ_DISABLE_P)
+ paddr = 0;
+ else
+ paddr = addrs[0];
+
+ if (flags & DMA_PREP_PQ_DISABLE_Q)
+ qaddr = 0;
+ else
+ qaddr = addrs[1];
+
+ if (!paddr || !qaddr)
+ addr = paddr ? paddr : qaddr;
+
+ /*
+ * walk through the WXOR source list and set P/Q-destinations
+ * for each slot:
+ */
+ if (!test_bit(PPC460EX_DESC_RXOR, &sw_desc->flags)) {
+ /*
+ * This is WXOR-only chain; may have 1/2 zero descs
+ */
+ if (test_bit(PPC460EX_ZERO_P, &sw_desc->flags))
+ index++;
+ if (test_bit(PPC460EX_ZERO_Q, &sw_desc->flags))
+ index++;
+
+ iter = ppc460ex_get_group_entry(sw_desc, index);
+ if (addr) {
+ /*
+ * one destination
+ */
+ list_for_each_entry_from(iter,
+ &sw_desc->group_list, chain_node)
+ ppc460ex_desc_set_dest_addr(iter, chan,
+ DMA_CUED_XOR_BASE, addr, 0);
+ } else {
+ /*
+ * two destinations
+ */
+ list_for_each_entry_from(iter,
+ &sw_desc->group_list, chain_node) {
+ ppc460ex_desc_set_dest_addr(iter, chan,
+ DMA_CUED_XOR_BASE, paddr, 0);
+ ppc460ex_desc_set_dest_addr(iter, chan,
+ DMA_CUED_XOR_BASE, qaddr, 1);
+ }
+ }
+
+ if (index) {
+ /* To clear destinations update the descriptor
+ * (1st,2nd, or both depending on flags)
+ */
+ index = 0;
+ if (test_bit(PPC460EX_ZERO_P,
+ &sw_desc->flags)) {
+ iter = ppc460ex_get_group_entry(
+ sw_desc, index++);
+ ppc460ex_adma_pq_zero_op(iter, chan,
+ paddr);
+ }
+
+ if (test_bit(PPC460EX_ZERO_Q,
+ &sw_desc->flags)) {
+ iter = ppc460ex_get_group_entry(
+ sw_desc, index++);
+ ppc460ex_adma_pq_zero_op(iter, chan,
+ qaddr);
+ }
+
+ return;
+ }
+ } else {
+ /*
+ * This is RXOR-only or RXOR/WXOR mixed chain
+ *
+ * If we want to include destination into calculations,
+ * then make dest addresses cued with mult=1 (XOR).
+ */
+ ppath = test_bit(PPC460EX_ZERO_P, &sw_desc->flags) ?
+ DMA_CUED_XOR_HB :
+ DMA_CUED_XOR_BASE |
+ (1 << DMA_CUED_MULT1_OFF);
+ qpath = test_bit(PPC460EX_ZERO_Q, &sw_desc->flags) ?
+ DMA_CUED_XOR_HB :
+ DMA_CUED_XOR_BASE |
+ (1 << DMA_CUED_MULT1_OFF);
+
+ /*
+ * Setup destination(s) in RXOR slot(s)
+ */
+ iter = ppc460ex_get_group_entry(sw_desc, index++);
+ ppc460ex_desc_set_dest_addr(iter, chan,
+ paddr ? ppath : qpath,
+ paddr ? paddr : qaddr, 0);
+ if (!addr) {
+ /*
+ * two destinations
+ */
+ iter = ppc460ex_get_group_entry(sw_desc,
+ index++);
+ ppc460ex_desc_set_dest_addr(iter, chan,
+ qpath, qaddr, 0);
+ }
+
+ if (test_bit(PPC460EX_DESC_WXOR, &sw_desc->flags)) {
+ /* Setup destination(s) in remaining WXOR
+ * slots
+ */
+ iter = ppc460ex_get_group_entry(sw_desc,
+ index);
+ if (addr) {
+ /*
+ * one destination
+ */
+ list_for_each_entry_from(iter,
+ &sw_desc->group_list,
+ chain_node)
+ ppc460ex_desc_set_dest_addr(
+ iter, chan,
+ DMA_CUED_XOR_BASE,
+ addr, 0);
+
+ } else {
+ /*
+ * two destinations
+ */
+ list_for_each_entry_from(iter,
+ &sw_desc->group_list,
+ chain_node) {
+ ppc460ex_desc_set_dest_addr(
+ iter, chan,
+ DMA_CUED_XOR_BASE,
+ paddr, 0);
+ ppc460ex_desc_set_dest_addr(
+ iter, chan,
+ DMA_CUED_XOR_BASE,
+ qaddr, 1);
+ }
+ }
+ }
+
+ }
+}
+
+/*
+ * ppc460ex_adma_pqxor_set_src - set source address into descriptor
+ */
+static void ppc460ex_adma_pqxor_set_src(struct ppc460ex_adma_desc_slot *sw_desc,
+ dma_addr_t addr, int index)
+{
+ struct ppc460ex_adma_chan *chan;
+ dma_addr_t haddr = 0;
+ struct ppc460ex_adma_desc_slot *iter = NULL;
+
+ chan = to_ppc460ex_adma_chan(sw_desc->async_tx.chan);
+ /* DMA0,1 may do: WXOR, RXOR, RXOR+WXORs chain
+ */
+ if (test_bit(PPC460EX_DESC_RXOR, &sw_desc->flags)) {
+ /*
+ * RXOR-only or RXOR/WXOR operation
+ */
+ int iskip = test_bit(PPC460EX_DESC_RXOR12,
+ &sw_desc->flags) ? 2 : 3;
+
+ if (index == 0) {
+ /* 1st slot (RXOR)
+ * setup sources region (R1-2-3, R1-2-4,or R1-2-5)
+ */
+ if (test_bit(PPC460EX_DESC_RXOR12,
+ &sw_desc->flags))
+ haddr = DMA_RXOR12 <<
+ DMA_CUED_REGION_OFF;
+ else if (test_bit(PPC460EX_DESC_RXOR123,
+ &sw_desc->flags))
+ haddr = DMA_RXOR123 <<
+ DMA_CUED_REGION_OFF;
+ else if (test_bit(PPC460EX_DESC_RXOR124,
+ &sw_desc->flags))
+ haddr = DMA_RXOR124 <<
+ DMA_CUED_REGION_OFF;
+ else if (test_bit(PPC460EX_DESC_RXOR125,
+ &sw_desc->flags))
+ haddr = DMA_RXOR125 <<
+ DMA_CUED_REGION_OFF;
+ else
+ BUG();
+ haddr |= DMA_CUED_XOR_BASE;
+ sw_desc = sw_desc->group_head;
+ } else if (index < iskip) {
+ /* 1st slot (RXOR)
+ * shall actually set source address only once
+ * instead of first <iskip>
+ */
+ iter = NULL;
+ } else {
+ /* second and next slots (WXOR);
+ * skip first slot with RXOR
+ */
+ haddr = DMA_CUED_XOR_HB;
+ sw_desc = ppc460ex_get_group_entry(sw_desc,
+ index - iskip + 1);
+ }
+ } else {
+ int znum = 0;
+ /* WXOR-only operation;
+ * skip first slots with destinations
+ */
+ if (test_bit(PPC460EX_ZERO_P, &sw_desc->flags))
+ znum++;
+ if (test_bit(PPC460EX_ZERO_Q, &sw_desc->flags))
+ znum++;
+
+ haddr = DMA_CUED_XOR_HB;
+ iter = ppc460ex_get_group_entry(sw_desc,
+ index + znum);
+ }
+
+ if (likely(iter)) {
+ ppc460ex_desc_set_src_addr(iter, chan, 0, haddr, addr);
+ if (!index &&
+ test_bit(PPC460EX_DESC_RXOR, &sw_desc->flags) &&
+ sw_desc->dst_cnt == 2) {
+ /* if we have two destinations for RXOR, then
+ * setup source in the second descr too
+ */
+ iter = ppc460ex_get_group_entry(sw_desc, 1);
+ ppc460ex_desc_set_src_addr(iter, chan, 0,
+ haddr, addr);
+ }
+ }
+}
+/*
+ * ppc460ex_adma_pqxor_set_src_mult - set multiplication coefficient into
+ * descriptor for the PQXOR operation
+ */
+static void ppc460ex_adma_pqxor_set_src_mult(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ unsigned char mult, int index, int dst_pos)
+{
+ struct ppc460ex_adma_chan *chan;
+ u32 mult_idx, mult_dst;
+ struct ppc460ex_adma_desc_slot *iter = NULL, *iter1 = NULL;
+
+ chan = to_ppc460ex_adma_chan(sw_desc->async_tx.chan);
+ if (test_bit(PPC460EX_DESC_RXOR, &sw_desc->flags)) {
+ int region = test_bit(PPC460EX_DESC_RXOR12,
+ &sw_desc->flags) ? 2 : 3;
+
+ if (index < region) {
+ /*
+ * RXOR multipliers
+ */
+
+ iter = ppc460ex_get_group_entry(sw_desc,
+ sw_desc->dst_cnt - 1);
+ if (sw_desc->dst_cnt == 2)
+ iter1 = ppc460ex_get_group_entry(sw_desc, 0);
+ mult_idx = DMA_CUED_MULT1_OFF + (index << 3);
+ mult_dst = DMA_CDB_SG_SRC;
+ } else {
+ /*
+ * WXOR multiplier
+ */
+ iter = ppc460ex_get_group_entry(sw_desc,
+ index - region +
+ sw_desc->dst_cnt);
+ mult_idx = DMA_CUED_MULT1_OFF;
+ mult_dst = dst_pos ? DMA_CDB_SG_DST2 :
+ DMA_CDB_SG_DST1;
+ }
+ } else {
+ int znum = 0;
+
+ /* WXOR-only;
+ * skip first slots with destinations (if ZERO_DST has
+ * place)
+ */
+ if (test_bit(PPC460EX_ZERO_P, &sw_desc->flags))
+ znum++;
+ if (test_bit(PPC460EX_ZERO_Q, &sw_desc->flags))
+ znum++;
+ iter = ppc460ex_get_group_entry(sw_desc, index + znum);
+ mult_idx = DMA_CUED_MULT1_OFF;
+ mult_dst = dst_pos ? DMA_CDB_SG_DST2 : DMA_CDB_SG_DST1;
+ }
+
+ if (likely(iter)) {
+ ppc460ex_desc_set_src_mult(iter, chan,
+ mult_idx, mult_dst, mult);
+
+ if (unlikely(iter1)) {
+ /* if we have two destinations for RXOR, then
+ * we've just set Q mult. Set-up P now.
+ */
+ ppc460ex_desc_set_src_mult(iter1, chan,
+ mult_idx, mult_dst, 1);
+ }
+
+ }
+}
+
+
+static inline struct ppc460ex_adma_desc_slot *ppc460ex_dma_prep_pq(
+ struct ppc460ex_adma_chan *ppc460ex_chan,
+ dma_addr_t *dst, unsigned int dst_cnt,
+ dma_addr_t *src, unsigned int src_cnt, unsigned char *scf,
+ size_t len, unsigned long flags)
+{
+ int slot_cnt;
+ struct ppc460ex_adma_desc_slot *sw_desc = NULL, *iter;
+ unsigned long op = 0;
+ unsigned char mult = 1;
+
+ /* select operations WXOR/RXOR depending on the
+ * source addresses of operators and the number
+ * of destinations (RXOR support only Q-parity calculations)
+ */
+ set_bit(PPC460EX_DESC_WXOR, &op);
+ if (!test_and_set_bit(PPC460EX_RXOR_RUN, &ppc460ex_rxor_state)) {
+ /* no active RXOR;
+ * do RXOR if:
+ * - destination os only one,
+ * - there are more than 1 source,
+ * - len is aligned on 512-byte boundary,
+ * - source addresses fit to one of 4 possible regions.
+ */
+ if (dst_cnt == 3 && src_cnt > 1 &&
+ !(len & ~MQ0_CF2H_RXOR_BS_MASK) &&
+ (src[0] + len) == src[1]) {
+ /* may do RXOR R1 R2 */
+ set_bit(PPC460EX_DESC_RXOR, &op);
+ if (src_cnt != 2) {
+ /* may try to enhance region of RXOR */
+ if ((src[1] + len) == src[2]) {
+ /* do RXOR R1 R2 R3 */
+ set_bit(PPC460EX_DESC_RXOR123,
+ &op);
+ } else if ((src[1] + len * 2) == src[2]) {
+ /* do RXOR R1 R2 R4 */
+ set_bit(PPC460EX_DESC_RXOR124, &op);
+ } else if ((src[1] + len * 3) == src[2]) {
+ /* do RXOR R1 R2 R5 */
+ set_bit(PPC460EX_DESC_RXOR125,
+ &op);
+ } else {
+ /* do RXOR R1 R2 */
+ set_bit(PPC460EX_DESC_RXOR12,
+ &op);
+ }
+ } else {
+ /* do RXOR R1 R2 */
+ set_bit(PPC460EX_DESC_RXOR12, &op);
+ }
+ }
+
+ if (!test_bit(PPC460EX_DESC_RXOR, &op)) {
+ /* can not do this operation with RXOR */
+ clear_bit(PPC460EX_RXOR_RUN,
+ &ppc460ex_rxor_state);
+ } else {
+ /* can do; set block size right now */
+ ppc460ex_desc_set_rxor_block_size(len);
+ }
+ }
+
+ /*
+ * Number of necessary slots depends on operation type selected
+ */
+ if (!test_bit(PPC460EX_DESC_RXOR, &op)) {
+ /* This is a WXOR only chain. Need descriptors for each
+ * source to GF-XOR them with WXOR, and need descriptors
+ * for each destination to zero them with WXOR
+ */
+ slot_cnt = src_cnt;
+
+ if (flags & DMA_PREP_ZERO_P) {
+ slot_cnt++;
+ set_bit(PPC460EX_ZERO_P, &op);
+ }
+ if (flags & DMA_PREP_ZERO_Q) {
+ slot_cnt++;
+ set_bit(PPC460EX_ZERO_Q, &op);
+ }
+ } else {
+ /* Need 1/2 descriptor for RXOR operation, and
+ * need (src_cnt - (2 or 3)) for WXOR of sources
+ * remained (if any)
+ */
+ slot_cnt = dst_cnt;
+
+ if (flags & DMA_PREP_ZERO_P)
+ set_bit(PPC460EX_ZERO_P, &op);
+ if (flags & DMA_PREP_ZERO_Q)
+ set_bit(PPC460EX_ZERO_Q, &op);
+
+ if (test_bit(PPC460EX_DESC_RXOR12, &op))
+ slot_cnt += src_cnt - 2;
+ else
+ slot_cnt += src_cnt - 3;
+
+ /*
+ * Thus we have either RXOR only chain or
+ * mixed RXOR/WXOR
+ */
+ if (slot_cnt == dst_cnt) {
+ /* RXOR only chain */
+ clear_bit(PPC460EX_DESC_WXOR, &op);
+ }
+ }
+
+ spin_lock_bh(&ppc460ex_chan->lock);
+ /*
+ * for both RXOR/WXOR each descriptor occupies one slot
+ */
+ sw_desc = ppc460ex_adma_alloc_slots(ppc460ex_chan, slot_cnt, 1);
+ if (sw_desc) {
+ ppc460ex_desc_init_pq(sw_desc, dst_cnt, src_cnt,
+ flags, op);
+
+ /*
+ * setup dst/src/mult
+ */
+ ppc460ex_adma_pqxor_set_dest(sw_desc,
+ dst, flags);
+ while (src_cnt--) {
+ ppc460ex_adma_pqxor_set_src(sw_desc, src[src_cnt],
+ src_cnt);
+ if (!(flags & DMA_PREP_PQ_DISABLE_Q))
+ mult = scf[src_cnt];
+ ppc460ex_adma_pqxor_set_src_mult(sw_desc, mult,
+ src_cnt, dst_cnt - 1);
+ }
+
+ /*
+ * Setup byte count foreach slot just allocated
+ */
+ sw_desc->async_tx.flags = flags;
+ list_for_each_entry(iter, &sw_desc->group_list,
+ chain_node) {
+ ppc460ex_desc_set_byte_count(iter,
+ ppc460ex_chan, len);
+ iter->unmap_len = len;
+ }
+ }
+ spin_unlock_bh(&ppc460ex_chan->lock);
+
+ return sw_desc;
+}
+/*
+ * ppc460ex_dma01_prep_mult -
+ * for Q operation where destination is also the source
+ */
+static struct ppc460ex_adma_desc_slot *ppc460ex_dma01_prep_mult(
+ struct ppc460ex_adma_chan *ppc460ex_chan,
+ dma_addr_t *dst, int dst_cnt, dma_addr_t *src, int src_cnt,
+ const unsigned char *scf, size_t len, unsigned long flags)
+{
+ struct ppc460ex_adma_desc_slot *sw_desc = NULL;
+ unsigned long op = 0;
+ int slot_cnt;
+
+ set_bit(PPC460EX_DESC_WXOR, &op);
+ slot_cnt = 2;
+
+ spin_lock_bh(&ppc460ex_chan->lock);
+
+ /*
+ * use WXOR, each descriptor occupies one slot
+ */
+ sw_desc = ppc460ex_adma_alloc_slots(ppc460ex_chan, slot_cnt, 1);
+ if (sw_desc) {
+ struct ppc460ex_adma_chan *chan;
+ struct ppc460ex_adma_desc_slot *iter;
+ struct dma_cdb *hw_desc;
+
+ chan = to_ppc460ex_adma_chan(sw_desc->async_tx.chan);
+ set_bits(op, &sw_desc->flags);
+ sw_desc->src_cnt = src_cnt;
+ sw_desc->dst_cnt = dst_cnt;
+ /*
+ * First descriptor, zero data in the destination and copy it
+ * to q page using MULTICAST transfer.
+ */
+ iter = list_first_entry(&sw_desc->group_list,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ memset(iter->hw_desc, 0, sizeof(struct dma_cdb));
+ /*
+ * set 'next' pointer
+ */
+ iter->hw_next = list_entry(iter->chain_node.next,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ clear_bit(PPC460EX_DESC_INT, &iter->flags);
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = DMA_CDB_OPC_MULTICAST;
+
+ ppc460ex_desc_set_dest_addr(iter, chan,
+ DMA_CUED_XOR_BASE, dst[0], 0);
+ ppc460ex_desc_set_dest_addr(iter, chan, 0, dst[1], 1);
+ ppc460ex_desc_set_src_addr(iter, chan, 0, DMA_CUED_XOR_HB,
+ src[0]);
+ ppc460ex_desc_set_byte_count(iter, ppc460ex_chan, len);
+ iter->unmap_len = len;
+
+ /*
+ * Second descriptor, multiply data from the q page
+ * and store the result in real destination.
+ */
+ iter = list_first_entry(&iter->chain_node,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ memset(iter->hw_desc, 0, sizeof(struct dma_cdb));
+ iter->hw_next = NULL;
+ if (flags & DMA_PREP_INTERRUPT)
+ set_bit(PPC460EX_DESC_INT, &iter->flags);
+ else
+ clear_bit(PPC460EX_DESC_INT, &iter->flags);
+
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = DMA_CDB_OPC_MV_SG1_SG2;
+ ppc460ex_desc_set_src_addr(iter, chan, 0,
+ DMA_CUED_XOR_HB, dst[1]);
+ ppc460ex_desc_set_dest_addr(iter, chan,
+ DMA_CUED_XOR_BASE, dst[0], 0);
+
+ ppc460ex_desc_set_src_mult(iter, chan, DMA_CUED_MULT1_OFF,
+ DMA_CDB_SG_DST1, scf[0]);
+ ppc460ex_desc_set_byte_count(iter, ppc460ex_chan, len);
+ iter->unmap_len = len;
+ sw_desc->async_tx.flags = flags;
+ }
+
+ spin_unlock_bh(&ppc460ex_chan->lock);
+
+ return sw_desc;
+}
+/*
+ * ppc460ex_dma01_prep_sum_product -
+ * Dx = A*(P+Pxy) + B*(Q+Qxy) operation where destination is also
+ * the source.
+ */
+static struct ppc460ex_adma_desc_slot *ppc460ex_dma01_prep_sum_product(
+ struct ppc460ex_adma_chan *ppc460ex_chan,
+ dma_addr_t *dst, dma_addr_t *src, int src_cnt,
+ const unsigned char *scf, size_t len, unsigned long flags)
+{
+ struct ppc460ex_adma_desc_slot *sw_desc = NULL;
+ unsigned long op = 0;
+ int slot_cnt;
+
+ set_bit(PPC460EX_DESC_WXOR, &op);
+ slot_cnt = 3;
+
+ spin_lock_bh(&ppc460ex_chan->lock);
+
+ /*
+ * WXOR, each descriptor occupies one slot
+ */
+ sw_desc = ppc460ex_adma_alloc_slots(ppc460ex_chan, slot_cnt, 1);
+ if (sw_desc) {
+ struct ppc460ex_adma_chan *chan;
+ struct ppc460ex_adma_desc_slot *iter;
+ struct dma_cdb *hw_desc;
+
+ chan = to_ppc460ex_adma_chan(sw_desc->async_tx.chan);
+ set_bits(op, &sw_desc->flags);
+ sw_desc->src_cnt = src_cnt;
+ sw_desc->dst_cnt = 1;
+ /*
+ * 1st descriptor, src[1] data to q page and zero destination
+ */
+ iter = list_first_entry(&sw_desc->group_list,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ memset(iter->hw_desc, 0, sizeof(struct dma_cdb));
+ iter->hw_next = list_entry(iter->chain_node.next,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ clear_bit(PPC460EX_DESC_INT, &iter->flags);
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = DMA_CDB_OPC_MULTICAST;
+
+ ppc460ex_desc_set_dest_addr(iter, chan, DMA_CUED_XOR_BASE,
+ *dst, 0);
+ ppc460ex_desc_set_dest_addr(iter, chan, 0,
+ ppc460ex_chan->qdest, 1);
+ ppc460ex_desc_set_src_addr(iter, chan, 0, DMA_CUED_XOR_HB,
+ src[1]);
+ ppc460ex_desc_set_byte_count(iter, ppc460ex_chan, len);
+ iter->unmap_len = len;
+
+ /*
+ * 2nd descriptor, multiply src[1] data and store the
+ * result in destination */
+ iter = list_first_entry(&iter->chain_node,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ memset(iter->hw_desc, 0, sizeof(struct dma_cdb));
+ /* set 'next' pointer */
+ iter->hw_next = list_entry(iter->chain_node.next,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ if (flags & DMA_PREP_INTERRUPT)
+ set_bit(PPC460EX_DESC_INT, &iter->flags);
+ else
+ clear_bit(PPC460EX_DESC_INT, &iter->flags);
+
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = DMA_CDB_OPC_MV_SG1_SG2;
+ ppc460ex_desc_set_src_addr(iter, chan, 0, DMA_CUED_XOR_HB,
+ ppc460ex_chan->qdest);
+ ppc460ex_desc_set_dest_addr(iter, chan, DMA_CUED_XOR_BASE,
+ *dst, 0);
+ ppc460ex_desc_set_src_mult(iter, chan, DMA_CUED_MULT1_OFF,
+ DMA_CDB_SG_DST1, scf[1]);
+ ppc460ex_desc_set_byte_count(iter, ppc460ex_chan, len);
+ iter->unmap_len = len;
+
+ /*
+ * 3rd descriptor, multiply src[0] data and xor it
+ * with destination
+ */
+ iter = list_first_entry(&iter->chain_node,
+ struct ppc460ex_adma_desc_slot,
+ chain_node);
+ memset(iter->hw_desc, 0, sizeof(struct dma_cdb));
+ iter->hw_next = NULL;
+ if (flags & DMA_PREP_INTERRUPT)
+ set_bit(PPC460EX_DESC_INT, &iter->flags);
+ else
+ clear_bit(PPC460EX_DESC_INT, &iter->flags);
+
+ hw_desc = iter->hw_desc;
+ hw_desc->opc = DMA_CDB_OPC_MV_SG1_SG2;
+ ppc460ex_desc_set_src_addr(iter, chan, 0, DMA_CUED_XOR_HB,
+ src[0]);
+ ppc460ex_desc_set_dest_addr(iter, chan, DMA_CUED_XOR_BASE,
+ *dst, 0);
+ ppc460ex_desc_set_src_mult(iter, chan, DMA_CUED_MULT1_OFF,
+ DMA_CDB_SG_DST1, scf[0]);
+ ppc460ex_desc_set_byte_count(iter, ppc460ex_chan, len);
+ iter->unmap_len = len;
+ sw_desc->async_tx.flags = flags;
+ }
+
+ spin_unlock_bh(&ppc460ex_chan->lock);
+
+ return sw_desc;
+}
+/*
+ * ppc460ex_adma_prep_dma_pq- prepare CDB (group) for a GF-XOR operation
+ */
+static struct dma_async_tx_descriptor *ppc460ex_adma_prep_dma_pq(
+ struct dma_chan *chan, dma_addr_t *dst, dma_addr_t *src,
+ unsigned int src_cnt, unsigned char *scf,
+ size_t len, unsigned long flags)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ struct ppc460ex_adma_desc_slot *sw_desc = NULL;
+ int dst_cnt = 0;
+
+ BUG_ON(!len);
+ BUG_ON(unlikely(len > PPC460EX_ADMA_XOR_MAX_BYTE_COUNT));
+ BUG_ON(!src_cnt);
+
+ if (src_cnt == 1 && dst[1] == src[0]) {
+ dma_addr_t dest[2];
+
+ /* dst[1] is real destination (Q) */
+ dest[0] = dst[1];
+ /* this is the page to multicast source data to */
+ dest[1] = ppc460ex_chan->qdest;
+ sw_desc = ppc460ex_dma01_prep_mult(ppc460ex_chan,
+ dest, 2, src, src_cnt, scf, len, flags);
+ return sw_desc ? &sw_desc->async_tx : NULL;
+ }
+
+ if (src_cnt == 2 && dst[1] == src[1]) {
+ sw_desc = ppc460ex_dma01_prep_sum_product(ppc460ex_chan,
+ &dst[1], src, 2, scf, len, flags);
+ return sw_desc ? &sw_desc->async_tx : NULL;
+ }
+ if (!(flags & DMA_PREP_PQ_DISABLE_P)) {
+ BUG_ON(!dst[0]);
+ dst_cnt++;
+ flags |= DMA_PREP_ZERO_P;
+ }
+
+ if (!(flags & DMA_PREP_PQ_DISABLE_Q)) {
+ BUG_ON(!dst[1]);
+ dst_cnt++;
+ flags |= DMA_PREP_ZERO_Q;
+ }
+ BUG_ON(!dst_cnt);
+
+ dev_dbg(ppc460ex_chan->device->common.dev,
+ "ppc460ex adma%d: %s src_cnt: %d len: %u int_en: %d\n",
+ ppc460ex_chan->device->id, __func__, src_cnt, len,
+ flags & DMA_PREP_INTERRUPT ? 1 : 0);
+
+ sw_desc = ppc460ex_dma_prep_pq(ppc460ex_chan,
+ dst, dst_cnt, src, src_cnt, scf,
+ len, flags);
+
+
+ return sw_desc ? &sw_desc->async_tx : NULL;
+}
+/*
+ * ppc460ex_adma_prep_dma_p- prepare CDB (group) for a GF-XOR operation
+ */
+static struct dma_async_tx_descriptor *ppc460ex_adma_prep_dma_p(
+ struct dma_chan *chan, dma_addr_t *dst, dma_addr_t *src,
+ unsigned int src_cnt, unsigned char *scf,
+ size_t len, unsigned long flags)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ struct ppc460ex_adma_desc_slot *sw_desc = NULL;
+ int dst_cnt = 0;
+
+ BUG_ON(!len);
+ BUG_ON(unlikely(len > PPC460EX_ADMA_XOR_MAX_BYTE_COUNT));
+ BUG_ON(!src_cnt);
+
+ if (src_cnt == 1 && dst[1] == src[0]) {
+ dma_addr_t dest[2];
+
+ /* dst[1] is real destination (Q) */
+ dest[0] = dst[1];
+ /* this is the page to multicast source data to */
+ dest[1] = ppc460ex_chan->qdest;
+ sw_desc = ppc460ex_dma01_prep_mult(ppc460ex_chan,
+ dest, 2, src, src_cnt, scf, len, flags);
+ return sw_desc ? &sw_desc->async_tx : NULL;
+ }
+
+ if (src_cnt == 2 && dst[1] == src[1]) {
+ sw_desc = ppc460ex_dma01_prep_sum_product(ppc460ex_chan,
+ &dst[1], src, 2, scf, len, flags);
+ return sw_desc ? &sw_desc->async_tx : NULL;
+ }
+ if (!(flags & DMA_PREP_PQ_DISABLE_P)) {
+ BUG_ON(!dst[0]);
+ dst_cnt++;
+ if (flags & DMA_ZERO_P)
+ flags |= DMA_PREP_ZERO_P;
+ }
+
+ if (!(flags & DMA_PREP_PQ_DISABLE_Q)) {
+ BUG_ON(!dst[1]);
+ dst_cnt++;
+ flags |= DMA_PREP_ZERO_Q;
+ }
+ BUG_ON(!dst_cnt);
+
+ dev_dbg(ppc460ex_chan->device->common.dev,
+ "ppc460ex adma%d: %s src_cnt: %d len: %u int_en: %d\n",
+ ppc460ex_chan->device->id, __func__, src_cnt, len,
+ flags & DMA_PREP_INTERRUPT ? 1 : 0);
+
+ sw_desc = ppc460ex_dma_prep_pq(ppc460ex_chan,
+ dst, dst_cnt, src, src_cnt, scf,
+ len, flags);
+
+ return sw_desc ? &sw_desc->async_tx : NULL;
+}
+/*
+ * ppc460ex_adma_prep_dma_mq_xor - prepare CDB (group) for a GF-XOR operation
+ */
+static struct dma_async_tx_descriptor *ppc460ex_adma_prep_dma_mq_xor(
+ struct dma_chan *chan, dma_addr_t dst,
+ dma_addr_t *src, unsigned int src_cnt,
+ size_t len, unsigned long flags)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ struct dma_async_tx_descriptor *tx;
+ dma_addr_t pq[2];
+ unsigned char scf = 0;
+
+ /* validate P, disable Q */
+ pq[0] = dst;
+ pq[1] = 0;
+ flags |= DMA_PREP_PQ_DISABLE_Q;
+
+ BUG_ON(!len);
+ BUG_ON(unlikely(len > PPC460EX_ADMA_XOR_MAX_BYTE_COUNT));
+ BUG_ON(!src_cnt);
+
+ dev_dbg(ppc460ex_chan->device->common.dev,
+ "ppc460ex adma%d: %s src_cnt: %d len: %u int_en: %d\n",
+ ppc460ex_chan->device->id, __func__, src_cnt, len,
+ flags & DMA_PREP_INTERRUPT ? 1 : 0);
+
+ tx = ppc460ex_adma_prep_dma_p(chan, &pq[0], src,
+ src_cnt, &scf, len, flags);
+ return tx;
+
+}
+/*
+ * ppc460ex_adma_memcpy_xor_set_src - set source address into descriptor
+ */
+static inline void ppc460ex_adma_memcpy_xor_set_src(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ dma_addr_t addr,
+ int index)
+{
+ struct ppc460ex_adma_chan *chan;
+
+ sw_desc = sw_desc->group_head;
+ chan = to_ppc460ex_adma_chan(sw_desc->async_tx.chan);
+
+ if (likely(sw_desc))
+ ppc460ex_desc_set_src_addr(sw_desc, chan, index, 0, addr);
+}
+/*
+ * ppc460ex_adma_pqzero_sum_set_src_mult - set multiplication coefficient
+ * into descriptor for the PQZERO_SUM operation
+ */
+static void ppc460ex_adma_pqzero_sum_set_src_mult(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ unsigned char mult, int index, int dst_pos)
+{
+ struct ppc460ex_adma_chan *chan;
+ u32 mult_idx, mult_dst;
+
+ chan = to_ppc460ex_adma_chan(sw_desc->async_tx.chan);
+ /* set mult for sources only */
+ BUG_ON(index >= sw_desc->src_cnt);
+
+ /* get pointed slot */
+ sw_desc = ppc460ex_get_group_entry(sw_desc, index);
+
+ mult_idx = DMA_CUED_MULT1_OFF;
+ mult_dst = dst_pos ? DMA_CDB_SG_DST2 : DMA_CDB_SG_DST1;
+
+ if (likely(sw_desc))
+ ppc460ex_desc_set_src_mult(sw_desc, chan, mult_idx, mult_dst,
+ mult);
+}
+/*
+ * ppc460ex_adma_pq_set_src_mult - set multiplication coefficient into
+ * descriptor for the PQXOR operation
+ */
+static void ppc460ex_adma_pq_set_src_mult(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ unsigned char mult, int index, int dst_pos)
+{
+ struct ppc460ex_adma_chan *chan;
+ u32 mult_idx, mult_dst;
+ struct ppc460ex_adma_desc_slot *iter = NULL, *iter1 = NULL;
+
+ chan = to_ppc460ex_adma_chan(sw_desc->async_tx.chan);
+
+ if (test_bit(PPC460EX_DESC_RXOR, &sw_desc->flags)) {
+ int region = test_bit(PPC460EX_DESC_RXOR12,
+ &sw_desc->flags) ? 2 : 3;
+
+ if (index < region) {
+ /* RXOR multipliers */
+ iter = ppc460ex_get_group_entry(sw_desc,
+ sw_desc->dst_cnt - 1);
+ if (sw_desc->dst_cnt == 2)
+ iter1 = ppc460ex_get_group_entry(
+ sw_desc, 0);
+
+ mult_idx = DMA_CUED_MULT1_OFF + (index << 3);
+ mult_dst = DMA_CDB_SG_SRC;
+ } else {
+ /* WXOR multiplier */
+ iter = ppc460ex_get_group_entry(sw_desc,
+ index - region +
+ sw_desc->dst_cnt);
+ mult_idx = DMA_CUED_MULT1_OFF;
+ mult_dst = dst_pos ? DMA_CDB_SG_DST2 :
+ DMA_CDB_SG_DST1;
+ }
+ } else {
+ int znum = 0;
+
+ /* WXOR-only;
+ * skip first slots with destinations (if ZERO_DST has
+ * place)
+ */
+ if (test_bit(PPC460EX_ZERO_P, &sw_desc->flags))
+ znum++;
+ if (test_bit(PPC460EX_ZERO_Q, &sw_desc->flags))
+ znum++;
+
+ iter = ppc460ex_get_group_entry(sw_desc, index + znum);
+ mult_idx = DMA_CUED_MULT1_OFF;
+ mult_dst = dst_pos ? DMA_CDB_SG_DST2 : DMA_CDB_SG_DST1;
+ }
+
+ if (likely(iter)) {
+ ppc460ex_desc_set_src_mult(iter, chan,
+ mult_idx, mult_dst, mult);
+
+ if (unlikely(iter1)) {
+ /* if we have two destinations for RXOR, then
+ * we've just set Q mult. Set-up P now.
+ */
+ ppc460ex_desc_set_src_mult(iter1, chan,
+ mult_idx, mult_dst, 1);
+ }
+
+ }
+
+}
+/*
+ * ppc460ex_adma_pq_zero_sum_set_dest - set destination address into descriptor
+ * for the PQ_ZERO_SUM operation
+ */
+void ppc460ex_adma_pqzero_sum_set_dest(
+ struct ppc460ex_adma_desc_slot *sw_desc,
+ dma_addr_t paddr, dma_addr_t qaddr)
+{
+ struct ppc460ex_adma_desc_slot *iter, *end;
+ struct ppc460ex_adma_chan *chan;
+ dma_addr_t addr = 0;
+ int idx;
+
+ chan = to_ppc460ex_adma_chan(sw_desc->async_tx.chan);
+
+ /* walk through the WXOR source list and set P/Q-destinations
+ * for each slot
+ */
+ idx = (paddr && qaddr) ? 2 : 1;
+ /* set end */
+ list_for_each_entry_reverse(end, &sw_desc->group_list,
+ chain_node) {
+ if (!(--idx))
+ break;
+ }
+ /* set start */
+ idx = (paddr && qaddr) ? 2 : 1;
+ iter = ppc460ex_get_group_entry(sw_desc, idx);
+
+ if (paddr && qaddr) {
+ /* two destinations */
+ list_for_each_entry_from(iter, &sw_desc->group_list,
+ chain_node) {
+ if (unlikely(iter == end))
+ break;
+ ppc460ex_desc_set_dest_addr(iter, chan,
+ DMA_CUED_XOR_BASE, paddr, 0);
+ ppc460ex_desc_set_dest_addr(iter, chan,
+ DMA_CUED_XOR_BASE, qaddr, 1);
+ }
+ } else {
+ /* one destination */
+ addr = paddr ? paddr : qaddr;
+ list_for_each_entry_from(iter, &sw_desc->group_list,
+ chain_node) {
+ if (unlikely(iter == end))
+ break;
+ ppc460ex_desc_set_dest_addr(iter, chan,
+ DMA_CUED_XOR_BASE, addr, 0);
+ }
+ }
+
+ /* The remaining descriptors are DATACHECK. These have no need in
+ * destination. Actually, these destinations are used there
+ * as sources for check operation. So, set addr as source.
+ */
+ ppc460ex_desc_set_src_addr(end, chan, 0, 0, addr ? addr : paddr);
+
+ if (!addr) {
+ end = list_entry(end->chain_node.next,
+ struct ppc460ex_adma_desc_slot, chain_node);
+ ppc460ex_desc_set_src_addr(end, chan, 0, 0, qaddr);
+ }
+}
+/*
+ * ppc460ex_adma_free_chan_resources - free the resources allocated
+ */
+void ppc460ex_adma_free_chan_resources(struct dma_chan *chan)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ struct ppc460ex_adma_desc_slot *iter, *_iter;
+ int in_use_descs = 0;
+
+ ppc460ex_adma_slot_cleanup(ppc460ex_chan);
+
+ spin_lock_bh(&ppc460ex_chan->lock);
+ list_for_each_entry_safe(iter, _iter, &ppc460ex_chan->chain,
+ chain_node) {
+ in_use_descs++;
+ list_del(&iter->chain_node);
+ }
+ list_for_each_entry_safe_reverse(iter, _iter,
+ &ppc460ex_chan->all_slots, slot_node) {
+ list_del(&iter->slot_node);
+ kfree(iter);
+ ppc460ex_chan->slots_allocated--;
+ }
+ ppc460ex_chan->last_used = NULL;
+
+ dev_dbg(ppc460ex_chan->device->common.dev,
+ "ppc460ex adma%d %s slots_allocated %d\n",
+ ppc460ex_chan->device->id,
+ __func__, ppc460ex_chan->slots_allocated);
+ spin_unlock_bh(&ppc460ex_chan->lock);
+
+ /* one is ok since we left it on there on purpose */
+ if (in_use_descs > 1)
+ dev_dbg(ppc460ex_chan->device->common.dev,
+ "GT: Freeing %d in use descriptors!\n",
+ in_use_descs - 1);
+}
+
+/**
+ * ppc460ex_adma_tx_status - poll the status of an ADMA transaction
+ * @chan: ADMA channel handle
+ * @cookie: ADMA transaction identifier
+ * @txstate: a holder for the current state of the channel
+ */
+static enum dma_status ppc460ex_adma_tx_status(struct dma_chan *chan,
+ dma_cookie_t cookie, struct dma_tx_state *txstate)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan;
+ dma_cookie_t last_used;
+ dma_cookie_t last_complete;
+ enum dma_status ret;
+
+ ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ last_used = chan->cookie;
+ last_complete = ppc460ex_chan->completed_cookie;
+
+ dma_set_tx_state(txstate, last_complete, last_used, 0);
+
+ ret = dma_async_is_complete(cookie, last_complete, last_used);
+ if (ret == DMA_SUCCESS)
+ return ret;
+
+ ppc460ex_adma_slot_cleanup(ppc460ex_chan);
+
+ last_used = chan->cookie;
+ last_complete = ppc460ex_chan->completed_cookie;
+
+ dma_set_tx_state(txstate, last_complete, last_used, 0);
+
+ return dma_async_is_complete(cookie, last_complete, last_used);
+}
+/*
+ * ppc460ex_adma_is_complete - poll the status of an ADMA transaction
+ * @chan: ADMA channel handle
+ * @cookie: ADMA transaction identifier
+ */
+enum dma_status ppc460ex_adma_is_complete(struct dma_chan *chan,
+ dma_cookie_t cookie, dma_cookie_t *done, dma_cookie_t *used)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ dma_cookie_t last_used;
+ dma_cookie_t last_complete;
+ enum dma_status ret;
+
+ last_used = chan->cookie;
+ last_complete = ppc460ex_chan->completed_cookie;
+
+ if (done)
+ *done = last_complete;
+ if (used)
+ *used = last_used;
+
+ ret = dma_async_is_complete(cookie, last_complete, last_used);
+ if (ret == DMA_SUCCESS)
+ return ret;
+
+ ppc460ex_adma_slot_cleanup(ppc460ex_chan);
+
+ last_used = chan->cookie;
+ last_complete = ppc460ex_chan->completed_cookie;
+
+ if (done)
+ *done = last_complete;
+ if (used)
+ *used = last_used;
+
+ return dma_async_is_complete(cookie, last_complete, last_used);
+}
+/*
+ * ppc460ex_adma_eot_handler - end of transfer interrupt handler
+ */
+static irqreturn_t ppc460ex_adma_eot_handler(int irq, void *data)
+{
+ struct ppc460ex_adma_chan *chan = data;
+
+ dev_dbg(chan->device->common.dev,
+ "ppc460ex adma%d: %s\n", chan->device->id, __func__);
+
+ tasklet_schedule(&chan->irq_tasklet);
+ ppc460ex_adma_device_clear_eot_status(chan);
+
+ return IRQ_HANDLED;
+}
+
+/*
+ * ppc460ex_adma_err_handler - DMA error interrupt handler;
+ * do the same things as a eot handler
+ */
+static irqreturn_t ppc460ex_adma_err_handler(int irq, void *data)
+{
+ struct ppc460ex_adma_chan *chan = data;
+ dev_dbg(chan->device->common.dev,
+ "ppc460ex adma%d: %s\n", chan->device->id, __func__);
+ tasklet_schedule(&chan->irq_tasklet);
+ ppc460ex_adma_device_clear_eot_status(chan);
+
+ return IRQ_HANDLED;
+}
+
+static void ppc460ex_test_rad6_callback(void *unused)
+{
+ complete(&ppc460ex_r6_test_comp);
+}
+/*
+ * ppc460ex_test_callback - called when test operation has been done
+ */
+static void ppc460ex_test_raid5_callback(void *unused)
+{
+ complete(&ppc460ex_r5_test_comp);
+}
+/*
+ * ppc460ex_adma_issue_pending - flush all pending descriptors to h/w
+ */
+static void ppc460ex_adma_issue_pending(struct dma_chan *chan)
+{
+ struct ppc460ex_adma_chan *ppc460ex_chan;
+
+ ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ dev_dbg(ppc460ex_chan->device->common.dev,
+ "ppc460ex adma%d: %s %d\n", ppc460ex_chan->device->id, __func__,
+ ppc460ex_chan->pending);
+ if (ppc460ex_chan->pending) {
+ ppc460ex_chan->pending = 0;
+ ppc460ex_chan_append(ppc460ex_chan);
+ }
+}
+/*
+ * ppc460ex_test_raid6 - test are RAID-6 capabilities enabled successfully.
+ * For this we just perform one WXOR operation with the same source
+ * and destination addresses, the GF-multiplier is 1; so if RAID-6
+ o/of_platform_driver_unregister(&ppc460ex_adma_driver);
+ * capabilities are enabled then we'll get src/dst filled with zero.
+ */
+static int ppc460ex_test_raid6(struct ppc460ex_adma_chan *chan)
+{
+ struct ppc460ex_adma_desc_slot *sw_desc, *iter;
+ struct page *pg;
+ char *a;
+ unsigned long op = 0;
+ int rval = 0;
+ dma_addr_t dma_addr, addrs[2];;
+
+ if (!ppc460ex_r6_tchan)
+ return -1;
+
+ set_bit(PPC460EX_DESC_WXOR, &op);
+
+ pg = alloc_page(GFP_KERNEL);
+ if (!pg)
+ return -ENOMEM;
+
+ spin_lock_bh(&chan->lock);
+ sw_desc = ppc460ex_adma_alloc_slots(chan, 1, 1);
+ if (sw_desc) {
+ /* 1 src, 1 dsr, int_ena, WXOR */
+ ppc460ex_desc_init_pq(sw_desc, 1, 1, 1, op);
+ list_for_each_entry(iter, &sw_desc->group_list, chain_node) {
+ ppc460ex_desc_set_byte_count(iter, chan, PAGE_SIZE);
+ iter->unmap_len = PAGE_SIZE;
+ }
+ } else {
+ rval = -EFAULT;
+ spin_unlock_bh(&chan->lock);
+ goto exit;
+ }
+ spin_unlock_bh(&chan->lock);
+
+ /* Fill the test page with ones */
+ memset(page_address(pg), 0xFF, PAGE_SIZE);
+ dma_addr = dma_map_page(chan->device->dev, pg, 0, PAGE_SIZE,
+ DMA_BIDIRECTIONAL);
+
+ /* Setup adresses */
+ ppc460ex_adma_pqxor_set_src(sw_desc, dma_addr, 0);
+ ppc460ex_adma_pqxor_set_src_mult(sw_desc, 1, 0, 0);
+ addrs[0] = dma_addr;
+ addrs[1] = 0;
+ ppc460ex_adma_pqxor_set_dest(sw_desc, addrs, DMA_PREP_PQ_DISABLE_Q);
+
+ async_tx_ack(&sw_desc->async_tx);
+ sw_desc->async_tx.callback = ppc460ex_test_rad6_callback;
+ sw_desc->async_tx.callback_param = NULL;
+
+ init_completion(&ppc460ex_r6_test_comp);
+
+ ppc460ex_adma_tx_submit(&sw_desc->async_tx);
+ ppc460ex_adma_issue_pending(&chan->common);
+
+ wait_for_completion(&ppc460ex_r6_test_comp);
+
+ /* Now check is the test page zeroed */
+ a = page_address(pg);
+ if ((*(u32 *)a) == 0 && memcmp(a, a+4, PAGE_SIZE-4) == 0) {
+ /* page is zero - RAID-6 enabled */
+ rval = 0;
+ } else {
+ /* RAID-6 was not enabled */
+ rval = -EINVAL;
+ }
+exit:
+ __free_page(pg);
+ return rval;
+}
+/*
+ * ppc460ex_test_raid5 - test are RAID-5 capabilities enabled successfully.
+ * For this we just perform one WXOR operation with the same source
+ * and destination addresses, the GF-multiplier is 1; so if RAID-5
+ o/of_platform_driver_unregister(&ppc460ex_adma_driver);
+ * capabilities are enabled then we'll get src/dst filled with zero.
+ */
+static int ppc460ex_test_raid5(struct ppc460ex_adma_chan *chan)
+{
+ struct ppc460ex_adma_desc_slot *sw_desc, *iter;
+ struct page *pg;
+ char *a;
+ dma_addr_t dma_addr, addrs[2];
+ unsigned long op = 0;
+ int rval = 0;
+
+ if (!ppc460ex_r5_tchan)
+ return -1;
+
+ set_bit(PPC460EX_DESC_WXOR, &op);
+
+ pg = alloc_page(GFP_KERNEL);
+ if (!pg)
+ return -ENOMEM;
+
+ spin_lock_bh(&chan->lock);
+ sw_desc = ppc460ex_adma_alloc_slots(chan, 1, 1);
+ if (sw_desc) {
+ /* 1 src, 1 dsr, int_ena, WXOR */
+ ppc460ex_desc_init_pq(sw_desc, 1, 1, 1, op);
+ list_for_each_entry(iter, &sw_desc->group_list, chain_node) {
+ ppc460ex_desc_set_byte_count(iter, chan, PAGE_SIZE);
+ iter->unmap_len = PAGE_SIZE;
+ }
+ } else {
+ rval = -EFAULT;
+ spin_unlock_bh(&chan->lock);
+ goto exit;
+ }
+ spin_unlock_bh(&chan->lock);
+
+ /* Fill the test page with ones */
+ memset(page_address(pg), 0xFF, PAGE_SIZE);
+ dma_addr = dma_map_page(chan->device->dev, pg, 0, PAGE_SIZE,
+ DMA_BIDIRECTIONAL);
+
+ /* Setup adresses */
+ ppc460ex_adma_pqxor_set_src(sw_desc, dma_addr, 0);
+ ppc460ex_adma_pqxor_set_src_mult(sw_desc, 1, 0, 0);
+ addrs[0] = dma_addr;
+ addrs[1] = 0;
+ ppc460ex_adma_pqxor_set_dest(sw_desc, addrs, DMA_PREP_PQ_DISABLE_Q);
+
+ async_tx_ack(&sw_desc->async_tx);
+ sw_desc->async_tx.callback = ppc460ex_test_raid5_callback;
+ sw_desc->async_tx.callback_param = NULL;
+
+ init_completion(&ppc460ex_r5_test_comp);
+
+ ppc460ex_adma_tx_submit(&sw_desc->async_tx);
+ ppc460ex_adma_issue_pending(&chan->common);
+
+ wait_for_completion(&ppc460ex_r5_test_comp);
+
+ /*Make sure cache is flushed to memory*/
+ dma_addr = dma_map_page(chan->device->dev, pg, 0, PAGE_SIZE,
+ DMA_BIDIRECTIONAL);
+ /* Now check is the test page zeroed */
+ a = page_address(pg);
+ if ((*(u32 *)a) == 0 && memcmp(a, a+4, PAGE_SIZE-4) == 0) {
+ /* page is zero - RAID-5 enabled */
+ rval = 0;
+ } else {
+ /* RAID-5 was not enabled */
+ rval = -EINVAL;
+ }
+exit:
+ __free_page(pg);
+ return rval;
+}
+/*
+ * /sysfs interface
+ */
+static ssize_t ppc460ex_poly_read(struct device_driver *dev, char *buf)
+{
+ ssize_t size = 0;
+ u32 reg;
+
+ reg = dcr_read(ppc460ex_mq_dcr_host, DCRN_MQ0_CFBHL);
+ reg >>= MQ0_CFBHL_POLY;
+ reg &= 0xFF;
+
+ size = sprintf(buf, PAGE_SIZE,
+ "PPC460EX RAID-6 driver uses 0x1%02x polynomial.\n",
+ reg);
+ return size;
+}
+
+static ssize_t ppc460ex_poly_write(struct device_driver *dev,
+ const char *buf, size_t count)
+{
+ unsigned long val, rval;
+
+ if (!count || count > 6)
+ return -EINVAL;
+
+ sscanf(buf, "%lx", &val);
+ if (val & ~0x1FF)
+ return -EINVAL;
+
+ val &= 0xFF;
+ rval = dcr_read(ppc460ex_mq_dcr_host, DCRN_MQ0_CFBHL);
+ rval &= ~(0xFF << MQ0_CFBHL_POLY);
+ rval |= val << MQ0_CFBHL_POLY;
+ dcr_write(ppc460ex_mq_dcr_host, DCRN_MQ0_CFBHL, rval);
+
+ return count;
+}
+static ssize_t show_ppc460ex_devices(struct device_driver *dev, char *buf)
+{
+ ssize_t size = 0;
+ int i;
+
+ for (i = 0; i < PPC460EX_ADMA_ENGINES_NUM; i++) {
+ if (ppc460ex_adma_devices[i] == -1)
+ continue;
+ size += snprintf(buf + size, PAGE_SIZE - size,
+ "PPC460EX-ADMA.%d: %s\n", i,
+ ppc_adma_errors[ppc460ex_adma_devices[i]]);
+ }
+ return size;
+}
+static ssize_t ppc460ex_r6ena_read(struct device_driver *dev, char *buf)
+{
+ ssize_t size = 0;
+ size = snprintf(buf, PAGE_SIZE,
+ "PPC460EX RAID-6 capabilities are %sABLED.\n",
+ ppc460ex_r6_enabled ? "EN" : "DIS");
+ return size;
+}
+
+static ssize_t ppc460ex_r6ena_write(struct device_driver *dev,
+ const char *buf, size_t count)
+{
+ unsigned long val;
+
+ if (!count || count > 11)
+ return -EINVAL;
+
+ if (!ppc460ex_r6_tchan)
+ return -EFAULT;
+
+ /* Write a key */
+ sscanf(buf, "%lx", &val);
+ dcr_write(ppc460ex_mq_dcr_host, DCRN_MQ0_XORBA, val);
+ isync();
+
+ /* Verify does it really work now */
+ if (ppc460ex_test_raid6(ppc460ex_r6_tchan) == 0) {
+ /*
+ * PPC460Ex RAID-6 has been activated successfully
+ */;
+ dev_dbg(dev, "PPC460Ex RAID-6 has been activated "
+ "successfully\n");
+ ppc460ex_r6_enabled = 1;
+ } else {
+ /*
+ * PPC460Ex RAID-6 hasn't been activated! Error key ?
+ */
+ dev_dbg(dev, "PPC460Ex RAID-6 hasn't been activated!"
+ " Error key ?\n");
+ ppc460ex_r6_enabled = 0;
+ }
+
+ return count;
+}
+static ssize_t ppc460ex_r5ena_read(struct device_driver *dev, char *buf)
+{
+ ssize_t size = 0;
+ u32 reg;
+ reg = dcr_read(ppc460ex_mq_dcr_host, DCRN_MQ0_CFBHL);
+ reg >>= MQ0_CFBHL_POLY;
+ reg &= 0xFF;
+ size = snprintf(buf, PAGE_SIZE,
+ "PPC460EX RAID-5 capabilities are %sABLED.\n",
+ ppc460ex_r6_enabled ? "EN" : "DIS");
+ return size;
+}
+
+static ssize_t ppc460ex_r5ena_write(struct device_driver *dev,
+ const char *buf, size_t count)
+{
+ unsigned long val;
+
+ if (!count || count > 11)
+ return -EINVAL;
+
+ if (!ppc460ex_r6_tchan)
+ return -EFAULT;
+
+ /* Write a key */
+ sscanf(buf, "%lx", &val);
+ dcr_write(ppc460ex_mq_dcr_host, DCRN_MQ0_XORBA, val);
+ isync();
+
+ /* Verify does it really work now */
+ if (ppc460ex_test_raid5(ppc460ex_r5_tchan) == 0) {
+ /*
+ * PPC460Ex RAID-5 has been activated successfully
+ */
+ dev_dbg(dev, "PPC460Ex RAID-5 has been activated "
+ "successfully\n");
+ ppc460ex_r5_enabled = 1;
+ } else {
+ /*
+ * PPC460Ex RAID-5 hasn't been activated! Error key ?
+ */
+ dev_dbg(dev, "PPC460Ex RAID-5 hasn't been activated!"
+ " Error key ?\n");
+ ppc460ex_r5_enabled = 0;
+ }
+
+ return count;
+}
+static DRIVER_ATTR(devices, S_IRUGO, show_ppc460ex_devices, NULL);
+static DRIVER_ATTR(raid6_enable, S_IRUGO | S_IWUSR, ppc460ex_r6ena_read,
+ ppc460ex_r6ena_write);
+static DRIVER_ATTR(poly, S_IRUGO | S_IWUSR, ppc460ex_poly_read,
+ ppc460ex_poly_write);
+static DRIVER_ATTR(raid5_enable, S_IRUGO | S_IWUSR, ppc460ex_r5ena_read,
+ ppc460ex_r5ena_write);
+static void ppc460ex_adma_init_capabilities(struct ppc460ex_adma_device *adev)
+{
+ dma_cap_set(DMA_MEMCPY, adev->common.cap_mask);
+ dma_cap_set(DMA_INTERRUPT, adev->common.cap_mask);
+ dma_cap_set(DMA_MEMSET, adev->common.cap_mask);
+ dma_cap_set(DMA_PQ, adev->common.cap_mask);
+ dma_cap_set(DMA_PQ_VAL, adev->common.cap_mask);
+ dma_cap_set(DMA_XOR_VAL, adev->common.cap_mask);
+
+ /* Set base routines */
+
+ adev->common.device_alloc_chan_resources =
+ ppc460ex_adma_alloc_chan_resources;
+ adev->common.device_free_chan_resources =
+ ppc460ex_adma_free_chan_resources;
+ adev->common.device_tx_status = ppc460ex_adma_tx_status;
+ adev->common.device_issue_pending =
+ ppc460ex_adma_issue_pending;
+
+
+ /*Setup routines based on capability*/
+ if (dma_has_cap(DMA_MEMCPY, adev->common.cap_mask)) {
+ adev->common.device_prep_dma_memcpy =
+ ppc460ex_adma_prep_dma_memcpy;
+ }
+ if (dma_has_cap(DMA_MEMSET, adev->common.cap_mask)) {
+ adev->common.device_prep_dma_memset =
+ ppc460ex_adma_prep_dma_memset;
+ }
+ if (dma_has_cap(DMA_XOR, adev->common.cap_mask)) {
+ adev->common.max_xor = XOR_MAX_OPS;
+ adev->common.device_prep_dma_xor =
+ ppc460ex_adma_prep_dma_mq_xor;
+ }
+ if (dma_has_cap(DMA_PQ, adev->common.cap_mask)) {
+ dma_set_maxpq(&adev->common,
+ DMA0_FIFO_SIZE / sizeof(struct dma_cdb), 0);
+ adev->common.device_prep_dma_pq =
+ ppc460ex_adma_prep_dma_pq;
+ }
+ if (dma_has_cap(DMA_PQ_VAL, adev->common.cap_mask)) {
+ adev->common.max_pq = DMA0_FIFO_SIZE / sizeof(struct dma_cdb);
+ adev->common.device_prep_dma_pq_val =
+ ppc460ex_adma_prep_dma_pqzero_sum;
+ }
+ if (dma_has_cap(DMA_XOR_VAL, adev->common.cap_mask)) {
+ adev->common.max_xor = DMA0_FIFO_SIZE /
+ sizeof(struct dma_cdb);
+ adev->common.device_prep_dma_xor_val =
+ ppc460ex_adma_prep_dma_xor_zero_sum;
+ }
+ if (dma_has_cap(DMA_INTERRUPT, adev->common.cap_mask)) {
+ adev->common.device_prep_dma_interrupt =
+ ppc460ex_adma_prep_dma_interrupt;
+ }
+ pr_info("%s: APM ppc460ex ADMA engine:"
+ "( %s%s%s%s%s%s%s)\n",
+ dev_name(adev->dev),
+ dma_has_cap(DMA_PQ, adev->common.cap_mask) ?
+ "pq " : "",
+ dma_has_cap(DMA_PQ_VAL, adev->common.cap_mask) ?
+ "pq_val " : "",
+ dma_has_cap(DMA_XOR, adev->common.cap_mask) ?
+ "xor " : "",
+ dma_has_cap(DMA_XOR_VAL, adev->common.cap_mask) ?
+ "xor_val " : "",
+ dma_has_cap(DMA_MEMCPY, adev->common.cap_mask) ?
+ "memcpy " : "",
+ dma_has_cap(DMA_MEMSET, adev->common.cap_mask) ?
+ "memset " : "",
+ dma_has_cap(DMA_INTERRUPT, adev->common.cap_mask) ?
+ "intr " : "");
+}
+
+static int ppc460ex_setup_irqs(struct ppc460ex_adma_device *adev,
+ struct ppc460ex_adma_chan *chan, int *initcode)
+{
+ struct device_node *np;
+ int ret;
+
+ np = adev->dev->of_node;
+
+ adev->err_irq = irq_of_parse_and_map(np, 2);
+ if (adev->err_irq == NO_IRQ) {
+ dev_warn(adev->dev, "no err irq resource?\n");
+ *initcode = PPC_ADMA_INIT_IRQ2;
+ adev->err_irq = -ENXIO;
+ } else
+ atomic_inc(&ppc460ex_adma_err_irq_ref);
+
+ adev->irq = irq_of_parse_and_map(np, 0);
+ if (adev->irq == NO_IRQ) {
+ dev_err(adev->dev, "no irq resource\n");
+ *initcode = PPC_ADMA_INIT_IRQ1;
+ ret = -ENXIO;
+ goto err_irq_map;
+ }
+ dev_dbg(adev->dev, "irq %d, err irq %d\n",
+ adev->irq, adev->err_irq);
+ ret = request_irq(adev->irq, ppc460ex_adma_eot_handler,
+ 0, dev_driver_string(adev->dev), chan);
+ if (ret) {
+ dev_err(adev->dev, "can't request irq %d\n",
+ adev->irq);
+ *initcode = PPC_ADMA_INIT_IRQ1;
+ ret = -EIO;
+ goto err_req1;
+ }
+ if (adev->err_irq > 0) {
+ u32 mask, enable;
+ ret = request_irq(adev->err_irq,
+ ppc460ex_adma_err_handler,
+ IRQF_SHARED,
+ dev_driver_string(adev->dev),
+ chan);
+ if (ret) {
+ dev_err(adev->dev, "can't request irq %d\n",
+ adev->err_irq);
+ *initcode = PPC_ADMA_INIT_IRQ2;
+ ret = -EIO;
+ goto err_req2;
+ }
+ np = of_find_compatible_node(NULL, NULL, "ibm,i2o-460ex");
+ if (!np) {
+ pr_err("%s: can't find I2O device tree node\n",
+ __func__);
+ return -ENODEV;
+ }
+ adev->i2o_reg = of_iomap(np, 0);
+ if (!adev->i2o_reg) {
+ pr_err("%s: failed to map I2O registers\n", __func__);
+ of_node_put(np);
+ ret = -EINVAL;
+ goto err_req2;
+ }
+ of_node_put(np);
+ /* Unmask 'CS FIFO Attention' interrupts and
+ * enable generating interrupts on errors
+ */
+ enable = ~(I2O_IOPIM_P1EM | I2O_IOPIM_P1SNE);
+ mask = ioread32(&adev->i2o_reg->iopim) & enable;
+ iowrite32(mask, &adev->i2o_reg->iopim);
+ }
+ return 0;
+
+err_req2:
+ free_irq(adev->irq, chan);
+err_req1:
+ irq_dispose_mapping(adev->irq);
+err_irq_map:
+ if (adev->err_irq > 0) {
+ if (atomic_dec_and_test(&ppc460ex_adma_err_irq_ref))
+ irq_dispose_mapping(adev->err_irq);
+ }
+ return ret;
+}
+static void ppc460ex_adma_release_irqs(struct ppc460ex_adma_device *adev,
+ struct ppc460ex_adma_chan *chan)
+{
+ u32 mask;
+
+ /* disable DMAx engine interrupts */
+ mask = ioread32(&adev->i2o_reg->iopim) | I2O_IOPIM_P1SNE
+ | I2O_IOPIM_P1EM ;
+ iowrite32(mask, &adev->i2o_reg->iopim);
+
+ free_irq(adev->irq, chan);
+ irq_dispose_mapping(adev->irq);
+ if (adev->err_irq > 0) {
+ free_irq(adev->err_irq, chan);
+ if (atomic_dec_and_test(&ppc460ex_adma_err_irq_ref)) {
+ irq_dispose_mapping(adev->err_irq);
+ iounmap(adev->i2o_reg);
+ }
+ }
+}
+/*
+ * ppc460ex_adma_remove - remove the asynch device
+ */
+static int __devexit ppc460ex_adma_remove(struct of_device *ofdev)
+{
+ struct ppc460ex_adma_device *device = dev_get_drvdata(&ofdev->dev);
+ struct device_node *np = ofdev->dev.of_node;
+ struct resource res;
+ struct dma_chan *chan, *_chan;
+ struct ppc460ex_adma_chan *ppc460ex_chan;
+
+ dev_set_drvdata(&ofdev->dev, NULL);
+
+ dma_async_device_unregister(&device->common);
+
+ dma_free_coherent(device->dev, device->pool_size,
+ device->dma_desc_pool_virt, device->dma_desc_pool);
+
+ list_for_each_entry_safe(chan, _chan, &device->common.channels,
+ device_node) {
+ ppc460ex_chan = to_ppc460ex_adma_chan(chan);
+ dma_unmap_page(&ofdev->dev, ppc460ex_chan->pdest,
+ PAGE_SIZE, DMA_BIDIRECTIONAL);
+ dma_unmap_page(&ofdev->dev, ppc460ex_chan->qdest,
+ PAGE_SIZE, DMA_BIDIRECTIONAL);
+ __free_page(ppc460ex_chan->pdest_page);
+ __free_page(ppc460ex_chan->qdest_page);
+ list_del(&chan->device_node);
+ kfree(ppc460ex_chan);
+ }
+
+ dma_free_coherent(device->dev, device->pool_size,
+ device->dma_desc_pool_virt, device->dma_desc_pool);
+
+
+ iounmap(device->dma_reg);
+ of_address_to_resource(np, 0, &res);
+ release_mem_region(res.start, resource_size(&res));
+ kfree(device);
+ return 0;
+}
+/*
+ * ppc460ex_adma_probe - Probe the DMA engine for features.
+ */
+static int __devinit ppc460ex_adma_probe(struct of_device *ofdev,
+ const struct of_device_id *match)
+{
+ struct device_node *np = ofdev->dev.of_node;
+ struct resource res;
+ struct ppc_dma_chan_ref *ref, *_ref;
+ struct ppc460ex_adma_device *adev;
+ struct ppc460ex_adma_chan *chan;
+ int ret = 0, initcode = PPC_ADMA_INIT_OK;
+ void *regs;
+
+ if (of_address_to_resource(np, 0, &res)) {
+ dev_err(&ofdev->dev, "Failed to get memory resource\n");
+ ret = -ENODEV;
+ goto out;
+ }
+
+ if (!request_mem_region(res.start, resource_size(&res),
+ dev_driver_string(&ofdev->dev))) {
+ dev_err(&ofdev->dev, "failed to request memory region "
+ "(0x%016llx-0x%016llx)\n",
+ (u64)res.start, (u64)res.end);
+ ret = -EBUSY;
+ goto out;
+ }
+
+ /* create a device */
+ adev = kzalloc(sizeof(*adev), GFP_KERNEL);
+ if (!adev) {
+ dev_err(&ofdev->dev, "failed to allocate device\n");
+ initcode = PPC_ADMA_INIT_ALLOC;
+ ret = -ENOMEM;
+ goto err_adev_alloc;
+ }
+
+ adev->id = 0;
+ adev->pool_size = DMA_FIFO_SIZE << 2;
+ /*
+ * allocate coherent memory for hardware descriptors
+ */
+ adev->dma_desc_pool_virt = dma_alloc_coherent(&ofdev->dev,
+ adev->pool_size, &adev->dma_desc_pool,
+ GFP_KERNEL);
+ if (adev->dma_desc_pool_virt == NULL) {
+ dev_err(&ofdev->dev, "failed to allocate %d bytes of coherent "
+ "memory for hardware descriptors\n",
+ adev->pool_size);
+ ret = -ENOMEM;
+ goto err_dma_alloc;
+ }
+
+ regs = ioremap(res.start, resource_size(&res));
+ if (!regs) {
+ dev_err(&ofdev->dev, "failed to ioremap regs!\n");
+ goto err_regs_alloc;
+ }
+
+ adev->dma_reg = regs;
+ /*
+ *DMA FIFO length = CSlength + CPlength;
+ */
+ iowrite32(DMA_FIFO_ENABLE | ((DMA_FIFO_SIZE >> 3) - 2),
+ &adev->dma_reg->fsiz);
+ /* Configure DMA engine */
+ iowrite32(DMA_CFG_DXEPR_HP | DMA_CFG_DFMPP_HP | DMA_CFG_FALGN,
+ &adev->dma_reg->cfg);
+ /* Clear Status */
+ iowrite32(~0, &adev->dma_reg->dsts);
+
+ adev->dev = &ofdev->dev;
+ adev->common.dev = &ofdev->dev;
+ INIT_LIST_HEAD(&adev->common.channels);
+ dev_set_drvdata(&ofdev->dev, adev);
+
+ /* create a channel */
+ chan = kzalloc(sizeof(*chan), GFP_KERNEL);
+ if (!chan) {
+ dev_err(&ofdev->dev, "can't allocate channel structure\n");
+ ret = -ENOMEM;
+ goto err_chan_alloc;
+ }
+ spin_lock_init(&chan->lock);
+ INIT_LIST_HEAD(&chan->chain);
+ INIT_LIST_HEAD(&chan->all_slots);
+ chan->device = adev;
+ chan->common.device = &adev->common;
+ list_add_tail(&chan->common.device_node, &adev->common.channels);
+ tasklet_init(&chan->irq_tasklet, ppc460ex_adma_tasklet,
+ (unsigned long)chan);
+ /*
+ * allocate and map helper pages for async validation or
+ * async_mult/async_sum_product operations on DMA0/1.
+ */
+ chan->pdest_page = alloc_page(GFP_KERNEL);
+ chan->qdest_page = alloc_page(GFP_KERNEL);
+ if (!chan->pdest_page ||
+ !chan->qdest_page) {
+ if (chan->pdest_page)
+ __free_page(chan->pdest_page);
+ if (chan->qdest_page)
+ __free_page(chan->qdest_page);
+ ret = -ENOMEM;
+ goto err_page_alloc;
+ }
+ chan->pdest = dma_map_page(&ofdev->dev, chan->pdest_page, 0,
+ PAGE_SIZE, DMA_BIDIRECTIONAL);
+ chan->qdest = dma_map_page(&ofdev->dev, chan->qdest_page, 0,
+ PAGE_SIZE, DMA_BIDIRECTIONAL);
+
+ ref = kmalloc(sizeof(*ref), GFP_KERNEL);
+ if (ref) {
+ ref->chan = &chan->common;
+ INIT_LIST_HEAD(&ref->node);
+ list_add_tail(&ref->node, &ppc460ex_adma_chan_list);
+ } else {
+ dev_err(&ofdev->dev, "failed to allocate channel reference!\n");
+ ret = -ENOMEM;
+ goto err_ref_alloc;
+ }
+
+
+ ret = ppc460ex_setup_irqs(adev, chan, &initcode);
+ if (ret)
+ goto err_irq;
+ ppc460ex_adma_init_capabilities(adev);
+
+ ret = dma_async_device_register(&adev->common);
+ if (ret) {
+ dev_err(&ofdev->dev, "failed to register dma device\n");
+ goto err_dev_reg;
+ }
+ goto out;
+
+err_dev_reg:
+ ppc460ex_adma_release_irqs(adev, chan);
+err_irq:
+ list_for_each_entry_safe(ref, _ref, &ppc460ex_adma_chan_list, node) {
+ if (chan == to_ppc460ex_adma_chan(ref->chan)) {
+ list_del(&ref->node);
+ kfree(ref);
+ }
+ }
+
+err_ref_alloc:
+ dma_unmap_page(&ofdev->dev, chan->pdest,
+ PAGE_SIZE, DMA_BIDIRECTIONAL);
+ dma_unmap_page(&ofdev->dev, chan->qdest,
+ PAGE_SIZE, DMA_BIDIRECTIONAL);
+ __free_page(chan->pdest_page);
+ __free_page(chan->qdest_page);
+
+err_page_alloc:
+ kfree(chan);
+err_chan_alloc:
+ iounmap(adev->dma_reg);
+err_regs_alloc:
+ dma_free_coherent(adev->dev, adev->pool_size,
+ adev->dma_desc_pool_virt,
+ adev->dma_desc_pool);
+err_dma_alloc:
+ kfree(adev);
+err_adev_alloc:
+ release_mem_region(res.start, resource_size(&res));
+out:
+ return ret;
+
+}
+/*
+ * Create sys fs entries to enable DMA engine and select RAID-6 Poly
+ * nomial
+ */
+/*
+ *One time initialization which would decide some proparties like
+ *FIO depth, priority LL HB buses etc.
+ */
+static int ppc460ex_configure_raid_devices(void)
+{
+ struct device_node *np;
+ struct resource i2o_res;
+ struct i2o_regs __iomem *i2o_reg;
+ dcr_host_t i2o_dcr_host;
+ unsigned int dcr_base, dcr_len;
+ int i, ret;
+
+ np = of_find_compatible_node(NULL, NULL, "ibm,i2o-460ex");
+ if (!np) {
+ pr_err("%s: can't find I2O device tree node\n",
+ __func__);
+ return -ENODEV;
+ }
+
+ if (of_address_to_resource(np, 0, &i2o_res)) {
+ of_node_put(np);
+ return -EINVAL;
+ }
+
+ i2o_reg = of_iomap(np, 0);
+ if (!i2o_reg) {
+ pr_err("%s: failed to map I2O registers\n", __func__);
+ of_node_put(np);
+ return -EINVAL;
+ }
+
+ /* Get I2O DCRs base */
+ dcr_base = dcr_resource_start(np, 0);
+ dcr_len = dcr_resource_len(np, 0);
+ if (!dcr_base && !dcr_len) {
+ pr_err("%s: can't get DCR registers base/len!\n",
+ np->full_name);
+ of_node_put(np);
+ iounmap(i2o_reg);
+ return -ENODEV;
+ }
+
+ i2o_dcr_host = dcr_map(np, dcr_base, dcr_len);
+ if (!DCR_MAP_OK(i2o_dcr_host)) {
+ pr_err("%s: failed to map DCRs!\n", np->full_name);
+ of_node_put(np);
+ iounmap(i2o_reg);
+ return -ENODEV;
+ }
+ of_node_put(np);
+
+ ppc460ex_dma_fifo_buf = kmalloc(DMA_FIFO_SIZE, GFP_KERNEL);
+ if (!ppc460ex_dma_fifo_buf) {
+ pr_err("%s: DMA FIFO buffer allocation failed\n", __func__);
+ iounmap(i2o_reg);
+ dcr_unmap(i2o_dcr_host, dcr_len);
+ return -ENOMEM;
+ }
+
+ /*
+ * Confgiure HW
+ */
+ /* reset DMA */
+ mtdcri(SDR0, DCRN_SDR0_SRST, DCRN_SDR0_SRST_I2ODMA);
+ mtdcri(SDR0, DCRN_SDR0_SRST, 0);
+
+ /* Setup the base address of mmapped registers */
+ dcr_write(i2o_dcr_host, DCRN_I2O0_IBAH, (u32)(i2o_res.start >> 32));
+ dcr_write(i2o_dcr_host, DCRN_I2O0_IBAL, (u32)(i2o_res.start) |
+ I2O_REG_ENABLE);
+ dcr_unmap(i2o_dcr_host, dcr_len);
+
+ /* Setup FIFO memory space base address */
+ iowrite32(0, &i2o_reg->ifbah);
+ iowrite32(((u32)__pa(ppc460ex_dma_fifo_buf)), &i2o_reg->ifbal);
+
+ /* set zero FIFO size for I2O, so the whole
+ * ppc460ex_dma_fifo_buf is used by DMAs.
+ * DMAx_FIFOs will be configured while probe.
+ */
+ iowrite32(0, &i2o_reg->ifsiz);
+ iounmap(i2o_reg);
+
+ /* To prepare WXOR/RXOR functionality we need access to
+ * Memory Queue Module DCRs (finally it will be enabled
+ * via /sys interface of the ppc460ex ADMA driver).
+ */
+ np = of_find_compatible_node(NULL, NULL, "ibm,mq-460ex");
+ if (!np) {
+ pr_err("%s: can't find MQ device tree node\n",
+ __func__);
+ ret = -ENODEV;
+ goto out_free;
+ }
+
+ /* Get MQ DCRs base */
+ dcr_base = dcr_resource_start(np, 0);
+ dcr_len = dcr_resource_len(np, 0);
+ if (!dcr_base && !dcr_len) {
+ pr_err("%s: can't get DCR registers base/len!\n",
+ np->full_name);
+ ret = -ENODEV;
+ goto out_mq;
+ }
+
+ ppc460ex_mq_dcr_host = dcr_map(np, dcr_base, dcr_len);
+ if (!DCR_MAP_OK(ppc460ex_mq_dcr_host)) {
+ pr_err("%s: failed to map DCRs!\n", np->full_name);
+ ret = -ENODEV;
+ goto out_mq;
+ }
+ of_node_put(np);
+ ppc460ex_mq_dcr_len = dcr_len;
+
+ /* Set HB alias */
+ dcr_write(ppc460ex_mq_dcr_host, DCRN_MQ0_BAUH, DMA_CUED_XOR_HB);
+
+ /* Set:
+ * - LL transaction passing limit to 1;
+ * - Memory controller cycle limit to 1;
+ * - Galois Polynomial to 0x14d (default)
+ */
+ dcr_write(ppc460ex_mq_dcr_host, DCRN_MQ0_CFBHL,
+ (1 << MQ0_CFBHL_TPLM) | (1 << MQ0_CFBHL_HBCL) |
+ (PPC460EX_DEFAULT_POLY << MQ0_CFBHL_POLY));
+
+ atomic_set(&ppc460ex_adma_err_irq_ref, 0);
+ for (i = 0; i < PPC460EX_ADMA_ENGINES_NUM; i++)
+ ppc460ex_adma_devices[i] = -1;
+
+ return 0;
+
+out_mq:
+ of_node_put(np);
+out_free:
+ kfree(ppc460ex_dma_fifo_buf);
+ return ret;
+
+}
+
+static struct of_device_id adma_match[] = {
+ {
+ .compatible = "amcc,dma-460ex",
+ },
+ {},
+};
+static struct of_platform_driver ppc460ex_adma_driver = {
+ .probe = ppc460ex_adma_probe,
+ .remove = ppc460ex_adma_remove,
+ .driver = {
+ .name = "PPC460Ex-ADMA",
+ .owner = THIS_MODULE,
+ .of_match_table = adma_match,
+ },
+};
+static int __init ppc460ex_adma_init(void)
+{
+ int rval;
+
+ rval = ppc460ex_configure_raid_devices();
+ if (rval)
+ return rval;
+
+ rval = of_register_platform_driver(&ppc460ex_adma_driver);
+ if (rval) {
+ pr_err("%s: Driver reigstration failed\n", __func__);
+ goto out_reg;
+ }
+ /* Initialization status */
+ rval = driver_create_file(&ppc460ex_adma_driver.driver,
+ &driver_attr_devices);
+ if (rval)
+ goto out_dev;
+
+ /* RAID-6 h/w enable entry */
+ rval = driver_create_file(&ppc460ex_adma_driver.driver,
+ &driver_attr_raid6_enable);
+ if (rval)
+ goto out_en;
+ /* RAID-5 h/w enable entry */
+ rval = driver_create_file(&ppc460ex_adma_driver.driver,
+ &driver_attr_raid5_enable);
+ if (rval)
+ goto out_en;
+
+ /* GF polynomial to use */
+ rval = driver_create_file(&ppc460ex_adma_driver.driver,
+ &driver_attr_poly);
+ if (!rval)
+ return rval;
+
+out_en:
+ driver_remove_file(&ppc460ex_adma_driver.driver,
+ &driver_attr_devices);
+out_dev:
+ /* User will not be able to enable h/w RAID-6 */
+ pr_err("%s: failed to create RAID-6 driver interface\n",
+ __func__);
+ of_unregister_platform_driver(&ppc460ex_adma_driver);
+out_reg:
+ dcr_unmap(ppc460ex_mq_dcr_host , ppc460ex_mq_dcr_len);
+ kfree(ppc460ex_dma_fifo_buf);
+ return rval;
+
+
+}
+static void __exit ppc460ex_adma_exit(void)
+{
+ driver_remove_file(&ppc460ex_adma_driver.driver,
+ &driver_attr_poly);
+ driver_remove_file(&ppc460ex_adma_driver.driver,
+ &driver_attr_raid5_enable);
+ driver_remove_file(&ppc460ex_adma_driver.driver,
+ &driver_attr_raid6_enable);
+ driver_remove_file(&ppc460ex_adma_driver.driver,
+ &driver_attr_devices);
+ of_unregister_platform_driver(&ppc460ex_adma_driver);
+ dcr_unmap(ppc460ex_mq_dcr_host, ppc460ex_mq_dcr_len);
+}
+arch_initcall(ppc460ex_adma_init);
+module_exit(ppc460ex_adma_exit);
+
+MODULE_AUTHOR(" Tirumala R Marri <tmarri@apm.com>");
+MODULE_DESCRIPTION(" PPC460Ex ADMA Engine Driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/dma/ppc4xx/adma1.h b/drivers/dma/ppc4xx/adma1.h
new file mode 100644
index 0000000..7a71f8d
--- /dev/null
+++ b/drivers/dma/ppc4xx/adma1.h
@@ -0,0 +1,192 @@
+/*
+ * 2010 (C) Applied Micro(APM).
+ *
+ * Author: Tirumala R Marri <tmarri@apm.com>
+ *
+ * This file is licensed under the terms of the GNU General Public License
+ * version 2. This program is licensed "as is" without any warranty of
+ * any kind, whether express or implied.
+ */
+#ifndef PPC460EX_ADMA_H
+#define PPC460EX_ADMA_H
+
+#include <linux/types.h>
+#include "dma.h"
+
+#define to_ppc460ex_adma_chan(chan) \
+ container_of(chan, struct ppc460ex_adma_chan, common)
+#define to_ppc460ex_adma_device(dev) \
+ container_of(dev, struct ppc460ex_adma_device, common)
+#define tx_to_ppc460ex_adma_slot(tx) \
+ container_of(tx, struct ppc460ex_adma_desc_slot, async_tx)
+
+#define PPC460EX_R6_PROC_ROOT "driver/460ex_raid6"
+#define PPC460EX_R5_PROC_ROOT "driver/460ex_raid5"
+
+#define PPC460EX_DEFAULT_POLY 0x4d
+
+#define PPC460EX_ADMA_ENGINES_NUM 1
+#define PPC460EX_ADMA_WATCHDOG_MSEC 3
+#define PPC460EX_ADMA_THRESHOLD 1
+
+#define XOR_MAX_OPS 16
+
+
+#define PPC460EX_ADMA_DMA_MAX_BYTE_COUNT 0xFFFFFFUL
+/* this is the XOR_CBBCR width */
+#define PPC460EX_ADMA_XOR_MAX_BYTE_COUNT (1 << 31)
+#define PPC460EX_ADMA_ZERO_SUM_MAX_BYTE_COUNT PPC460EX_ADMA_XOR_MAX_BYTE_COUNT
+
+#define DMA_FIFO_SIZE 0x1000
+
+#define PPC460EX_RXOR_RUN 0
+#define MQ0_CF2H_RXOR_BS_MASK 0x1FF
+
+#define DMA_ZERO_P 7
+
+/**
+ * struct ppc460ex_adma_device - internal representation of an ADMA device
+ * @dev: device
+ * @dma_reg: DMA register base
+ * @i2o_reg: I2O register base
+ * @id: HW ADMA Device selector
+ * @dma_desc_pool_virt: base of DMA descriptor region (CPU address)
+ * @dma_desc_pool: base of DMA descriptor region (DMA address)
+ * @pool_size: Size of the descriptor pool.
+ * @irq: DMA completion interrupt
+ * @err_irq: DMA error interrupt
+ * @common: embedded struct dma_device
+ */
+struct ppc460ex_adma_device {
+ struct device *dev;
+ struct dma_regs __iomem *dma_reg;
+ struct i2o_regs __iomem *i2o_reg;
+ int id;
+ void *dma_desc_pool_virt;
+ dma_addr_t dma_desc_pool;
+ size_t pool_size;
+ int irq;
+ int err_irq;
+ struct dma_device common;
+};
+
+/**
+ * struct ppc460ex_adma_chan - internal representation of an ADMA channel
+ * @lock: serializes enqueue/dequeue operations to the slot pool
+ * @device: parent device
+ * @chain: device chain view of the descriptors
+ * @common: common dmaengine channel object members
+ * @all_slots: complete domain of slots usable by the channel
+ * @last_used:
+ * @pending: allows batching of hardware operations
+ * @completed_cookie: identifier for the most recently completed operation
+ * @slots_allocated: records the actual size of the descriptor slot pool
+ * @hw_chain_inited: h/w descriptor chain initialization flag
+ * @irq_tasklet: bottom half where ppc460ex_adma_slot_cleanup runs
+ * @needs_unmap: if buffers should not be unmapped upon final processing
+ * @pdest_page: P destination page for async validate operation
+ * @qdest_page: Q destination page for async validate operation
+ * @pdest: P dma addr for async validate operation
+ * @qdest: Q dma addr for async validate operation
+ */
+struct ppc460ex_adma_chan {
+ spinlock_t lock;
+ struct ppc460ex_adma_device *device;
+ struct list_head chain;
+ struct dma_chan common;
+ struct list_head all_slots;
+ struct ppc460ex_adma_desc_slot *last_used;
+ int pending;
+ dma_cookie_t completed_cookie;
+ int slots_allocated;
+ int hw_chain_inited;
+ struct tasklet_struct irq_tasklet;
+ u8 needs_unmap;
+ struct page *pdest_page;
+ struct page *qdest_page;
+ dma_addr_t pdest;
+ dma_addr_t qdest;
+};
+
+struct ppc460ex_rxor {
+ u32 addrl;
+ u32 addrh;
+ int len;
+ int xor_count;
+ int addr_count;
+ int desc_count;
+ int state;
+};
+
+/**
+ * struct ppc460ex_adma_desc_slot - PPC460EX-ADMA software descriptor
+ * @phys: hardware address of the hardware descriptor chain
+ * @group_head: first operation in a transaction
+ * @hw_next: pointer to the next descriptor in chain
+ * @async_tx: support for the async_tx api
+ * @slot_node: node on the iop_adma_chan.all_slots list
+ * @chain_node: node on the op_adma_chan.chain list
+ * @group_list: list of slots that make up a multi-descriptor transaction
+ * for example transfer lengths larger than the supported hw max
+ * @unmap_len: transaction bytecount
+ * @hw_desc: virtual address of the hardware descriptor chain
+ * @stride: currently chained or not
+ * @idx: pool index
+ * @slot_cnt: total slots used in an transaction (group of operations)
+ * @src_cnt: number of sources set in this descriptor
+ * @dst_cnt: number of destinations set in the descriptor
+ * @slots_per_op: number of slots per operation
+ * @descs_per_op: number of slot per P/Q operation see comment
+ * for ppc460ex_prep_dma_pqxor function
+ * @flags: desc state/type
+ * @reverse_flags: 1 if a corresponding rxor address uses reversed address order
+ * @xor_check_result: result of zero sum
+ * @crc32_result: result crc calculation
+ */
+struct ppc460ex_adma_desc_slot {
+ dma_addr_t phys;
+ struct ppc460ex_adma_desc_slot *group_head;
+ struct ppc460ex_adma_desc_slot *hw_next;
+ struct dma_async_tx_descriptor async_tx;
+ struct list_head slot_node;
+ struct list_head chain_node; /* node in channel ops list */
+ struct list_head group_list; /* list */
+ unsigned int unmap_len;
+ void *hw_desc;
+ u16 stride;
+ u16 idx;
+ u16 slot_cnt;
+ u8 src_cnt;
+ u8 dst_cnt;
+ u8 slots_per_op;
+ u8 descs_per_op;
+ unsigned long flags;
+ unsigned long reverse_flags[8];
+
+#define PPC460EX_DESC_INT 0 /* generate interrupt on complete */
+#define PPC460EX_ZERO_P 1 /* clear P destionaion */
+#define PPC460EX_ZERO_Q 2 /* clear Q destination */
+#define PPC460EX_COHERENT 3 /* src/dst are coherent */
+
+#define PPC460EX_DESC_WXOR 4 /* WXORs are in chain */
+#define PPC460EX_DESC_RXOR 5 /* RXOR is in chain */
+
+#define PPC460EX_DESC_RXOR123 8 /* CDB for RXOR123 operation */
+#define PPC460EX_DESC_RXOR124 9 /* CDB for RXOR124 operation */
+#define PPC460EX_DESC_RXOR125 10 /* CDB for RXOR125 operation */
+#define PPC460EX_DESC_RXOR12 11 /* CDB for RXOR12 operation */
+#define PPC460EX_DESC_RXOR_REV 12 /* CDB cont srcs in reversed order*/
+
+#define PPC460EX_DESC_PCHECK 13
+#define PPC460EX_DESC_QCHECK 14
+
+#define PPC460EX_DESC_RXOR_MSK 0x3
+
+
+ union {
+ u32 *xor_check_result;
+ u32 *crc32_result;
+ };
+};
+
+#endif /* PPC460EX_ADMA_H*/
diff --git a/drivers/dma/ppc4xx/dma.h b/drivers/dma/ppc4xx/dma.h
index bcde2df..9c05b1f 100644
--- a/drivers/dma/ppc4xx/dma.h
+++ b/drivers/dma/ppc4xx/dma.h
@@ -10,11 +10,23 @@
* kind, whether express or implied.
*/
-#ifndef _PPC440SPE_DMA_H
-#define _PPC440SPE_DMA_H
+#ifndef _PPC4XX_DMA_H
+#define _PPC4XX_DMA_H
#include <linux/types.h>
+#if defined(CONFIG_PPC460EX)
+
+/* Number of elements in the array with statical CDBs */
+#define MAX_STAT_DMA_CDBS 16
+/* Number of DMA engines available on the contoller */
+#define DMA_ENGINES_NUM 1
+/* Maximum h/w supported number of destinations */
+#define DMA_DEST_MAX_NUM 2
+
+#define DMA_FIFO_SIZE 0x1000
+#else
+
/* Number of elements in the array with statical CDBs */
#define MAX_STAT_DMA_CDBS 16
/* Number of DMA engines available on the contoller */
@@ -57,6 +69,8 @@
#define I2O_IOPIM_P1SNE (1<<6)
#define I2O_IOPIM_P1EM (1<<8)
+#endif /*defined(CONFIG_460EX)*/
+
/* DMA CDB fields */
#define DMA_CDB_MSK (0xF)
#define DMA_CDB_64B_ADDR (1<<2)
@@ -220,4 +234,4 @@ struct i2o_regs {
u32 iopt;
};
-#endif /* _PPC440SPE_DMA_H */
+#endif /* _PPC4XX_DMA_H */
--
1.6.1.rc3
^ permalink raw reply related
* [PATCH 4/5] powerpc/mm: Add some debug output when hash insertion fails
From: Benjamin Herrenschmidt @ 2010-07-23 0:41 UTC (permalink / raw)
To: linuxppc-dev
In-Reply-To: <1279845704-14848-3-git-send-email-benh@kernel.crashing.org>
This adds some debug output to our MMU hash code to print out some
useful debug data if the hypervisor refuses the insertion (which
should normally never happen).
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
---
arch/powerpc/include/asm/mmu-hash64.h | 4 ++-
arch/powerpc/mm/hash_utils_64.c | 34 ++++++++++++++++++++++++++++----
arch/powerpc/mm/hugetlbpage-hash64.c | 2 +
3 files changed, 34 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/include/asm/mmu-hash64.h b/arch/powerpc/include/asm/mmu-hash64.h
index 2102b21..0e398cf 100644
--- a/arch/powerpc/include/asm/mmu-hash64.h
+++ b/arch/powerpc/include/asm/mmu-hash64.h
@@ -250,7 +250,9 @@ extern int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid,
pte_t *ptep, unsigned long trap, int local, int ssize,
unsigned int shift, unsigned int mmu_psize);
-
+extern void hash_failure_debug(unsigned long ea, unsigned long access,
+ unsigned long vsid, unsigned long trap,
+ int ssize, int psize, unsigned long pte);
extern int htab_bolt_mapping(unsigned long vstart, unsigned long vend,
unsigned long pstart, unsigned long prot,
int psize, int ssize);
diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c
index 40847a9..09dffe6 100644
--- a/arch/powerpc/mm/hash_utils_64.c
+++ b/arch/powerpc/mm/hash_utils_64.c
@@ -871,6 +871,18 @@ static inline int subpage_protection(struct mm_struct *mm, unsigned long ea)
}
#endif
+void hash_failure_debug(unsigned long ea, unsigned long access,
+ unsigned long vsid, unsigned long trap,
+ int ssize, int psize, unsigned long pte)
+{
+ if (!printk_ratelimit())
+ return;
+ pr_info("mm: Hashing failure ! EA=0x%lx access=0x%lx current=%s\n",
+ ea, access, current->comm);
+ pr_info(" trap=0x%lx vsid=0x%lx ssize=%d psize=%d pte=0x%lx\n",
+ trap, vsid, ssize, psize, pte);
+}
+
/* Result code is:
* 0 - handled
* 1 - normal page fault
@@ -1036,6 +1048,12 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
local, ssize, spp);
}
+ /* Dump some info in case of hash insertion failure, they should
+ * never happen so it is really useful to know if/when they do
+ */
+ if (rc == -1)
+ hash_failure_debug(ea, access, vsid, trap, ssize, psize,
+ pte_val(*ptep));
#ifndef CONFIG_PPC_64K_PAGES
DBG_LOW(" o-pte: %016lx\n", pte_val(*ptep));
#else
@@ -1054,8 +1072,7 @@ void hash_preload(struct mm_struct *mm, unsigned long ea,
void *pgdir;
pte_t *ptep;
unsigned long flags;
- int local = 0;
- int ssize;
+ int rc, ssize, local = 0;
BUG_ON(REGION_ID(ea) != USER_REGION_ID);
@@ -1101,11 +1118,18 @@ void hash_preload(struct mm_struct *mm, unsigned long ea,
/* Hash it in */
#ifdef CONFIG_PPC_HAS_HASH_64K
if (mm->context.user_psize == MMU_PAGE_64K)
- __hash_page_64K(ea, access, vsid, ptep, trap, local, ssize);
+ rc = __hash_page_64K(ea, access, vsid, ptep, trap, local, ssize);
else
#endif /* CONFIG_PPC_HAS_HASH_64K */
- __hash_page_4K(ea, access, vsid, ptep, trap, local, ssize,
- subpage_protection(pgdir, ea));
+ rc = __hash_page_4K(ea, access, vsid, ptep, trap, local, ssize,
+ subpage_protection(pgdir, ea));
+
+ /* Dump some info in case of hash insertion failure, they should
+ * never happen so it is really useful to know if/when they do
+ */
+ if (rc == -1)
+ hash_failure_debug(ea, access, vsid, trap, ssize,
+ mm->context.user_psize, pte_val(*ptep));
local_irq_restore(flags);
}
diff --git a/arch/powerpc/mm/hugetlbpage-hash64.c b/arch/powerpc/mm/hugetlbpage-hash64.c
index fd0a15e..db70c0d 100644
--- a/arch/powerpc/mm/hugetlbpage-hash64.c
+++ b/arch/powerpc/mm/hugetlbpage-hash64.c
@@ -127,6 +127,8 @@ repeat:
*/
if (unlikely(slot == -2)) {
*ptep = __pte(old_pte);
+ hash_failure_debug(ea, access, vsid, trap, ssize,
+ mmu_psize, old_pte);
return -1;
}
--
1.6.3.3
^ permalink raw reply related
* [PATCH 2/5] powerpc/mm: Move around testing of _PAGE_PRESENT in hash code
From: Benjamin Herrenschmidt @ 2010-07-23 0:41 UTC (permalink / raw)
To: linuxppc-dev
In-Reply-To: <1279845704-14848-1-git-send-email-benh@kernel.crashing.org>
Instead of adding _PAGE_PRESENT to the access permission mask
in each low level routine independently, we add it once from
hash_page().
We also move the preliminary access check (the racy one before
the PTE is locked) up so it applies to the huge page case. This
duplicates code in __hash_page_huge() which we'll remove in a
subsequent patch to fix a race in there.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/mm/hash_low_64.S | 9 ---------
arch/powerpc/mm/hash_utils_64.c | 19 +++++++++++--------
2 files changed, 11 insertions(+), 17 deletions(-)
diff --git a/arch/powerpc/mm/hash_low_64.S b/arch/powerpc/mm/hash_low_64.S
index a719f53..3079f6b 100644
--- a/arch/powerpc/mm/hash_low_64.S
+++ b/arch/powerpc/mm/hash_low_64.S
@@ -68,9 +68,6 @@ _GLOBAL(__hash_page_4K)
std r8,STK_PARM(r8)(r1)
std r9,STK_PARM(r9)(r1)
- /* Add _PAGE_PRESENT to access */
- ori r4,r4,_PAGE_PRESENT
-
/* Save non-volatile registers.
* r31 will hold "old PTE"
* r30 is "new PTE"
@@ -347,9 +344,6 @@ _GLOBAL(__hash_page_4K)
std r8,STK_PARM(r8)(r1)
std r9,STK_PARM(r9)(r1)
- /* Add _PAGE_PRESENT to access */
- ori r4,r4,_PAGE_PRESENT
-
/* Save non-volatile registers.
* r31 will hold "old PTE"
* r30 is "new PTE"
@@ -687,9 +681,6 @@ _GLOBAL(__hash_page_64K)
std r8,STK_PARM(r8)(r1)
std r9,STK_PARM(r9)(r1)
- /* Add _PAGE_PRESENT to access */
- ori r4,r4,_PAGE_PRESENT
-
/* Save non-volatile registers.
* r31 will hold "old PTE"
* r30 is "new PTE"
diff --git a/arch/powerpc/mm/hash_utils_64.c b/arch/powerpc/mm/hash_utils_64.c
index 98f262d..40847a9 100644
--- a/arch/powerpc/mm/hash_utils_64.c
+++ b/arch/powerpc/mm/hash_utils_64.c
@@ -955,6 +955,17 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
return 1;
}
+ /* Add _PAGE_PRESENT to the required access perm */
+ access |= _PAGE_PRESENT;
+
+ /* Pre-check access permissions (will be re-checked atomically
+ * in __hash_page_XX but this pre-check is a fast path
+ */
+ if (access & ~pte_val(*ptep)) {
+ DBG_LOW(" no access !\n");
+ return 1;
+ }
+
#ifdef CONFIG_HUGETLB_PAGE
if (hugeshift)
return __hash_page_huge(ea, access, vsid, ptep, trap, local,
@@ -967,14 +978,6 @@ int hash_page(unsigned long ea, unsigned long access, unsigned long trap)
DBG_LOW(" i-pte: %016lx %016lx\n", pte_val(*ptep),
pte_val(*(ptep + PTRS_PER_PTE)));
#endif
- /* Pre-check access permissions (will be re-checked atomically
- * in __hash_page_XX but this pre-check is a fast path
- */
- if (access & ~pte_val(*ptep)) {
- DBG_LOW(" no access !\n");
- return 1;
- }
-
/* Do actual hashing */
#ifdef CONFIG_PPC_64K_PAGES
/* If _PAGE_4K_PFN is set, make sure this is a 4k segment */
--
1.6.3.3
^ permalink raw reply related
* [PATCH 3/5] powerpc/mm: Fix bugs in huge page hashing
From: Benjamin Herrenschmidt @ 2010-07-23 0:41 UTC (permalink / raw)
To: linuxppc-dev
In-Reply-To: <1279845704-14848-2-git-send-email-benh@kernel.crashing.org>
There's a couple of nasty bugs lurking in our huge page hashing code.
First, we don't check the access permission atomically with setting
the _PAGE_BUSY bit, which means that the PTE value we end up using
for the hashing might be different than the one we have checked
the access permissions for.
We've seen cases where that leads us to try to use an invalidated
PTE for hashing, causing all sort of "interesting" issues.
Then, we also failed to set _PAGE_DIRTY on a write access.
This fixes both, while also simplifying the code a bit.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
---
arch/powerpc/mm/hugetlbpage-hash64.c | 31 +++++++++++++------------------
1 files changed, 13 insertions(+), 18 deletions(-)
diff --git a/arch/powerpc/mm/hugetlbpage-hash64.c b/arch/powerpc/mm/hugetlbpage-hash64.c
index c9acd79..fd0a15e 100644
--- a/arch/powerpc/mm/hugetlbpage-hash64.c
+++ b/arch/powerpc/mm/hugetlbpage-hash64.c
@@ -21,21 +21,13 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid,
unsigned long old_pte, new_pte;
unsigned long va, rflags, pa, sz;
long slot;
- int err = 1;
BUG_ON(shift != mmu_psize_defs[mmu_psize].shift);
/* Search the Linux page table for a match with va */
va = hpt_va(ea, vsid, ssize);
- /*
- * Check the user's access rights to the page. If access should be
- * prevented then send the problem up to do_page_fault.
- */
- if (unlikely(access & ~pte_val(*ptep)))
- goto out;
- /*
- * At this point, we have a pte (old_pte) which can be used to build
+ /* At this point, we have a pte (old_pte) which can be used to build
* or update an HPTE. There are 2 cases:
*
* 1. There is a valid (present) pte with no associated HPTE (this is
@@ -49,9 +41,17 @@ int __hash_page_huge(unsigned long ea, unsigned long access, unsigned long vsid,
do {
old_pte = pte_val(*ptep);
- if (old_pte & _PAGE_BUSY)
- goto out;
+ /* If PTE busy, retry the access */
+ if (unlikely(old_pte & _PAGE_BUSY))
+ return 0;
+ /* If PTE permissions don't match, take page fault */
+ if (unlikely(access & ~pte_val(*ptep)))
+ return 1;
+ /* Try to lock the PTE, add ACCESSED and DIRTY if it was
+ * a write access */
new_pte = old_pte | _PAGE_BUSY | _PAGE_ACCESSED;
+ if (access & _PAGE_RW)
+ new_pte |= _PAGE_DIRTY;
} while(old_pte != __cmpxchg_u64((unsigned long *)ptep,
old_pte, new_pte));
@@ -127,8 +127,7 @@ repeat:
*/
if (unlikely(slot == -2)) {
*ptep = __pte(old_pte);
- err = -1;
- goto out;
+ return -1;
}
new_pte |= (slot << 12) & (_PAGE_F_SECOND | _PAGE_F_GIX);
@@ -138,9 +137,5 @@ repeat:
* No need to use ldarx/stdcx here
*/
*ptep = __pte(new_pte & ~_PAGE_BUSY);
-
- err = 0;
-
- out:
- return err;
+ return 0;
}
--
1.6.3.3
^ permalink raw reply related
* [PATCH 5/5] powerpc: Fix erroneous lmb->memblock conversions
From: Benjamin Herrenschmidt @ 2010-07-23 0:41 UTC (permalink / raw)
To: linuxppc-dev
In-Reply-To: <1279845704-14848-4-git-send-email-benh@kernel.crashing.org>
Oooops... we missed these. We incorrectly converted strings
used when parsing the device-tree on pseries, thus breaking
access to drconf memory and hotplug memory.
While at it, also revert some variable names that represent
something the FW calls "lmb" and thus don't need to be converted
to "memblock".
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
---
arch/powerpc/kernel/prom.c | 2 +-
arch/powerpc/mm/numa.c | 24 +++++++++++-----------
arch/powerpc/platforms/pseries/hotplug-memory.c | 22 ++++++++++----------
3 files changed, 24 insertions(+), 24 deletions(-)
diff --git a/arch/powerpc/kernel/prom.c b/arch/powerpc/kernel/prom.c
index 9d39539..fed9bf6 100644
--- a/arch/powerpc/kernel/prom.c
+++ b/arch/powerpc/kernel/prom.c
@@ -414,7 +414,7 @@ static int __init early_init_dt_scan_drconf_memory(unsigned long node)
u64 base, size, memblock_size;
unsigned int is_kexec_kdump = 0, rngs;
- ls = of_get_flat_dt_prop(node, "ibm,memblock-size", &l);
+ ls = of_get_flat_dt_prop(node, "ibm,lmb-size", &l);
if (ls == NULL || l < dt_root_size_cells * sizeof(__be32))
return 0;
memblock_size = dt_mem_next_cell(dt_root_size_cells, &ls);
diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index f473645..aa731af 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -398,15 +398,15 @@ static int of_get_drconf_memory(struct device_node *memory, const u32 **dm)
}
/*
- * Retreive and validate the ibm,memblock-size property for drconf memory
+ * Retreive and validate the ibm,lmb-size property for drconf memory
* from the device tree.
*/
-static u64 of_get_memblock_size(struct device_node *memory)
+static u64 of_get_lmb_size(struct device_node *memory)
{
const u32 *prop;
u32 len;
- prop = of_get_property(memory, "ibm,memblock-size", &len);
+ prop = of_get_property(memory, "ibm,lmb-size", &len);
if (!prop || len < sizeof(unsigned int))
return 0;
@@ -562,7 +562,7 @@ static unsigned long __init numa_enforce_memory_limit(unsigned long start,
static inline int __init read_usm_ranges(const u32 **usm)
{
/*
- * For each memblock in ibm,dynamic-memory a corresponding
+ * For each lmb in ibm,dynamic-memory a corresponding
* entry in linux,drconf-usable-memory property contains
* a counter followed by that many (base, size) duple.
* read the counter from linux,drconf-usable-memory
@@ -578,7 +578,7 @@ static void __init parse_drconf_memory(struct device_node *memory)
{
const u32 *dm, *usm;
unsigned int n, rc, ranges, is_kexec_kdump = 0;
- unsigned long memblock_size, base, size, sz;
+ unsigned long lmb_size, base, size, sz;
int nid;
struct assoc_arrays aa;
@@ -586,8 +586,8 @@ static void __init parse_drconf_memory(struct device_node *memory)
if (!n)
return;
- memblock_size = of_get_memblock_size(memory);
- if (!memblock_size)
+ lmb_size = of_get_lmb_size(memory);
+ if (!lmb_size)
return;
rc = of_get_assoc_arrays(memory, &aa);
@@ -611,7 +611,7 @@ static void __init parse_drconf_memory(struct device_node *memory)
continue;
base = drmem.base_addr;
- size = memblock_size;
+ size = lmb_size;
ranges = 1;
if (is_kexec_kdump) {
@@ -1072,7 +1072,7 @@ static int hot_add_drconf_scn_to_nid(struct device_node *memory,
{
const u32 *dm;
unsigned int drconf_cell_cnt, rc;
- unsigned long memblock_size;
+ unsigned long lmb_size;
struct assoc_arrays aa;
int nid = -1;
@@ -1080,8 +1080,8 @@ static int hot_add_drconf_scn_to_nid(struct device_node *memory,
if (!drconf_cell_cnt)
return -1;
- memblock_size = of_get_memblock_size(memory);
- if (!memblock_size)
+ lmb_size = of_get_lmb_size(memory);
+ if (!lmb_size)
return -1;
rc = of_get_assoc_arrays(memory, &aa);
@@ -1100,7 +1100,7 @@ static int hot_add_drconf_scn_to_nid(struct device_node *memory,
continue;
if ((scn_addr < drmem.base_addr)
- || (scn_addr >= (drmem.base_addr + memblock_size)))
+ || (scn_addr >= (drmem.base_addr + lmb_size)))
continue;
nid = of_drconf_to_nid_single(&drmem, &aa);
diff --git a/arch/powerpc/platforms/pseries/hotplug-memory.c b/arch/powerpc/platforms/pseries/hotplug-memory.c
index deab5f9..bc88036 100644
--- a/arch/powerpc/platforms/pseries/hotplug-memory.c
+++ b/arch/powerpc/platforms/pseries/hotplug-memory.c
@@ -69,7 +69,7 @@ static int pseries_remove_memory(struct device_node *np)
const char *type;
const unsigned int *regs;
unsigned long base;
- unsigned int memblock_size;
+ unsigned int lmb_size;
int ret = -EINVAL;
/*
@@ -87,9 +87,9 @@ static int pseries_remove_memory(struct device_node *np)
return ret;
base = *(unsigned long *)regs;
- memblock_size = regs[3];
+ lmb_size = regs[3];
- ret = pseries_remove_memblock(base, memblock_size);
+ ret = pseries_remove_memblock(base, lmb_size);
return ret;
}
@@ -98,7 +98,7 @@ static int pseries_add_memory(struct device_node *np)
const char *type;
const unsigned int *regs;
unsigned long base;
- unsigned int memblock_size;
+ unsigned int lmb_size;
int ret = -EINVAL;
/*
@@ -116,36 +116,36 @@ static int pseries_add_memory(struct device_node *np)
return ret;
base = *(unsigned long *)regs;
- memblock_size = regs[3];
+ lmb_size = regs[3];
/*
* Update memory region to represent the memory add
*/
- ret = memblock_add(base, memblock_size);
+ ret = memblock_add(base, lmb_size);
return (ret < 0) ? -EINVAL : 0;
}
static int pseries_drconf_memory(unsigned long *base, unsigned int action)
{
struct device_node *np;
- const unsigned long *memblock_size;
+ const unsigned long *lmb_size;
int rc;
np = of_find_node_by_path("/ibm,dynamic-reconfiguration-memory");
if (!np)
return -EINVAL;
- memblock_size = of_get_property(np, "ibm,memblock-size", NULL);
- if (!memblock_size) {
+ lmb_size = of_get_property(np, "ibm,lmb-size", NULL);
+ if (!lmb_size) {
of_node_put(np);
return -EINVAL;
}
if (action == PSERIES_DRCONF_MEM_ADD) {
- rc = memblock_add(*base, *memblock_size);
+ rc = memblock_add(*base, *lmb_size);
rc = (rc < 0) ? -EINVAL : 0;
} else if (action == PSERIES_DRCONF_MEM_REMOVE) {
- rc = pseries_remove_memblock(*base, *memblock_size);
+ rc = pseries_remove_memblock(*base, *lmb_size);
} else {
rc = -EINVAL;
}
--
1.6.3.3
^ permalink raw reply related
* [PATCH 1/5] powerpc/mm: Handle hypervisor pte insert failure in __hash_page_huge
From: Benjamin Herrenschmidt @ 2010-07-23 0:41 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Anton Blanchard
From: Anton Blanchard <anton@samba.org>
If the hypervisor gives us an error on a hugepage insert we panic. The
normal page code already handles this by returning an error instead and we end
calling low_hash_fault which will just kill the task if possible.
The patch below does a similar thing for the hugepage case.
Signed-off-by: Anton Blanchard <anton@samba.org>
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/mm/hugetlbpage-hash64.c | 11 +++++++++--
1 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/mm/hugetlbpage-hash64.c b/arch/powerpc/mm/hugetlbpage-hash64.c
index 1995398..c9acd79 100644
--- a/arch/powerpc/mm/hugetlbpage-hash64.c
+++ b/arch/powerpc/mm/hugetlbpage-hash64.c
@@ -121,8 +121,15 @@ repeat:
}
}
- if (unlikely(slot == -2))
- panic("hash_huge_page: pte_insert failed\n");
+ /*
+ * Hypervisor failure. Restore old pte and return -1
+ * similar to __hash_page_*
+ */
+ if (unlikely(slot == -2)) {
+ *ptep = __pte(old_pte);
+ err = -1;
+ goto out;
+ }
new_pte |= (slot << 12) & (_PAGE_F_SECOND | _PAGE_F_GIX);
}
--
1.6.3.3
^ permalink raw reply related
* Re: [PATCH 1/9 v1.02] Add Synopsys DesignWare HS USB OTG Controller driver.
From: Greg KH @ 2010-07-23 0:34 UTC (permalink / raw)
To: David Daney; +Cc: linuxppc-dev, linux-usb, Mark Miesfeld, gregkh, Fushen Chen
In-Reply-To: <4C48DE1C.4040008@caviumnetworks.com>
On Thu, Jul 22, 2010 at 05:11:08PM -0700, David Daney wrote:
> Also the subject line for all nine patches seems to be identical,
> yet the patches are distinct. Perhaps you could find better subject
> lines.
s/Perhaps/Must/
As noted, I can't accept this series, please work to resolve these
issues.
thanks,
greg k-h
^ permalink raw reply
* linux-next: OOPS at bot time
From: Stephen Rothwell @ 2010-07-23 0:22 UTC (permalink / raw)
To: ppc-dev; +Cc: Ben Hutchings, LKML, Jesse Barnes
[-- Attachment #1: Type: text/plain, Size: 3532 bytes --]
Hi all,
My Power7 boot test paniced like this: (next-20100722)
%GQLogic Fibre Channel HBA Driver: 8.03.03-k0
qla2xxx 0002:01:00.2: enabling device (0144 -> 0146)
qla2xxx 0002:01:00.2: Found an ISP8001, irq 35, iobase 0xd000080080014000
------------[ cut here ]------------
kernel BUG at drivers/pci/msi.c:205!
Oops: Exception in kernel mode, sig: 5 [#1]
SMP NR_CPUS=128 NUMA pSeries
last sysfs file: /sys/devices/virtual/tty/ptyz8/uevent
Modules linked in: qla2xxx(+)
NIP: c0000000002fcd54 LR: c000000000048d9c CTR: 0000000000000001
REGS: c00000000278aff0 TRAP: 0700 Not tainted (2.6.35-rc5-autokern1-next-20100721)
MSR: 8000000000029032 <EE,ME,CE,IR,DR> CR: 28422488 XER: 20000008
TASK = c000000002008000[2226] 'modprobe' THREAD: c000000002788000 CPU: 12
GPR00: 0000000000000001 c00000000278b270 c0000000009a36d0 c0000000009b8900
GPR04: c00000000278b2e8 ffffffffffffffff 0000000000000000 0000000000020000
GPR08: 00000000000033e7 c00000000a38b280 0000000000000000 0000000000000000
GPR12: 0000000088422488 c00000000f331800 00000fff921750a0 0000000000000000
GPR16: 0000000010033110 00000000100334b8 0000000000000000 0000000000000000
GPR20: d000080080018000 0000000000022225 c0000000009f7bb4 0000000000010200
GPR24: 000000002000020d 0000000000000025 c00000000278b2e0 c00000000278b2e8
GPR28: 0000000000000001 c00000000d0ac5f8 c000000000af8f00 c00000000a38b280
NIP [c0000000002fcd54] .read_msi_msg_desc+0x24/0x3c
LR [c000000000048d9c] .rtas_setup_msi_irqs+0x1d8/0x254
Call Trace:
[c00000000278b270] [c000000000048d9c] .rtas_setup_msi_irqs+0x1d8/0x254 (unreliable)
[c00000000278b360] [c00000000002a9cc] .arch_setup_msi_irqs+0x34/0x4c
[c00000000278b3e0] [c0000000002fd3fc] .pci_enable_msix+0x49c/0x4ac
[c00000000278b4c0] [d0000000001a5e30] .qla2x00_request_irqs+0x158/0x5b4 [qla2xxx]
[c00000000278b580] [d0000000001cb41c] .qla2x00_probe_one+0xeac/0x63b0 [qla2xxx]
[c00000000278b6f0] [c0000000002f5c4c] .local_pci_probe+0x34/0x48
[c00000000278b760] [c0000000002f6078] .pci_device_probe+0xe8/0x130
[c00000000278b810] [c00000000036e648] .driver_probe_device+0xdc/0x1a4
[c00000000278b8a0] [c00000000036e7a4] .__driver_attach+0x94/0xd8
[c00000000278b930] [c00000000036dabc] .bus_for_each_dev+0x7c/0xe0
[c00000000278b9e0] [c00000000036e410] .driver_attach+0x28/0x40
[c00000000278ba60] [c00000000036d134] .bus_add_driver+0x144/0x310
[c00000000278bb10] [c00000000036ec28] .driver_register+0xd8/0x198
[c00000000278bbb0] [c0000000002f63d0] .__pci_register_driver+0x60/0x10c
[c00000000278bc50] [d0000000001ca520] .qla2x00_module_init+0x150/0x1a0 [qla2xxx]
[c00000000278bce0] [c00000000000947c] .do_one_initcall+0x80/0x1a8
[c00000000278bd90] [c0000000000a4364] .SyS_init_module+0xd8/0x244
[c00000000278be30] [c00000000000852c] syscall_exit+0x0/0x40
Instruction dump:
ebe1fff8 7c0803a6 4e800020 e9230028 81490030 80090034 81690038 7d400378
7c005b78 7c000034 5400d97e 78000020 <0b000000> 81690038 e8090030 91640008
---[ end trace f67a78811ed47c60 ]---
%Gudevd-work[1379]: '/sbin/modprobe -b pci:v00001077d00008001sv00001077sd0000017Fbc0Csc04i00' unexpected exit with status 0x0005
That line number is this:
BUG_ON(!(entry->msg.address_hi | entry->msg.address_lo |
entry->msg.data));
in read_msi_msg_desc(). That BUG_ON was added by commit
2ca1af9aa3285c6a5f103ed31ad09f7399fc65d7 ("PCI: MSI: Remove unsafe and
unnecessary hardware access") from the pci tree.
--
Cheers,
Stephen Rothwell sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/
[-- Attachment #2: Type: application/pgp-signature, Size: 490 bytes --]
^ permalink raw reply
* Re: [PATCH 1/9 v1.02] Add Synopsys DesignWare HS USB OTG Controller driver.
From: David Daney @ 2010-07-23 0:11 UTC (permalink / raw)
To: Fushen Chen; +Cc: linuxppc-dev, linux-usb, Mark Miesfeld, gregkh
In-Reply-To: <12798369431775-git-send-email-fchen@apm.com>
On 07/22/2010 03:15 PM, Fushen Chen wrote:
> The DWC OTG driver module provides the initialization and cleanup
> entry points for the DWC OTG USB driver.
>
> Signed-off-by: Fushen Chen<fchen@apm.com>
> Signed-off-by: Mark Miesfeld<mmiesfeld@apm.com>
> ---
> drivers/Makefile | 1 +
> drivers/usb/Kconfig | 2 +
> drivers/usb/dwc_otg/Kconfig | 96 +++
> drivers/usb/dwc_otg/Makefile | 13 +
> drivers/usb/dwc_otg/dwc_otg_driver.c | 1246 ++++++++++++++++++++++++++++++++++
> drivers/usb/dwc_otg/dwc_otg_driver.h | 97 +++
> drivers/usb/gadget/Kconfig | 21 +
> drivers/usb/gadget/gadget_chips.h | 7 +
> 8 files changed, 1483 insertions(+), 0 deletions(-)
> create mode 100644 drivers/usb/dwc_otg/Kconfig
> create mode 100644 drivers/usb/dwc_otg/Makefile
> create mode 100644 drivers/usb/dwc_otg/dwc_otg_driver.c
> create mode 100644 drivers/usb/dwc_otg/dwc_otg_driver.h
>
> diff --git a/drivers/Makefile b/drivers/Makefile
> index 20dcced..f3fc7c7 100644
> --- a/drivers/Makefile
> +++ b/drivers/Makefile
> @@ -67,6 +67,7 @@ obj-$(CONFIG_UWB) += uwb/
> obj-$(CONFIG_USB_OTG_UTILS) += usb/otg/
> obj-$(CONFIG_USB) += usb/
> obj-$(CONFIG_USB_MUSB_HDRC) += usb/musb/
> +obj-$(CONFIG_USB_DWC_OTG) += usb/dwc_otg/
> obj-$(CONFIG_PCI) += usb/
> obj-$(CONFIG_USB_GADGET) += usb/gadget/
> obj-$(CONFIG_SERIO) += input/serio/
> diff --git a/drivers/usb/Kconfig b/drivers/usb/Kconfig
> index 6a58cb1..f48920b 100644
> --- a/drivers/usb/Kconfig
> +++ b/drivers/usb/Kconfig
> @@ -113,6 +113,8 @@ source "drivers/usb/host/Kconfig"
>
> source "drivers/usb/musb/Kconfig"
>
> +source "drivers/usb/dwc_otg/Kconfig"
> +
> source "drivers/usb/class/Kconfig"
>
> source "drivers/usb/storage/Kconfig"
> diff --git a/drivers/usb/dwc_otg/Kconfig b/drivers/usb/dwc_otg/Kconfig
> new file mode 100644
> index 0000000..27ae0d5
> --- /dev/null
> +++ b/drivers/usb/dwc_otg/Kconfig
> @@ -0,0 +1,96 @@
> +#
> +# USB Dual Role (OTG-ready) Controller Drivers
> +# for silicon based on Synopsys DesignWare IP
> +#
[...]
> diff --git a/drivers/usb/dwc_otg/Makefile b/drivers/usb/dwc_otg/Makefile
> new file mode 100644
> index 0000000..337ff81
> --- /dev/null
> +++ b/drivers/usb/dwc_otg/Makefile
> @@ -0,0 +1,13 @@
> +#
> +# OTG infrastructure and transceiver drivers
> +#
> +obj-$(CONFIG_USB_DWC_OTG) += dwc_otg.o
> +
> +dwc_otg-objs := dwc_otg_driver.o dwc_otg_cil.o dwc_otg_cil_intr.o
> +ifneq ($(CONFIG_DWC_DEVICE_ONLY),y)
> +dwc_otg-objs += dwc_otg_hcd.o dwc_otg_hcd_intr.o \
> + dwc_otg_hcd_queue.o
> +endif
> +ifneq ($(CONFIG_DWC_HOST_ONLY),y)
> +dwc_otg-objs += dwc_otg_pcd.o dwc_otg_pcd_intr.o
> +endif
> diff --git a/drivers/usb/dwc_otg/dwc_otg_driver.c b/drivers/usb/dwc_otg/dwc_otg_driver.c
> new file mode 100644
> index 0000000..3aae30e
Look at all those files you reference in your Makefile. Most of them
don't exist. This will cause the kernel to be unbuildable and break the
ability to use git bisect.
One way to remedy this situation would be to make your Kconfig changes
the last patch in the series.
Also the subject line for all nine patches seems to be identical, yet
the patches are distinct. Perhaps you could find better subject lines.
The very last patch contains exactly one file (dwc_otg_regs.h), but this
file is required by most of the preceding patches. This indicates that
the ordering of the patches and the way that the files were distributed
among the patches could improve. Could you just fold most of the file
addition patches into a single patch?
Or if that is untenable, put the core files in one patch, and then maybe
hcd and pcd seperatly.
This patch contains many lines that are indented with spaces instead of
tabs. How did it manage to pass checkpatch.pl formatted like that?
And finally I would like to suggest taking all the glue-logic functions
in dwc_otg_driver.c and putting them in a separate file (perhaps
something like dwc_otg_amppc.c or something like that). It could be
that initially you just rename dwc_otg_driver.c to dwc_otg_amppc.c.
That would make it easy for me to then add my dwc_otg_octeon.c and use
the driver with OCTEON (in arch/mips/cavium-octeon).
See: http://marc.info/?l=linux-mips&m=125502126531841&w=2
Thanks,
David Daney
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox