Documentation request for MP warp error 0x10

Documentation request for MP warp error 0x10

[Ilia Mirkin]

Hello,

I've recently come across an error reported by the GPU and would like
to know what it means and especially what causes it to be triggered.
Any information would be very useful:

I'm seeing MP warp error 0x10 (appears in MP register 0x48). This is
what we currently have in nouveau:

<reg32 offset="0x048" name="TRAP_WARP_ERROR"> <!-- ctx-switched -->
<bitfield high="15" low="0" name="ID">
<value value="1" name="STACK_MISMATCH"/>
<value value="5" name="MISALIGNED_PC"/>
<value value="8" name="MISALIGNED_GPR"/>
<value value="9" name="INVALID_OPCODE"/>
<value value="13" name="GPR_OUT_OF_BOUNDS"/>
<value value="14" name="MEM_OUT_OF_BOUNDS"/>
<value value="17" name="INVALID_PARAM"/>
</bitfield>
</reg32>

[Additionally it seems like 15 = UNALIGNED_MEM_ACCESS]

It seems to happen whenever I try to access global memory on kepler
(potentially only atomics, not sure at this point). Knowing precisely
what causes the error to get triggered (and esp what we need to do in
order not to trigger it) would be most useful.

For reference, my shader looks something like this (for SM35):

        /*0018*/                   LDC.64 R0, c[0xf][0x1c0];
  /* 0x7ca80780e01ffc02 */
        /*0020*/                   ATOM.E.ADD R2, [R0], R2;
  /* 0x68080000011c000a */
        /*0028*/                   LD.E.CG R0, [R0];
  /* 0xcc800000001c0000 */

I know that the proprietary drivers are a lot more sophisticated and
only do the atomic add from a single lane, but I was assuming that was
not required.

Thanks,

  -ilia
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: Documentation request for MP warp error 0x10

[Robert Morell]

Hi Ilia,

Error 0x10 is INVALID_ADDR_SPACE. It is triggered when an ATOM or RED [1]
instruction accesses local or shared memory. Global memory accesses are the
only allowed accesses for ATOM and RED instructions.


Note that SM30 also has this restriction that ATOM and RED should only be used
on global memory, but it is not error-checked in hardware until SM35.


[1] What is documented as RED internally appears to be called SUREDP in Mesa.


- Robert

On Wed, Sep 30, 2015 at 03:14:47PM -0400, Ilia Mirkin wrote:
> Hello,
> 
> I've recently come across an error reported by the GPU and would like
> to know what it means and especially what causes it to be triggered.
> Any information would be very useful:
> 
> I'm seeing MP warp error 0x10 (appears in MP register 0x48). This is
> what we currently have in nouveau:
> 
> <reg32 offset="0x048" name="TRAP_WARP_ERROR"> <!-- ctx-switched -->
> <bitfield high="15" low="0" name="ID">
> <value value="1" name="STACK_MISMATCH"/>
> <value value="5" name="MISALIGNED_PC"/>
> <value value="8" name="MISALIGNED_GPR"/>
> <value value="9" name="INVALID_OPCODE"/>
> <value value="13" name="GPR_OUT_OF_BOUNDS"/>
> <value value="14" name="MEM_OUT_OF_BOUNDS"/>
> <value value="17" name="INVALID_PARAM"/>
> </bitfield>
> </reg32>
> 
> [Additionally it seems like 15 = UNALIGNED_MEM_ACCESS]
> 
> It seems to happen whenever I try to access global memory on kepler
> (potentially only atomics, not sure at this point). Knowing precisely
> what causes the error to get triggered (and esp what we need to do in
> order not to trigger it) would be most useful.
> 
> For reference, my shader looks something like this (for SM35):
> 
>         /*0018*/                   LDC.64 R0, c[0xf][0x1c0];
>   /* 0x7ca80780e01ffc02 */
>         /*0020*/                   ATOM.E.ADD R2, [R0], R2;
>   /* 0x68080000011c000a */
>         /*0028*/                   LD.E.CG R0, [R0];
>   /* 0xcc800000001c0000 */
> 
> I know that the proprietary drivers are a lot more sophisticated and
> only do the atomic add from a single lane, but I was assuming that was
> not required.
> 
> Thanks,
> 
>   -ilia
> 
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: Documentation request for MP warp error 0x10

[Ilia Mirkin]

Hi Robert,

Thanks for the quick response! That goes in line with my observations
which is that these things happen when using an ATOM/RED instruction.
I've checked and rechecked that I'm generating ops with identical bits
as what the proprietary driver does, however (and nvdisasm prints
identical output). Could you advise what the proper way of indicating
that the memory is "global" to the op? I'm sure I'm just missing
something simple. If you show me what to look for in SM35 I can
probably find it on my own for SM20/SM30/SM50.

In case you're interested in looking at the mesa code, It's available
on my atomic2 branch at:
https://github.com/imirkin/mesa/commits/atomic2 . However I hardly
expect you to debug my buggy code :) The SUREDP stuff is about surface
RED ops, the existing code uses it but I'm going to leave it for image
support and break direct buffer accesses directly into OP_ATOM (and in
NVIDIA terminology RED is just ATOM without a destination).

Thanks,

  -ilia

On Fri, Oct 2, 2015 at 5:48 PM, Robert Morell <rmorell@nvidia.com> wrote:
> Hi Ilia,
>
> Error 0x10 is INVALID_ADDR_SPACE. It is triggered when an ATOM or RED [1]
> instruction accesses local or shared memory. Global memory accesses are the
> only allowed accesses for ATOM and RED instructions.
>
>
> Note that SM30 also has this restriction that ATOM and RED should only be used
> on global memory, but it is not error-checked in hardware until SM35.
>
>
> [1] What is documented as RED internally appears to be called SUREDP in Mesa.
>
>
> - Robert
>
> On Wed, Sep 30, 2015 at 03:14:47PM -0400, Ilia Mirkin wrote:
>> Hello,
>>
>> I've recently come across an error reported by the GPU and would like
>> to know what it means and especially what causes it to be triggered.
>> Any information would be very useful:
>>
>> I'm seeing MP warp error 0x10 (appears in MP register 0x48). This is
>> what we currently have in nouveau:
>>
>> <reg32 offset="0x048" name="TRAP_WARP_ERROR"> <!-- ctx-switched -->
>> <bitfield high="15" low="0" name="ID">
>> <value value="1" name="STACK_MISMATCH"/>
>> <value value="5" name="MISALIGNED_PC"/>
>> <value value="8" name="MISALIGNED_GPR"/>
>> <value value="9" name="INVALID_OPCODE"/>
>> <value value="13" name="GPR_OUT_OF_BOUNDS"/>
>> <value value="14" name="MEM_OUT_OF_BOUNDS"/>
>> <value value="17" name="INVALID_PARAM"/>
>> </bitfield>
>> </reg32>
>>
>> [Additionally it seems like 15 = UNALIGNED_MEM_ACCESS]
>>
>> It seems to happen whenever I try to access global memory on kepler
>> (potentially only atomics, not sure at this point). Knowing precisely
>> what causes the error to get triggered (and esp what we need to do in
>> order not to trigger it) would be most useful.
>>
>> For reference, my shader looks something like this (for SM35):
>>
>>         /*0018*/                   LDC.64 R0, c[0xf][0x1c0];
>>   /* 0x7ca80780e01ffc02 */
>>         /*0020*/                   ATOM.E.ADD R2, [R0], R2;
>>   /* 0x68080000011c000a */
>>         /*0028*/                   LD.E.CG R0, [R0];
>>   /* 0xcc800000001c0000 */
>>
>> I know that the proprietary drivers are a lot more sophisticated and
>> only do the atomic add from a single lane, but I was assuming that was
>> not required.
>>
>> Thanks,
>>
>>   -ilia
>>
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: Documentation request for MP warp error 0x10

[Robert Morell]

Hi Ilia,

On Fri, Oct 02, 2015 at 06:05:21PM -0400, Ilia Mirkin wrote:
> Hi Robert,
> 
> Thanks for the quick response! That goes in line with my observations
> which is that these things happen when using an ATOM/RED instruction.
> I've checked and rechecked that I'm generating ops with identical bits
> as what the proprietary driver does, however (and nvdisasm prints
> identical output). Could you advise what the proper way of indicating
> that the memory is "global" to the op? I'm sure I'm just missing
> something simple. If you show me what to look for in SM35 I can
> probably find it on my own for SM20/SM30/SM50.

Unfortunately this isn't something I know a lot about, so I'm going to
have do some research and get back to you, hopefully within a few days.

> In case you're interested in looking at the mesa code, It's available
> on my atomic2 branch at:
> https://github.com/imirkin/mesa/commits/atomic2 . However I hardly
> expect you to debug my buggy code :) The SUREDP stuff is about surface
> RED ops, the existing code uses it but I'm going to leave it for image
> support and break direct buffer accesses directly into OP_ATOM (and in
> NVIDIA terminology RED is just ATOM without a destination).

Neat.  I'll take a look at it.

Thanks,
Robert

> Thanks,
> 
>   -ilia
> 
> On Fri, Oct 2, 2015 at 5:48 PM, Robert Morell <rmorell@nvidia.com> wrote:
> > Hi Ilia,
> >
> > Error 0x10 is INVALID_ADDR_SPACE. It is triggered when an ATOM or RED [1]
> > instruction accesses local or shared memory. Global memory accesses are the
> > only allowed accesses for ATOM and RED instructions.
> >
> >
> > Note that SM30 also has this restriction that ATOM and RED should only be used
> > on global memory, but it is not error-checked in hardware until SM35.
> >
> >
> > [1] What is documented as RED internally appears to be called SUREDP in Mesa.
> >
> >
> > - Robert
> >
> > On Wed, Sep 30, 2015 at 03:14:47PM -0400, Ilia Mirkin wrote:
> >> Hello,
> >>
> >> I've recently come across an error reported by the GPU and would like
> >> to know what it means and especially what causes it to be triggered.
> >> Any information would be very useful:
> >>
> >> I'm seeing MP warp error 0x10 (appears in MP register 0x48). This is
> >> what we currently have in nouveau:
> >>
> >> <reg32 offset="0x048" name="TRAP_WARP_ERROR"> <!-- ctx-switched -->
> >> <bitfield high="15" low="0" name="ID">
> >> <value value="1" name="STACK_MISMATCH"/>
> >> <value value="5" name="MISALIGNED_PC"/>
> >> <value value="8" name="MISALIGNED_GPR"/>
> >> <value value="9" name="INVALID_OPCODE"/>
> >> <value value="13" name="GPR_OUT_OF_BOUNDS"/>
> >> <value value="14" name="MEM_OUT_OF_BOUNDS"/>
> >> <value value="17" name="INVALID_PARAM"/>
> >> </bitfield>
> >> </reg32>
> >>
> >> [Additionally it seems like 15 = UNALIGNED_MEM_ACCESS]
> >>
> >> It seems to happen whenever I try to access global memory on kepler
> >> (potentially only atomics, not sure at this point). Knowing precisely
> >> what causes the error to get triggered (and esp what we need to do in
> >> order not to trigger it) would be most useful.
> >>
> >> For reference, my shader looks something like this (for SM35):
> >>
> >>         /*0018*/                   LDC.64 R0, c[0xf][0x1c0];
> >>   /* 0x7ca80780e01ffc02 */
> >>         /*0020*/                   ATOM.E.ADD R2, [R0], R2;
> >>   /* 0x68080000011c000a */
> >>         /*0028*/                   LD.E.CG R0, [R0];
> >>   /* 0xcc800000001c0000 */
> >>
> >> I know that the proprietary drivers are a lot more sophisticated and
> >> only do the atomic add from a single lane, but I was assuming that was
> >> not required.
> >>
> >> Thanks,
> >>
> >>   -ilia
> >>
> 
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: Documentation request for MP warp error 0x10

[Ilia Mirkin]

On Fri, Oct 2, 2015 at 6:14 PM, Robert Morell <rmorell@nvidia.com> wrote:
> Hi Ilia,
>
> On Fri, Oct 02, 2015 at 06:05:21PM -0400, Ilia Mirkin wrote:
>> Hi Robert,
>>
>> Thanks for the quick response! That goes in line with my observations
>> which is that these things happen when using an ATOM/RED instruction.
>> I've checked and rechecked that I'm generating ops with identical bits
>> as what the proprietary driver does, however (and nvdisasm prints
>> identical output). Could you advise what the proper way of indicating
>> that the memory is "global" to the op? I'm sure I'm just missing
>> something simple. If you show me what to look for in SM35 I can
>> probably find it on my own for SM20/SM30/SM50.
>
> Unfortunately this isn't something I know a lot about, so I'm going to
> have do some research and get back to you, hopefully within a few days.

Hi Robert,

Were you able to find any further information out about this? Happy to
provide with any traces or additional details as to what I'm doing
(and which is failing).

Thanks,

  -ilia
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: Documentation request for MP warp error 0x10

[Robert Morell]

On Mon, Oct 26, 2015 at 03:28:59PM -0400, Ilia Mirkin wrote:
> On Fri, Oct 2, 2015 at 6:14 PM, Robert Morell <rmorell@nvidia.com> wrote:
> > Hi Ilia,
> >
> > On Fri, Oct 02, 2015 at 06:05:21PM -0400, Ilia Mirkin wrote:
> >> Hi Robert,
> >>
> >> Thanks for the quick response! That goes in line with my observations
> >> which is that these things happen when using an ATOM/RED instruction.
> >> I've checked and rechecked that I'm generating ops with identical bits
> >> as what the proprietary driver does, however (and nvdisasm prints
> >> identical output). Could you advise what the proper way of indicating
> >> that the memory is "global" to the op? I'm sure I'm just missing
> >> something simple. If you show me what to look for in SM35 I can
> >> probably find it on my own for SM20/SM30/SM50.
> >
> > Unfortunately this isn't something I know a lot about, so I'm going to
> > have do some research and get back to you, hopefully within a few days.
> 
> Hi Robert,
> 
> Were you able to find any further information out about this? Happy to
> provide with any traces or additional details as to what I'm doing
> (and which is failing).

I'm still waiting for clarification on some stuff I'm not sure about --
I'll try pinging a little harder.

Sorry for the delay.

- Robert
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: Documentation request for MP warp error 0x10

[Robert Morell]

On Fri, Oct 02, 2015 at 06:05:21PM -0400, Ilia Mirkin wrote:
> Could you advise what the proper way of indicating
> that the memory is "global" to the op? I'm sure I'm just missing
> something simple. If you show me what to look for in SM35 I can
> probably find it on my own for SM20/SM30/SM50.

Sorry again for the delay.  Here's what I've been able to find out about
the generic thread address space (used by the SMs) and what types of
memory it contains.  Hopefully this clears things up.


Local memory is a per-thread space.
Shared memory is a per-CTA space (compute shaders only).

LDL and STL instructions access local memory with a zero offset.
LDS, LSDLK, STS, and STSCUL instructions access shared memory with a zero
offset.

LD, ST, RED, ATOM, and CCTL.D instructions access the generic thread address
space, which is layered on top of the channel's virtual address space.

In the generic thread address space, there are 16MB windows into local and
shared memory; everything not in a Local or Shared address window accesses
global virtual memory.

The local window offset within the generic thread address space is determined
by the SetShaderLocalMemoryWindow class method (offset 0x77c in classes *97 and
*c0).

The shared window offset within the generic thread address space is determined
by the SetShaderSharedMemoryWindow class method (offset 0x214 in classes *c0).

For both methods, the offset is in bytes, but the window must be aligned to a
16MB boundary (so the lower 24 bits of the data must be zero). The upper 32
bits of the windows are hard-coded to 0 (so they must be placed within the
lower 4GB of address space).

Generally, it is expected that software will reserve ranges in the global
virtual address space where these windows will be placed. (Otherwise anything
mapped there will be inaccessible to shaders.)

For graphics shaders, the shared address space logic does not exist, so there
is no need to reserve virtual memory for it.


- Robert
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: Documentation request for MP warp error 0x10

[Ilia Mirkin]

On Fri, Nov 6, 2015 at 3:59 PM, Robert Morell <rmorell@nvidia.com> wrote:
> On Fri, Oct 02, 2015 at 06:05:21PM -0400, Ilia Mirkin wrote:
>> Could you advise what the proper way of indicating
>> that the memory is "global" to the op? I'm sure I'm just missing
>> something simple. If you show me what to look for in SM35 I can
>> probably find it on my own for SM20/SM30/SM50.
>
> Sorry again for the delay.  Here's what I've been able to find out about
> the generic thread address space (used by the SMs) and what types of
> memory it contains.  Hopefully this clears things up.
>
>
> Local memory is a per-thread space.
> Shared memory is a per-CTA space (compute shaders only).
>
> LDL and STL instructions access local memory with a zero offset.
> LDS, LSDLK, STS, and STSCUL instructions access shared memory with a zero
> offset.
>
> LD, ST, RED, ATOM, and CCTL.D instructions access the generic thread address
> space, which is layered on top of the channel's virtual address space.
>
> In the generic thread address space, there are 16MB windows into local and
> shared memory; everything not in a Local or Shared address window accesses
> global virtual memory.
>
> The local window offset within the generic thread address space is determined
> by the SetShaderLocalMemoryWindow class method (offset 0x77c in classes *97 and
> *c0).
>
> The shared window offset within the generic thread address space is determined
> by the SetShaderSharedMemoryWindow class method (offset 0x214 in classes *c0).
>
> For both methods, the offset is in bytes, but the window must be aligned to a
> 16MB boundary (so the lower 24 bits of the data must be zero). The upper 32
> bits of the windows are hard-coded to 0 (so they must be placed within the
> lower 4GB of address space).
>
> Generally, it is expected that software will reserve ranges in the global
> virtual address space where these windows will be placed. (Otherwise anything
> mapped there will be inaccessible to shaders.)
>
> For graphics shaders, the shared address space logic does not exist, so there
> is no need to reserve virtual memory for it.

Hi Robert, thanks so much for getting back to me. I believe I've
understood what you've said, but please confirm:

In order for ATOM.*/RED.* to work, the addresses in question must
*NOT* be inside of the 16MB local/shared windows. So if I'm getting
that error, the address must be inside.

If so, this may be a reasonable explanation for what I'm seeing --
while I knew about the local/shared windows, I didn't realize that the
windows were 16MB-sized.

Thanks again,

  -ilia
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: Documentation request for MP warp error 0x10

[Robert Morell]

On Fri, Nov 06, 2015 at 04:15:29PM -0500, Ilia Mirkin wrote:
> In order for ATOM.*/RED.* to work, the addresses in question must
> *NOT* be inside of the 16MB local/shared windows. So if I'm getting
> that error, the address must be inside.

Yes, that's my understanding.

> If so, this may be a reasonable explanation for what I'm seeing --

Cool, I'm happy it helps.


Thanks,
Robert
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: Documentation request for MP warp error 0x10

[Ilia Mirkin]

On Fri, Nov 6, 2015 at 4:19 PM, Robert Morell <rmorell@nvidia.com> wrote:
> On Fri, Nov 06, 2015 at 04:15:29PM -0500, Ilia Mirkin wrote:
>> In order for ATOM.*/RED.* to work, the addresses in question must
>> *NOT* be inside of the 16MB local/shared windows. So if I'm getting
>> that error, the address must be inside.
>
> Yes, that's my understanding.
>
>> If so, this may be a reasonable explanation for what I'm seeing --
>
> Cool, I'm happy it helps.

Looks like we were setting LOCAL_BASE (0x077c) to 0, which was
effectively shadowing the low 16M of g[] space, which is where our
buffers were ending up too. Setting it to some high-up far-off land
makes everything work!

Obviously I'll need some cleverer way to deal with this, but looks
like it's all exactly as you described. Documentation does wonders :)
Looks like I should be able to make progress on my atomics/ssbo work
now.

Thanks again,

  -ilia
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau