Re: SERIALIZE command

Re: SERIALIZE command

[Albert Vilella]

Hi all,

Has anything changed since this email that will make OpenCL support in
nouveau easier?

I would like to compile some information about the different steps
needed to have OpenCL support for Nouveau (and the ATI open-source
drivers for that matters) so that people interested in this issue can
have a starting document to read.

Cheers,

Albert.

> Kernel allows you to create the relevant object, but other than that there's
> no support. You can indeed run your gpu code under nouveau if you're happy
> with writing it in nv50 machine code and submitting via the pushbuffer
> interface. Also our current memory managment is totally unfit for GPGPU
> applications and will need to be redone before there's any hope of seeing CUDA
> or OpenCL support in nouveau.
>
> Also, if you have any more questions and don't hate IRC, you can come to
> #nouveau channel on freenode. Makes it easier to answer.
>
> > Thanks,
> > - Shinpei
>
> Marcin KoÅcielnicki

Re: SERIALIZE command

["C. Bergström"]

Albert Vilella wrote:
> Hi all,
>
> Has anything changed since this email that will make OpenCL support in
> nouveau easier?
>
> I would like to compile some information about the different steps
> needed to have OpenCL support for Nouveau (and the ATI open-source
> drivers for that matters) so that people interested in this issue can
> have a starting document to read.
>   
Hi Albert,

Keep a very close eye on the GPGPU space this week.  There's going to be 
some announcements that I think will help answer your questions.  I 
can't repeat enough though that I think OpenCL's programming model sucks 
and hopefully everyone will rejoice once something better is available.

./C

SERIALIZE command

[Shinpei KATO]

Hi all,

I am trying to use the 0x110 SERIALIZE (or 0x108 WAIT_FOR_IDLE?) command to
barrier the PGRAPH commands.
Unfortunately, I do not find any examples either in nouveau or gallium.

If it plays just a barrier, I guess it should work by something like (in
case of m2mf):
BEGIN_RING(chan, m2mf, 0x110, 1);
OUT_RING(chan, 1);

Otherwise, it may actually play to serialize some commands, then I would
rather use as follows:
BEGIN_RING(chan, m2mf, 0x110, 1);
OUT_RING(chan, NUMBER_OF_FOLLOWING_COMMANDS);

I have tried both approaches, and the application runs without any problems,
but I am not sure if barriers have happened or not ;-)
I would appreciate if anyone can tell me how to use the barrier command.

Best,
- Shinpei

Re: SERIALIZE command

[Marcin Kościelnicki]

> I am trying to use the 0x110 SERIALIZE (or 0x108 WAIT_FOR_IDLE?) command to
> barrier the PGRAPH commands.
> Unfortunately, I do not find any examples either in nouveau or gallium.

There are some examples in DDX, but they don't use the symbolic name

> If it plays just a barrier, I guess it should work by something like (in
> case of m2mf):
> BEGIN_RING(chan, m2mf, 0x110, 1);
> OUT_RING(chan, 1);
>
> Otherwise, it may actually play to serialize some commands, then I would
> rather use as follows:
> BEGIN_RING(chan, m2mf, 0x110, 1);
> OUT_RING(chan, NUMBER_OF_FOLLOWING_COMMANDS);
>
> I have tried both approaches, and the application runs without any
> problems, but I am not sure if barriers have happened or not ;-)
> I would appreciate if anyone can tell me how to use the barrier command.

The WAIT_FOR_IDLE/SERIALIZE methods [they're really the same, I just didn't 
know it when I named the 0x110 on nv50] ignore the data value. Whenever 
they're on the tip of the internal PGRAPH command FIFO, they'll stall all 
processing until all PGRAPH subunits are idled, effectively acting as a full 
barrier.

Yes, SERIALIZE should be renamed to WAIT_FOR_IDLE.

> Best,
> - Shinpei

Marcin Kościelnicki
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: SERIALIZE command

[Shinpei KATO]

Hi Marcin,

Thanks for your quick response.
I found some examples in DDX!
These examples use with Nv2D (subc=2).
Do you think that the SERIALIZE command also works for other objects, like M2MF?

Thanks,
- Shinpei

> -----Original Message-----
> From: Marcin Kościelnicki [mailto:koriakin@0x04.net]
> Sent: Saturday, April 10, 2010 12:11 AM
> To: Shinpei KATO
> Cc: nouveau@lists.freedesktop.org
> Subject: Re: [Nouveau] SERIALIZE command
> 
> > I am trying to use the 0x110 SERIALIZE (or 0x108 WAIT_FOR_IDLE?) command
> to
> > barrier the PGRAPH commands.
> > Unfortunately, I do not find any examples either in nouveau or gallium.
> 
> There are some examples in DDX, but they don't use the symbolic name
> 
> > If it plays just a barrier, I guess it should work by something like (in
> > case of m2mf):
> > BEGIN_RING(chan, m2mf, 0x110, 1);
> > OUT_RING(chan, 1);
> >
> > Otherwise, it may actually play to serialize some commands, then I would
> > rather use as follows:
> > BEGIN_RING(chan, m2mf, 0x110, 1);
> > OUT_RING(chan, NUMBER_OF_FOLLOWING_COMMANDS);
> >
> > I have tried both approaches, and the application runs without any
> > problems, but I am not sure if barriers have happened or not ;-)
> > I would appreciate if anyone can tell me how to use the barrier command.
> 
> The WAIT_FOR_IDLE/SERIALIZE methods [they're really the same, I just didn't
> know it when I named the 0x110 on nv50] ignore the data value. Whenever
> they're on the tip of the internal PGRAPH command FIFO, they'll stall all
> processing until all PGRAPH subunits are idled, effectively acting as a
> full
> barrier.
> 
> Yes, SERIALIZE should be renamed to WAIT_FOR_IDLE.
> 
> > Best,
> > - Shinpei
> 
> Marcin Kościelnicki

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: SERIALIZE command

[Marcin Kościelnicki]

> Hi Marcin,
>
> Thanks for your quick response.
> I found some examples in DDX!
> These examples use with Nv2D (subc=2).
> Do you think that the SERIALIZE command also works for other objects, like
> M2MF?

The command is known to work on all NV50 objects. Pre-NV50, only a few objects 
support it, and I'm not fully certain if the address is 0x108 or 0x110... it's 
0x110 at least on the NV40 3d object.

> Thanks,
> - Shinpei

Marcin Kościelnicki
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: SERIALIZE command

[Shinpei KATO]

Marcin,

That's nice to work on all nv50 objects. Thanks for telling me this. It is very helpful!

Best,
- Shinpei

> -----Original Message-----
> From: Marcin Kościelnicki [mailto:koriakin@0x04.net]
> Sent: Saturday, April 10, 2010 1:06 AM
> To: Shinpei KATO
> Cc: nouveau@lists.freedesktop.org
> Subject: Re: [Nouveau] SERIALIZE command
> 
> > Hi Marcin,
> >
> > Thanks for your quick response.
> > I found some examples in DDX!
> > These examples use with Nv2D (subc=2).
> > Do you think that the SERIALIZE command also works for other objects,
> like
> > M2MF?
> 
> The command is known to work on all NV50 objects. Pre-NV50, only a few objects
> support it, and I'm not fully certain if the address is 0x108 or 0x110...
> it's
> 0x110 at least on the NV40 3d object.
> 
> > Thanks,
> > - Shinpei
> 
> Marcin Kościelnicki

_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: SERIALIZE command

[Shinpei KATO]

This is a follow-up.

> The command is known to work on all NV50 objects. Pre-NV50, only a few objects
> support it, and I'm not fully certain if the address is 0x108 or 0x110...
> it's
> 0x110 at least on the NV40 3d object.

I tried inserting the 0x110 commands with NvSubM2MF (subc=0), MvSubSw (subc=1), NvSub2D (subc=2), NvSubGdiRect (subc=3), NvSubImageBlit (subc=4).
As a result, I got the PFIFO_CACHE_ERROR interrupts, with respect to those with NvSubSw and NvSubImageBlit.
I am really not certain what PFIFO_CACHE_ERROR indicates, but seems they do not work correctly?
Well, those with NvSubM2MF, NvSub2D and NvSubGdiRect are still very helpful though.

Thanks,
- Shinpei

Re: SERIALIZE command

[Marcin Kościelnicki]

> I tried inserting the 0x110 commands with NvSubM2MF (subc=0), MvSubSw
> (subc=1), NvSub2D (subc=2), NvSubGdiRect (subc=3), NvSubImageBlit (subc=4).
> As a result, I got the PFIFO_CACHE_ERROR interrupts, with respect to those
> with NvSubSw and NvSubImageBlit. I am really not certain what
> PFIFO_CACHE_ERROR indicates, but seems they do not work correctly? Well,
> those with NvSubM2MF, NvSub2D and NvSubGdiRect are still very helpful
> though.

This is not how it works.

The command channels on nvidia hardware are split into 8 identical so-called 
subchannels. The consts you're refering to are subchannel numbers.

However, these subchannels aren't constantly bound to the same objects - you 
bind an object to a subchannel by executing command 0 with data being the 
RAMHT handle of an object.

The Nv* consts you're referring to are only valid in kernel, and represent the 
bindings it sets up at start. Most of them don't even apply to NV50. Also, 
NvSubSw is bound to a software object.

PFIFO_CACHE_ERROR is an error that happens when the PFIFO puller encounters 
some problems. One of the causes is submitting a command on a subchannel which 
isn't actually bound to anything, which is what happens here.

Software objects are fake objects where you don't actually bind any object to 
a subchannel, and instead trap the PFIFO_CACHE_ERROR this causes on command 
submission and do something on the host side. We use this for syncing with 
vblank on nv50. These don't support the SERIALIZE command because they're not 
real objects.

Basically, there exist only 4 types of real objects you can submit commands on 
and bind to subchans:

- m2mf [5039]: memory copies
- eng2d [502d]: 2d engine
- tesla [5097, 8297, 8397, 8597]: 3d engine
- turing [50c0, 85c0]: computation engine: CUDA and the like

Very likely there are more types, using different engine than PGRAPH, but we 
don't know anything about them yet. At the very least there should be the 
video decoding engine.

Marcin Kościelnicki
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: SERIALIZE command

[Shinpei KATO]

Hi,

Thanks again for your reply!

> > I tried inserting the 0x110 commands with NvSubM2MF (subc=0), MvSubSw
> > (subc=1), NvSub2D (subc=2), NvSubGdiRect (subc=3), NvSubImageBlit
> (subc=4).
> > As a result, I got the PFIFO_CACHE_ERROR interrupts, with respect to those
> > with NvSubSw and NvSubImageBlit. I am really not certain what
> > PFIFO_CACHE_ERROR indicates, but seems they do not work correctly? Well,
> > those with NvSubM2MF, NvSub2D and NvSubGdiRect are still very helpful
> > though.
> 
> This is not how it works.
> 
> The command channels on nvidia hardware are split into 8 identical so-called
> subchannels. The consts you're refering to are subchannel numbers.
> 
> However, these subchannels aren't constantly bound to the same objects -
> you
> bind an object to a subchannel by executing command 0 with data being the
> RAMHT handle of an object.
> 
> The Nv* consts you're referring to are only valid in kernel, and represent
> the
> bindings it sets up at start. Most of them don't even apply to NV50. Also,
> NvSubSw is bound to a software object.

Oops... I thought that each subchannel is always bound to the same object, and I was actually looking at only the kernel driver ;-)
I have been trying to understand the behavior of GPU operations, using Gallium demo applications.
To do so, I was dumping the commands sent from the user space (Nouveau/Gallium libdrm), by mapping the pushbuf to the kernel virtual address  in the kernel driver, to see what is going on.
I was also inserting another command in the kernel driver, like in nouveau_gem_ioctl_pushbuf(), to see what happens. 
Since several Gallium demo applications seem to use the subchannels in the same manner, I misunderstood that they are constantly bound to the same objects... 
I should have looked into the libdrm code in detail. Now I see them, thanks.
By the way, when I was dumping the commands, I sometimes saw 0x00000000 without any data.
Do you have any idea what this value means? (just a NOP?)

> PFIFO_CACHE_ERROR is an error that happens when the PFIFO puller encounters
> some problems. One of the causes is submitting a command on a subchannel
> which
> isn't actually bound to anything, which is what happens here.
> 
> Software objects are fake objects where you don't actually bind any object
> to
> a subchannel, and instead trap the PFIFO_CACHE_ERROR this causes on command
> submission and do something on the host side. We use this for syncing with
> vblank on nv50. These don't support the SERIALIZE command because they're
> not
> real objects.

So I was sending those commands with unbounded subchannels, that is why I got the PFIFO_CACHE_ERROR interrupts.

> Basically, there exist only 4 types of real objects you can submit commands
> on
> and bind to subchans:
> 
> - m2mf [5039]: memory copies
> - eng2d [502d]: 2d engine
> - tesla [5097, 8297, 8397, 8597]: 3d engine
> - turing [50c0, 85c0]: computation engine: CUDA and the like
> 
> Very likely there are more types, using different engine than PGRAPH, but
> we
> don't know anything about them yet. At the very least there should be the
> video decoding engine.

Got it.
Regarding turing, you mean that Nouveau now supports CUDA?

Thanks,
- Shinpei

Re: SERIALIZE command

[Younes Manton]

2010/4/9 Shinpei KATO <shinpei-a3aLqY3FAeqkRP3n4FtbsiGn5s73+xxf@public.gmane.org>:
> Hi,
>
> Thanks again for your reply!
>
>> > I tried inserting the 0x110 commands with NvSubM2MF (subc=0), MvSubSw
>> > (subc=1), NvSub2D (subc=2), NvSubGdiRect (subc=3), NvSubImageBlit
>> (subc=4).
>> > As a result, I got the PFIFO_CACHE_ERROR interrupts, with respect to those
>> > with NvSubSw and NvSubImageBlit. I am really not certain what
>> > PFIFO_CACHE_ERROR indicates, but seems they do not work correctly? Well,
>> > those with NvSubM2MF, NvSub2D and NvSubGdiRect are still very helpful
>> > though.
>>
>> This is not how it works.
>>
>> The command channels on nvidia hardware are split into 8 identical so-called
>> subchannels. The consts you're refering to are subchannel numbers.
>>
>> However, these subchannels aren't constantly bound to the same objects -
>> you
>> bind an object to a subchannel by executing command 0 with data being the
>> RAMHT handle of an object.
>>
>> The Nv* consts you're referring to are only valid in kernel, and represent
>> the
>> bindings it sets up at start. Most of them don't even apply to NV50. Also,
>> NvSubSw is bound to a software object.
>
> Oops... I thought that each subchannel is always bound to the same object, and I was actually looking at only the kernel driver ;-)
> I have been trying to understand the behavior of GPU operations, using Gallium demo applications.
> To do so, I was dumping the commands sent from the user space (Nouveau/Gallium libdrm), by mapping the pushbuf to the kernel virtual address  in the kernel driver, to see what is going on.
> I was also inserting another command in the kernel driver, like in nouveau_gem_ioctl_pushbuf(), to see what happens.
> Since several Gallium demo applications seem to use the subchannels in the same manner, I misunderstood that they are constantly bound to the same objects...
> I should have looked into the libdrm code in detail. Now I see them, thanks.
> By the way, when I was dumping the commands, I sometimes saw 0x00000000 without any data.
> Do you have any idea what this value means? (just a NOP?)

That's probably the null object (handle=0, class=0x30). It's created
for every channel, see libdrm/nouveau_channel.c:86.

>> PFIFO_CACHE_ERROR is an error that happens when the PFIFO puller encounters
>> some problems. One of the causes is submitting a command on a subchannel
>> which
>> isn't actually bound to anything, which is what happens here.
>>
>> Software objects are fake objects where you don't actually bind any object
>> to
>> a subchannel, and instead trap the PFIFO_CACHE_ERROR this causes on command
>> submission and do something on the host side. We use this for syncing with
>> vblank on nv50. These don't support the SERIALIZE command because they're
>> not
>> real objects.
>
> So I was sending those commands with unbounded subchannels, that is why I got the PFIFO_CACHE_ERROR interrupts.
>
>> Basically, there exist only 4 types of real objects you can submit commands
>> on
>> and bind to subchans:
>>
>> - m2mf [5039]: memory copies
>> - eng2d [502d]: 2d engine
>> - tesla [5097, 8297, 8397, 8597]: 3d engine
>> - turing [50c0, 85c0]: computation engine: CUDA and the like
>>
>> Very likely there are more types, using different engine than PGRAPH, but
>> we
>> don't know anything about them yet. At the very least there should be the
>> video decoding engine.
>
> Got it.
> Regarding turing, you mean that Nouveau now supports CUDA?

I think the kernel allows that object to be created, but that's all.
We don't have any userspace GPGPU library yet.

Re: SERIALIZE command

[Shinpei KATO]

> > By the way, when I was dumping the commands, I sometimes saw 0x00000000
> without any data.
> > Do you have any idea what this value means? (just a NOP?)
> 
> That's probably the null object (handle=0, class=0x30). It's created
> for every channel, see libdrm/nouveau_channel.c:86.

Do we write data (0x00000000) to the pushbuf (chan->cur) when we create an
object?.
Because I see that mostly every pushbuf flush, by the pushbuf_flush()
function, includes multiple 0x00000000 values, which are not written through
BEGIN_RING().
I am not certain who writes these values to the pushbuf.
They may be written through OUT_RING(), but I am sure that they are not
following any header data written through BEGIN_RING().
Do you think it is possible that someone just writes 0x00000000 to the
pushbuf?
FYI, I am dumping the pushbuf in such a way that (1) outputs the data A
(packet header) at the start address of the pushbuf, (2) gets the size X by
(A >> 18 & 0x3fff), (3) skips the address by X * 32, and (4) outputs the
data again.

> >> - m2mf [5039]: memory copies
> >> - eng2d [502d]: 2d engine
> >> - tesla [5097, 8297, 8397, 8597]: 3d engine
> >> - turing [50c0, 85c0]: computation engine: CUDA and the like
> >>
> >> Very likely there are more types, using different engine than PGRAPH,
> but
> >> we
> >> don't know anything about them yet. At the very least there should be
> the
> >> video decoding engine.
> >
> > Got it.
> > Regarding turing, you mean that Nouveau now supports CUDA?
> 
> I think the kernel allows that object to be created, but that's all.
> We don't have any userspace GPGPU library yet.

Ok. Well, it is wonderful if Nouveau supports CUDA in the user space someday
;-)

Thanks,
- Shinpei

Re: SERIALIZE command

[Marcin Kościelnicki]

> Oops... I thought that each subchannel is always bound to the same object,
> and I was actually looking at only the kernel driver ;-) I have been trying
> to understand the behavior of GPU operations, using Gallium demo
> applications. To do so, I was dumping the commands sent from the user space
> (Nouveau/Gallium libdrm), by mapping the pushbuf to the kernel virtual
> address  in the kernel driver, to see what is going on. I was also
> inserting another command in the kernel driver, like in
> nouveau_gem_ioctl_pushbuf(), to see what happens. Since several Gallium
> demo applications seem to use the subchannels in the same manner, I
> misunderstood that they are constantly bound to the same objects... I
> should have looked into the libdrm code in detail. Now I see them, thanks.
> By the way, when I was dumping the commands, I sometimes saw 0x00000000
> without any data. Do you have any idea what this value means? (just a NOP?)

Plain 0 in a pushbuffer is just a nop, yes. It's inserted as a placeholder for 
unused relocations sometimes.

The known pushbuf commands are roughly:

- bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 0: submit methods, increasing 
kind. method is in bits 2-12, subchan in 13-15, method count in 18-28. 
Subsequent <method count> words in FIFO get pushed into sequential methods 
starting from the given one.
- bits 0-1 = 0, bits 29-31 = 1: JMP to data & 0x1ffffffc
- bits 0-1 = 0, bits 29-31 = 2, bits 16-17 = 0: submit methods, non-increasing 
kind. method is in bits 2-12, subchan in 13-15, method count in 18-28. 
Subsequent <method count> words in FIFO get pushed into that method. [NV10+]
- bits 0-1 = 2: CALL to data & 0xfffffffc [NV11+]
- bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 2: RET [NV11+]
- bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 1: conditional. disables method 
submission if bits 4-15 of this command AND contents of PFIFO reg 2058 == 0. 
Used for selecting individual cards in SLI configurations for part of the 
command stream [NV40+]
- bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 3: big non-increasing method 
submission. like the other non-increasing method submission, but method count 
is now taken from next word after the command and can be much larger. Used for 
pushing big amounts of data through the PFIFO, like nv50-style index buffers. 
[NV50+... I think.]

There are likely more, but I don't recall any right now.

> > PFIFO_CACHE_ERROR is an error that happens when the PFIFO puller
> > encounters some problems. One of the causes is submitting a command on a
> > subchannel which
> > isn't actually bound to anything, which is what happens here.
> >
> > Software objects are fake objects where you don't actually bind any
> > object to
> > a subchannel, and instead trap the PFIFO_CACHE_ERROR this causes on
> > command submission and do something on the host side. We use this for
> > syncing with vblank on nv50. These don't support the SERIALIZE command
> > because they're not
> > real objects.
>
> So I was sending those commands with unbounded subchannels, that is why I
> got the PFIFO_CACHE_ERROR interrupts.

Yup. Or maybe you sent them on out software object, and nouveau didn't 
recognise it.

As I already mentioned, there are two kinds of objects you can bind to 
subchannels... software and PGRAPH objects. There are likely more for video 
dec, but meh.

For software objects, all methods >= 0x100 cause PFIFO_CACHE_ERRORs, which 
driver can trap and emulate, or not.

For PGRAPH objects, all methods >= 0x100 are passed on to PGRAPH and are 
handled there, perhaps causing PGRAPH_ILLEGAL_MTHD error if they happen to not 
exist.

Methods < 0x100 are handled directly by the PFIFO, and are independent of the 
current object. There are a couple of known ones...

> > Basically, there exist only 4 types of real objects you can submit
> > commands on
> > and bind to subchans:
> >
> > - m2mf [5039]: memory copies
> > - eng2d [502d]: 2d engine
> > - tesla [5097, 8297, 8397, 8597]: 3d engine
> > - turing [50c0, 85c0]: computation engine: CUDA and the like
> >
> > Very likely there are more types, using different engine than PGRAPH, but
> > we
> > don't know anything about them yet. At the very least there should be the
> > video decoding engine.
>
> Got it.
> Regarding turing, you mean that Nouveau now supports CUDA?

Kernel allows you to create the relevant object, but other than that there's 
no support. You can indeed run your gpu code under nouveau if you're happy 
with writing it in nv50 machine code and submitting via the pushbuffer 
interface. Also our current memory managment is totally unfit for GPGPU 
applications and will need to be redone before there's any hope of seeing CUDA 
or OpenCL support in nouveau.

Also, if you have any more questions and don't hate IRC, you can come to 
#nouveau channel on freenode. Makes it easier to answer.

> Thanks,
> - Shinpei

Marcin Kościelnicki
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: SERIALIZE command

[Shinpei KATO]

> Plain 0 in a pushbuffer is just a nop, yes. It's inserted as a placeholder
> for unused relocations sometimes.

Ok, thanks.

> The known pushbuf commands are roughly:
> 
> - bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 0: submit methods, increasing
> kind. method is in bits 2-12, subchan in 13-15, method count in 18-28.
> Subsequent <method count> words in FIFO get pushed into sequential methods
> starting from the given one.
> - bits 0-1 = 0, bits 29-31 = 1: JMP to data & 0x1ffffffc
> - bits 0-1 = 0, bits 29-31 = 2, bits 16-17 = 0: submit methods, non-increasing
> kind. method is in bits 2-12, subchan in 13-15, method count in 18-28.
> Subsequent <method count> words in FIFO get pushed into that method. [NV10+]
> - bits 0-1 = 2: CALL to data & 0xfffffffc [NV11+]
> - bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 2: RET [NV11+]
> - bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 1: conditional. disables method
> submission if bits 4-15 of this command AND contents of PFIFO reg 2058 ==
> 0.
> Used for selecting individual cards in SLI configurations for part of the
> command stream [NV40+]
> - bits 0-1 = 0, bits 29-31 = 0, bits 16-17 = 3: big non-increasing method
> submission. like the other non-increasing method submission, but method
> count
> is now taken from next word after the command and can be much larger. Used
> for
> pushing big amounts of data through the PFIFO, like nv50-style index
> buffers.
> [NV50+... I think.]
> 
> There are likely more, but I don't recall any right now.

This is very enough!

> > > PFIFO_CACHE_ERROR is an error that happens when the PFIFO puller
> > > encounters some problems. One of the causes is submitting a command
> on a
> > > subchannel which
> > > isn't actually bound to anything, which is what happens here.
> > >
> > > Software objects are fake objects where you don't actually bind any
> > > object to
> > > a subchannel, and instead trap the PFIFO_CACHE_ERROR this causes on
> > > command submission and do something on the host side. We use this for
> > > syncing with vblank on nv50. These don't support the SERIALIZE command
> > > because they're not
> > > real objects.
> >
> > So I was sending those commands with unbounded subchannels, that is why
> I
> > got the PFIFO_CACHE_ERROR interrupts.
> 
> Yup. Or maybe you sent them on out software object, and nouveau didn't
> recognise it.
> 
> As I already mentioned, there are two kinds of objects you can bind to
> subchannels... software and PGRAPH objects. There are likely more for video
> dec, but meh.
> 
> For software objects, all methods >= 0x100 cause PFIFO_CACHE_ERRORs, which
> driver can trap and emulate, or not.
> 
> For PGRAPH objects, all methods >= 0x100 are passed on to PGRAPH and are
> handled there, perhaps causing PGRAPH_ILLEGAL_MTHD error if they happen
> to not exist.

Yeah I saw both PFIFO_CACHE_ERROR and PGRAPH>ILLEGAL_MTHD while I was trying some commands ;-)
Now things are more clear to me!

> Methods < 0x100 are handled directly by the PFIFO, and are independent of
> the
> current object. There are a couple of known ones...
>
> > > Basically, there exist only 4 types of real objects you can submit
> > > commands on
> > > and bind to subchans:
> > >
> > > - m2mf [5039]: memory copies
> > > - eng2d [502d]: 2d engine
> > > - tesla [5097, 8297, 8397, 8597]: 3d engine
> > > - turing [50c0, 85c0]: computation engine: CUDA and the like
> > >
> > > Very likely there are more types, using different engine than PGRAPH,
> but
> > > we
> > > don't know anything about them yet. At the very least there should be
> the
> > > video decoding engine.
> >
> > Got it.
> > Regarding turing, you mean that Nouveau now supports CUDA?
> 
> Kernel allows you to create the relevant object, but other than that there's
> no support. You can indeed run your gpu code under nouveau if you're happy
> with writing it in nv50 machine code and submitting via the pushbuffer
> interface. Also our current memory managment is totally unfit for GPGPU
> applications and will need to be redone before there's any hope of seeing
> CUDA
> or OpenCL support in nouveau.

Ok, since I read some article saying Nouveau is not yet ready for CUDA applications, I just wondered.
Hope to see the CUDA support someday!

> Also, if you have any more questions and don't hate IRC, you can come to
> #nouveau channel on freenode. Makes it easier to answer.

Now I am in ;-)
Thank you very much for your support.

Shinpei