[PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Maarten Maathuis]

[-- Attachment #1: Type: text/plain, Size: 1371 bytes --]

Many people using nv50+ hardware are aware of gpu lockups when a fifo
closes under certain conditions. Based on a mmio-trace and some trail
and error testing i've come up with a patch that improves the
situation on my NV96.

This patch needs testing on NV50+ hardware and regression testing on
older hardware, since i did change some of the common codepaths. This
is very much a work in progress, and if you have anything to
add/correct, please share it.

I've also attached a 2 test apps, once is bitscan-fail from mwk, use
it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
version only emits NOPs (method 0x100) and represents the no error
situation.

For me, i can run the NOP program in loops of 10000 iterations with no
problems (i've done so several times), the bitscan-fail survives 10000
iterations sometimes, but can also fail after a few thousand. In
comparison, a single run of bitscan-fail could cause a gpu lockup for
me in the past.

Please try the gallium driver, the test apps, suspend to ram. Suspend
to ram isn't 100% reliable yet for me (this was always the case after
strange experiments/hammering/etc), but should not regress. This goes
for older hw as well, whatever worked should still work, but i
wouldn't expect serious improvements there.

As always, feedback is appreciated, especially since this is a touchy subject.

Maarten.

[-- Attachment #2: test_better_context_handling_v4.patch --]
[-- Type: text/x-patch, Size: 9043 bytes --]

diff --git a/drivers/gpu/drm/nouveau/nouveau_channel.c b/drivers/gpu/drm/nouveau/nouveau_channel.c
index 4f378b6..e5a077d 100644
--- a/drivers/gpu/drm/nouveau/nouveau_channel.c
+++ b/drivers/gpu/drm/nouveau/nouveau_channel.c
@@ -204,12 +204,14 @@ nouveau_channel_alloc(struct drm_device *dev, struct nouveau_channel **chan_ret,
 	/* disable the fifo caches */
 	pfifo->reassign(dev, false);
 
+	pgraph->fifo_access(dev, false);
 	/* Create a graphics context for new channel */
 	ret = pgraph->create_context(chan);
 	if (ret) {
 		nouveau_channel_free(chan);
 		return ret;
 	}
+	pgraph->fifo_access(dev, true);
 
 	/* Construct inital RAMFC for new channel */
 	ret = pfifo->create_context(chan);
@@ -284,30 +286,23 @@ nouveau_channel_free(struct nouveau_channel *chan)
 	struct drm_nouveau_private *dev_priv = dev->dev_private;
 	struct nouveau_pgraph_engine *pgraph = &dev_priv->engine.graph;
 	struct nouveau_fifo_engine *pfifo = &dev_priv->engine.fifo;
-	unsigned long flags;
+	struct nouveau_fence *fence = NULL;
 	int ret;
 
 	NV_INFO(dev, "%s: freeing fifo %d\n", __func__, chan->id);
 
 	nouveau_debugfs_channel_fini(chan);
 
-	/* Give outstanding push buffers a chance to complete */
-	spin_lock_irqsave(&chan->fence.lock, flags);
-	nouveau_fence_update(chan);
-	spin_unlock_irqrestore(&chan->fence.lock, flags);
-	if (chan->fence.sequence != chan->fence.sequence_ack) {
-		struct nouveau_fence *fence = NULL;
-
-		ret = nouveau_fence_new(chan, &fence, true);
-		if (ret == 0) {
-			ret = nouveau_fence_wait(fence, NULL, false, false);
-			nouveau_fence_unref((void *)&fence);
-		}
-
-		if (ret)
-			NV_ERROR(dev, "Failed to idle channel %d.\n", chan->id);
+	/* Unconditionally emit a fence, to be sure it's done. */
+	ret = nouveau_fence_new(chan, &fence, true);
+	if (ret == 0) {
+		ret = nouveau_fence_wait(fence, NULL, false, false);
+		nouveau_fence_unref((void *)&fence);
 	}
 
+	if (ret)
+		NV_ERROR(dev, "Failed to idle channel %d.\n", chan->id);
+
 	/* Ensure all outstanding fences are signaled.  They should be if the
 	 * above attempts at idling were OK, but if we failed this'll tell TTM
 	 * we're done with the buffers.
@@ -317,13 +312,6 @@ nouveau_channel_free(struct nouveau_channel *chan)
 	/* Ensure the channel is no longer active on the GPU */
 	pfifo->reassign(dev, false);
 
-	if (pgraph->channel(dev) == chan) {
-		pgraph->fifo_access(dev, false);
-		pgraph->unload_context(dev);
-		pgraph->fifo_access(dev, true);
-	}
-	pgraph->destroy_context(chan);
-
 	if (pfifo->channel_id(dev) == chan->id) {
 		pfifo->disable(dev);
 		pfifo->unload_context(dev);
@@ -331,6 +319,13 @@ nouveau_channel_free(struct nouveau_channel *chan)
 	}
 	pfifo->destroy_context(chan);
 
+	if (pgraph->channel(dev) == chan) {
+		pgraph->fifo_access(dev, false);
+		pgraph->unload_context(dev);
+		pgraph->fifo_access(dev, true);
+	}
+	pgraph->destroy_context(chan);
+
 	pfifo->reassign(dev, true);
 
 	/* Release the channel's resources */
diff --git a/drivers/gpu/drm/nouveau/nouveau_drv.c b/drivers/gpu/drm/nouveau/nouveau_drv.c
index 58c7172..bdb9e3b 100644
--- a/drivers/gpu/drm/nouveau/nouveau_drv.c
+++ b/drivers/gpu/drm/nouveau/nouveau_drv.c
@@ -185,6 +185,10 @@ nouveau_pci_suspend(struct pci_dev *pdev, pm_message_t pm_state)
 	pfifo->disable(dev);
 	pfifo->unload_context(dev);
 	pgraph->unload_context(dev);
+	/* This may seem odd, but without it nv50 doesn't resume again.
+	 * It's specifically bit 0 from register 0x00400500 that needs to be set.
+	 */
+	pgraph->fifo_access(dev, true);
 
 	NV_INFO(dev, "Suspending GPU objects...\n");
 	ret = nouveau_gpuobj_suspend(dev);
diff --git a/drivers/gpu/drm/nouveau/nouveau_irq.c b/drivers/gpu/drm/nouveau/nouveau_irq.c
index 370c72c..43ac594 100644
--- a/drivers/gpu/drm/nouveau/nouveau_irq.c
+++ b/drivers/gpu/drm/nouveau/nouveau_irq.c
@@ -579,11 +579,14 @@ nv50_pgraph_irq_handler(struct drm_device *dev)
 	}
 
 	if (status & 0x00001000) {
-		nv_wr32(dev, 0x400500, 0x00000000);
+		struct drm_nouveau_private *dev_priv = dev->dev_private;
+		struct nouveau_pgraph_engine *pgraph = &dev_priv->engine.graph;
+
+		pgraph->fifo_access(dev, false);
 		nv_wr32(dev, NV03_PGRAPH_INTR, NV_PGRAPH_INTR_CONTEXT_SWITCH);
 		nv_wr32(dev, NV40_PGRAPH_INTR_EN, nv_rd32(dev,
 			NV40_PGRAPH_INTR_EN) & ~NV_PGRAPH_INTR_CONTEXT_SWITCH);
-		nv_wr32(dev, 0x400500, 0x00010001);
+		pgraph->fifo_access(dev, true);
 
 		nv50_graph_context_switch(dev);
 
diff --git a/drivers/gpu/drm/nouveau/nv50_fifo.c b/drivers/gpu/drm/nouveau/nv50_fifo.c
index b728228..3eea0d5 100644
--- a/drivers/gpu/drm/nouveau/nv50_fifo.c
+++ b/drivers/gpu/drm/nouveau/nv50_fifo.c
@@ -89,18 +89,39 @@ static void
 nv50_fifo_channel_disable(struct drm_device *dev, int channel, bool nt)
 {
 	struct drm_nouveau_private *dev_priv = dev->dev_private;
+	struct nouveau_pgraph_engine *pgraph = &dev_priv->engine.graph;
 	uint32_t inst;
 
 	NV_DEBUG(dev, "ch%d, nt=%d\n", channel, nt);
 
+	if (!nt) {
+		nv50_fifo_init_thingo(dev);
+
+		nv_wr32(dev, 0x2504, nv_rd32(dev, 0x2504) | 1);
+		if (!nv_wait(0x2504, 0x11, 0x11))
+			NV_ERROR(dev, "0x2504 timed out, 0x%08X\n",
+				nv_rd32(dev, 0x2504));
+
+		/* This is PGRAPH, but the blob does it "here".
+		 * I've seen both cases where the blob only reads and where
+		 * it also writes. I can only assume it's related to the pgraph
+		 * channel. It's done in pgraph unload context if it doesn't
+		 * happen here.
+		 */
+		if (pgraph->channel(dev) == dev_priv->fifos[channel]) {
+			inst = nv_rd32(dev, NV50_PGRAPH_CTXCTL_CUR)
+					& NV50_PGRAPH_CTXCTL_CUR_INSTANCE;
+			nv_wr32(dev, NV50_PGRAPH_CTXCTL_CUR, inst);
+		}
+
+		nv_wr32(dev, 0x2504, nv_rd32(dev, 0x2504) & ~1);
+	}
+
 	if (IS_G80)
 		inst = NV50_PFIFO_CTX_TABLE_INSTANCE_MASK_G80;
 	else
 		inst = NV50_PFIFO_CTX_TABLE_INSTANCE_MASK_G84;
 	nv_wr32(dev, NV50_PFIFO_CTX_TABLE(channel), inst);
-
-	if (!nt)
-		nv50_fifo_init_thingo(dev);
 }
 
 static void
diff --git a/drivers/gpu/drm/nouveau/nv50_graph.c b/drivers/gpu/drm/nouveau/nv50_graph.c
index 444a46b..583d2eb 100644
--- a/drivers/gpu/drm/nouveau/nv50_graph.c
+++ b/drivers/gpu/drm/nouveau/nv50_graph.c
@@ -152,10 +152,21 @@ nv50_graph_fifo_access(struct drm_device *dev, bool enabled)
 {
 	const uint32_t mask = 0x00010001;
 
-	if (enabled)
+	if (enabled) {
 		nv_wr32(dev, 0x400500, nv_rd32(dev, 0x400500) | mask);
-	else
+			nv_wr32(dev, 0x100c80, 0x00000001);
+		if (!nv_wait(0x100c80, 0x00000001, 0x00000000)) {
+			NV_ERROR(dev, "timeout: (0x100c80 & 1) == 0 (2)\n");
+			NV_ERROR(dev, "0x100c80 = 0x%08x\n", nv_rd32(dev, 0x100c80));
+		}
+	} else {
+		nv_wr32(dev, 0x100c80, 0x00050001);
+		if (!nv_wait(0x100c80, 0x00000001, 0x00000000)) {
+			NV_ERROR(dev, "timeout: (0x100c80 & 1) == 0 (2)\n");
+			NV_ERROR(dev, "0x100c80 = 0x%08x\n", nv_rd32(dev, 0x100c80));
+		}
 		nv_wr32(dev, 0x400500, nv_rd32(dev, 0x400500) & ~mask);
+	}
 }
 
 struct nouveau_channel *
@@ -275,29 +286,46 @@ nv50_graph_load_context(struct nouveau_channel *chan)
 int
 nv50_graph_unload_context(struct drm_device *dev)
 {
-	uint32_t inst, fifo = nv_rd32(dev, 0x400500);
-
-	inst  = nv_rd32(dev, NV50_PGRAPH_CTXCTL_CUR);
-	if (!(inst & NV50_PGRAPH_CTXCTL_CUR_LOADED))
-		return 0;
-	inst &= NV50_PGRAPH_CTXCTL_CUR_INSTANCE;
-
-	nv_wr32(dev, 0x400500, fifo & ~1);
+	struct drm_nouveau_private *dev_priv = dev->dev_private;
+	struct nouveau_pgraph_engine *pgraph = &dev_priv->engine.graph;
+	struct nouveau_fifo_engine *pfifo = &dev_priv->engine.fifo;
+	uint32_t tmp, inst;
+
+	inst = nv_rd32(dev, NV50_PGRAPH_CTXCTL_CUR)
+				& NV50_PGRAPH_CTXCTL_CUR_INSTANCE;
+
+	/* The blob clearly puts this in the "middle" of pfifo context unload,
+	 * but do it here in case it didn't happen in pfifo unload context.
+	 */
+	if (pfifo->channel_id(dev) != pgraph->channel(dev)->id)
+		nv_wr32(dev, NV50_PGRAPH_CTXCTL_CUR, inst);
+
+	/* These 3 registers are important for suspend.
+	 * They don't seem to be needed for normal running.
+	 */
 	nv_wr32(dev, 0x400784, inst);
 	nv_wr32(dev, 0x400824, nv_rd32(dev, 0x400824) | 0x20);
 	nv_wr32(dev, 0x400304, nv_rd32(dev, 0x400304) | 0x01);
 	nouveau_wait_for_idle(dev);
-	nv_wr32(dev, 0x400500, fifo);
 
-	nv_wr32(dev, NV50_PGRAPH_CTXCTL_CUR, inst);
+	if ((tmp = nv_rd32(dev, 0x400380)) != 0)
+		NV_ERROR(dev, "0x400380 has nonzero value 0x%08X\n", tmp);
+	if ((tmp = nv_rd32(dev, 0x400384)) != 0)
+		NV_ERROR(dev, "0x400384 has nonzero value 0x%08X\n", tmp);
+	if ((tmp = nv_rd32(dev, 0x400388)) != 0)
+		NV_ERROR(dev, "0x400388 has nonzero value 0x%08X\n", tmp);
+
 	return 0;
 }
 
 void
 nv50_graph_context_switch(struct drm_device *dev)
 {
+	struct drm_nouveau_private *dev_priv = dev->dev_private;
+	struct nouveau_pgraph_engine *pgraph = &dev_priv->engine.graph;
 	uint32_t inst;
 
+	pgraph->fifo_access(dev, false);
 	nv50_graph_unload_context(dev);
 
 	inst  = nv_rd32(dev, NV50_PGRAPH_CTXCTL_NEXT);
@@ -306,6 +334,7 @@ nv50_graph_context_switch(struct drm_device *dev)
 
 	nv_wr32(dev, NV40_PGRAPH_INTR_EN, nv_rd32(dev,
 		NV40_PGRAPH_INTR_EN) | NV_PGRAPH_INTR_CONTEXT_SWITCH);
+	pgraph->fifo_access(dev, true);
 }
 
 static int

[-- Attachment #3: nop.c --]
[-- Type: application/octet-stream, Size: 1237 bytes --]

#include <xf86drm.h>
#include "nouveau_drm.h"
#include "nouveau_device.h"
#include "nouveau_notifier.h"
#include "nouveau_channel.h"
#include "nouveau_pushbuf.h"
#include "nouveau_bo.h"
#include "nouveau_class.h"
#include <stdio.h>
#include <string.h>
#include <stdint.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char **argv) {
	struct nouveau_device *dev = 0;
	int err;
	int i;
	if (err = nouveau_device_open(&dev, 0)) {
		printf ("open: %s\n", strerror(-err));
		return 1;
	}
	struct nouveau_channel *chan = 0;
	if (err = nouveau_channel_alloc(dev, 0xdeadbeef, 0xdeadbeef, &chan)) {
		printf ("chan: %s\n", strerror(-err));
		return 1;
	}
	struct nouveau_grobj *turing = 0;
	if (err = nouveau_grobj_alloc(chan, 0xdeadd00d, 0x50c0, &turing)) {
		printf ("turing: %s\n", strerror(-err));
		return 1;
	}
	struct nouveau_notifier *sync = 0;
	if (err = nouveau_notifier_alloc(chan, 0xbeef0301, 1, &sync)) {
		printf ("sync: %s\n", strerror(-err));
		return 1;
	}

	BEGIN_RING(chan, turing, 0x180, 1);
	OUT_RING(chan, sync->handle);
	BEGIN_RING(chan, turing, 0x60, 1);
	OUT_RING(chan, sync->handle);
	for (i = 0; i < 33; i++) {
		BEGIN_RING(chan, turing, 0x100, 1);
		OUT_RING(chan, 0);
	}
	FIRE_RING(chan);

	return 0;
}

[-- Attachment #4: bitscan-fail.c --]
[-- Type: application/octet-stream, Size: 1265 bytes --]

#include <xf86drm.h>
#include "nouveau_drm.h"
#include "nouveau_device.h"
#include "nouveau_notifier.h"
#include "nouveau_channel.h"
#include "nouveau_pushbuf.h"
#include "nouveau_bo.h"
#include "nouveau_class.h"
#include <stdio.h>
#include <string.h>
#include <stdint.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char **argv) {
	struct nouveau_device *dev = 0;
	int err;
	int i;
	if (err = nouveau_device_open(&dev, 0)) {
		printf ("open: %s\n", strerror(-err));
		return 1;
	}
	struct nouveau_channel *chan = 0;
	if (err = nouveau_channel_alloc(dev, 0xdeadbeef, 0xdeadbeef, &chan)) {
		printf ("chan: %s\n", strerror(-err));
		return 1;
	}
	struct nouveau_grobj *turing = 0;
	if (err = nouveau_grobj_alloc(chan, 0xdeadd00d, 0x50c0, &turing)) {
		printf ("turing: %s\n", strerror(-err));
		return 1;
	}
	struct nouveau_notifier *sync = 0;
	if (err = nouveau_notifier_alloc(chan, 0xbeef0301, 1, &sync)) {
		printf ("sync: %s\n", strerror(-err));
		return 1;
	}

	BEGIN_RING(chan, turing, 0x180, 1);
	OUT_RING(chan, sync->handle);
	BEGIN_RING(chan, turing, 0x60, 1);
	OUT_RING(chan, sync->handle);
	for (i = 0; i < 33; i++) {
		BEGIN_RING(chan, turing, strtol(argv[1], 0, 0), 1);
		OUT_RING(chan, (i<32?1<<i:0));
	}
	FIRE_RING(chan);

	return 0;
}

[-- Attachment #5: Makefile --]
[-- Type: application/octet-stream, Size: 340 bytes --]

HEADERS=
CFLAGS =`pkg-config --cflags libdrm_nouveau`
LDFLAGS=`pkg-config --libs libdrm_nouveau`

all: bitscan-fail nop

bitscan-fail: bitscan-fail.c $(HEADERS)
	gcc -ggdb3 $(CFLAGS) -o bitscan-fail bitscan-fail.c $(LDFLAGS)

nop: nop.c $(HEADERS)
	gcc -ggdb3 $(CFLAGS) -o nop nop.c $(LDFLAGS)

clean:
	rm -f bitscan-fail

.PHONY: clean run

[-- Attachment #6: Type: text/plain, Size: 181 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Maarten Maathuis]

Please do report your successes, and not only failures.

On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Many people using nv50+ hardware are aware of gpu lockups when a fifo
> closes under certain conditions. Based on a mmio-trace and some trail
> and error testing i've come up with a patch that improves the
> situation on my NV96.
>
> This patch needs testing on NV50+ hardware and regression testing on
> older hardware, since i did change some of the common codepaths. This
> is very much a work in progress, and if you have anything to
> add/correct, please share it.
>
> I've also attached a 2 test apps, once is bitscan-fail from mwk, use
> it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
> version only emits NOPs (method 0x100) and represents the no error
> situation.
>
> For me, i can run the NOP program in loops of 10000 iterations with no
> problems (i've done so several times), the bitscan-fail survives 10000
> iterations sometimes, but can also fail after a few thousand. In
> comparison, a single run of bitscan-fail could cause a gpu lockup for
> me in the past.
>
> Please try the gallium driver, the test apps, suspend to ram. Suspend
> to ram isn't 100% reliable yet for me (this was always the case after
> strange experiments/hammering/etc), but should not regress. This goes
> for older hw as well, whatever worked should still work, but i
> wouldn't expect serious improvements there.
>
> As always, feedback is appreciated, especially since this is a touchy subject.
>
> Maarten.
>

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Johannes Obermayr]

Am Samstag, 2. Januar 2010 16:39:55 schrieb Maarten Maathuis:
> Please do report your successes, and not only failures.

Hi,

I included it in my openSUSE Build Service project. I think some people will update und try it...

Johannes

> 
> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > Many people using nv50+ hardware are aware of gpu lockups when a fifo
> > closes under certain conditions. Based on a mmio-trace and some trail
> > and error testing i've come up with a patch that improves the
> > situation on my NV96.
> >
> > This patch needs testing on NV50+ hardware and regression testing on
> > older hardware, since i did change some of the common codepaths. This
> > is very much a work in progress, and if you have anything to
> > add/correct, please share it.
> >
> > I've also attached a 2 test apps, once is bitscan-fail from mwk, use
> > it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
> > version only emits NOPs (method 0x100) and represents the no error
> > situation.
> >
> > For me, i can run the NOP program in loops of 10000 iterations with no
> > problems (i've done so several times), the bitscan-fail survives 10000
> > iterations sometimes, but can also fail after a few thousand. In
> > comparison, a single run of bitscan-fail could cause a gpu lockup for
> > me in the past.
> >
> > Please try the gallium driver, the test apps, suspend to ram. Suspend
> > to ram isn't 100% reliable yet for me (this was always the case after
> > strange experiments/hammering/etc), but should not regress. This goes
> > for older hw as well, whatever worked should still work, but i
> > wouldn't expect serious improvements there.
> >
> > As always, feedback is appreciated, especially since this is a touchy subject.
> >
> > Maarten.
> >
> _______________________________________________
> Nouveau mailing list
> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> http://lists.freedesktop.org/mailman/listinfo/nouveau
> 

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Maarten Maathuis]

I hope you did add a note for your victims :-)

On Sun, Jan 3, 2010 at 1:37 AM, Johannes Obermayr
<johannesobermayr-Mmb7MZpHnFY@public.gmane.org> wrote:
> Am Samstag, 2. Januar 2010 16:39:55 schrieb Maarten Maathuis:
>> Please do report your successes, and not only failures.
>
> Hi,
>
> I included it in my openSUSE Build Service project. I think some people will update und try it...
>
> Johannes
>
>>
>> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > Many people using nv50+ hardware are aware of gpu lockups when a fifo
>> > closes under certain conditions. Based on a mmio-trace and some trail
>> > and error testing i've come up with a patch that improves the
>> > situation on my NV96.
>> >
>> > This patch needs testing on NV50+ hardware and regression testing on
>> > older hardware, since i did change some of the common codepaths. This
>> > is very much a work in progress, and if you have anything to
>> > add/correct, please share it.
>> >
>> > I've also attached a 2 test apps, once is bitscan-fail from mwk, use
>> > it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
>> > version only emits NOPs (method 0x100) and represents the no error
>> > situation.
>> >
>> > For me, i can run the NOP program in loops of 10000 iterations with no
>> > problems (i've done so several times), the bitscan-fail survives 10000
>> > iterations sometimes, but can also fail after a few thousand. In
>> > comparison, a single run of bitscan-fail could cause a gpu lockup for
>> > me in the past.
>> >
>> > Please try the gallium driver, the test apps, suspend to ram. Suspend
>> > to ram isn't 100% reliable yet for me (this was always the case after
>> > strange experiments/hammering/etc), but should not regress. This goes
>> > for older hw as well, whatever worked should still work, but i
>> > wouldn't expect serious improvements there.
>> >
>> > As always, feedback is appreciated, especially since this is a touchy subject.
>> >
>> > Maarten.
>> >
>> _______________________________________________
>> Nouveau mailing list
>> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
>> http://lists.freedesktop.org/mailman/listinfo/nouveau
>>
> _______________________________________________
> Nouveau mailing list
> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> http://lists.freedesktop.org/mailman/listinfo/nouveau
>

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Maarten Maathuis]

[-- Attachment #1: Type: text/plain, Size: 1813 bytes --]

I've narrowed it down further, the "pgraph->fifo_access" bit is still
cleanup (register 0x400500 represents pgraph fifo access), the rest
appears needed for the desired effect. The reordering of pfifo and
pgraph destroy is needed. As usual, feedback is appreciated.

Maarten.

On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Many people using nv50+ hardware are aware of gpu lockups when a fifo
> closes under certain conditions. Based on a mmio-trace and some trail
> and error testing i've come up with a patch that improves the
> situation on my NV96.
>
> This patch needs testing on NV50+ hardware and regression testing on
> older hardware, since i did change some of the common codepaths. This
> is very much a work in progress, and if you have anything to
> add/correct, please share it.
>
> I've also attached a 2 test apps, once is bitscan-fail from mwk, use
> it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
> version only emits NOPs (method 0x100) and represents the no error
> situation.
>
> For me, i can run the NOP program in loops of 10000 iterations with no
> problems (i've done so several times), the bitscan-fail survives 10000
> iterations sometimes, but can also fail after a few thousand. In
> comparison, a single run of bitscan-fail could cause a gpu lockup for
> me in the past.
>
> Please try the gallium driver, the test apps, suspend to ram. Suspend
> to ram isn't 100% reliable yet for me (this was always the case after
> strange experiments/hammering/etc), but should not regress. This goes
> for older hw as well, whatever worked should still work, but i
> wouldn't expect serious improvements there.
>
> As always, feedback is appreciated, especially since this is a touchy subject.
>
> Maarten.
>

[-- Attachment #2: test_better_context_handling_v5.patch --]
[-- Type: text/x-patch, Size: 4447 bytes --]

diff --git a/drivers/gpu/drm/nouveau/nouveau_channel.c b/drivers/gpu/drm/nouveau/nouveau_channel.c
index d1c7f8c..74d73fa 100644
--- a/drivers/gpu/drm/nouveau/nouveau_channel.c
+++ b/drivers/gpu/drm/nouveau/nouveau_channel.c
@@ -276,13 +276,6 @@ nouveau_channel_free(struct nouveau_channel *chan)
 	/* Ensure the channel is no longer active on the GPU */
 	pfifo->reassign(dev, false);
 
-	if (pgraph->channel(dev) == chan) {
-		pgraph->fifo_access(dev, false);
-		pgraph->unload_context(dev);
-		pgraph->fifo_access(dev, true);
-	}
-	pgraph->destroy_context(chan);
-
 	if (pfifo->channel_id(dev) == chan->id) {
 		pfifo->disable(dev);
 		pfifo->unload_context(dev);
@@ -290,6 +283,13 @@ nouveau_channel_free(struct nouveau_channel *chan)
 	}
 	pfifo->destroy_context(chan);
 
+	if (pgraph->channel(dev) == chan) {
+		pgraph->fifo_access(dev, false);
+		pgraph->unload_context(dev);
+		pgraph->fifo_access(dev, true);
+	}
+	pgraph->destroy_context(chan);
+
 	pfifo->reassign(dev, true);
 
 	/* Release the channel's resources */
diff --git a/drivers/gpu/drm/nouveau/nouveau_drv.c b/drivers/gpu/drm/nouveau/nouveau_drv.c
index 58c7172..bdb9e3b 100644
--- a/drivers/gpu/drm/nouveau/nouveau_drv.c
+++ b/drivers/gpu/drm/nouveau/nouveau_drv.c
@@ -185,6 +185,10 @@ nouveau_pci_suspend(struct pci_dev *pdev, pm_message_t pm_state)
 	pfifo->disable(dev);
 	pfifo->unload_context(dev);
 	pgraph->unload_context(dev);
+	/* This may seem odd, but without it nv50 doesn't resume again.
+	 * It's specifically bit 0 from register 0x00400500 that needs to be set.
+	 */
+	pgraph->fifo_access(dev, true);
 
 	NV_INFO(dev, "Suspending GPU objects...\n");
 	ret = nouveau_gpuobj_suspend(dev);
diff --git a/drivers/gpu/drm/nouveau/nv50_fifo.c b/drivers/gpu/drm/nouveau/nv50_fifo.c
index b728228..4279f9d 100644
--- a/drivers/gpu/drm/nouveau/nv50_fifo.c
+++ b/drivers/gpu/drm/nouveau/nv50_fifo.c
@@ -89,18 +89,32 @@ static void
 nv50_fifo_channel_disable(struct drm_device *dev, int channel, bool nt)
 {
 	struct drm_nouveau_private *dev_priv = dev->dev_private;
+	struct nouveau_pgraph_engine *pgraph = &dev_priv->engine.graph;
 	uint32_t inst;
 
 	NV_DEBUG(dev, "ch%d, nt=%d\n", channel, nt);
 
+	if (!nt) {
+		nv50_fifo_init_thingo(dev);
+
+		/* This is PGRAPH, but the blob does it "here".
+		 * I've seen both cases where the blob only reads and where
+		 * it also writes. I can only assume it's related to the pgraph
+		 * channel. It's done in pgraph unload context if it doesn't
+		 * happen here.
+		 */
+		if (pgraph->channel(dev) == dev_priv->fifos[channel]) {
+			inst = nv_rd32(dev, NV50_PGRAPH_CTXCTL_CUR)
+					& NV50_PGRAPH_CTXCTL_CUR_INSTANCE;
+			nv_wr32(dev, NV50_PGRAPH_CTXCTL_CUR, inst);
+		}
+	}
+
 	if (IS_G80)
 		inst = NV50_PFIFO_CTX_TABLE_INSTANCE_MASK_G80;
 	else
 		inst = NV50_PFIFO_CTX_TABLE_INSTANCE_MASK_G84;
 	nv_wr32(dev, NV50_PFIFO_CTX_TABLE(channel), inst);
-
-	if (!nt)
-		nv50_fifo_init_thingo(dev);
 }
 
 static void
diff --git a/drivers/gpu/drm/nouveau/nv50_graph.c b/drivers/gpu/drm/nouveau/nv50_graph.c
index 444a46b..de8d5c7 100644
--- a/drivers/gpu/drm/nouveau/nv50_graph.c
+++ b/drivers/gpu/drm/nouveau/nv50_graph.c
@@ -275,21 +275,28 @@ nv50_graph_load_context(struct nouveau_channel *chan)
 int
 nv50_graph_unload_context(struct drm_device *dev)
 {
-	uint32_t inst, fifo = nv_rd32(dev, 0x400500);
+	struct drm_nouveau_private *dev_priv = dev->dev_private;
+	struct nouveau_pgraph_engine *pgraph = &dev_priv->engine.graph;
+	struct nouveau_fifo_engine *pfifo = &dev_priv->engine.fifo;
+	uint32_t inst;
 
-	inst  = nv_rd32(dev, NV50_PGRAPH_CTXCTL_CUR);
-	if (!(inst & NV50_PGRAPH_CTXCTL_CUR_LOADED))
-		return 0;
-	inst &= NV50_PGRAPH_CTXCTL_CUR_INSTANCE;
+	inst = nv_rd32(dev, NV50_PGRAPH_CTXCTL_CUR)
+				& NV50_PGRAPH_CTXCTL_CUR_INSTANCE;
 
-	nv_wr32(dev, 0x400500, fifo & ~1);
+	/* The blob clearly puts this in the "middle" of pfifo context unload,
+	 * but do it here in case it didn't happen in pfifo unload context.
+	 */
+	if (pfifo->channel_id(dev) != pgraph->channel(dev)->id)
+		nv_wr32(dev, NV50_PGRAPH_CTXCTL_CUR, inst);
+
+	/* These 3 registers are important for suspend.
+	 * They don't seem to be needed for normal running.
+	 */
 	nv_wr32(dev, 0x400784, inst);
 	nv_wr32(dev, 0x400824, nv_rd32(dev, 0x400824) | 0x20);
 	nv_wr32(dev, 0x400304, nv_rd32(dev, 0x400304) | 0x01);
 	nouveau_wait_for_idle(dev);
-	nv_wr32(dev, 0x400500, fifo);
 
-	nv_wr32(dev, NV50_PGRAPH_CTXCTL_CUR, inst);
 	return 0;
 }
 

[-- Attachment #3: Type: text/plain, Size: 181 bytes --]

_______________________________________________
Nouveau mailing list
Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Ben Skeggs]

On Mon, 2010-01-04 at 20:29 +0100, Maarten Maathuis wrote:
> I've narrowed it down further, the "pgraph->fifo_access" bit is still
> cleanup (register 0x400500 represents pgraph fifo access), the rest
> appears needed for the desired effect. The reordering of pfifo and
> pgraph destroy is needed. As usual, feedback is appreciated.
I played a bit yesterday and have the gr/fifoctx unload ordering swap
and queued up already, as well as unconditionally waiting on a fence at
channel destroy (not really needed, but served as a bit of a cleanup
anyway).

I'll try and look at the rest of the changes.

Ben.
> 
> Maarten.
> 
> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > Many people using nv50+ hardware are aware of gpu lockups when a fifo
> > closes under certain conditions. Based on a mmio-trace and some trail
> > and error testing i've come up with a patch that improves the
> > situation on my NV96.
> >
> > This patch needs testing on NV50+ hardware and regression testing on
> > older hardware, since i did change some of the common codepaths. This
> > is very much a work in progress, and if you have anything to
> > add/correct, please share it.
> >
> > I've also attached a 2 test apps, once is bitscan-fail from mwk, use
> > it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
> > version only emits NOPs (method 0x100) and represents the no error
> > situation.
> >
> > For me, i can run the NOP program in loops of 10000 iterations with no
> > problems (i've done so several times), the bitscan-fail survives 10000
> > iterations sometimes, but can also fail after a few thousand. In
> > comparison, a single run of bitscan-fail could cause a gpu lockup for
> > me in the past.
> >
> > Please try the gallium driver, the test apps, suspend to ram. Suspend
> > to ram isn't 100% reliable yet for me (this was always the case after
> > strange experiments/hammering/etc), but should not regress. This goes
> > for older hw as well, whatever worked should still work, but i
> > wouldn't expect serious improvements there.
> >
> > As always, feedback is appreciated, especially since this is a touchy subject.
> >
> > Maarten.
> >
> _______________________________________________
> Nouveau mailing list
> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> http://lists.freedesktop.org/mailman/listinfo/nouveau

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[okias]

Lastest patch work for me. NV96

2010/1/4 Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> I've narrowed it down further, the "pgraph->fifo_access" bit is still
> cleanup (register 0x400500 represents pgraph fifo access), the rest
> appears needed for the desired effect. The reordering of pfifo and
> pgraph destroy is needed. As usual, feedback is appreciated.
>
> Maarten.
>
> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Many people using nv50+ hardware are aware of gpu lockups when a fifo
>> closes under certain conditions. Based on a mmio-trace and some trail
>> and error testing i've come up with a patch that improves the
>> situation on my NV96.
>>
>> This patch needs testing on NV50+ hardware and regression testing on
>> older hardware, since i did change some of the common codepaths. This
>> is very much a work in progress, and if you have anything to
>> add/correct, please share it.
>>
>> I've also attached a 2 test apps, once is bitscan-fail from mwk, use
>> it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
>> version only emits NOPs (method 0x100) and represents the no error
>> situation.
>>
>> For me, i can run the NOP program in loops of 10000 iterations with no
>> problems (i've done so several times), the bitscan-fail survives 10000
>> iterations sometimes, but can also fail after a few thousand. In
>> comparison, a single run of bitscan-fail could cause a gpu lockup for
>> me in the past.
>>
>> Please try the gallium driver, the test apps, suspend to ram. Suspend
>> to ram isn't 100% reliable yet for me (this was always the case after
>> strange experiments/hammering/etc), but should not regress. This goes
>> for older hw as well, whatever worked should still work, but i
>> wouldn't expect serious improvements there.
>>
>> As always, feedback is appreciated, especially since this is a touchy subject.
>>
>> Maarten.
>>
>
> _______________________________________________
> Nouveau mailing list
> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> http://lists.freedesktop.org/mailman/listinfo/nouveau
>
>



-- 
Jabber/XMPP: okias-O1n184kmSCviB9QmIjCX8w@public.gmane.org
SIP VoIP: sip:17474537254-rne+wJe3e/u3Jhga2bR/9VaTQe2KTcn/@public.gmane.org

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Maarten Maathuis]

I forgot to mention that you should run nop from fbcon without X
running for reliable lockups.

On Mon, Jan 4, 2010 at 11:39 PM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Mon, 2010-01-04 at 20:29 +0100, Maarten Maathuis wrote:
>> I've narrowed it down further, the "pgraph->fifo_access" bit is still
>> cleanup (register 0x400500 represents pgraph fifo access), the rest
>> appears needed for the desired effect. The reordering of pfifo and
>> pgraph destroy is needed. As usual, feedback is appreciated.
> I played a bit yesterday and have the gr/fifoctx unload ordering swap
> and queued up already, as well as unconditionally waiting on a fence at
> channel destroy (not really needed, but served as a bit of a cleanup
> anyway).
>
> I'll try and look at the rest of the changes.
>
> Ben.
>>
>> Maarten.
>>
>> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > Many people using nv50+ hardware are aware of gpu lockups when a fifo
>> > closes under certain conditions. Based on a mmio-trace and some trail
>> > and error testing i've come up with a patch that improves the
>> > situation on my NV96.
>> >
>> > This patch needs testing on NV50+ hardware and regression testing on
>> > older hardware, since i did change some of the common codepaths. This
>> > is very much a work in progress, and if you have anything to
>> > add/correct, please share it.
>> >
>> > I've also attached a 2 test apps, once is bitscan-fail from mwk, use
>> > it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
>> > version only emits NOPs (method 0x100) and represents the no error
>> > situation.
>> >
>> > For me, i can run the NOP program in loops of 10000 iterations with no
>> > problems (i've done so several times), the bitscan-fail survives 10000
>> > iterations sometimes, but can also fail after a few thousand. In
>> > comparison, a single run of bitscan-fail could cause a gpu lockup for
>> > me in the past.
>> >
>> > Please try the gallium driver, the test apps, suspend to ram. Suspend
>> > to ram isn't 100% reliable yet for me (this was always the case after
>> > strange experiments/hammering/etc), but should not regress. This goes
>> > for older hw as well, whatever worked should still work, but i
>> > wouldn't expect serious improvements there.
>> >
>> > As always, feedback is appreciated, especially since this is a touchy subject.
>> >
>> > Maarten.
>> >
>> _______________________________________________
>> Nouveau mailing list
>> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
>> http://lists.freedesktop.org/mailman/listinfo/nouveau
>
>
>

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Xavier]

On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Many people using nv50+ hardware are aware of gpu lockups when a fifo
> closes under certain conditions. Based on a mmio-trace and some trail
> and error testing i've come up with a patch that improves the
> situation on my NV96.
>
> This patch needs testing on NV50+ hardware and regression testing on
> older hardware, since i did change some of the common codepaths. This
> is very much a work in progress, and if you have anything to
> add/correct, please share it.
>
> I've also attached a 2 test apps, once is bitscan-fail from mwk, use
> it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
> version only emits NOPs (method 0x100) and represents the no error
> situation.
>
> For me, i can run the NOP program in loops of 10000 iterations with no
> problems (i've done so several times), the bitscan-fail survives 10000
> iterations sometimes, but can also fail after a few thousand. In
> comparison, a single run of bitscan-fail could cause a gpu lockup for
> me in the past.
>
> Please try the gallium driver, the test apps, suspend to ram. Suspend
> to ram isn't 100% reliable yet for me (this was always the case after
> strange experiments/hammering/etc), but should not regress. This goes
> for older hw as well, whatever worked should still work, but i
> wouldn't expect serious improvements there.
>
> As always, feedback is appreciated, especially since this is a touchy subject.
>

I tried patch v5 on nv84. I get similar results for bitscan-fail and
nop, i.e. a huge improvement compared to before (with or without X).
No apparent regressions on gallium and suspend/resume.

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Ben Skeggs]

On Mon, 2010-01-04 at 23:54 +0100, Maarten Maathuis wrote:
> I forgot to mention that you should run nop from fbcon without X
> running for reliable lockups.
Yup, that's what I've been doing.

> 
> On Mon, Jan 4, 2010 at 11:39 PM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > On Mon, 2010-01-04 at 20:29 +0100, Maarten Maathuis wrote:
> >> I've narrowed it down further, the "pgraph->fifo_access" bit is still
> >> cleanup (register 0x400500 represents pgraph fifo access), the rest
> >> appears needed for the desired effect. The reordering of pfifo and
> >> pgraph destroy is needed. As usual, feedback is appreciated.
> > I played a bit yesterday and have the gr/fifoctx unload ordering swap
> > and queued up already, as well as unconditionally waiting on a fence at
> > channel destroy (not really needed, but served as a bit of a cleanup
> > anyway).
> >
> > I'll try and look at the rest of the changes.
> >
Mmm OK.  The gr/fifoctx swap appears to just achieve a little extra
delay before we hit the grctx unload, some of the other changes (the
PGRAPH stuff in fifo channel disable specifically) work around the
changed ordering.

For an identical effect, add a nice mdelay(50) right before the
pgraph->fifo_access(dev, false) in nouveau_channel_free()..  We have a
race.

Ben.
> > Ben.
> >>
> >> Maarten.
> >>
> >> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >> > Many people using nv50+ hardware are aware of gpu lockups when a fifo
> >> > closes under certain conditions. Based on a mmio-trace and some trail
> >> > and error testing i've come up with a patch that improves the
> >> > situation on my NV96.
> >> >
> >> > This patch needs testing on NV50+ hardware and regression testing on
> >> > older hardware, since i did change some of the common codepaths. This
> >> > is very much a work in progress, and if you have anything to
> >> > add/correct, please share it.
> >> >
> >> > I've also attached a 2 test apps, once is bitscan-fail from mwk, use
> >> > it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
> >> > version only emits NOPs (method 0x100) and represents the no error
> >> > situation.
> >> >
> >> > For me, i can run the NOP program in loops of 10000 iterations with no
> >> > problems (i've done so several times), the bitscan-fail survives 10000
> >> > iterations sometimes, but can also fail after a few thousand. In
> >> > comparison, a single run of bitscan-fail could cause a gpu lockup for
> >> > me in the past.
> >> >
> >> > Please try the gallium driver, the test apps, suspend to ram. Suspend
> >> > to ram isn't 100% reliable yet for me (this was always the case after
> >> > strange experiments/hammering/etc), but should not regress. This goes
> >> > for older hw as well, whatever worked should still work, but i
> >> > wouldn't expect serious improvements there.
> >> >
> >> > As always, feedback is appreciated, especially since this is a touchy subject.
> >> >
> >> > Maarten.
> >> >
> >> _______________________________________________
> >> Nouveau mailing list
> >> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> >> http://lists.freedesktop.org/mailman/listinfo/nouveau
> >
> >
> >

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Maarten Maathuis]

On Tue, Jan 5, 2010 at 4:20 AM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Mon, 2010-01-04 at 23:54 +0100, Maarten Maathuis wrote:
>> I forgot to mention that you should run nop from fbcon without X
>> running for reliable lockups.
> Yup, that's what I've been doing.
>
>>
>> On Mon, Jan 4, 2010 at 11:39 PM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> > On Mon, 2010-01-04 at 20:29 +0100, Maarten Maathuis wrote:
>> >> I've narrowed it down further, the "pgraph->fifo_access" bit is still
>> >> cleanup (register 0x400500 represents pgraph fifo access), the rest
>> >> appears needed for the desired effect. The reordering of pfifo and
>> >> pgraph destroy is needed. As usual, feedback is appreciated.
>> > I played a bit yesterday and have the gr/fifoctx unload ordering swap
>> > and queued up already, as well as unconditionally waiting on a fence at
>> > channel destroy (not really needed, but served as a bit of a cleanup
>> > anyway).
>> >
>> > I'll try and look at the rest of the changes.
>> >
> Mmm OK.  The gr/fifoctx swap appears to just achieve a little extra
> delay before we hit the grctx unload, some of the other changes (the
> PGRAPH stuff in fifo channel disable specifically) work around the
> changed ordering.
>
> For an identical effect, add a nice mdelay(50) right before the
> pgraph->fifo_access(dev, false) in nouveau_channel_free()..  We have a
> race.

So what do you propose as the preferred solution?

>
> Ben.
>> > Ben.
>> >>
>> >> Maarten.
>> >>
>> >> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8@public.gmane.orgm> wrote:
>> >> > Many people using nv50+ hardware are aware of gpu lockups when a fifo
>> >> > closes under certain conditions. Based on a mmio-trace and some trail
>> >> > and error testing i've come up with a patch that improves the
>> >> > situation on my NV96.
>> >> >
>> >> > This patch needs testing on NV50+ hardware and regression testing on
>> >> > older hardware, since i did change some of the common codepaths. This
>> >> > is very much a work in progress, and if you have anything to
>> >> > add/correct, please share it.
>> >> >
>> >> > I've also attached a 2 test apps, once is bitscan-fail from mwk, use
>> >> > it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
>> >> > version only emits NOPs (method 0x100) and represents the no error
>> >> > situation.
>> >> >
>> >> > For me, i can run the NOP program in loops of 10000 iterations with no
>> >> > problems (i've done so several times), the bitscan-fail survives 10000
>> >> > iterations sometimes, but can also fail after a few thousand. In
>> >> > comparison, a single run of bitscan-fail could cause a gpu lockup for
>> >> > me in the past.
>> >> >
>> >> > Please try the gallium driver, the test apps, suspend to ram. Suspend
>> >> > to ram isn't 100% reliable yet for me (this was always the case after
>> >> > strange experiments/hammering/etc), but should not regress. This goes
>> >> > for older hw as well, whatever worked should still work, but i
>> >> > wouldn't expect serious improvements there.
>> >> >
>> >> > As always, feedback is appreciated, especially since this is a touchy subject.
>> >> >
>> >> > Maarten.
>> >> >
>> >> _______________________________________________
>> >> Nouveau mailing list
>> >> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
>> >> http://lists.freedesktop.org/mailman/listinfo/nouveau
>> >
>> >
>> >
>
>
>

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Maarten Maathuis]

On Tue, Jan 5, 2010 at 9:41 AM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Tue, Jan 5, 2010 at 4:20 AM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On Mon, 2010-01-04 at 23:54 +0100, Maarten Maathuis wrote:
>>> I forgot to mention that you should run nop from fbcon without X
>>> running for reliable lockups.
>> Yup, that's what I've been doing.
>>
>>>
>>> On Mon, Jan 4, 2010 at 11:39 PM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> > On Mon, 2010-01-04 at 20:29 +0100, Maarten Maathuis wrote:
>>> >> I've narrowed it down further, the "pgraph->fifo_access" bit is still
>>> >> cleanup (register 0x400500 represents pgraph fifo access), the rest
>>> >> appears needed for the desired effect. The reordering of pfifo and
>>> >> pgraph destroy is needed. As usual, feedback is appreciated.
>>> > I played a bit yesterday and have the gr/fifoctx unload ordering swap
>>> > and queued up already, as well as unconditionally waiting on a fence at
>>> > channel destroy (not really needed, but served as a bit of a cleanup
>>> > anyway).
>>> >
>>> > I'll try and look at the rest of the changes.
>>> >
>> Mmm OK.  The gr/fifoctx swap appears to just achieve a little extra
>> delay before we hit the grctx unload, some of the other changes (the
>> PGRAPH stuff in fifo channel disable specifically) work around the
>> changed ordering.
>>
>> For an identical effect, add a nice mdelay(50) right before the
>> pgraph->fifo_access(dev, false) in nouveau_channel_free()..  We have a
>> race.
>
> So what do you propose as the preferred solution?
>
>>
>> Ben.
>>> > Ben.
>>> >>
>>> >> Maarten.
>>> >>
>>> >> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003@gmail.com> wrote:
>>> >> > Many people using nv50+ hardware are aware of gpu lockups when a fifo
>>> >> > closes under certain conditions. Based on a mmio-trace and some trail
>>> >> > and error testing i've come up with a patch that improves the
>>> >> > situation on my NV96.
>>> >> >
>>> >> > This patch needs testing on NV50+ hardware and regression testing on
>>> >> > older hardware, since i did change some of the common codepaths. This
>>> >> > is very much a work in progress, and if you have anything to
>>> >> > add/correct, please share it.
>>> >> >
>>> >> > I've also attached a 2 test apps, once is bitscan-fail from mwk, use
>>> >> > it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
>>> >> > version only emits NOPs (method 0x100) and represents the no error
>>> >> > situation.
>>> >> >
>>> >> > For me, i can run the NOP program in loops of 10000 iterations with no
>>> >> > problems (i've done so several times), the bitscan-fail survives 10000
>>> >> > iterations sometimes, but can also fail after a few thousand. In
>>> >> > comparison, a single run of bitscan-fail could cause a gpu lockup for
>>> >> > me in the past.
>>> >> >
>>> >> > Please try the gallium driver, the test apps, suspend to ram. Suspend
>>> >> > to ram isn't 100% reliable yet for me (this was always the case after
>>> >> > strange experiments/hammering/etc), but should not regress. This goes
>>> >> > for older hw as well, whatever worked should still work, but i
>>> >> > wouldn't expect serious improvements there.
>>> >> >
>>> >> > As always, feedback is appreciated, especially since this is a touchy subject.
>>> >> >
>>> >> > Maarten.
>>> >> >
>>> >> _______________________________________________
>>> >> Nouveau mailing list
>>> >> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
>>> >> http://lists.freedesktop.org/mailman/listinfo/nouveau
>>> >
>>> >
>>> >
>>
>>
>>
>

I've isolated a small part of a mmiotrace, which is one of the few
cases where bit28 of 0x40032c is unset. The end is most interesting,
the beginning is just to be sure everything is there. Maybe it helps.

W 4 543.049438 3 0xc6100c80 0x50001 0x0 0
R 4 543.049496 3 0xc6100c80 0x50000 0x0 0
R 4 543.049548 3 0xc6400500 0x10010001 0x0 0
R 4 543.049596 3 0xc6400500 0x10010001 0x0 0
W 4 543.049644 3 0xc6400500 0x10010000 0x0 0
R 4 543.049693 3 0xc6400700 0x0 0x0 0
R 4 543.049741 3 0xc6400380 0x0 0x0 0
R 4 543.049797 3 0xc6400384 0x0 0x0 0
R 4 543.049845 3 0xc6400388 0x0 0x0 0
W 4 543.049900 3 0xc6100c80 0x1 0x0 0
R 4 543.049958 3 0xc6100c80 0x0 0x0 0
W 4 543.050009 3 0xc6400500 0x10010001 0x0 0
W 4 543.050150 10 0xc41f04c8 0x1 0x0 0
W 4 543.050175 10 0xc41f04cc 0x4 0x0 0
W 4 543.050282 3 0xc6070000 0x1 0x0 0
R 4 543.050358 3 0xc6070000 0x0 0x0 0
R 4 543.050418 3 0xc661002c 0x370 0x0 0
R 4 543.050462 3 0xc661002c 0x370 0x0 0
W 4 543.050588 10 0xc41f0440 0x1 0x0 0
W 4 543.050614 10 0xc41f0444 0x4 0x0 0
W 4 543.050719 3 0xc6070000 0x1 0x0 0
R 4 543.050793 3 0xc6070000 0x0 0x0 0
W 4 543.050896 10 0xc41f03c0 0x1 0x0 0
W 4 543.050922 10 0xc41f03c4 0x4 0x0 0
W 4 543.051028 3 0xc6070000 0x1 0x0 0
R 4 543.051101 3 0xc6070000 0x0 0x0 0
W 4 543.051227 10 0xc41f05e0 0x1 0x0 0
W 4 543.051253 10 0xc41f05e4 0x4 0x0 0
W 4 543.051360 3 0xc6070000 0x1 0x0 0
R 4 543.051434 3 0xc6070000 0x0 0x0 0
W 4 543.051529 10 0xc41f0200 0x1 0x0 0
W 4 543.051554 10 0xc41f0204 0x4 0x0 0
W 4 543.051659 3 0xc6070000 0x1 0x0 0
R 4 543.051732 3 0xc6070000 0x0 0x0 0
W 4 543.051784 10 0xc439e000 0x7e 0x0 0
W 4 543.051807 10 0xc439e004 0x7e 0x0 0
W 4 543.051829 10 0xc439e008 0x1 0x0 0
W 4 543.051851 10 0xc439e00c 0x2 0x0 0
W 4 543.051926 3 0xc6070000 0x1 0x0 0
R 4 543.051999 3 0xc6070000 0x0 0x0 0
W 4 543.052158 3 0xc60032f4 0x1ff64 0x0 0
W 4 543.052228 3 0xc60032ec 0x4 0x0 0
R 4 543.052296 3 0xc60032ec 0x4 0x0 0
R 4 543.052377 3 0xc6002504 0x0 0x0 0
W 4 543.052451 3 0xc6002504 0x1 0x0 0
R 4 543.052745 3 0xc6000100 0x0 0x0 0
R 4 543.052849 3 0xc6002080 0x0 0x0 0
R 4 543.053007 3 0xc6003220 0xd06191 0x0 0
R 4 543.053075 3 0xc6003250 0x90000001 0x0 0
R 4 543.053154 3 0xc6002504 0x11 0x0 0
R 4 543.053226 3 0xc6002508 0x340 0x0 0
R 4 543.053295 3 0xc6003220 0xd06191 0x0 0
R 4 543.053365 3 0xc6003250 0x90000001 0x0 0
R 4 543.053444 3 0xc6000200 0xdff3d113 0x0 0
R 4 543.053516 3 0xc600251c 0x3f 0x0 0
R 4 543.053581 3 0xc640032c 0x8001fd9a 0x0 0
R 4 543.053630 3 0xc640032c 0x8001fd9a 0x0 0
W 4 543.053678 3 0xc640032c 0x1fd9a 0x0 0
R 4 543.053753 3 0xc60032f0 0x3 0x0 0
W 4 543.053843 3 0xc60032f0 0x7f 0x0 0
R 4 543.053921 3 0xc6003220 0xd06191 0x0 0
W 4 543.053990 3 0xc6003220 0xd06191 0x0 0
R 4 543.054054 3 0xc6002504 0x11 0x0 0
W 4 543.054123 3 0xc6002504 0x10 0x0 0
R 4 543.054195 3 0xc600260c 0x801fd99f 0x0 0
W 4 543.054268 3 0xc600260c 0x1ff68 0x0 0
W 4 543.054371 10 0xc43cdd10 0x0 0x0 0
W 4 543.054393 10 0xc43cdd14 0x0 0x0 0
W 4 543.054415 10 0xc43cdd18 0x0 0x0 0
W 4 543.054437 10 0xc43cdd1c 0x0 0x0 0
W 4 543.054460 10 0xc43cdd20 0x0 0x0 0
W 4 543.054482 10 0xc43cdd24 0x0 0x0 0
W 4 543.054504 10 0xc43cdd28 0x0 0x0 0
W 4 543.054526 10 0xc43cdd2c 0x0 0x0 0
W 4 543.054549 10 0xc43cdd30 0x0 0x0 0
W 4 543.054571 10 0xc43cdd34 0x0 0x0 0
W 4 543.054593 10 0xc43cdd38 0x0 0x0 0
W 4 543.054616 10 0xc43cdd3c 0x0 0x0 0
W 4 543.054638 10 0xc43cdd40 0x0 0x0 0
W 4 543.054660 10 0xc43cdd44 0x0 0x0 0
W 4 543.054823 3 0xc6070000 0x1 0x0 0
R 4 543.054921 3 0xc6070000 0x0 0x0 0

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Maarten Maathuis]

On Tue, Jan 5, 2010 at 10:19 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Tue, Jan 5, 2010 at 9:41 AM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On Tue, Jan 5, 2010 at 4:20 AM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> On Mon, 2010-01-04 at 23:54 +0100, Maarten Maathuis wrote:
>>>> I forgot to mention that you should run nop from fbcon without X
>>>> running for reliable lockups.
>>> Yup, that's what I've been doing.
>>>
>>>>
>>>> On Mon, Jan 4, 2010 at 11:39 PM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> > On Mon, 2010-01-04 at 20:29 +0100, Maarten Maathuis wrote:
>>>> >> I've narrowed it down further, the "pgraph->fifo_access" bit is still
>>>> >> cleanup (register 0x400500 represents pgraph fifo access), the rest
>>>> >> appears needed for the desired effect. The reordering of pfifo and
>>>> >> pgraph destroy is needed. As usual, feedback is appreciated.
>>>> > I played a bit yesterday and have the gr/fifoctx unload ordering swap
>>>> > and queued up already, as well as unconditionally waiting on a fence at
>>>> > channel destroy (not really needed, but served as a bit of a cleanup
>>>> > anyway).
>>>> >
>>>> > I'll try and look at the rest of the changes.
>>>> >
>>> Mmm OK.  The gr/fifoctx swap appears to just achieve a little extra
>>> delay before we hit the grctx unload, some of the other changes (the
>>> PGRAPH stuff in fifo channel disable specifically) work around the
>>> changed ordering.
>>>
>>> For an identical effect, add a nice mdelay(50) right before the
>>> pgraph->fifo_access(dev, false) in nouveau_channel_free()..  We have a
>>> race.
>>
>> So what do you propose as the preferred solution?
>>
>>>
>>> Ben.
>>>> > Ben.
>>>> >>
>>>> >> Maarten.
>>>> >>
>>>> >> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003@gmail.com> wrote:
>>>> >> > Many people using nv50+ hardware are aware of gpu lockups when a fifo
>>>> >> > closes under certain conditions. Based on a mmio-trace and some trail
>>>> >> > and error testing i've come up with a patch that improves the
>>>> >> > situation on my NV96.
>>>> >> >
>>>> >> > This patch needs testing on NV50+ hardware and regression testing on
>>>> >> > older hardware, since i did change some of the common codepaths. This
>>>> >> > is very much a work in progress, and if you have anything to
>>>> >> > add/correct, please share it.
>>>> >> >
>>>> >> > I've also attached a 2 test apps, once is bitscan-fail from mwk, use
>>>> >> > it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
>>>> >> > version only emits NOPs (method 0x100) and represents the no error
>>>> >> > situation.
>>>> >> >
>>>> >> > For me, i can run the NOP program in loops of 10000 iterations with no
>>>> >> > problems (i've done so several times), the bitscan-fail survives 10000
>>>> >> > iterations sometimes, but can also fail after a few thousand. In
>>>> >> > comparison, a single run of bitscan-fail could cause a gpu lockup for
>>>> >> > me in the past.
>>>> >> >
>>>> >> > Please try the gallium driver, the test apps, suspend to ram. Suspend
>>>> >> > to ram isn't 100% reliable yet for me (this was always the case after
>>>> >> > strange experiments/hammering/etc), but should not regress. This goes
>>>> >> > for older hw as well, whatever worked should still work, but i
>>>> >> > wouldn't expect serious improvements there.
>>>> >> >
>>>> >> > As always, feedback is appreciated, especially since this is a touchy subject.
>>>> >> >
>>>> >> > Maarten.
>>>> >> >
>>>> >> _______________________________________________
>>>> >> Nouveau mailing list
>>>> >> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
>>>> >> http://lists.freedesktop.org/mailman/listinfo/nouveau
>>>> >
>>>> >
>>>> >
>>>
>>>
>>>
>>
>
> I've isolated a small part of a mmiotrace, which is one of the few
> cases where bit28 of 0x40032c is unset. The end is most interesting,
> the beginning is just to be sure everything is there. Maybe it helps.

I meant to say bit31.

>
> W 4 543.049438 3 0xc6100c80 0x50001 0x0 0
> R 4 543.049496 3 0xc6100c80 0x50000 0x0 0
> R 4 543.049548 3 0xc6400500 0x10010001 0x0 0
> R 4 543.049596 3 0xc6400500 0x10010001 0x0 0
> W 4 543.049644 3 0xc6400500 0x10010000 0x0 0
> R 4 543.049693 3 0xc6400700 0x0 0x0 0
> R 4 543.049741 3 0xc6400380 0x0 0x0 0
> R 4 543.049797 3 0xc6400384 0x0 0x0 0
> R 4 543.049845 3 0xc6400388 0x0 0x0 0
> W 4 543.049900 3 0xc6100c80 0x1 0x0 0
> R 4 543.049958 3 0xc6100c80 0x0 0x0 0
> W 4 543.050009 3 0xc6400500 0x10010001 0x0 0
> W 4 543.050150 10 0xc41f04c8 0x1 0x0 0
> W 4 543.050175 10 0xc41f04cc 0x4 0x0 0
> W 4 543.050282 3 0xc6070000 0x1 0x0 0
> R 4 543.050358 3 0xc6070000 0x0 0x0 0
> R 4 543.050418 3 0xc661002c 0x370 0x0 0
> R 4 543.050462 3 0xc661002c 0x370 0x0 0
> W 4 543.050588 10 0xc41f0440 0x1 0x0 0
> W 4 543.050614 10 0xc41f0444 0x4 0x0 0
> W 4 543.050719 3 0xc6070000 0x1 0x0 0
> R 4 543.050793 3 0xc6070000 0x0 0x0 0
> W 4 543.050896 10 0xc41f03c0 0x1 0x0 0
> W 4 543.050922 10 0xc41f03c4 0x4 0x0 0
> W 4 543.051028 3 0xc6070000 0x1 0x0 0
> R 4 543.051101 3 0xc6070000 0x0 0x0 0
> W 4 543.051227 10 0xc41f05e0 0x1 0x0 0
> W 4 543.051253 10 0xc41f05e4 0x4 0x0 0
> W 4 543.051360 3 0xc6070000 0x1 0x0 0
> R 4 543.051434 3 0xc6070000 0x0 0x0 0
> W 4 543.051529 10 0xc41f0200 0x1 0x0 0
> W 4 543.051554 10 0xc41f0204 0x4 0x0 0
> W 4 543.051659 3 0xc6070000 0x1 0x0 0
> R 4 543.051732 3 0xc6070000 0x0 0x0 0
> W 4 543.051784 10 0xc439e000 0x7e 0x0 0
> W 4 543.051807 10 0xc439e004 0x7e 0x0 0
> W 4 543.051829 10 0xc439e008 0x1 0x0 0
> W 4 543.051851 10 0xc439e00c 0x2 0x0 0
> W 4 543.051926 3 0xc6070000 0x1 0x0 0
> R 4 543.051999 3 0xc6070000 0x0 0x0 0
> W 4 543.052158 3 0xc60032f4 0x1ff64 0x0 0
> W 4 543.052228 3 0xc60032ec 0x4 0x0 0
> R 4 543.052296 3 0xc60032ec 0x4 0x0 0
> R 4 543.052377 3 0xc6002504 0x0 0x0 0
> W 4 543.052451 3 0xc6002504 0x1 0x0 0
> R 4 543.052745 3 0xc6000100 0x0 0x0 0
> R 4 543.052849 3 0xc6002080 0x0 0x0 0
> R 4 543.053007 3 0xc6003220 0xd06191 0x0 0
> R 4 543.053075 3 0xc6003250 0x90000001 0x0 0
> R 4 543.053154 3 0xc6002504 0x11 0x0 0
> R 4 543.053226 3 0xc6002508 0x340 0x0 0
> R 4 543.053295 3 0xc6003220 0xd06191 0x0 0
> R 4 543.053365 3 0xc6003250 0x90000001 0x0 0
> R 4 543.053444 3 0xc6000200 0xdff3d113 0x0 0
> R 4 543.053516 3 0xc600251c 0x3f 0x0 0
> R 4 543.053581 3 0xc640032c 0x8001fd9a 0x0 0
> R 4 543.053630 3 0xc640032c 0x8001fd9a 0x0 0
> W 4 543.053678 3 0xc640032c 0x1fd9a 0x0 0
> R 4 543.053753 3 0xc60032f0 0x3 0x0 0
> W 4 543.053843 3 0xc60032f0 0x7f 0x0 0
> R 4 543.053921 3 0xc6003220 0xd06191 0x0 0
> W 4 543.053990 3 0xc6003220 0xd06191 0x0 0
> R 4 543.054054 3 0xc6002504 0x11 0x0 0
> W 4 543.054123 3 0xc6002504 0x10 0x0 0
> R 4 543.054195 3 0xc600260c 0x801fd99f 0x0 0
> W 4 543.054268 3 0xc600260c 0x1ff68 0x0 0
> W 4 543.054371 10 0xc43cdd10 0x0 0x0 0
> W 4 543.054393 10 0xc43cdd14 0x0 0x0 0
> W 4 543.054415 10 0xc43cdd18 0x0 0x0 0
> W 4 543.054437 10 0xc43cdd1c 0x0 0x0 0
> W 4 543.054460 10 0xc43cdd20 0x0 0x0 0
> W 4 543.054482 10 0xc43cdd24 0x0 0x0 0
> W 4 543.054504 10 0xc43cdd28 0x0 0x0 0
> W 4 543.054526 10 0xc43cdd2c 0x0 0x0 0
> W 4 543.054549 10 0xc43cdd30 0x0 0x0 0
> W 4 543.054571 10 0xc43cdd34 0x0 0x0 0
> W 4 543.054593 10 0xc43cdd38 0x0 0x0 0
> W 4 543.054616 10 0xc43cdd3c 0x0 0x0 0
> W 4 543.054638 10 0xc43cdd40 0x0 0x0 0
> W 4 543.054660 10 0xc43cdd44 0x0 0x0 0
> W 4 543.054823 3 0xc6070000 0x1 0x0 0
> R 4 543.054921 3 0xc6070000 0x0 0x0 0
>

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Maarten Maathuis]

On Tue, Jan 5, 2010 at 10:19 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Tue, Jan 5, 2010 at 9:41 AM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On Tue, Jan 5, 2010 at 4:20 AM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> On Mon, 2010-01-04 at 23:54 +0100, Maarten Maathuis wrote:
>>>> I forgot to mention that you should run nop from fbcon without X
>>>> running for reliable lockups.
>>> Yup, that's what I've been doing.
>>>
>>>>
>>>> On Mon, Jan 4, 2010 at 11:39 PM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> > On Mon, 2010-01-04 at 20:29 +0100, Maarten Maathuis wrote:
>>>> >> I've narrowed it down further, the "pgraph->fifo_access" bit is still
>>>> >> cleanup (register 0x400500 represents pgraph fifo access), the rest
>>>> >> appears needed for the desired effect. The reordering of pfifo and
>>>> >> pgraph destroy is needed. As usual, feedback is appreciated.
>>>> > I played a bit yesterday and have the gr/fifoctx unload ordering swap
>>>> > and queued up already, as well as unconditionally waiting on a fence at
>>>> > channel destroy (not really needed, but served as a bit of a cleanup
>>>> > anyway).
>>>> >
>>>> > I'll try and look at the rest of the changes.
>>>> >
>>> Mmm OK.  The gr/fifoctx swap appears to just achieve a little extra
>>> delay before we hit the grctx unload, some of the other changes (the
>>> PGRAPH stuff in fifo channel disable specifically) work around the
>>> changed ordering.
>>>
>>> For an identical effect, add a nice mdelay(50) right before the
>>> pgraph->fifo_access(dev, false) in nouveau_channel_free()..  We have a
>>> race.
>>
>> So what do you propose as the preferred solution?
>>
>>>
>>> Ben.
>>>> > Ben.
>>>> >>
>>>> >> Maarten.
>>>> >>
>>>> >> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003@gmail.com> wrote:
>>>> >> > Many people using nv50+ hardware are aware of gpu lockups when a fifo
>>>> >> > closes under certain conditions. Based on a mmio-trace and some trail
>>>> >> > and error testing i've come up with a patch that improves the
>>>> >> > situation on my NV96.
>>>> >> >
>>>> >> > This patch needs testing on NV50+ hardware and regression testing on
>>>> >> > older hardware, since i did change some of the common codepaths. This
>>>> >> > is very much a work in progress, and if you have anything to
>>>> >> > add/correct, please share it.
>>>> >> >
>>>> >> > I've also attached a 2 test apps, once is bitscan-fail from mwk, use
>>>> >> > it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
>>>> >> > version only emits NOPs (method 0x100) and represents the no error
>>>> >> > situation.
>>>> >> >
>>>> >> > For me, i can run the NOP program in loops of 10000 iterations with no
>>>> >> > problems (i've done so several times), the bitscan-fail survives 10000
>>>> >> > iterations sometimes, but can also fail after a few thousand. In
>>>> >> > comparison, a single run of bitscan-fail could cause a gpu lockup for
>>>> >> > me in the past.
>>>> >> >
>>>> >> > Please try the gallium driver, the test apps, suspend to ram. Suspend
>>>> >> > to ram isn't 100% reliable yet for me (this was always the case after
>>>> >> > strange experiments/hammering/etc), but should not regress. This goes
>>>> >> > for older hw as well, whatever worked should still work, but i
>>>> >> > wouldn't expect serious improvements there.
>>>> >> >
>>>> >> > As always, feedback is appreciated, especially since this is a touchy subject.
>>>> >> >
>>>> >> > Maarten.
>>>> >> >
>>>> >> _______________________________________________
>>>> >> Nouveau mailing list
>>>> >> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
>>>> >> http://lists.freedesktop.org/mailman/listinfo/nouveau
>>>> >
>>>> >
>>>> >
>>>
>>>
>>>
>>
>
> I've isolated a small part of a mmiotrace, which is one of the few
> cases where bit28 of 0x40032c is unset. The end is most interesting,
> the beginning is just to be sure everything is there. Maybe it helps.
>
> W 4 543.049438 3 0xc6100c80 0x50001 0x0 0
> R 4 543.049496 3 0xc6100c80 0x50000 0x0 0
> R 4 543.049548 3 0xc6400500 0x10010001 0x0 0
> R 4 543.049596 3 0xc6400500 0x10010001 0x0 0
> W 4 543.049644 3 0xc6400500 0x10010000 0x0 0
> R 4 543.049693 3 0xc6400700 0x0 0x0 0
> R 4 543.049741 3 0xc6400380 0x0 0x0 0
> R 4 543.049797 3 0xc6400384 0x0 0x0 0
> R 4 543.049845 3 0xc6400388 0x0 0x0 0
> W 4 543.049900 3 0xc6100c80 0x1 0x0 0
> R 4 543.049958 3 0xc6100c80 0x0 0x0 0
> W 4 543.050009 3 0xc6400500 0x10010001 0x0 0
> W 4 543.050150 10 0xc41f04c8 0x1 0x0 0
> W 4 543.050175 10 0xc41f04cc 0x4 0x0 0
> W 4 543.050282 3 0xc6070000 0x1 0x0 0
> R 4 543.050358 3 0xc6070000 0x0 0x0 0
> R 4 543.050418 3 0xc661002c 0x370 0x0 0
> R 4 543.050462 3 0xc661002c 0x370 0x0 0
> W 4 543.050588 10 0xc41f0440 0x1 0x0 0
> W 4 543.050614 10 0xc41f0444 0x4 0x0 0
> W 4 543.050719 3 0xc6070000 0x1 0x0 0
> R 4 543.050793 3 0xc6070000 0x0 0x0 0
> W 4 543.050896 10 0xc41f03c0 0x1 0x0 0
> W 4 543.050922 10 0xc41f03c4 0x4 0x0 0
> W 4 543.051028 3 0xc6070000 0x1 0x0 0
> R 4 543.051101 3 0xc6070000 0x0 0x0 0
> W 4 543.051227 10 0xc41f05e0 0x1 0x0 0
> W 4 543.051253 10 0xc41f05e4 0x4 0x0 0
> W 4 543.051360 3 0xc6070000 0x1 0x0 0
> R 4 543.051434 3 0xc6070000 0x0 0x0 0
> W 4 543.051529 10 0xc41f0200 0x1 0x0 0
> W 4 543.051554 10 0xc41f0204 0x4 0x0 0
> W 4 543.051659 3 0xc6070000 0x1 0x0 0
> R 4 543.051732 3 0xc6070000 0x0 0x0 0
> W 4 543.051784 10 0xc439e000 0x7e 0x0 0
> W 4 543.051807 10 0xc439e004 0x7e 0x0 0
> W 4 543.051829 10 0xc439e008 0x1 0x0 0
> W 4 543.051851 10 0xc439e00c 0x2 0x0 0
> W 4 543.051926 3 0xc6070000 0x1 0x0 0
> R 4 543.051999 3 0xc6070000 0x0 0x0 0
> W 4 543.052158 3 0xc60032f4 0x1ff64 0x0 0
> W 4 543.052228 3 0xc60032ec 0x4 0x0 0
> R 4 543.052296 3 0xc60032ec 0x4 0x0 0
> R 4 543.052377 3 0xc6002504 0x0 0x0 0
> W 4 543.052451 3 0xc6002504 0x1 0x0 0
> R 4 543.052745 3 0xc6000100 0x0 0x0 0
> R 4 543.052849 3 0xc6002080 0x0 0x0 0
> R 4 543.053007 3 0xc6003220 0xd06191 0x0 0
> R 4 543.053075 3 0xc6003250 0x90000001 0x0 0
> R 4 543.053154 3 0xc6002504 0x11 0x0 0
> R 4 543.053226 3 0xc6002508 0x340 0x0 0
> R 4 543.053295 3 0xc6003220 0xd06191 0x0 0
> R 4 543.053365 3 0xc6003250 0x90000001 0x0 0
> R 4 543.053444 3 0xc6000200 0xdff3d113 0x0 0
> R 4 543.053516 3 0xc600251c 0x3f 0x0 0
> R 4 543.053581 3 0xc640032c 0x8001fd9a 0x0 0
> R 4 543.053630 3 0xc640032c 0x8001fd9a 0x0 0
> W 4 543.053678 3 0xc640032c 0x1fd9a 0x0 0
> R 4 543.053753 3 0xc60032f0 0x3 0x0 0
> W 4 543.053843 3 0xc60032f0 0x7f 0x0 0
> R 4 543.053921 3 0xc6003220 0xd06191 0x0 0
> W 4 543.053990 3 0xc6003220 0xd06191 0x0 0
> R 4 543.054054 3 0xc6002504 0x11 0x0 0
> W 4 543.054123 3 0xc6002504 0x10 0x0 0
> R 4 543.054195 3 0xc600260c 0x801fd99f 0x0 0
> W 4 543.054268 3 0xc600260c 0x1ff68 0x0 0
> W 4 543.054371 10 0xc43cdd10 0x0 0x0 0
> W 4 543.054393 10 0xc43cdd14 0x0 0x0 0
> W 4 543.054415 10 0xc43cdd18 0x0 0x0 0
> W 4 543.054437 10 0xc43cdd1c 0x0 0x0 0
> W 4 543.054460 10 0xc43cdd20 0x0 0x0 0
> W 4 543.054482 10 0xc43cdd24 0x0 0x0 0
> W 4 543.054504 10 0xc43cdd28 0x0 0x0 0
> W 4 543.054526 10 0xc43cdd2c 0x0 0x0 0
> W 4 543.054549 10 0xc43cdd30 0x0 0x0 0
> W 4 543.054571 10 0xc43cdd34 0x0 0x0 0
> W 4 543.054593 10 0xc43cdd38 0x0 0x0 0
> W 4 543.054616 10 0xc43cdd3c 0x0 0x0 0
> W 4 543.054638 10 0xc43cdd40 0x0 0x0 0
> W 4 543.054660 10 0xc43cdd44 0x0 0x0 0
> W 4 543.054823 3 0xc6070000 0x1 0x0 0
> R 4 543.054921 3 0xc6070000 0x0 0x0 0
>

This chunk comes after it, very similar to the one before it. But i
forgot to add it.

W 4 543.055001 3 0xc6100c80 0x50001 0x0 0
R 4 543.055059 3 0xc6100c80 0x50000 0x0 0
R 4 543.055111 3 0xc6400500 0x10010001 0x0 0
R 4 543.055159 3 0xc6400500 0x10010001 0x0 0
W 4 543.055207 3 0xc6400500 0x10010000 0x0 0
R 4 543.055256 3 0xc6400700 0x0 0x0 0
R 4 543.055304 3 0xc6400380 0x0 0x0 0
R 4 543.055352 3 0xc6400384 0x0 0x0 0
R 4 543.055400 3 0xc6400388 0x0 0x0 0
W 4 543.055454 3 0xc6100c80 0x1 0x0 0
R 4 543.055511 3 0xc6100c80 0x0 0x0 0
W 4 543.055562 3 0xc6400500 0x10010001 0x0 0
W 4 543.055657 3 0xc600260c 0x1ff680 0x0 0
W 4 543.055745 3 0xc6000140 0x1 0x0 0
W 4 543.055954 3 0xc6000140 0x0 0x0 0
W 4 543.055996 10 0xc43cdd48 0x0 0x0 0
W 4 543.056019 10 0xc43cdd4c 0x0 0x0 0
W 4 543.056041 10 0xc43cdd50 0x0 0x0 0
W 4 543.056064 10 0xc43cdd54 0x0 0x0 0
W 4 543.056167 3 0xc6070000 0x1 0x0 0
R 4 543.056246 3 0xc6070000 0x0 0x0 0

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Maarten Maathuis]

Patch v5 remains necessary (a simple swap of pfifo and pgraph unload
isn't enough) even on a current kernel, the change is that it's now
possible to generate pgraph errors without locking up. Without the
patch even nop fails in loops, while running under fbcon.

Maarten.

On Tue, Jan 5, 2010 at 11:55 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> On Tue, Jan 5, 2010 at 10:19 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> On Tue, Jan 5, 2010 at 9:41 AM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>> On Tue, Jan 5, 2010 at 4:20 AM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>> On Mon, 2010-01-04 at 23:54 +0100, Maarten Maathuis wrote:
>>>>> I forgot to mention that you should run nop from fbcon without X
>>>>> running for reliable lockups.
>>>> Yup, that's what I've been doing.
>>>>
>>>>>
>>>>> On Mon, Jan 4, 2010 at 11:39 PM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>>>>> > On Mon, 2010-01-04 at 20:29 +0100, Maarten Maathuis wrote:
>>>>> >> I've narrowed it down further, the "pgraph->fifo_access" bit is still
>>>>> >> cleanup (register 0x400500 represents pgraph fifo access), the rest
>>>>> >> appears needed for the desired effect. The reordering of pfifo and
>>>>> >> pgraph destroy is needed. As usual, feedback is appreciated.
>>>>> > I played a bit yesterday and have the gr/fifoctx unload ordering swap
>>>>> > and queued up already, as well as unconditionally waiting on a fence at
>>>>> > channel destroy (not really needed, but served as a bit of a cleanup
>>>>> > anyway).
>>>>> >
>>>>> > I'll try and look at the rest of the changes.
>>>>> >
>>>> Mmm OK.  The gr/fifoctx swap appears to just achieve a little extra
>>>> delay before we hit the grctx unload, some of the other changes (the
>>>> PGRAPH stuff in fifo channel disable specifically) work around the
>>>> changed ordering.
>>>>
>>>> For an identical effect, add a nice mdelay(50) right before the
>>>> pgraph->fifo_access(dev, false) in nouveau_channel_free()..  We have a
>>>> race.
>>>
>>> So what do you propose as the preferred solution?
>>>
>>>>
>>>> Ben.
>>>>> > Ben.
>>>>> >>
>>>>> >> Maarten.
>>>>> >>
>>>>> >> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003@gmail.com> wrote:
>>>>> >> > Many people using nv50+ hardware are aware of gpu lockups when a fifo
>>>>> >> > closes under certain conditions. Based on a mmio-trace and some trail
>>>>> >> > and error testing i've come up with a patch that improves the
>>>>> >> > situation on my NV96.
>>>>> >> >
>>>>> >> > This patch needs testing on NV50+ hardware and regression testing on
>>>>> >> > older hardware, since i did change some of the common codepaths. This
>>>>> >> > is very much a work in progress, and if you have anything to
>>>>> >> > add/correct, please share it.
>>>>> >> >
>>>>> >> > I've also attached a 2 test apps, once is bitscan-fail from mwk, use
>>>>> >> > it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
>>>>> >> > version only emits NOPs (method 0x100) and represents the no error
>>>>> >> > situation.
>>>>> >> >
>>>>> >> > For me, i can run the NOP program in loops of 10000 iterations with no
>>>>> >> > problems (i've done so several times), the bitscan-fail survives 10000
>>>>> >> > iterations sometimes, but can also fail after a few thousand. In
>>>>> >> > comparison, a single run of bitscan-fail could cause a gpu lockup for
>>>>> >> > me in the past.
>>>>> >> >
>>>>> >> > Please try the gallium driver, the test apps, suspend to ram. Suspend
>>>>> >> > to ram isn't 100% reliable yet for me (this was always the case after
>>>>> >> > strange experiments/hammering/etc), but should not regress. This goes
>>>>> >> > for older hw as well, whatever worked should still work, but i
>>>>> >> > wouldn't expect serious improvements there.
>>>>> >> >
>>>>> >> > As always, feedback is appreciated, especially since this is a touchy subject.
>>>>> >> >
>>>>> >> > Maarten.
>>>>> >> >
>>>>> >> _______________________________________________
>>>>> >> Nouveau mailing list
>>>>> >> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
>>>>> >> http://lists.freedesktop.org/mailman/listinfo/nouveau
>>>>> >
>>>>> >
>>>>> >
>>>>
>>>>
>>>>
>>>
>>
>> I've isolated a small part of a mmiotrace, which is one of the few
>> cases where bit28 of 0x40032c is unset. The end is most interesting,
>> the beginning is just to be sure everything is there. Maybe it helps.
>>
>> W 4 543.049438 3 0xc6100c80 0x50001 0x0 0
>> R 4 543.049496 3 0xc6100c80 0x50000 0x0 0
>> R 4 543.049548 3 0xc6400500 0x10010001 0x0 0
>> R 4 543.049596 3 0xc6400500 0x10010001 0x0 0
>> W 4 543.049644 3 0xc6400500 0x10010000 0x0 0
>> R 4 543.049693 3 0xc6400700 0x0 0x0 0
>> R 4 543.049741 3 0xc6400380 0x0 0x0 0
>> R 4 543.049797 3 0xc6400384 0x0 0x0 0
>> R 4 543.049845 3 0xc6400388 0x0 0x0 0
>> W 4 543.049900 3 0xc6100c80 0x1 0x0 0
>> R 4 543.049958 3 0xc6100c80 0x0 0x0 0
>> W 4 543.050009 3 0xc6400500 0x10010001 0x0 0
>> W 4 543.050150 10 0xc41f04c8 0x1 0x0 0
>> W 4 543.050175 10 0xc41f04cc 0x4 0x0 0
>> W 4 543.050282 3 0xc6070000 0x1 0x0 0
>> R 4 543.050358 3 0xc6070000 0x0 0x0 0
>> R 4 543.050418 3 0xc661002c 0x370 0x0 0
>> R 4 543.050462 3 0xc661002c 0x370 0x0 0
>> W 4 543.050588 10 0xc41f0440 0x1 0x0 0
>> W 4 543.050614 10 0xc41f0444 0x4 0x0 0
>> W 4 543.050719 3 0xc6070000 0x1 0x0 0
>> R 4 543.050793 3 0xc6070000 0x0 0x0 0
>> W 4 543.050896 10 0xc41f03c0 0x1 0x0 0
>> W 4 543.050922 10 0xc41f03c4 0x4 0x0 0
>> W 4 543.051028 3 0xc6070000 0x1 0x0 0
>> R 4 543.051101 3 0xc6070000 0x0 0x0 0
>> W 4 543.051227 10 0xc41f05e0 0x1 0x0 0
>> W 4 543.051253 10 0xc41f05e4 0x4 0x0 0
>> W 4 543.051360 3 0xc6070000 0x1 0x0 0
>> R 4 543.051434 3 0xc6070000 0x0 0x0 0
>> W 4 543.051529 10 0xc41f0200 0x1 0x0 0
>> W 4 543.051554 10 0xc41f0204 0x4 0x0 0
>> W 4 543.051659 3 0xc6070000 0x1 0x0 0
>> R 4 543.051732 3 0xc6070000 0x0 0x0 0
>> W 4 543.051784 10 0xc439e000 0x7e 0x0 0
>> W 4 543.051807 10 0xc439e004 0x7e 0x0 0
>> W 4 543.051829 10 0xc439e008 0x1 0x0 0
>> W 4 543.051851 10 0xc439e00c 0x2 0x0 0
>> W 4 543.051926 3 0xc6070000 0x1 0x0 0
>> R 4 543.051999 3 0xc6070000 0x0 0x0 0
>> W 4 543.052158 3 0xc60032f4 0x1ff64 0x0 0
>> W 4 543.052228 3 0xc60032ec 0x4 0x0 0
>> R 4 543.052296 3 0xc60032ec 0x4 0x0 0
>> R 4 543.052377 3 0xc6002504 0x0 0x0 0
>> W 4 543.052451 3 0xc6002504 0x1 0x0 0
>> R 4 543.052745 3 0xc6000100 0x0 0x0 0
>> R 4 543.052849 3 0xc6002080 0x0 0x0 0
>> R 4 543.053007 3 0xc6003220 0xd06191 0x0 0
>> R 4 543.053075 3 0xc6003250 0x90000001 0x0 0
>> R 4 543.053154 3 0xc6002504 0x11 0x0 0
>> R 4 543.053226 3 0xc6002508 0x340 0x0 0
>> R 4 543.053295 3 0xc6003220 0xd06191 0x0 0
>> R 4 543.053365 3 0xc6003250 0x90000001 0x0 0
>> R 4 543.053444 3 0xc6000200 0xdff3d113 0x0 0
>> R 4 543.053516 3 0xc600251c 0x3f 0x0 0
>> R 4 543.053581 3 0xc640032c 0x8001fd9a 0x0 0
>> R 4 543.053630 3 0xc640032c 0x8001fd9a 0x0 0
>> W 4 543.053678 3 0xc640032c 0x1fd9a 0x0 0
>> R 4 543.053753 3 0xc60032f0 0x3 0x0 0
>> W 4 543.053843 3 0xc60032f0 0x7f 0x0 0
>> R 4 543.053921 3 0xc6003220 0xd06191 0x0 0
>> W 4 543.053990 3 0xc6003220 0xd06191 0x0 0
>> R 4 543.054054 3 0xc6002504 0x11 0x0 0
>> W 4 543.054123 3 0xc6002504 0x10 0x0 0
>> R 4 543.054195 3 0xc600260c 0x801fd99f 0x0 0
>> W 4 543.054268 3 0xc600260c 0x1ff68 0x0 0
>> W 4 543.054371 10 0xc43cdd10 0x0 0x0 0
>> W 4 543.054393 10 0xc43cdd14 0x0 0x0 0
>> W 4 543.054415 10 0xc43cdd18 0x0 0x0 0
>> W 4 543.054437 10 0xc43cdd1c 0x0 0x0 0
>> W 4 543.054460 10 0xc43cdd20 0x0 0x0 0
>> W 4 543.054482 10 0xc43cdd24 0x0 0x0 0
>> W 4 543.054504 10 0xc43cdd28 0x0 0x0 0
>> W 4 543.054526 10 0xc43cdd2c 0x0 0x0 0
>> W 4 543.054549 10 0xc43cdd30 0x0 0x0 0
>> W 4 543.054571 10 0xc43cdd34 0x0 0x0 0
>> W 4 543.054593 10 0xc43cdd38 0x0 0x0 0
>> W 4 543.054616 10 0xc43cdd3c 0x0 0x0 0
>> W 4 543.054638 10 0xc43cdd40 0x0 0x0 0
>> W 4 543.054660 10 0xc43cdd44 0x0 0x0 0
>> W 4 543.054823 3 0xc6070000 0x1 0x0 0
>> R 4 543.054921 3 0xc6070000 0x0 0x0 0
>>
>
> This chunk comes after it, very similar to the one before it. But i
> forgot to add it.
>
> W 4 543.055001 3 0xc6100c80 0x50001 0x0 0
> R 4 543.055059 3 0xc6100c80 0x50000 0x0 0
> R 4 543.055111 3 0xc6400500 0x10010001 0x0 0
> R 4 543.055159 3 0xc6400500 0x10010001 0x0 0
> W 4 543.055207 3 0xc6400500 0x10010000 0x0 0
> R 4 543.055256 3 0xc6400700 0x0 0x0 0
> R 4 543.055304 3 0xc6400380 0x0 0x0 0
> R 4 543.055352 3 0xc6400384 0x0 0x0 0
> R 4 543.055400 3 0xc6400388 0x0 0x0 0
> W 4 543.055454 3 0xc6100c80 0x1 0x0 0
> R 4 543.055511 3 0xc6100c80 0x0 0x0 0
> W 4 543.055562 3 0xc6400500 0x10010001 0x0 0
> W 4 543.055657 3 0xc600260c 0x1ff680 0x0 0
> W 4 543.055745 3 0xc6000140 0x1 0x0 0
> W 4 543.055954 3 0xc6000140 0x0 0x0 0
> W 4 543.055996 10 0xc43cdd48 0x0 0x0 0
> W 4 543.056019 10 0xc43cdd4c 0x0 0x0 0
> W 4 543.056041 10 0xc43cdd50 0x0 0x0 0
> W 4 543.056064 10 0xc43cdd54 0x0 0x0 0
> W 4 543.056167 3 0xc6070000 0x1 0x0 0
> R 4 543.056246 3 0xc6070000 0x0 0x0 0
>

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Ben Skeggs]

On Wed, 2010-01-06 at 18:58 +0100, Maarten Maathuis wrote:
> Patch v5 remains necessary (a simple swap of pfifo and pgraph unload
> isn't enough) even on a current kernel, the change is that it's now
> possible to generate pgraph errors without locking up. Without the
> patch even nop fails in loops, while running under fbcon.
Yes, the commit fixing the ctxprog hang wasn't intended to fix the
entire problem.  I actually came across that issue while working on
something else, it just turns out to be one of the issues that effects
channel destruction too.

Adding a simple nouveau_wait_for_idle() after pgraph->fifo_access(dev,
false) is enough now to make it work *almost* all the time.  Still
something else we're not waiting for, mdelay(50) lets me run
bitscan-fail in a loop for as long as I like without issue.  I don't
really have any ideas atm of what it could be yet, but i'd *really*
rather fix it properly instead of hiding the problem away...

Ben.
> 
> Maarten.
> 
> On Tue, Jan 5, 2010 at 11:55 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > On Tue, Jan 5, 2010 at 10:19 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >> On Tue, Jan 5, 2010 at 9:41 AM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >>> On Tue, Jan 5, 2010 at 4:20 AM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >>>> On Mon, 2010-01-04 at 23:54 +0100, Maarten Maathuis wrote:
> >>>>> I forgot to mention that you should run nop from fbcon without X
> >>>>> running for reliable lockups.
> >>>> Yup, that's what I've been doing.
> >>>>
> >>>>>
> >>>>> On Mon, Jan 4, 2010 at 11:39 PM, Ben Skeggs <skeggsb-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >>>>> > On Mon, 2010-01-04 at 20:29 +0100, Maarten Maathuis wrote:
> >>>>> >> I've narrowed it down further, the "pgraph->fifo_access" bit is still
> >>>>> >> cleanup (register 0x400500 represents pgraph fifo access), the rest
> >>>>> >> appears needed for the desired effect. The reordering of pfifo and
> >>>>> >> pgraph destroy is needed. As usual, feedback is appreciated.
> >>>>> > I played a bit yesterday and have the gr/fifoctx unload ordering swap
> >>>>> > and queued up already, as well as unconditionally waiting on a fence at
> >>>>> > channel destroy (not really needed, but served as a bit of a cleanup
> >>>>> > anyway).
> >>>>> >
> >>>>> > I'll try and look at the rest of the changes.
> >>>>> >
> >>>> Mmm OK.  The gr/fifoctx swap appears to just achieve a little extra
> >>>> delay before we hit the grctx unload, some of the other changes (the
> >>>> PGRAPH stuff in fifo channel disable specifically) work around the
> >>>> changed ordering.
> >>>>
> >>>> For an identical effect, add a nice mdelay(50) right before the
> >>>> pgraph->fifo_access(dev, false) in nouveau_channel_free()..  We have a
> >>>> race.
> >>>
> >>> So what do you propose as the preferred solution?
> >>>
> >>>>
> >>>> Ben.
> >>>>> > Ben.
> >>>>> >>
> >>>>> >> Maarten.
> >>>>> >>
> >>>>> >> On Sat, Jan 2, 2010 at 4:36 PM, Maarten Maathuis <madman2003-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >>>>> >> > Many people using nv50+ hardware are aware of gpu lockups when a fifo
> >>>>> >> > closes under certain conditions. Based on a mmio-trace and some trail
> >>>>> >> > and error testing i've come up with a patch that improves the
> >>>>> >> > situation on my NV96.
> >>>>> >> >
> >>>>> >> > This patch needs testing on NV50+ hardware and regression testing on
> >>>>> >> > older hardware, since i did change some of the common codepaths. This
> >>>>> >> > is very much a work in progress, and if you have anything to
> >>>>> >> > add/correct, please share it.
> >>>>> >> >
> >>>>> >> > I've also attached a 2 test apps, once is bitscan-fail from mwk, use
> >>>>> >> > it like ./bitscan-fail 0x200 to trigger PGRAPH errors. A modified
> >>>>> >> > version only emits NOPs (method 0x100) and represents the no error
> >>>>> >> > situation.
> >>>>> >> >
> >>>>> >> > For me, i can run the NOP program in loops of 10000 iterations with no
> >>>>> >> > problems (i've done so several times), the bitscan-fail survives 10000
> >>>>> >> > iterations sometimes, but can also fail after a few thousand. In
> >>>>> >> > comparison, a single run of bitscan-fail could cause a gpu lockup for
> >>>>> >> > me in the past.
> >>>>> >> >
> >>>>> >> > Please try the gallium driver, the test apps, suspend to ram. Suspend
> >>>>> >> > to ram isn't 100% reliable yet for me (this was always the case after
> >>>>> >> > strange experiments/hammering/etc), but should not regress. This goes
> >>>>> >> > for older hw as well, whatever worked should still work, but i
> >>>>> >> > wouldn't expect serious improvements there.
> >>>>> >> >
> >>>>> >> > As always, feedback is appreciated, especially since this is a touchy subject.
> >>>>> >> >
> >>>>> >> > Maarten.
> >>>>> >> >
> >>>>> >> _______________________________________________
> >>>>> >> Nouveau mailing list
> >>>>> >> Nouveau-PD4FTy7X32lNgt0PjOBp9y5qC8QIuHrW@public.gmane.org
> >>>>> >> http://lists.freedesktop.org/mailman/listinfo/nouveau
> >>>>> >
> >>>>> >
> >>>>> >
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >> I've isolated a small part of a mmiotrace, which is one of the few
> >> cases where bit28 of 0x40032c is unset. The end is most interesting,
> >> the beginning is just to be sure everything is there. Maybe it helps.
> >>
> >> W 4 543.049438 3 0xc6100c80 0x50001 0x0 0
> >> R 4 543.049496 3 0xc6100c80 0x50000 0x0 0
> >> R 4 543.049548 3 0xc6400500 0x10010001 0x0 0
> >> R 4 543.049596 3 0xc6400500 0x10010001 0x0 0
> >> W 4 543.049644 3 0xc6400500 0x10010000 0x0 0
> >> R 4 543.049693 3 0xc6400700 0x0 0x0 0
> >> R 4 543.049741 3 0xc6400380 0x0 0x0 0
> >> R 4 543.049797 3 0xc6400384 0x0 0x0 0
> >> R 4 543.049845 3 0xc6400388 0x0 0x0 0
> >> W 4 543.049900 3 0xc6100c80 0x1 0x0 0
> >> R 4 543.049958 3 0xc6100c80 0x0 0x0 0
> >> W 4 543.050009 3 0xc6400500 0x10010001 0x0 0
> >> W 4 543.050150 10 0xc41f04c8 0x1 0x0 0
> >> W 4 543.050175 10 0xc41f04cc 0x4 0x0 0
> >> W 4 543.050282 3 0xc6070000 0x1 0x0 0
> >> R 4 543.050358 3 0xc6070000 0x0 0x0 0
> >> R 4 543.050418 3 0xc661002c 0x370 0x0 0
> >> R 4 543.050462 3 0xc661002c 0x370 0x0 0
> >> W 4 543.050588 10 0xc41f0440 0x1 0x0 0
> >> W 4 543.050614 10 0xc41f0444 0x4 0x0 0
> >> W 4 543.050719 3 0xc6070000 0x1 0x0 0
> >> R 4 543.050793 3 0xc6070000 0x0 0x0 0
> >> W 4 543.050896 10 0xc41f03c0 0x1 0x0 0
> >> W 4 543.050922 10 0xc41f03c4 0x4 0x0 0
> >> W 4 543.051028 3 0xc6070000 0x1 0x0 0
> >> R 4 543.051101 3 0xc6070000 0x0 0x0 0
> >> W 4 543.051227 10 0xc41f05e0 0x1 0x0 0
> >> W 4 543.051253 10 0xc41f05e4 0x4 0x0 0
> >> W 4 543.051360 3 0xc6070000 0x1 0x0 0
> >> R 4 543.051434 3 0xc6070000 0x0 0x0 0
> >> W 4 543.051529 10 0xc41f0200 0x1 0x0 0
> >> W 4 543.051554 10 0xc41f0204 0x4 0x0 0
> >> W 4 543.051659 3 0xc6070000 0x1 0x0 0
> >> R 4 543.051732 3 0xc6070000 0x0 0x0 0
> >> W 4 543.051784 10 0xc439e000 0x7e 0x0 0
> >> W 4 543.051807 10 0xc439e004 0x7e 0x0 0
> >> W 4 543.051829 10 0xc439e008 0x1 0x0 0
> >> W 4 543.051851 10 0xc439e00c 0x2 0x0 0
> >> W 4 543.051926 3 0xc6070000 0x1 0x0 0
> >> R 4 543.051999 3 0xc6070000 0x0 0x0 0
> >> W 4 543.052158 3 0xc60032f4 0x1ff64 0x0 0
> >> W 4 543.052228 3 0xc60032ec 0x4 0x0 0
> >> R 4 543.052296 3 0xc60032ec 0x4 0x0 0
> >> R 4 543.052377 3 0xc6002504 0x0 0x0 0
> >> W 4 543.052451 3 0xc6002504 0x1 0x0 0
> >> R 4 543.052745 3 0xc6000100 0x0 0x0 0
> >> R 4 543.052849 3 0xc6002080 0x0 0x0 0
> >> R 4 543.053007 3 0xc6003220 0xd06191 0x0 0
> >> R 4 543.053075 3 0xc6003250 0x90000001 0x0 0
> >> R 4 543.053154 3 0xc6002504 0x11 0x0 0
> >> R 4 543.053226 3 0xc6002508 0x340 0x0 0
> >> R 4 543.053295 3 0xc6003220 0xd06191 0x0 0
> >> R 4 543.053365 3 0xc6003250 0x90000001 0x0 0
> >> R 4 543.053444 3 0xc6000200 0xdff3d113 0x0 0
> >> R 4 543.053516 3 0xc600251c 0x3f 0x0 0
> >> R 4 543.053581 3 0xc640032c 0x8001fd9a 0x0 0
> >> R 4 543.053630 3 0xc640032c 0x8001fd9a 0x0 0
> >> W 4 543.053678 3 0xc640032c 0x1fd9a 0x0 0
> >> R 4 543.053753 3 0xc60032f0 0x3 0x0 0
> >> W 4 543.053843 3 0xc60032f0 0x7f 0x0 0
> >> R 4 543.053921 3 0xc6003220 0xd06191 0x0 0
> >> W 4 543.053990 3 0xc6003220 0xd06191 0x0 0
> >> R 4 543.054054 3 0xc6002504 0x11 0x0 0
> >> W 4 543.054123 3 0xc6002504 0x10 0x0 0
> >> R 4 543.054195 3 0xc600260c 0x801fd99f 0x0 0
> >> W 4 543.054268 3 0xc600260c 0x1ff68 0x0 0
> >> W 4 543.054371 10 0xc43cdd10 0x0 0x0 0
> >> W 4 543.054393 10 0xc43cdd14 0x0 0x0 0
> >> W 4 543.054415 10 0xc43cdd18 0x0 0x0 0
> >> W 4 543.054437 10 0xc43cdd1c 0x0 0x0 0
> >> W 4 543.054460 10 0xc43cdd20 0x0 0x0 0
> >> W 4 543.054482 10 0xc43cdd24 0x0 0x0 0
> >> W 4 543.054504 10 0xc43cdd28 0x0 0x0 0
> >> W 4 543.054526 10 0xc43cdd2c 0x0 0x0 0
> >> W 4 543.054549 10 0xc43cdd30 0x0 0x0 0
> >> W 4 543.054571 10 0xc43cdd34 0x0 0x0 0
> >> W 4 543.054593 10 0xc43cdd38 0x0 0x0 0
> >> W 4 543.054616 10 0xc43cdd3c 0x0 0x0 0
> >> W 4 543.054638 10 0xc43cdd40 0x0 0x0 0
> >> W 4 543.054660 10 0xc43cdd44 0x0 0x0 0
> >> W 4 543.054823 3 0xc6070000 0x1 0x0 0
> >> R 4 543.054921 3 0xc6070000 0x0 0x0 0
> >>
> >
> > This chunk comes after it, very similar to the one before it. But i
> > forgot to add it.
> >
> > W 4 543.055001 3 0xc6100c80 0x50001 0x0 0
> > R 4 543.055059 3 0xc6100c80 0x50000 0x0 0
> > R 4 543.055111 3 0xc6400500 0x10010001 0x0 0
> > R 4 543.055159 3 0xc6400500 0x10010001 0x0 0
> > W 4 543.055207 3 0xc6400500 0x10010000 0x0 0
> > R 4 543.055256 3 0xc6400700 0x0 0x0 0
> > R 4 543.055304 3 0xc6400380 0x0 0x0 0
> > R 4 543.055352 3 0xc6400384 0x0 0x0 0
> > R 4 543.055400 3 0xc6400388 0x0 0x0 0
> > W 4 543.055454 3 0xc6100c80 0x1 0x0 0
> > R 4 543.055511 3 0xc6100c80 0x0 0x0 0
> > W 4 543.055562 3 0xc6400500 0x10010001 0x0 0
> > W 4 543.055657 3 0xc600260c 0x1ff680 0x0 0
> > W 4 543.055745 3 0xc6000140 0x1 0x0 0
> > W 4 543.055954 3 0xc6000140 0x0 0x0 0
> > W 4 543.055996 10 0xc43cdd48 0x0 0x0 0
> > W 4 543.056019 10 0xc43cdd4c 0x0 0x0 0
> > W 4 543.056041 10 0xc43cdd50 0x0 0x0 0
> > W 4 543.056064 10 0xc43cdd54 0x0 0x0 0
> > W 4 543.056167 3 0xc6070000 0x1 0x0 0
> > R 4 543.056246 3 0xc6070000 0x0 0x0 0
> >

Re: [PATCH/TESTING(all hw)/DISCUSSION] FIFO (minor) create and (major) destroy instabilities on nv50+

[Marcin Slusarz]

On Mon, Jan 04, 2010 at 11:42:08PM +0100, okias wrote:
> Lastest patch work for me. NV96

NV92 works too.