Re: [Qemu-devel] SPARC iommu mapping

Re: [Qemu-devel] SPARC iommu mapping

[Blue Swirl]

[-- Attachment #1: Type: text/plain, Size: 1009 bytes --]

Thank you for the excellent analysis of the problem! Looks like Qemu doesn't 
handle the case where DMA length and SCSI length are not equal.

>Until here, the addresses are subsequent. Then the next address is much 
>lower:

So Linux splits the transfer into two parts to use two separate pieces of 
buffers. Funny, I thought this situation was exactly the one which virtually 
addressed DMA would prevent from happening.

>What is expected, when the number of bytes from the transfer register have
>been read? Should the hardware raise an interrupt? And if yes, which flags
>must be set?

Yes, there will be an interrupt. I didn't find this situation in the 
NCR53C9X.txt file documenting the ESP, though.

There are two cases:
1 - DMA length shorter than SCSI (your case)
2 - SCSI length shorter (could happen in command replies etc.)

Maybe this patch helps?

_________________________________________________________________
Don't just search. Find. Check out the new MSN Search! 
http://search.msn.com/

[-- Attachment #2: qemu-sparc.patch-52.bz2 --]
[-- Type: application/octet-stream, Size: 924 bytes --]

Re: [Qemu-devel] SPARC iommu mapping

[Joerg Platte]

Am Freitag, 7. April 2006 17:44 schrieb Blue Swirl:
Hi!
> Thank you for the excellent analysis of the problem! Looks like Qemu
> doesn't handle the case where DMA length and SCSI length are not equal.

Exact.

> So Linux splits the transfer into two parts to use two separate pieces of
> buffers. Funny, I thought this situation was exactly the one which
> virtually addressed DMA would prevent from happening.

I read somewhere, that Solaris and Linux unfortunately do not try to allocate 
subsequent virtual addresses.

> Yes, there will be an interrupt. I didn't find this situation in the
> NCR53C9X.txt file documenting the ESP, though.

Me too. I tried to understand what's going wrong, but it is not that easy to 
understand the Linux source, if you don't know much about the hardware and 
the esp/iommu/dma/sbus architecture. 

> There are two cases:
> 1 - DMA length shorter than SCSI (your case)
> 2 - SCSI length shorter (could happen in command replies etc.)
>
> Maybe this patch helps?

Unfortunately not. I tried a similar approach, but it didn't work. Here is the 
dump:

First the mapping, similar to the one described in the last mail:

IOMMU: page flush f0039000
IOMMU: page flush f003a000
IOMMU: page flush f003b000
IOMMU: page flush f003c000
IOMMU: page flush f003d000
IOMMU: page flush f003e000
IOMMU: page flush f003f000
IOMMU: page flush f0040000
IOMMU: page flush f0041000
IOMMU: page flush f0042000
IOMMU: page flush f0043000
IOMMU: page flush f0044000
IOMMU: page flush f0045000
IOMMU: page flush f000e000

Then the transfer, which stops after three pages, since the Transfer 
information len is 12288:

ESP: Transfer Information len 12288
ESP: DMA Direction: r, addr 0x0bdea000 0000e000
ESP: DMA address p 0bdea000 v f0039000
ESP: DMA address p 0bdeb000 v f003a000
ESP: DMA address p 0bdec000 v f003b000
ESP: Write callback (offset 224 len 112 size 57344 trans_size 12288)

Now I would expect, that "somebody" (maybe the esp inerrupt handler) updates 
the dma registers to map the remaining pages. But this does not happen. Linux 
simply unmaps the old mapping:

IOMMU: page flush f0039000
IOMMU: page flush f003a000
IOMMU: page flush f003b000
IOMMU: page flush f003c000
IOMMU: page flush f003d000
IOMMU: page flush f003e000
IOMMU: page flush f003f000
IOMMU: page flush f0040000
IOMMU: page flush f0041000
IOMMU: page flush f0042000
IOMMU: page flush f0043000
IOMMU: page flush f0044000
IOMMU: page flush f0045000
IOMMU: page flush f000e000

and configures a new one:
IOMMU: page flush f000f000
IOMMU: page flush f0010000
IOMMU: page flush f0011000
IOMMU: page flush f0012000
IOMMU: page flush f0013000
IOMMU: page flush f0014000
IOMMU: page flush f0015000
IOMMU: page flush f0016000
IOMMU: page flush f0017000
IOMMU: page flush f0018000
IOMMU: page flush f0019000
IOMMU: page flush f001a000
IOMMU: page flush f001b000
IOMMU: page flush f001c000

Then, a few pages of the new mapping are written to disk:
ESP: Transfer Information len 36864
ESP: DMA Direction: r, addr 0x0bdfa000 0000e000
ESP: DMA address p 0bdfa000 v f000f000
ESP: DMA address p 0bdfb000 v f0010000
ESP: DMA address p 0bdfc000 v f0011000
ESP: DMA address p 0bdfd000 v f0012000
ESP: DMA address p 0bdfe000 v f0013000
ESP: DMA address p 0bdff000 v f0014000
ESP: DMA address p 0be00000 v f0015000
ESP: DMA address p 0be01000 v f0016000
ESP: DMA address p 0be02000 v f0017000
ESP: Write callback (offset 336 len 112 size 57344 trans_size 36864)

Then this ara is unmapped, and Linux continues with the next area.

Hence, your patch avoids the zero mapping, but it does not provide the 
required information which should result in a new dma addressing to write the 
remaining mapped data. Tomorrow, I'll try to enable more debbugging in the 
Linux driver to try to find out what's going wrong. But with a real hardware 
it's more difficult to get an idea of the internal functions :( 

regards,
Jörg

Re: [Qemu-devel] SPARC iommu mapping

[Joerg Platte]

[-- Attachment #1: Type: text/plain, Size: 703 bytes --]

Am Freitag, 7. April 2006 17:44 schrieb Blue Swirl:
Hi!

> Maybe this patch helps?

The attached patch is an updated version of your patch. With whis patch I was 
able to copy files from one directory to another on a disk image. After 
unmounting the image, e2fsck reported no errors on this image. But 
unfortunately, booting from this image was not possible. INIT reported a 
segmentation violation. Maybe there is something else wrong. But now writing 
works much better than before :-)

regards,
Jörg

PS: Is there any reason, why qemu copies disk data byte by byte? Calling 
iommu_translate only once per page and copying a whole page could speed up 
disk access significantly...

[-- Attachment #2: qemu-write.patch --]
[-- Type: text/x-diff, Size: 5321 bytes --]

--- esp.c.orig	2006-04-09 18:20:38.000000000 +0200
+++ esp.c	2006-04-09 18:04:15.000000000 +0200
@@ -63,6 +63,8 @@
     ESPDMAFunc *dma_cb;
     int64_t offset, len;
     int target;
+    int blocksize;
+    int ti_bufstart;
 };
 
 #define STAT_DO 0x00
@@ -229,12 +231,12 @@
                             target_phys_addr_t phys_addr, 
                             int transfer_size1)
 {
+    int len = transfer_size1/s->blocksize;
     DPRINTF("Write callback (offset %lld len %lld size %d trans_size %d)\n",
             s->offset, s->len, s->ti_size, transfer_size1);
-    bdrv_write(s->bd[s->target], s->offset, s->ti_buf, s->len);
-    s->offset = 0;
-    s->len = 0;
-    s->target = 0;
+
+    bdrv_write(s->bd[s->target], s->offset, s->ti_buf+s->ti_bufstart, len);
+    s->offset+=len;
     return 0;
 }
 
@@ -265,6 +267,7 @@
     s->ti_size = 0;
     s->ti_rptr = 0;
     s->ti_wptr = 0;
+    s->ti_bufstart = 0;
 
     if (target >= 4 || !s->bd[target]) { // No such drive
 	s->rregs[4] = STAT_IN;
@@ -293,6 +296,7 @@
 	s->ti_buf[3] = 2;
 	s->ti_buf[4] = 32;
 	s->ti_dir = 1;
+        s->ti_bufstart = 0;
 	s->ti_size = 36;
 	break;
     case 0x1a:
@@ -314,6 +318,7 @@
 	    s->ti_buf[6] = 2; // sector size 512
 	s->ti_buf[7] = 0;
 	s->ti_dir = 1;
+        s->ti_bufstart = 0;
 	s->ti_size = 8;
 	break;
     case 0x28:
@@ -336,6 +341,7 @@
 	    bdrv_read(s->bd[target], offset, s->ti_buf, len);
 	    // XXX error handling
 	    s->ti_dir = 1;
+	    s->ti_bufstart = 0;
 	    break;
 	}
     case 0x2a:
@@ -346,10 +352,12 @@
 		offset = ((buf[3] << 24) | (buf[4] << 16) | (buf[5] << 8) | buf[6]) * 4;
 		len = ((buf[8] << 8) | buf[9]) * 4;
 		s->ti_size = len * 2048;
+		s->blocksize=2048;
 	    } else {
 		offset = (buf[3] << 24) | (buf[4] << 16) | (buf[5] << 8) | buf[6];
 		len = (buf[8] << 8) | buf[9];
 		s->ti_size = len * 512;
+		s->blocksize=512;
 	    }
 	    DPRINTF("Write (10) (offset %lld len %lld)\n", offset, len);
             if (s->ti_size > TI_BUFSZ) {
@@ -359,6 +367,7 @@
             s->offset = offset;
             s->len = len;
             s->target = target;
+            s->ti_bufstart = 0;
 	    // XXX error handling
 	    s->ti_dir = 0;
 	    break;
@@ -400,6 +409,7 @@
                 break;
             }
 	    s->ti_dir = 1;
+	    s->ti_bufstart = 0;
             break;
         }
     default:
@@ -415,10 +425,9 @@
 
 static void dma_write(ESPState *s, const uint8_t *buf, uint32_t len)
 {
-    uint32_t dmaptr, dmalen;
+    uint32_t dmaptr;
 
-    dmalen = s->wregs[0] | (s->wregs[1] << 8);
-    DPRINTF("Transfer status len %d\n", dmalen);
+    DPRINTF("Transfer status len %d\n", len);
     if (s->dma) {
 	dmaptr = iommu_translate(s->espdmaregs[1]);
 	DPRINTF("DMA Direction: %c\n", s->espdmaregs[0] & 0x100? 'w': 'r');
@@ -428,10 +437,10 @@
 	s->rregs[6] = SEQ_CD;
     } else {
 	memcpy(s->ti_buf, buf, len);
-	s->ti_size = dmalen;
+	s->ti_size = len;
 	s->ti_rptr = 0;
 	s->ti_wptr = 0;
-	s->rregs[7] = dmalen;
+	s->rregs[7] = len;
     }
     s->espdmaregs[0] |= DMA_INTR;
     pic_set_irq(s->irq, 1);
@@ -442,34 +451,57 @@
 
 static void handle_ti(ESPState *s)
 {
-    uint32_t dmaptr, dmalen;
+    uint32_t dmaptr, dmalen, minlen;
     unsigned int i;
 
     dmalen = s->wregs[0] | (s->wregs[1] << 8);
-    DPRINTF("Transfer Information len %d\n", dmalen);
+    if (dmalen==0) {
+      dmalen=0x10000;
+    }
+
+    minlen = (dmalen < s->ti_size) ? dmalen : s->ti_size;
+    DPRINTF("Transfer Information len %d\n", minlen);
     if (s->dma) {
 	dmaptr = iommu_translate(s->espdmaregs[1]);
-	DPRINTF("DMA Direction: %c, addr 0x%8.8x\n", s->espdmaregs[0] & 0x100? 'w': 'r', dmaptr);
-	for (i = 0; i < s->ti_size; i++) {
+	DPRINTF("DMA Direction: %c, addr 0x%8.8x %08x %d %d\n", s->espdmaregs[0] & 0x100? 'w': 'r', dmaptr, s->ti_size, s->ti_bufstart, s->ti_dir);
+	for (i = 0; i < minlen; i++) {
 	    dmaptr = iommu_translate(s->espdmaregs[1] + i);
 	    if (s->ti_dir)
-		cpu_physical_memory_write(dmaptr, &s->ti_buf[i], 1);
+		cpu_physical_memory_write(dmaptr, &s->ti_buf[s->ti_bufstart+i], 1);
 	    else
-		cpu_physical_memory_read(dmaptr, &s->ti_buf[i], 1);
+		cpu_physical_memory_read(dmaptr, &s->ti_buf[s->ti_bufstart+i], 1);
 	}
         if (s->dma_cb) {
-            s->dma_cb(s, s->espdmaregs[1], dmalen);
+            s->dma_cb(s, s->espdmaregs[1], minlen);
+        }
+        if (minlen<s->ti_size) {
+	    s->rregs[4] = STAT_IN | STAT_TC | (s->ti_dir?STAT_DO:STAT_DI);
+	    s->ti_size-=minlen;
+	    s->ti_bufstart+=minlen;
+        } else {
+	    s->rregs[4] = STAT_IN | STAT_TC | STAT_ST;
             s->dma_cb = NULL;
+            s->offset = 0;
+            s->len = 0;
+            s->target = 0;
+            s->ti_bufstart = 0;
         }
-	s->rregs[4] = STAT_IN | STAT_TC | STAT_ST;
-	s->rregs[5] = INTR_BS;
+        s->rregs[5] = INTR_BS;
 	s->rregs[6] = 0;
+	s->rregs[7] = 0;
 	s->espdmaregs[0] |= DMA_INTR;
     } else {
-	s->ti_size = dmalen;
+	s->ti_size = minlen;
 	s->ti_rptr = 0;
 	s->ti_wptr = 0;
-	s->rregs[7] = dmalen;
+	s->rregs[7] = minlen;
     }	
     pic_set_irq(s->irq, 1);
 }
@@ -485,8 +517,10 @@
     s->ti_rptr = 0;
     s->ti_wptr = 0;
     s->ti_dir = 0;
+    s->ti_bufstart = 0;
     s->dma = 0;
     s->dma_cb = NULL;
+    s->blocksize = 0;
 }
 
 static uint32_t esp_mem_readb(void *opaque, target_phys_addr_t addr)

Re: [Qemu-devel] SPARC iommu mapping

[Joerg Platte]

[-- Attachment #1: Type: text/plain, Size: 228 bytes --]

Am Freitag, 7. April 2006 17:44 schrieb Blue Swirl:
Hi!

The attached patch is an updated version of my previous patch. Now it applies 
cleanly to cvs head and the read and write performance is increased.

regards,
Jörg

[-- Attachment #2: qemu-write2.patch --]
[-- Type: text/x-diff, Size: 5691 bytes --]

--- esp.c	2006-04-10 14:51:37.000000000 +0200
+++ esp.c.new	2006-04-10 14:54:44.000000000 +0200
@@ -63,6 +63,8 @@
     ESPDMAFunc *dma_cb;
     int64_t offset, len;
     int target;
+    int blocksize;
+    int ti_bufstart;
 };
 
 #define STAT_DO 0x00
@@ -229,12 +231,12 @@
                             target_phys_addr_t phys_addr, 
                             int transfer_size1)
 {
+    int len = transfer_size1/s->blocksize;
     DPRINTF("Write callback (offset %lld len %lld size %d trans_size %d)\n",
             s->offset, s->len, s->ti_size, transfer_size1);
-    bdrv_write(s->bd[s->target], s->offset, s->ti_buf, s->len);
-    s->offset = 0;
-    s->len = 0;
-    s->target = 0;
+
+    bdrv_write(s->bd[s->target], s->offset, s->ti_buf+s->ti_bufstart, len);
+    s->offset+=len;
     return 0;
 }
 
@@ -265,6 +267,7 @@
     s->ti_size = 0;
     s->ti_rptr = 0;
     s->ti_wptr = 0;
+    s->ti_bufstart = 0;
 
     if (target >= 4 || !s->bd[target]) { // No such drive
 	s->rregs[4] = STAT_IN;
@@ -293,6 +296,7 @@
 	s->ti_buf[3] = 2;
 	s->ti_buf[4] = 32;
 	s->ti_dir = 1;
+	s->ti_bufstart = 0;
 	s->ti_size = 36;
 	break;
     case 0x1a:
@@ -314,6 +318,7 @@
 	    s->ti_buf[6] = 2; // sector size 512
 	s->ti_buf[7] = 0;
 	s->ti_dir = 1;
+	s->ti_bufstart = 0;
 	s->ti_size = 8;
 	break;
     case 0x28:
@@ -336,6 +341,7 @@
 	    bdrv_read(s->bd[target], offset, s->ti_buf, len);
 	    // XXX error handling
 	    s->ti_dir = 1;
+	    s->ti_bufstart = 0;
 	    break;
 	}
     case 0x2a:
@@ -346,10 +352,12 @@
 		offset = ((buf[3] << 24) | (buf[4] << 16) | (buf[5] << 8) | buf[6]) * 4;
 		len = ((buf[8] << 8) | buf[9]) * 4;
 		s->ti_size = len * 2048;
+		s->blocksize=2048;
 	    } else {
 		offset = (buf[3] << 24) | (buf[4] << 16) | (buf[5] << 8) | buf[6];
 		len = (buf[8] << 8) | buf[9];
 		s->ti_size = len * 512;
+		s->blocksize=512;
 	    }
 	    DPRINTF("Write (10) (offset %lld len %lld)\n", offset, len);
             if (s->ti_size > TI_BUFSZ) {
@@ -359,6 +367,7 @@
             s->offset = offset;
             s->len = len;
             s->target = target;
+            s->ti_bufstart = 0;
 	    // XXX error handling
 	    s->ti_dir = 0;
 	    break;
@@ -400,6 +409,7 @@
                 break;
             }
 	    s->ti_dir = 1;
+	    s->ti_bufstart = 0;
             break;
         }
     default:
@@ -415,10 +425,9 @@
 
 static void dma_write(ESPState *s, const uint8_t *buf, uint32_t len)
 {
-    uint32_t dmaptr, dmalen;
+    uint32_t dmaptr;
 
-    dmalen = s->wregs[0] | (s->wregs[1] << 8);
-    DPRINTF("Transfer status len %d\n", dmalen);
+    DPRINTF("Transfer status len %d\n", len);
     if (s->dma) {
 	dmaptr = iommu_translate(s->espdmaregs[1]);
 	DPRINTF("DMA Direction: %c\n", s->espdmaregs[0] & 0x100? 'w': 'r');
@@ -428,10 +437,10 @@
 	s->rregs[6] = SEQ_CD;
     } else {
 	memcpy(s->ti_buf, buf, len);
-	s->ti_size = dmalen;
+	s->ti_size = len;
 	s->ti_rptr = 0;
 	s->ti_wptr = 0;
-	s->rregs[7] = dmalen;
+	s->rregs[7] = len;
     }
     s->espdmaregs[0] |= DMA_INTR;
     pic_set_irq(s->irq, 1);
@@ -442,34 +451,58 @@
 
 static void handle_ti(ESPState *s)
 {
-    uint32_t dmaptr, dmalen;
+    uint32_t dmaptr, dmalen, minlen, len, from, to;
     unsigned int i;
 
     dmalen = s->wregs[0] | (s->wregs[1] << 8);
-    DPRINTF("Transfer Information len %d\n", dmalen);
+    if (dmalen==0) {
+      dmalen=0x10000;
+    }
+
+    minlen = (dmalen < s->ti_size) ? dmalen : s->ti_size;
+    DPRINTF("Transfer Information len %d\n", minlen);
     if (s->dma) {
 	dmaptr = iommu_translate(s->espdmaregs[1]);
-	DPRINTF("DMA Direction: %c, addr 0x%8.8x\n", s->espdmaregs[0] & 0x100? 'w': 'r', dmaptr);
-	for (i = 0; i < s->ti_size; i++) {
+	DPRINTF("DMA Direction: %c, addr 0x%8.8x %08x %d %d\n", s->espdmaregs[0] & 0x100? 'w': 'r', dmaptr, s->ti_size, s->ti_bufstart, s->ti_dir);
+	from=s->espdmaregs[1];
+	to=from+minlen;
+	for (i = 0; i < minlen; i+=len, from+=len) {
 	    dmaptr = iommu_translate(s->espdmaregs[1] + i);
+	    if ((from&TARGET_PAGE_MASK)!=(to&TARGET_PAGE_MASK)) {
+               len=TARGET_PAGE_SIZE-(from&~TARGET_PAGE_MASK);
+            } else {
+	       len=to-from;
+            }
+            DPRINTF("DMA address p %08x v %08x len %08x, from %08x, to %08x\n", dmaptr, s->espdmaregs[1] + i, len, from, to);
 	    if (s->ti_dir)
-		cpu_physical_memory_write(dmaptr, &s->ti_buf[i], 1);
+		cpu_physical_memory_write(dmaptr, &s->ti_buf[s->ti_bufstart+i], len);
 	    else
-		cpu_physical_memory_read(dmaptr, &s->ti_buf[i], 1);
+		cpu_physical_memory_read(dmaptr, &s->ti_buf[s->ti_bufstart+i], len);
 	}
         if (s->dma_cb) {
-            s->dma_cb(s, s->espdmaregs[1], dmalen);
+            s->dma_cb(s, s->espdmaregs[1], minlen);
+        }
+        if (minlen<s->ti_size) {
+	    s->rregs[4] = STAT_IN | STAT_TC | (s->ti_dir?STAT_DO:STAT_DI);
+	    s->ti_size-=minlen;
+	    s->ti_bufstart+=minlen;
+        } else {
+	    s->rregs[4] = STAT_IN | STAT_TC | STAT_ST;
             s->dma_cb = NULL;
+            s->offset = 0;
+            s->len = 0;
+            s->target = 0;
+            s->ti_bufstart = 0;
         }
-	s->rregs[4] = STAT_IN | STAT_TC | STAT_ST;
-	s->rregs[5] = INTR_BS;
+        s->rregs[5] = INTR_BS;
 	s->rregs[6] = 0;
+	s->rregs[7] = 0;
 	s->espdmaregs[0] |= DMA_INTR;
     } else {
-	s->ti_size = dmalen;
+	s->ti_size = minlen;
 	s->ti_rptr = 0;
 	s->ti_wptr = 0;
-	s->rregs[7] = dmalen;
+	s->rregs[7] = minlen;
     }	
     pic_set_irq(s->irq, 1);
 }
@@ -485,8 +518,10 @@
     s->ti_rptr = 0;
     s->ti_wptr = 0;
     s->ti_dir = 0;
+    s->ti_bufstart = 0;
     s->dma = 0;
     s->dma_cb = NULL;
+    s->blocksize = 0;
 }
 
 static uint32_t esp_mem_readb(void *opaque, target_phys_addr_t addr)

Re: [Qemu-devel] SPARC iommu mapping

[Blue Swirl]

>The attached patch is an updated version of my previous patch. Now it 
>applies
>cleanly to cvs head and the read and write performance is increased.

Great work! Now the Debian 3.0r4 installer with 2.6.11+tcx kernel almost 
finishes. Performance is also much better.

There is some problem with CDROM access, the files on CD look corrupt, 
whereas the same image used as hdb works perfectly.

Some comments on the code: Instead of adding s->bufstart, maybe you could 
use the s->rptr, as it is used for similar purpose for FIFO. Please add more 
spaces to conform to general code style.

For even better performance, you could try to avoid the memory to buffer 
copy and give bdrv_read pointer to host memory.

_________________________________________________________________
FREE pop-up blocking with the new MSN Toolbar - get it now! 
http://toolbar.msn.click-url.com/go/onm00200415ave/direct/01/

Re: [Qemu-devel] SPARC iommu mapping

[Joerg Platte]

[-- Attachment #1: Type: text/plain, Size: 1689 bytes --]

Am Dienstag, 11. April 2006 21:21 schrieb Blue Swirl:
Hi!

> Great work! Now the Debian 3.0r4 installer with 2.6.11+tcx kernel almost
> finishes. Performance is also much better.

I tried to boot a deb-bootstrapped SPARC image. But the first executed script 
segfaults when executing another command in backticks. The same happens when 
executing the bash and inside the bash another command. When the other 
command exits the bash segfaults. Other shells like the csh or the tcsh don't 
suffer from this problem.

After replacing the bash in the image with a minimal bourne compatible shell 
qemu was able to execute all scripts in rcS.

Then I tried to track down this problem with gdb and strace. But 
unfortunately, both are segfaulting, too. This makes debugging somewhat 
harder... Now, I'll try to analyze the core files of the crashing programs to 
get an idea what's going wrong. Or do you have any idea? What's the 
difference between bash compared to other shells?

> There is some problem with CDROM access, the files on CD look corrupt,
> whereas the same image used as hdb works perfectly.

Strange. I started qemu with "-cdrom /dev/cdrom" and could see all contents 
without a problem. What exactly did you try?

> Some comments on the code: Instead of adding s->bufstart, maybe you could
> use the s->rptr, as it is used for similar purpose for FIFO. Please add
> more spaces to conform to general code style.

OK, have a look at the attached patch.

> For even better performance, you could try to avoid the memory to buffer
> copy and give bdrv_read pointer to host memory.

Yes, I'll try this in the next patch :-)

regards,
Jörg

[-- Attachment #2: qemu-sparc-write-patch.diff --]
[-- Type: text/x-diff, Size: 4303 bytes --]

--- esp.c	2006-04-14 17:53:09.000000000 +0200
+++ esp.c.new	2006-04-14 17:49:55.000000000 +0200
@@ -229,12 +229,17 @@
                             target_phys_addr_t phys_addr, 
                             int transfer_size1)
 {
+    int len;
+    if (bdrv_get_type_hint(s->bd[s->target]) == BDRV_TYPE_CDROM) {
+        len = transfer_size1/2048;
+    } else {
+        len = transfer_size1/512;
+    }
     DPRINTF("Write callback (offset %lld len %lld size %d trans_size %d)\n",
             s->offset, s->len, s->ti_size, transfer_size1);
-    bdrv_write(s->bd[s->target], s->offset, s->ti_buf, s->len);
-    s->offset = 0;
-    s->len = 0;
-    s->target = 0;
+
+    bdrv_write(s->bd[s->target], s->offset, s->ti_buf+s->ti_rptr, len);
+    s->offset+=len;
     return 0;
 }
 
@@ -336,6 +341,7 @@
 	    bdrv_read(s->bd[target], offset, s->ti_buf, len);
 	    // XXX error handling
 	    s->ti_dir = 1;
+	    s->ti_rptr = 0;
 	    break;
 	}
     case 0x2a:
@@ -359,6 +365,7 @@
             s->offset = offset;
             s->len = len;
             s->target = target;
+            s->ti_rptr = 0;
 	    // XXX error handling
 	    s->ti_dir = 0;
 	    break;
@@ -415,10 +422,9 @@
 
 static void dma_write(ESPState *s, const uint8_t *buf, uint32_t len)
 {
-    uint32_t dmaptr, dmalen;
+    uint32_t dmaptr;
 
-    dmalen = s->wregs[0] | (s->wregs[1] << 8);
-    DPRINTF("Transfer status len %d\n", dmalen);
+    DPRINTF("Transfer status len %d\n", len);
     if (s->dma) {
 	dmaptr = iommu_translate(s->espdmaregs[1]);
 	DPRINTF("DMA Direction: %c\n", s->espdmaregs[0] & 0x100? 'w': 'r');
@@ -428,10 +434,10 @@
 	s->rregs[6] = SEQ_CD;
     } else {
 	memcpy(s->ti_buf, buf, len);
-	s->ti_size = dmalen;
+	s->ti_size = len;
 	s->ti_rptr = 0;
 	s->ti_wptr = 0;
-	s->rregs[7] = dmalen;
+	s->rregs[7] = len;
     }
     s->espdmaregs[0] |= DMA_INTR;
     pic_set_irq(s->irq, 1);
@@ -442,34 +448,58 @@
 
 static void handle_ti(ESPState *s)
 {
-    uint32_t dmaptr, dmalen;
+    uint32_t dmaptr, dmalen, minlen, len, from, to;
     unsigned int i;
 
     dmalen = s->wregs[0] | (s->wregs[1] << 8);
-    DPRINTF("Transfer Information len %d\n", dmalen);
+    if (dmalen==0) {
+      dmalen=0x10000;
+    }
+
+    minlen = (dmalen < s->ti_size) ? dmalen : s->ti_size;
+    DPRINTF("Transfer Information len %d\n", minlen);
     if (s->dma) {
 	dmaptr = iommu_translate(s->espdmaregs[1]);
-	DPRINTF("DMA Direction: %c, addr 0x%8.8x\n", s->espdmaregs[0] & 0x100? 'w': 'r', dmaptr);
-	for (i = 0; i < s->ti_size; i++) {
+	DPRINTF("DMA Direction: %c, addr 0x%8.8x %08x %d %d\n", s->espdmaregs[0] & 0x100? 'w': 'r', dmaptr, s->ti_size, s->ti_rptr, s->ti_dir);
+	from = s->espdmaregs[1];
+	to = from + minlen;
+	for (i = 0; i < minlen; i += len, from += len) {
 	    dmaptr = iommu_translate(s->espdmaregs[1] + i);
+	    if ((from & TARGET_PAGE_MASK) != (to & TARGET_PAGE_MASK)) {
+               len = TARGET_PAGE_SIZE - (from & ~TARGET_PAGE_MASK);
+            } else {
+	       len = to - from;
+            }
+            DPRINTF("DMA address p %08x v %08x len %08x, from %08x, to %08x\n", dmaptr, s->espdmaregs[1] + i, len, from, to);
 	    if (s->ti_dir)
-		cpu_physical_memory_write(dmaptr, &s->ti_buf[i], 1);
+		cpu_physical_memory_write(dmaptr, &s->ti_buf[s->ti_rptr + i], len);
 	    else
-		cpu_physical_memory_read(dmaptr, &s->ti_buf[i], 1);
+		cpu_physical_memory_read(dmaptr, &s->ti_buf[s->ti_rptr + i], len);
 	}
         if (s->dma_cb) {
-            s->dma_cb(s, s->espdmaregs[1], dmalen);
+            s->dma_cb(s, s->espdmaregs[1], minlen);
+        }
+        if (minlen < s->ti_size) {
+	    s->rregs[4] = STAT_IN | STAT_TC | (s->ti_dir ? STAT_DO : STAT_DI);
+	    s->ti_size -= minlen;
+	    s->ti_rptr += minlen;
+        } else {
+	    s->rregs[4] = STAT_IN | STAT_TC | STAT_ST;
             s->dma_cb = NULL;
+            s->offset = 0;
+            s->len = 0;
+            s->target = 0;
+            s->ti_rptr = 0;
         }
-	s->rregs[4] = STAT_IN | STAT_TC | STAT_ST;
-	s->rregs[5] = INTR_BS;
+        s->rregs[5] = INTR_BS;
 	s->rregs[6] = 0;
+	s->rregs[7] = 0;
 	s->espdmaregs[0] |= DMA_INTR;
     } else {
-	s->ti_size = dmalen;
+	s->ti_size = minlen;
 	s->ti_rptr = 0;
 	s->ti_wptr = 0;
-	s->rregs[7] = dmalen;
+	s->rregs[7] = minlen;
     }	
     pic_set_irq(s->irq, 1);
 }

Re: [Qemu-devel] SPARC iommu mapping

[Blue Swirl]

>Then I tried to track down this problem with gdb and strace. But
>unfortunately, both are segfaulting, too. This makes debugging somewhat
>harder... Now, I'll try to analyze the core files of the crashing programs 
>to
>get an idea what's going wrong. Or do you have any idea? What's the
>difference between bash compared to other shells?

I don't know,  I noticed the same problem earlier. I think it's something 
related to fork or clone.

> > There is some problem with CDROM access, the files on CD look corrupt,
> > whereas the same image used as hdb works perfectly.
>
>Strange. I started qemu with "-cdrom /dev/cdrom" and could see all contents
>without a problem. What exactly did you try?

This command fails when installing kernel and modules:
qemu-system-sparc -kernel vmlinux-2.6.11+tcx -cdrom 
debian-30r4-sparc-binary-1.iso -hda hd -initrd debian-30r4-sparc.initrd 
-nographic -append 'cdrom ramdisk_size=16384 devfs=mount rw root=/dev/rd/0'

"There was a problem installing the Drivers from  
/instmnt/dists/woody/main/disks-sparc/current/sun4cdm/drivers.tgz"
and some errors in background.

Replace -cdrom with -hdb and the installer has no problems until installing 
libc6.

>OK, have a look at the attached patch.

Much better, though a few spaces are still missing.

_________________________________________________________________
Don’t just search. Find. Check out the new MSN Search! 
http://search.msn.click-url.com/go/onm00200636ave/direct/01/

Re: [Qemu-devel] SPARC iommu mapping

[Joerg Platte]

Am Montag, 17. April 2006 10:49 schrieb Blue Swirl:
Hi!

> This command fails when installing kernel and modules:
> qemu-system-sparc -kernel vmlinux-2.6.11+tcx -cdrom
> debian-30r4-sparc-binary-1.iso -hda hd -initrd debian-30r4-sparc.initrd
> -nographic -append 'cdrom ramdisk_size=16384 devfs=mount rw root=/dev/rd/0'

After spending more time on this issue I tried to reproduce your problems. But 
I can't find a suitable initrd. The initrd from ftp.debian.org caused an 
Oops:

RAMDISK: ext2 filesystem found at block 0
RAMDISK: Loading 10028KiB [1 disk] into ram disk... done.
VFS: Mounted root (ext2 filesystem).
Root-NFS: No NFS server available, giving up.
Unable to handle kernel NULL pointer dereference
tsk->{mm,active_mm}->context = ffffffff
tsk->{mm,active_mm}->pgd = fc000000
              \|/ ____ \|/
              "@'/ ,. \`@"
              /_| \__/ |_\
                 \__U_/
swapper(1): Oops [#1]

Which initrd did you use?

regards,
Jörg

Re: [Qemu-devel] SPARC iommu mapping

[Blue Swirl]

>swapper(1): Oops [#1]
>
>Which initrd did you use?

It's the file
/dists/stable/main/disks-sparc/current/images-1.44/root.bin
as specified by /boot/silo.conf, from debian-30r4-sparc-binary-1.iso.

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

[Qemu-devel] SPARC iommu mapping

[Joerg Platte]

Hi!

I still have problems when using a disk image for the SPARC system emulator. 
Reading and writing now works better than in previous versions, but sometimes 
data written to image is simply corrupt. To find this problem I wrote a small 
program writing data to the beginning of a disk image. Afterwards I compared 
the written data in the image with the expected values. Most of the data was 
correct, but sometimes a few pages mostly contains only zeros. 

To find this problem, I enabled debugging in the esp.c file and printed the 
mapped address (after iommu mapping). In some cases the mapped address is 
zero:

Normal write:

ESP: DMA Direction: r, addr 0x008fd000 0000000b
ESP: DMA Direction: r, addr 0x0bea1000 0000e000
ESP: DMA address 0bea2000
ESP: DMA address 0bea4000
ESP: DMA address 0bea5000
ESP: DMA address 0bea6000
ESP: DMA address 0bea7000
ESP: DMA address 0bea8000
ESP: DMA address 0bea9000
ESP: DMA address 0beaa000
ESP: DMA address 0beab000
ESP: DMA address 0beac000
ESP: DMA address 0bead000
ESP: DMA address 0beaf000
ESP: DMA address 0beb0000

Faulty write:

ESP: DMA Direction: r
ESP: DMA Direction: r, addr 0x008fd000 0000000b
ESP: DMA Direction: r, addr 0x0beb1000 0000e000
ESP: DMA address 0beb2000
ESP: DMA address 0beb3000
ESP: DMA address 0beb4000
ESP: DMA address 0beb5000
ESP: DMA address 0beb6000
ESP: DMA address 0beb7000
ESP: DMA address 0beb8000
ESP: DMA address 00000000

I used this code in handle_ti to produce this output:

	    dmaptr = iommu_translate(s->espdmaregs[1] + i);
	    if ((olddma&0xfffff000) != (dmaptr&0xfffff000)) {
		    DPRINTF("DMA address %08x\n", dmaptr);
		    olddma=dmaptr;
	    }


Unfortunately, I'm no hardware or SPARC expert and it took a whole day to 
track down this problem. So I'm not able to find the reason for this 
behaviour. What may cause the wrong mapping? Is this some kind of page fault, 
which must be handled by the kernel? Or just a bug?

regards,
Jörg

Re: [Qemu-devel] SPARC iommu mapping

[Joerg Platte]

Am Dienstag, 4. April 2006 21:27 schrieb Joerg Platte:
Hi!

> To find this problem, I enabled debugging in the esp.c file and printed the
> mapped address (after iommu mapping). In some cases the mapped address is
> zero:

> ESP: DMA address 0beb8000
> ESP: DMA address 00000000

I didn't find a manual of the SBUS IOMMU. But if I understand the contents of 
newer implementations of an IOMMU, each entry in this MMU table has a valid 
bit. And therefore, a 0 entry is invalid. But the current implementation of 
qemu's IOMMU does not check for a valid bit. Where can I find more 
information on this topic? And is a besser IOMMU currently in development?

regards,
Jörg

Re: [Qemu-devel] SPARC iommu mapping

[Blue Swirl]

> > To find this problem, I enabled debugging in the esp.c file and printed 
>the
> > mapped address (after iommu mapping). In some cases the mapped address 
>is
> > zero:
>
> > ESP: DMA address 0beb8000
> > ESP: DMA address 00000000
>
>I didn't find a manual of the SBUS IOMMU. But if I understand the contents 
>of
>newer implementations of an IOMMU, each entry in this MMU table has a valid
>bit. And therefore, a 0 entry is invalid. But the current implementation of
>qemu's IOMMU does not check for a valid bit. Where can I find more
>information on this topic? And is a besser IOMMU currently in development?

I don't have any better documentation either, I just coded against what 
Proll and Linux expect. But the theory of operation is simple. Much like how 
the MMU translates CPU's virtual addresses to physical addresses for memory, 
IOMMU translates device virtual memory accesses to physical addresses. The 
VA to PA entries are found in a simple page table.

In the case of not finding a valid translation entry, IOMMU can't fault the 
device like normal MMU can easily fault the CPU. I don't know what should 
happen then, probably put the address to AFAR register and raise some 
interrupt, while the device (for example Ethernet controller) waiting for 
the data suffers in limbo. I think it would be strange for an OS to rely on 
this, so I guess it's a bug somewhere else. My guess for the valid bit is 
that it's used in a real IOMMU to select the entries that will be loaded to 
its internal translation buffer.

The DMA controllers for both ESP and Lance are within the same page. This 
means that in Qemu, DMA controller register accesses for either of them go 
to just one of these. It just happens to work, but maybe this causes the 
problem. You could try to confirm this by enabling also DEBUG_LANCE and see 
if there is troublesome activity in the Lance direction near the bad 
accesses.

Can you provide a test case so that I could try it as well?

_________________________________________________________________
Is your PC infected? Get a FREE online computer virus scan from McAfee® 
Security. http://clinic.mcafee.com/clinic/ibuy/campaign.asp?cid=3963

Re: [Qemu-devel] SPARC iommu mapping

[Joerg Platte]

[-- Attachment #1: Type: text/plain, Size: 2030 bytes --]

Am Mittwoch, 5. April 2006 19:36 schrieb Blue Swirl:
Hi!

> I don't have any better documentation either, I just coded against what
> Proll and Linux expect. But the theory of operation is simple. Much like

OK, I thought I was just too stupid to find good documentation :-)

> how the MMU translates CPU's virtual addresses to physical addresses for
> memory, IOMMU translates device virtual memory accesses to physical
> addresses. The VA to PA entries are found in a simple page table.

That's what I read from qemu's iommu functions.

> In the case of not finding a valid translation entry, IOMMU can't fault the
> device like normal MMU can easily fault the CPU. I don't know what should
> happen then, probably put the address to AFAR register and raise some
> interrupt, while the device (for example Ethernet controller) waiting for
> the data suffers in limbo. I think it would be strange for an OS to rely on
> this, so I guess it's a bug somewhere else. My guess for the valid bit is
> that it's used in a real IOMMU to select the entries that will be loaded to
> its internal translation buffer.

Maybe. 

> The DMA controllers for both ESP and Lance are within the same page. This
> means that in Qemu, DMA controller register accesses for either of them go
> to just one of these. It just happens to work, but maybe this causes the
> problem. You could try to confirm this by enabling also DEBUG_LANCE and see
> if there is troublesome activity in the Lance direction near the bad
> accesses.

Hmmm, I don't use the network. Just disk access. But I'll check this.

> Can you provide a test case so that I could try it as well?

I'm using the two attached programs. writetest is used inside qemu to directly 
write to /dev/sda (with "writetest /dev/sda"). readtest can then be used 
outside qemu to check the written data ("readtest imagefile"). 

Additionally, I logged the translated virtual address as mentioned in my first 
mail. The kernel is 2.6.13. 

regards,
Jörg

[-- Attachment #2: readtest.c --]
[-- Type: text/x-csrc, Size: 671 bytes --]

#include <stdio.h>


int main(int argc, char**argv) {
  FILE *fp;
  unsigned char buffer[4];
  unsigned int i,j;
  unsigned int doprint=1;
  unsigned int d=0;
  if ((fp=fopen(argv[1], "r"))) {
    for (i=0; i<10000000; i+=4) {
      if (fread(&buffer, 4, 1, fp)!=1) {
        printf("error writing at byte %d\n", i);
        break;
      }
      d=0;
      for(j=0;j<4; j++) {
        if (buffer[3-j]!=((i>>(j*8))&0xff)) {
          if (doprint) {
            printf("data differs at address %08x\n", i);
          }
          d=1;
          break;
        }
      }
      if (d) {
        doprint=0;
      } else {
        doprint=1;
      }
    }
    fclose(fp);
  }
}

[-- Attachment #3: writetest.c --]
[-- Type: text/x-csrc, Size: 289 bytes --]

#include <stdio.h>


int main(int argc, char**argv) {
  FILE *fp;
  unsigned int i;
  if ((fp=fopen(argv[1], "w"))) {
    for (i=0; i<10000000; i+=4) {
      if (fwrite(&i, 4, 1, fp)!=1) {
        printf("error writing at byte %d\n", i);
        break;
      }
    }
    fclose(fp);
  }
}

Re: [Qemu-devel] SPARC iommu mapping

[Joerg Platte]

[-- Attachment #1: Type: text/plain, Size: 2030 bytes --]

Am Mittwoch, 5. April 2006 19:36 schrieb Blue Swirl:
Hi!

> I don't have any better documentation either, I just coded against what
> Proll and Linux expect. But the theory of operation is simple. Much like

OK, I thought I was just too stupid to find good documentation :-)

> how the MMU translates CPU's virtual addresses to physical addresses for
> memory, IOMMU translates device virtual memory accesses to physical
> addresses. The VA to PA entries are found in a simple page table.

That's what I read from qemu's iommu functions.

> In the case of not finding a valid translation entry, IOMMU can't fault the
> device like normal MMU can easily fault the CPU. I don't know what should
> happen then, probably put the address to AFAR register and raise some
> interrupt, while the device (for example Ethernet controller) waiting for
> the data suffers in limbo. I think it would be strange for an OS to rely on
> this, so I guess it's a bug somewhere else. My guess for the valid bit is
> that it's used in a real IOMMU to select the entries that will be loaded to
> its internal translation buffer.

Maybe. 

> The DMA controllers for both ESP and Lance are within the same page. This
> means that in Qemu, DMA controller register accesses for either of them go
> to just one of these. It just happens to work, but maybe this causes the
> problem. You could try to confirm this by enabling also DEBUG_LANCE and see
> if there is troublesome activity in the Lance direction near the bad
> accesses.

Hmmm, I don't use the network. Just disk access. But I'll check this.

> Can you provide a test case so that I could try it as well?

I'm using the two attached programs. writetest is used inside qemu to directly 
write to /dev/sda (with "writetest /dev/sda"). readtest can then be used 
outside qemu to check the written data ("readtest imagefile"). 

Additionally, I logged the translated virtual address as mentioned in my first 
mail. The kernel is 2.6.13. 

regards,
Jörg

[-- Attachment #2: readtest.c --]
[-- Type: text/x-csrc, Size: 671 bytes --]

#include <stdio.h>


int main(int argc, char**argv) {
  FILE *fp;
  unsigned char buffer[4];
  unsigned int i,j;
  unsigned int doprint=1;
  unsigned int d=0;
  if ((fp=fopen(argv[1], "r"))) {
    for (i=0; i<10000000; i+=4) {
      if (fread(&buffer, 4, 1, fp)!=1) {
        printf("error writing at byte %d\n", i);
        break;
      }
      d=0;
      for(j=0;j<4; j++) {
        if (buffer[3-j]!=((i>>(j*8))&0xff)) {
          if (doprint) {
            printf("data differs at address %08x\n", i);
          }
          d=1;
          break;
        }
      }
      if (d) {
        doprint=0;
      } else {
        doprint=1;
      }
    }
    fclose(fp);
  }
}

[-- Attachment #3: writetest.c --]
[-- Type: text/x-csrc, Size: 289 bytes --]

#include <stdio.h>


int main(int argc, char**argv) {
  FILE *fp;
  unsigned int i;
  if ((fp=fopen(argv[1], "w"))) {
    for (i=0; i<10000000; i+=4) {
      if (fwrite(&i, 4, 1, fp)!=1) {
        printf("error writing at byte %d\n", i);
        break;
      }
    }
    fclose(fp);
  }
}

Re: [Qemu-devel] SPARC iommu mapping

[Joerg Platte]

Am Mittwoch, 5. April 2006 19:36 schrieb Blue Swirl:
Hi!

> The DMA controllers for both ESP and Lance are within the same page. This
> means that in Qemu, DMA controller register accesses for either of them go
> to just one of these. It just happens to work, but maybe this causes the
> problem. You could try to confirm this by enabling also DEBUG_LANCE and see
> if there is troublesome activity in the Lance direction near the bad
> accesses.

Lance is only used during system startup. Then there is no more debugging 
output. Looks like ethernet is not causing any problems. 

Can you reproduce my problems, or is it caused by the kernel? But the same 
kernel works without problems on a SS4...

regards,
Jörg

Re: [Qemu-devel] SPARC iommu mapping

[Joerg Platte]

Am Mittwoch, 5. April 2006 19:36 schrieb Blue Swirl:
Hi!

Today I did some more tests to determine, what's going wrong. At first, Linux 
flushes the page entries for the 14 buffers that are to be written to disk:
IOMMU: page flush f001c000
IOMMU: page flush f001d000
IOMMU: page flush f001e000
IOMMU: page flush f001f000
IOMMU: page flush f0020000
IOMMU: page flush f0021000
IOMMU: page flush f0022000
IOMMU: page flush f0023000
IOMMU: page flush f0024000

Until here, the addresses are subsequent. Then the next address is much lower:

IOMMU: page flush f000e000
IOMMU: page flush f000f000
IOMMU: page flush f0010000
IOMMU: page flush f0011000
IOMMU: page flush f0012000

Then, after sending the SCSI command, the DMA transfer starts:
ESP: DMA Transfer Information len 00009000
ESP: DMA Direction: r, addr 0x0bdbb000 0000e000

Here, 0000e000 is the number of bytes to write accoring to the SCSI command. 
In this case it equals the number of pages mentioned above (14). But Linux 
stores 00009000 in the SCSI controllers transfer counter, which is less than 
these 14 pages.

Qemu's SCSI driver implementation now reads 14 pages and ignores the number of 
bytes written to the transfer register and writes 14 pages to disk:

ESP: DMA address p 0bdbb000 v f001c000
ESP: DMA address p 0bdbc000 v f001d000
ESP: DMA address p 0bdbd000 v f001e000
ESP: DMA address p 0bdbe000 v f001f000
ESP: DMA address p 0bdbf000 v f0020000
ESP: DMA address p 0bdc0000 v f0021000
ESP: DMA address p 0bdc1000 v f0022000
ESP: DMA address p 0bdc2000 v f0023000
ESP: DMA address p 0bdc3000 v f0024000

No problem until here.

ESP: DMA address p 00000000 v f0025000

Here, after 9 pages qemu tries to access page f0025000. But this page has 
never been written to the page table and therefore the mapped address is 0. 
The next page is f000e000. And this problem occures after 00009000 bytes 
(which is the number of bytes written to the transfer register). Hence, the 
kernel must adjust the DMA controller to point to the next address, because 
starting from here, there is a "gap" in the virtual address space. Currently 
I'm trying to understand the Linux kernel to figure out, whats the expexted 
behaviour of this scsi controller. But it's not that easy, because I have no 
deep knowledge about the sparc hardware.

What is expected, when the number of bytes from the transfer register have 
been read? Should the hardware raise an interrupt? And if yes, which flags 
must be set?

regards,
Jörg