xen-swiotlb and linux swiotlb

xen-swiotlb and linux swiotlb

[Sander Eikelenboom]

Hi Konrad,

Don't know if this is logical, since it uses the same range.

[    0.000000] Placing 64MB software IO TLB between ffff880003d00000 - ffff880007d00000
[    0.000000] software IO TLB at phys 0x3d00000 - 0x7d00000

[    0.047391] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.047402] Placing 64MB software IO TLB between ffff880003d00000 - ffff880007d00000
[    0.047406] software IO TLB at phys 0x3d00000 - 0x7d00000


And with 2.6.36pre:

[    0.000000] DMA: Placing 64MB software IO TLB between ffff88000eb5d000 - ffff880012b5d000
[    0.000000] DMA: software IO TLB at phys 0xeb5d000 - 0x12b5d000
[    0.000000] xen_swiotlb_fixup: buf=ffff88000eb5d000 size=67108864
[    0.000000] xen_swiotlb_fixup: buf=ffff880012bbd000 size=32768

[    3.297348] DMA-API: preallocated 32768 debug entries
[    3.300022] DMA-API: debugging enabled by kernel config
[    3.300022] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    3.300022] DMA: Placing 64MB software IO TLB between ffff88000eb5d000 - ffff880012b5d000
[    3.300022] DMA: software IO TLB at phys 0xeb5d000 - 0x12b5d000


Seems xen-swiotlb splits the mem space, and the 32768 is probably for a translaction table ?
Since it gets overriden by the normal swiotlb later on, this mem could be used by devices as DMA mem, and since it is on the end, it could take a while ..  and finally overwrite mem it shouldn't and freeze everyting ?

--
Sander

Re: xen-swiotlb and linux swiotlb

[Konrad Rzeszutek Wilk]

[-- Attachment #1: Type: text/plain, Size: 2625 bytes --]

On Sun, Aug 15, 2010 at 07:02:25PM +0200, Sander Eikelenboom wrote:
> Hi Konrad,
> 
> Don't know if this is logical, since it uses the same range.
> 
> [    0.000000] Placing 64MB software IO TLB between ffff880003d00000 - ffff880007d00000
> [    0.000000] software IO TLB at phys 0x3d00000 - 0x7d00000
> 
> [    0.047391] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> [    0.047402] Placing 64MB software IO TLB between ffff880003d00000 - ffff880007d00000
> [    0.047406] software IO TLB at phys 0x3d00000 - 0x7d00000
> 
> 
> And with 2.6.36pre:
> 
> [    0.000000] DMA: Placing 64MB software IO TLB between ffff88000eb5d000 - ffff880012b5d000
> [    0.000000] DMA: software IO TLB at phys 0xeb5d000 - 0x12b5d000
> [    0.000000] xen_swiotlb_fixup: buf=ffff88000eb5d000 size=67108864
> [    0.000000] xen_swiotlb_fixup: buf=ffff880012bbd000 size=32768
> 
> [    3.297348] DMA-API: preallocated 32768 debug entries
> [    3.300022] DMA-API: debugging enabled by kernel config
> [    3.300022] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> [    3.300022] DMA: Placing 64MB software IO TLB between ffff88000eb5d000 - ffff880012b5d000
> [    3.300022] DMA: software IO TLB at phys 0xeb5d000 - 0x12b5d000
> 
> 
> Seems xen-swiotlb splits the mem space, and the 32768 is probably for a translaction table ?

I am not sure if you are referring to:
> [    3.297348] DMA-API: preallocated 32768 debug entries
or:
> [    0.000000] xen_swiotlb_fixup: buf=ffff880012bbd000 size=32768

The first is the entries for the DMA debug API. Every time your device
does an PCI map/unmap, that gets recorded in the API. And if you call
the 'debug_dma_dump_mappings' (see attached module). you get a nice
kernel dump of all the DMA kernel mappings. And you can count them up
to and see how much space is used.

The second is for an overflow buffer. That overflow buffer was used when
the main buffer (the 64MB) got completly used and we would use this 32kB
space as last ditch. It is going away and..

> Since it gets overriden by the normal swiotlb later on, this mem could be used by devices as DMA mem, and since it is on the end, it could take a while ..  and finally overwrite mem it shouldn't and freeze everyting ?

.. we don't end up using it in Xen-SWIOTLB. So no worry about overwrite.
thought we do lose 32768 bytes of memory b/c of this. 

In 2.6.38-timeframe I plan along with Fujita to remove this overflow_buffer
machination and make sure that most drivers used the 'pci_dma_mapping'
call to check if the region they got is DMA mappable instead of blindly
assuming that the pci_map_single call worked.


[-- Attachment #2: dump_dma.c --]
[-- Type: text/plain, Size: 1644 bytes --]

/*
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License v2.0 as published by
 * the Free Software Foundation
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 */

#include <linux/module.h>
#include <linux/string.h>
#include <linux/types.h>
#include <linux/init.h>
#include <linux/stat.h>
#include <linux/err.h>
#include <linux/ctype.h>
#include <linux/slab.h>
#include <linux/limits.h>
#include <linux/device.h>
#include <linux/pci.h>
#include <linux/blkdev.h>
#include <linux/device.h>

#include <linux/init.h>
#include <linux/mm.h>
#include <linux/fcntl.h>
#include <linux/slab.h>
#include <linux/kmod.h>
#include <linux/major.h>
#include <linux/smp_lock.h>
#include <linux/highmem.h>
#include <linux/blkdev.h>
#include <linux/module.h>
#include <linux/blkpg.h>
#include <linux/buffer_head.h>
#include <linux/mpage.h>
#include <linux/mount.h>
#include <linux/uio.h>
#include <linux/namei.h>
#include <asm/uaccess.h>

#include <linux/pagemap.h>
#include <linux/pagevec.h>

#include <linux/dma-debug.h>

#define DUMP_DMA_FUN  "0.1"

MODULE_AUTHOR("Konrad Rzeszutek Wilk <konrad@virtualiron>");
MODULE_DESCRIPTION("dump dma");
MODULE_LICENSE("GPL");
MODULE_VERSION(DUMP_DMA_FUN);

static int __init dump_dma_init(void)
{
	debug_dma_dump_mappings(NULL);
	return 0;
}

static void __exit dump_dma_exit(void)
{
}

module_init(dump_dma_init);
module_exit(dump_dma_exit);

[-- Attachment #3: Makefile --]
[-- Type: text/plain, Size: 752 bytes --]

# Comment/uncomment the following line to disable/enable debugging
#DEBUG = y

# Add your debugging flag (or not) to CFLAGS
ifeq ($(DEBUG),y)
  DEBFLAGS = -O -g # "-O" is needed to expand inlines
else
  DEBFLAGS = -O2
endif

EXTRA_CFLAGS += $(DEBFLAGS) -I$(LDDINCDIR)

ifneq ($(KERNELRELEASE),)
# call from kernel build system

obj-m	:= dump_dma.o

else

#KERNELDIR ?= /lib/modules/$(shell uname -r)/build
KERNELDIR ?= /home/konrad/git/neb.64/linux-build
PWD       := $(shell pwd)

default:
	$(MAKE) -C $(KERNELDIR) M=$(PWD) LDDINCDIR=$(PWD)/../include modules

endif

clean:
	rm -rf *.o *~ core .depend .*.cmd *.ko *.mod.c .tmp_versions

depend .depend dep:
	$(CC) $(CFLAGS) -M *.c > .depend


ifeq (.depend,$(wildcard .depend))
include .depend
endif

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel