From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Jan Beulich" Subject: Re: x86 swiotlb questions Date: Fri, 22 Dec 2006 14:49:53 +0000 Message-ID: <458BFEA1.76E4.0078.0@novell.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=__PartD7F31481.1__=" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: Muli Ben-Yehuda , xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org This is a MIME message. If you are reading this text, you may want to consider changing to a mail reader or gateway that understands how to properly handle MIME multipart messages. --=__PartD7F31481.1__= Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Content-Disposition: inline Patch update, fixing a bug on x86/PAE, and making include/xen/swiotlb.h look a lot nicer (but still not really nice). My plan is to submit the non-Xen ones to lkml right after New Year, unless I hear negative feedback. What are the plans on the Xen side - pull the non-Xen patches into patches/, or ignore everything until (hopefully) mainline has picked up some or all of the native ones? Jan >>Do we merge okay with lib/swiotlb.c then? One concern I had was with our >>preferred setup semantics -- we really want the user to be able to forcibly >>enable the swiotlb via a boot parameter *but* not have to suffer using it >>for every DMA operation. Last I looked the generic swiotlb didn't have that >>option. That and our very Xen-specific checks for whether to auto-enable the >>swiotlb led me to think that the very start-of-day setup of swiotlb would >>need to be overridable by architecture. > >I think I retained all of the semantics, attached the patches as I have them >by now. This is a submission for review only, as the first four patches will >need to go to kernel.org (and hopefully will get accepted). The Xen >customization is fairly ugly, but I didn't see anything nicer than that while >also keeping the amount of changes to lib/swiotlb.c reasonable. > >Patch order is >swiotlb-bugs.patch >swiotlb-bus.patch >swiotlb-cleanup.patch >swiotlb-split.patch >xen-swiotlb.patch --=__PartD7F31481.1__= Content-Type: text/plain; name="swiotlb-split.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="swiotlb-split.patch" This patch adds abstraction so that the file can be used by environments = other than IA64 and EM64T, namely for Xen. Signed-off-by: Jan Beulich Index: sle10-sp1-2006-12-21/include/asm-ia64/swiotlb.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ sle10-sp1-2006-12-21/include/asm-ia64/swiotlb.h 2006-12-21 = 16:13:18.000000000 +0100 @@ -0,0 +1,9 @@ +#ifndef _ASM_SWIOTLB_H +#define _ASM_SWIOTLB_H 1 + +#include + +#define SWIOTLB_ARCH_NEED_LATE_INIT +#define SWIOTLB_ARCH_NEED_ALLOC + +#endif /* _ASM_SWIOTLB_H */ Index: sle10-sp1-2006-12-21/include/asm-x86_64/swiotlb.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/include/asm-x86_64/swiotlb.h 2006-12-21 = 16:10:20.000000000 +0100 +++ sle10-sp1-2006-12-21/include/asm-x86_64/swiotlb.h 2006-12-21 = 16:13:03.000000000 +0100 @@ -52,6 +52,7 @@ extern void swiotlb_unmap_page(struct de #endif =20 #ifdef CONFIG_SWIOTLB +#define SWIOTLB_ARCH_NEED_ALLOC extern int swiotlb; #else #define swiotlb 0 Index: sle10-sp1-2006-12-21/lib/swiotlb.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/lib/swiotlb.c 2006-12-21 15:41:31.0000000= 00 +0100 +++ sle10-sp1-2006-12-21/lib/swiotlb.c 2006-12-21 16:09:44.000000000 = +0100 @@ -28,6 +28,7 @@ #include #include #include +#include =20 #include #include @@ -35,8 +36,10 @@ #define OFFSET(val,align) ((unsigned long) \ ( (val) & ( (align) - 1))) =20 +#ifndef SG_ENT_VIRT_ADDRESS #define SG_ENT_VIRT_ADDRESS(sg) (page_address((sg)->page) + = (sg)->offset) #define SG_ENT_PHYS_ADDRESS(sg) virt_to_bus(SG_ENT_VIRT_ADDRESS(sg)= ) +#endif =20 /* * Maximum allowable number of contiguous slabs to map, @@ -101,13 +104,25 @@ static unsigned int io_tlb_index; * We need to save away the original address corresponding to a mapped = entry * for the sync operations. */ -static unsigned char **io_tlb_orig_addr; +#ifndef SWIOTLB_ARCH_HAS_IO_TLB_ADDR_T +typedef char *io_tlb_addr_t; +#define swiotlb_orig_addr_null(buffer) (!(buffer)) +#define ptr_to_io_tlb_addr(ptr) (ptr) +#define page_to_io_tlb_addr(pg, off) (page_address(pg) + (off)) +#define sg_to_io_tlb_addr(sg) SG_ENT_VIRT_ADDRESS(sg) +#endif +static io_tlb_addr_t *io_tlb_orig_addr; =20 /* * Protect the above data structures in the map and unmap calls */ static DEFINE_SPINLOCK(io_tlb_lock); =20 +#ifdef SWIOTLB_EXTRA_VARIABLES +SWIOTLB_EXTRA_VARIABLES; +#endif + +#ifndef SWIOTLB_ARCH_HAS_SETUP_IO_TLB_NPAGES static int __init setup_io_tlb_npages(char *str) { @@ -122,9 +137,25 @@ setup_io_tlb_npages(char *str) swiotlb_force =3D 1; return 1; } +#endif __setup("swiotlb=3D", setup_io_tlb_npages); /* make io_tlb_overflow tunable too? */ =20 +#ifndef swiotlb_adjust_size +#define swiotlb_adjust_size(size) ((void)0) +#endif + +#ifndef swiotlb_adjust_seg +#define swiotlb_adjust_seg(start, size) ((void)0) +#endif + +#ifndef swiotlb_print_info +#define swiotlb_print_info(bytes) \ + printk(KERN_INFO "Placing %luMB software IO TLB between 0x%lx - " = \ + "0x%lx\n", bytes >> 20, \ + virt_to_bus(io_tlb_start), virt_to_bus(io_tlb_end)) +#endif + /* * Statically reserve bounce buffer space and initialize bounce buffer = data * structures for the software IO TLB used to implement the DMA API. @@ -138,6 +169,8 @@ swiotlb_init_with_default_size(size_t de io_tlb_nslabs =3D (default_size >> IO_TLB_SHIFT); io_tlb_nslabs =3D ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE); } + swiotlb_adjust_size(io_tlb_nslabs); + swiotlb_adjust_size(io_tlb_overflow); =20 bytes =3D io_tlb_nslabs << IO_TLB_SHIFT; =20 @@ -155,25 +188,33 @@ swiotlb_init_with_default_size(size_t de * between io_tlb_start and io_tlb_end. */ io_tlb_list =3D alloc_bootmem(io_tlb_nslabs * sizeof(int)); - for (i =3D 0; i < io_tlb_nslabs; i++) + for (i =3D 0; i < io_tlb_nslabs; i++) { + if ( !(i % IO_TLB_SEGSIZE) ) + swiotlb_adjust_seg(io_tlb_start + (i << IO_TLB_SHIF= T), + IO_TLB_SEGSIZE << IO_TLB_SHIFT); io_tlb_list[i] =3D IO_TLB_SEGSIZE - OFFSET(i, IO_TLB_SEGSIZ= E); + } io_tlb_index =3D 0; - io_tlb_orig_addr =3D alloc_bootmem(io_tlb_nslabs * sizeof(char = *)); + io_tlb_orig_addr =3D alloc_bootmem(io_tlb_nslabs * sizeof(io_tlb_ad= dr_t)); =20 /* * Get the overflow emergency buffer */ io_tlb_overflow_buffer =3D alloc_bootmem_low(io_tlb_overflow); - printk(KERN_INFO "Placing software IO TLB between 0x%lx - = 0x%lx\n", - virt_to_bus(io_tlb_start), virt_to_bus(io_tlb_end)); + swiotlb_adjust_seg(io_tlb_overflow_buffer, io_tlb_overflow); + swiotlb_print_info(bytes); } +#ifndef __swiotlb_init_with_default_size +#define __swiotlb_init_with_default_size swiotlb_init_with_default_size +#endif =20 void __init swiotlb_init(void) { - swiotlb_init_with_default_size(64 * (1<<20)); /* default to 64MB = */ + __swiotlb_init_with_default_size(64 * (1<<20)); /* default to 64MB = */ } =20 +#ifdef SWIOTLB_ARCH_NEED_LATE_INIT /* * Systems with larger DMA zones (those that don't support ISA) can * initialize the swiotlb later using the slab allocator if needed. @@ -231,12 +272,12 @@ swiotlb_late_init_with_default_size(size io_tlb_list[i] =3D IO_TLB_SEGSIZE - OFFSET(i, IO_TLB_SEGSIZ= E); io_tlb_index =3D 0; =20 - io_tlb_orig_addr =3D (unsigned char **)__get_free_pages(GFP_KERNEL,= - get_order(io_tlb_nslabs * sizeof(char = *))); + io_tlb_orig_addr =3D (io_tlb_addr_t *)__get_free_pages(GFP_KERNEL, + get_order(io_tlb_nslabs * sizeof(io_tlb_= addr_t))); if (!io_tlb_orig_addr) goto cleanup3; =20 - memset(io_tlb_orig_addr, 0, io_tlb_nslabs * sizeof(char *)); + memset(io_tlb_orig_addr, 0, io_tlb_nslabs * sizeof(io_tlb_addr_t));= =20 /* * Get the overflow emergency buffer @@ -246,19 +287,17 @@ swiotlb_late_init_with_default_size(size if (!io_tlb_overflow_buffer) goto cleanup4; =20 - printk(KERN_INFO "Placing %luMB software IO TLB between 0x%lx - " - "0x%lx\n", bytes >> 20, - virt_to_bus(io_tlb_start), virt_to_bus(io_tlb_end)); + swiotlb_print_info(bytes); =20 return 0; =20 cleanup4: - free_pages((unsigned long)io_tlb_orig_addr, get_order(io_tlb_nslabs= * - sizeof(char = *))); + free_pages((unsigned long)io_tlb_orig_addr, + get_order(io_tlb_nslabs * sizeof(io_tlb_addr_t))); io_tlb_orig_addr =3D NULL; cleanup3: - free_pages((unsigned long)io_tlb_list, get_order(io_tlb_nslabs * - sizeof(int))); + free_pages((unsigned long)io_tlb_list, + get_order(io_tlb_nslabs * sizeof(int))); io_tlb_list =3D NULL; cleanup2: io_tlb_end =3D NULL; @@ -268,7 +307,9 @@ cleanup1: io_tlb_nslabs =3D req_nslabs; return -ENOMEM; } +#endif =20 +#ifndef SWIOTLB_ARCH_HAS_NEEDS_MAPPING static inline int address_needs_mapping(struct device *hwdev, dma_addr_t addr) { @@ -279,11 +320,35 @@ address_needs_mapping(struct device *hwd return (addr & ~mask) !=3D 0; } =20 +static inline int range_needs_mapping(const void *ptr, size_t size) +{ + return swiotlb_force; +} + +static inline int order_needs_mapping(unsigned int order) +{ + return 0; +} +#endif + +static void +__sync_single(io_tlb_addr_t buffer, char *dma_addr, size_t size, int dir) +{ +#ifndef SWIOTLB_ARCH_HAS_SYNC_SINGLE + if (dir =3D=3D DMA_TO_DEVICE) + memcpy(dma_addr, buffer, size); + else + memcpy(buffer, dma_addr, size); +#else + __swiotlb_arch_sync_single(buffer, dma_addr, size, dir); +#endif +} + /* * Allocates bounce buffer and returns its kernel virtual address. */ static void * -map_single(struct device *hwdev, char *buffer, size_t size, int dir) +map_single(struct device *hwdev, io_tlb_addr_t buffer, size_t size, int = dir) { unsigned long flags; char *dma_addr; @@ -357,7 +422,7 @@ map_single(struct device *hwdev, char *b */ io_tlb_orig_addr[index] =3D buffer; if (dir =3D=3D DMA_TO_DEVICE || dir =3D=3D DMA_BIDIRECTIONAL) - memcpy(dma_addr, buffer, size); + __sync_single(buffer, dma_addr, size, DMA_TO_DEVICE); =20 return dma_addr; } @@ -371,17 +436,18 @@ unmap_single(struct device *hwdev, char=20 unsigned long flags; int i, count, nslots =3D ALIGN(size, 1 << IO_TLB_SHIFT) >> = IO_TLB_SHIFT; int index =3D (dma_addr - io_tlb_start) >> IO_TLB_SHIFT; - char *buffer =3D io_tlb_orig_addr[index]; + io_tlb_addr_t buffer =3D io_tlb_orig_addr[index]; =20 /* * First, sync the memory before unmapping the entry */ - if (buffer && ((dir =3D=3D DMA_FROM_DEVICE) || (dir =3D=3D = DMA_BIDIRECTIONAL))) + if (!swiotlb_orig_addr_null(buffer) + && ((dir =3D=3D DMA_FROM_DEVICE) || (dir =3D=3D DMA_BIDIRECTION= AL))) /* * bounce... copy the data back into the original buffer * = and * delete the bounce buffer. */ - memcpy(buffer, dma_addr, size); + __sync_single(buffer, dma_addr, size, DMA_FROM_DEVICE); =20 /* * Return the buffer to the free list by setting the corresponding @@ -414,18 +480,18 @@ sync_single(struct device *hwdev, char * int dir, int target) { int index =3D (dma_addr - io_tlb_start) >> IO_TLB_SHIFT; - char *buffer =3D io_tlb_orig_addr[index]; + io_tlb_addr_t buffer =3D io_tlb_orig_addr[index]; =20 switch (target) { case SYNC_FOR_CPU: if (likely(dir =3D=3D DMA_FROM_DEVICE || dir =3D=3D = DMA_BIDIRECTIONAL)) - memcpy(buffer, dma_addr, size); + __sync_single(buffer, dma_addr, size, DMA_FROM_DEVI= CE); else if (dir !=3D DMA_TO_DEVICE) BUG(); break; case SYNC_FOR_DEVICE: if (likely(dir =3D=3D DMA_TO_DEVICE || dir =3D=3D = DMA_BIDIRECTIONAL)) - memcpy(dma_addr, buffer, size); + __sync_single(buffer, dma_addr, size, DMA_TO_DEVICE= ); else if (dir !=3D DMA_FROM_DEVICE) BUG(); break; @@ -434,6 +500,8 @@ sync_single(struct device *hwdev, char * } } =20 +#ifdef SWIOTLB_ARCH_NEED_ALLOC + void * swiotlb_alloc_coherent(struct device *hwdev, size_t size, dma_addr_t *dma_handle, gfp_t flags) @@ -449,7 +517,10 @@ swiotlb_alloc_coherent(struct device *hw */ flags |=3D GFP_DMA; =20 - ret =3D (void *)__get_free_pages(flags, order); + if (!order_needs_mapping(order)) + ret =3D (void *)__get_free_pages(flags, order); + else + ret =3D NULL; if (ret && address_needs_mapping(hwdev, virt_to_bus(ret))) { /* * The allocated memory isn't reachable by the device. @@ -487,6 +558,7 @@ swiotlb_alloc_coherent(struct device *hw *dma_handle =3D dev_addr; return ret; } +EXPORT_SYMBOL(swiotlb_alloc_coherent); =20 void swiotlb_free_coherent(struct device *hwdev, size_t size, void *vaddr, @@ -499,6 +571,9 @@ swiotlb_free_coherent(struct device *hwd /* DMA_TO_DEVICE to avoid memcpy in unmap_single */ swiotlb_unmap_single (hwdev, dma_handle, size, DMA_TO_DEVIC= E); } +EXPORT_SYMBOL(swiotlb_free_coherent); + +#endif =20 static void swiotlb_full(struct device *dev, size_t size, int dir, int do_panic) @@ -536,18 +611,20 @@ swiotlb_map_single(struct device *hwdev, =20 if (dir =3D=3D DMA_NONE) BUG(); + /* * If the pointer passed in happens to be in the device's DMA = window, * we can safely return the device addr and not worry about bounce * buffering it. */ - if (!address_needs_mapping(hwdev, dev_addr) && !swiotlb_force) + if (!range_needs_mapping(ptr, size) + && !address_needs_mapping(hwdev, dev_addr)) return dev_addr; =20 /* * Oh well, have to allocate and map a bounce buffer. */ - map =3D map_single(hwdev, ptr, size, dir); + map =3D map_single(hwdev, ptr_to_io_tlb_addr(ptr), size, dir); if (!map) { swiotlb_full(hwdev, size, dir, 1); map =3D io_tlb_overflow_buffer; @@ -678,7 +755,6 @@ int swiotlb_map_sg(struct device *hwdev, struct scatterlist *sg, int nelems, int dir) { - void *addr; dma_addr_t dev_addr; int i; =20 @@ -686,10 +762,10 @@ swiotlb_map_sg(struct device *hwdev, str BUG(); =20 for (i =3D 0; i < nelems; i++, sg++) { - addr =3D SG_ENT_VIRT_ADDRESS(sg); - dev_addr =3D virt_to_bus(addr); - if (swiotlb_force || address_needs_mapping(hwdev, = dev_addr)) { - void *map =3D map_single(hwdev, addr, sg->length, = dir); + dev_addr =3D SG_ENT_PHYS_ADDRESS(sg); + if (range_needs_mapping(SG_ENT_VIRT_ADDRESS(sg), sg->length= ) + || address_needs_mapping(hwdev, dev_addr)) { + void *map =3D map_single(hwdev, sg_to_io_tlb_addr(s= g), sg->length, dir); if (!map) { /* Don't panic here, we expect map_sg = users to do proper error handling. */ @@ -765,6 +841,46 @@ swiotlb_sync_sg_for_device(struct device swiotlb_sync_sg(hwdev, sg, nelems, dir, SYNC_FOR_DEVICE); } =20 +#ifdef SWIOTLB_ARCH_NEED_MAP_PAGE + +dma_addr_t +swiotlb_map_page(struct device *hwdev, struct page *page, + unsigned long offset, size_t size, + enum dma_data_direction direction) +{ + dma_addr_t dev_addr; + char *map; + + dev_addr =3D page_to_bus(page) + offset; + if (address_needs_mapping(hwdev, dev_addr)) { + map =3D map_single(hwdev, page_to_io_tlb_addr(page, = offset), size, direction); + if (!map) { + swiotlb_full(hwdev, size, direction, 1); + map =3D io_tlb_overflow_buffer; + } + dev_addr =3D virt_to_bus(map); + } + + return dev_addr; +} +EXPORT_SYMBOL(swiotlb_map_page); + +void +swiotlb_unmap_page(struct device *hwdev, dma_addr_t dev_addr, + size_t size, enum dma_data_direction direction) +{ + char *dma_addr =3D bus_to_virt(dev_addr); + + BUG_ON(direction =3D=3D DMA_NONE); + if (dma_addr >=3D io_tlb_start && dma_addr < io_tlb_end) + unmap_single(hwdev, dma_addr, size, direction); + else if (direction =3D=3D DMA_FROM_DEVICE) + dma_mark_clean(dma_addr, size); +} +EXPORT_SYMBOL(swiotlb_unmap_page); + +#endif + int swiotlb_dma_mapping_error(dma_addr_t dma_addr) { @@ -780,7 +896,11 @@ swiotlb_dma_mapping_error(dma_addr_t dma int swiotlb_dma_supported(struct device *hwdev, u64 mask) { +#ifndef __swiotlb_dma_supported return (virt_to_bus(io_tlb_end) - 1) <=3D mask; +#else + return __swiotlb_dma_supported(hwdev, mask); +#endif } =20 EXPORT_SYMBOL(swiotlb_init); @@ -795,6 +915,4 @@ EXPORT_SYMBOL_GPL(swiotlb_sync_single_ra EXPORT_SYMBOL(swiotlb_sync_sg_for_cpu); EXPORT_SYMBOL(swiotlb_sync_sg_for_device); EXPORT_SYMBOL(swiotlb_dma_mapping_error); -EXPORT_SYMBOL(swiotlb_alloc_coherent); -EXPORT_SYMBOL(swiotlb_free_coherent); EXPORT_SYMBOL(swiotlb_dma_supported); --=__PartD7F31481.1__= Content-Type: text/plain; name="swiotlb-bus.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="swiotlb-bus.patch" Convert all phys_to_virt/virt_to_phys uses to bus_to_virt/virt_to_bus. Signed-off-by: Jan Beulich Index: sle10-sp1-2006-12-18/lib/swiotlb.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-18.orig/lib/swiotlb.c 2006-12-20 12:02:01.0000000= 00 +0100 +++ sle10-sp1-2006-12-18/lib/swiotlb.c 2006-12-20 12:09:03.000000000 = +0100 @@ -36,7 +36,7 @@ ( (val) & ( (align) - 1))) =20 #define SG_ENT_VIRT_ADDRESS(sg) (page_address((sg)->page) + = (sg)->offset) -#define SG_ENT_PHYS_ADDRESS(SG) virt_to_phys(SG_ENT_VIRT_ADDRESS(SG= )) +#define SG_ENT_PHYS_ADDRESS(sg) virt_to_bus(SG_ENT_VIRT_ADDRESS(sg)= ) =20 /* * Maximum allowable number of contiguous slabs to map, @@ -163,7 +163,7 @@ swiotlb_init_with_default_size (size_t d */ io_tlb_overflow_buffer =3D alloc_bootmem_low(io_tlb_overflow); printk(KERN_INFO "Placing software IO TLB between 0x%lx - = 0x%lx\n", - virt_to_phys(io_tlb_start), virt_to_phys(io_tlb_end)); + virt_to_bus(io_tlb_start), virt_to_bus(io_tlb_end)); } =20 void @@ -244,7 +244,7 @@ swiotlb_late_init_with_default_size (siz =20 printk(KERN_INFO "Placing %ldMB software IO TLB between 0x%lx - " "0x%lx\n", (io_tlb_nslabs * (1 << IO_TLB_SHIFT)) >> 20, - virt_to_phys(io_tlb_start), virt_to_phys(io_tlb_end)); + virt_to_bus(io_tlb_start), virt_to_bus(io_tlb_end)); =20 return 0; =20 @@ -446,7 +446,7 @@ swiotlb_alloc_coherent(struct device *hw flags |=3D GFP_DMA; =20 ret =3D (void *)__get_free_pages(flags, order); - if (ret && address_needs_mapping(hwdev, virt_to_phys(ret))) { + if (ret && address_needs_mapping(hwdev, virt_to_bus(ret))) { /* * The allocated memory isn't reachable by the device. * Fall back on swiotlb_map_single(). @@ -466,11 +466,11 @@ swiotlb_alloc_coherent(struct device *hw if (swiotlb_dma_mapping_error(handle)) return NULL; =20 - ret =3D phys_to_virt(handle); + ret =3D bus_to_virt(handle); } =20 memset(ret, 0, size); - dev_addr =3D virt_to_phys(ret); + dev_addr =3D virt_to_bus(ret); =20 /* Confirm address can be DMA'd by device */ if (address_needs_mapping(hwdev, dev_addr)) { @@ -526,7 +526,7 @@ swiotlb_full(struct device *dev, size_t=20 dma_addr_t swiotlb_map_single(struct device *hwdev, void *ptr, size_t size, int dir) { - unsigned long dev_addr =3D virt_to_phys(ptr); + unsigned long dev_addr =3D virt_to_bus(ptr); void *map; =20 if (dir =3D=3D DMA_NONE) @@ -548,7 +548,7 @@ swiotlb_map_single(struct device *hwdev, map =3D io_tlb_overflow_buffer; } =20 - dev_addr =3D virt_to_phys(map); + dev_addr =3D virt_to_bus(map); =20 /* * Ensure that the address returned is DMA'ble @@ -571,7 +571,7 @@ void swiotlb_unmap_single(struct device *hwdev, dma_addr_t dev_addr, size_t = size, int dir) { - char *dma_addr =3D phys_to_virt(dev_addr); + char *dma_addr =3D bus_to_virt(dev_addr); =20 if (dir =3D=3D DMA_NONE) BUG(); @@ -595,7 +595,7 @@ static inline void swiotlb_sync_single(struct device *hwdev, dma_addr_t dev_addr, size_t size, int dir, int target) { - char *dma_addr =3D phys_to_virt(dev_addr); + char *dma_addr =3D bus_to_virt(dev_addr); =20 if (dir =3D=3D DMA_NONE) BUG(); @@ -627,7 +627,7 @@ swiotlb_sync_single_range(struct device=20 unsigned long offset, size_t size, int dir, int target) { - char *dma_addr =3D phys_to_virt(dev_addr) + offset; + char *dma_addr =3D bus_to_virt(dev_addr) + offset; =20 if (dir =3D=3D DMA_NONE) BUG(); @@ -682,7 +682,7 @@ swiotlb_map_sg(struct device *hwdev, str =20 for (i =3D 0; i < nelems; i++, sg++) { addr =3D SG_ENT_VIRT_ADDRESS(sg); - dev_addr =3D virt_to_phys(addr); + dev_addr =3D virt_to_bus(addr); if (swiotlb_force || address_needs_mapping(hwdev, = dev_addr)) { void *map =3D map_single(hwdev, addr, sg->length, = dir); if (!map) { @@ -716,7 +716,8 @@ swiotlb_unmap_sg(struct device *hwdev, s =20 for (i =3D 0; i < nelems; i++, sg++) if (sg->dma_address !=3D SG_ENT_PHYS_ADDRESS(sg)) - unmap_single(hwdev, (void *) phys_to_virt(sg->dma_a= ddress), sg->dma_length, dir); + unmap_single(hwdev, bus_to_virt(sg->dma_address), + sg->dma_length, dir); else if (dir =3D=3D DMA_FROM_DEVICE) dma_mark_clean(SG_ENT_VIRT_ADDRESS(sg), sg->dma_len= gth); } @@ -739,7 +740,7 @@ swiotlb_sync_sg(struct device *hwdev, st =20 for (i =3D 0; i < nelems; i++, sg++) if (sg->dma_address !=3D SG_ENT_PHYS_ADDRESS(sg)) - sync_single(hwdev, phys_to_virt(sg->dma_address), + sync_single(hwdev, bus_to_virt(sg->dma_address), sg->dma_length, dir, target); else if (dir =3D=3D DMA_FROM_DEVICE) dma_mark_clean(SG_ENT_VIRT_ADDRESS(sg), sg->dma_len= gth); @@ -762,7 +763,7 @@ swiotlb_sync_sg_for_device(struct device int swiotlb_dma_mapping_error(dma_addr_t dma_addr) { - return (dma_addr =3D=3D virt_to_phys(io_tlb_overflow_buffer)); + return (dma_addr =3D=3D virt_to_bus(io_tlb_overflow_buffer)); } =20 /* @@ -774,7 +775,7 @@ swiotlb_dma_mapping_error(dma_addr_t dma int swiotlb_dma_supported (struct device *hwdev, u64 mask) { - return (virt_to_phys (io_tlb_end) - 1) <=3D mask; + return (virt_to_bus(io_tlb_end) - 1) <=3D mask; } =20 EXPORT_SYMBOL(swiotlb_init); --=__PartD7F31481.1__= Content-Type: text/plain; name="swiotlb-cleanup.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="swiotlb-cleanup.patch" This patch - adds proper __init decoration to swiotlb's init code (and the code = calling it, where not already the case) - replaces uses of 'unsigned long' with dma_addr_t where appropriate - does miscellaneous simplicfication and cleanup Signed-off-by: Jan Beulich Index: sle10-sp1-2006-12-21/arch/ia64/mm/init.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/arch/ia64/mm/init.c 2006-12-20 = 12:02:01.000000000 +0100 +++ sle10-sp1-2006-12-21/arch/ia64/mm/init.c 2006-12-20 12:09:22.0000000= 00 +0100 @@ -586,7 +586,7 @@ nolwsys_setup (char *s) =20 __setup("nolwsys", nolwsys_setup); =20 -void +void __init mem_init (void) { long reserved_pages, codesize, datasize, initsize; Index: sle10-sp1-2006-12-21/arch/x86_64/kernel/pci-swiotlb.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/arch/x86_64/kernel/pci-swiotlb.c 2006-12-21 = 11:10:55.000000000 +0100 +++ sle10-sp1-2006-12-21/arch/x86_64/kernel/pci-swiotlb.c 2006-12-20 = 12:09:22.000000000 +0100 @@ -28,7 +28,7 @@ struct dma_mapping_ops swiotlb_dma_ops =3D .dma_supported =3D NULL, }; =20 -void pci_swiotlb_init(void) +void __init pci_swiotlb_init(void) { /* don't initialize swiotlb if iommu=3Doff (no_iommu=3D1) */ if (!iommu_detected && !no_iommu && Index: sle10-sp1-2006-12-21/lib/swiotlb.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/lib/swiotlb.c 2006-12-20 12:09:03.0000000= 00 +0100 +++ sle10-sp1-2006-12-21/lib/swiotlb.c 2006-12-21 15:41:31.000000000 = +0100 @@ -1,7 +1,7 @@ /* * Dynamic DMA mapping support. * - * This implementation is for IA-64 and EM64T platforms that do not = support + * This implementation is a fallback for platforms that do not support * I/O TLBs (aka DMA address translation hardware). * Copyright (C) 2000 Asit Mallick * Copyright (C) 2000 Goutham Rao @@ -68,7 +68,7 @@ enum dma_sync_target { SYNC_FOR_DEVICE =3D 1, }; =20 -int swiotlb_force; +static int swiotlb_force; =20 /* * Used to do a quick range check in swiotlb_unmap_single and @@ -129,23 +129,25 @@ __setup("swiotlb=3D", setup_io_tlb_npages) * Statically reserve bounce buffer space and initialize bounce buffer = data * structures for the software IO TLB used to implement the DMA API. */ -void -swiotlb_init_with_default_size (size_t default_size) +void __init +swiotlb_init_with_default_size(size_t default_size) { - unsigned long i; + unsigned long i, bytes; =20 if (!io_tlb_nslabs) { io_tlb_nslabs =3D (default_size >> IO_TLB_SHIFT); io_tlb_nslabs =3D ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE); } =20 + bytes =3D io_tlb_nslabs << IO_TLB_SHIFT; + /* * Get IO TLB memory from the low pages */ - io_tlb_start =3D alloc_bootmem_low_pages(io_tlb_nslabs * (1 << = IO_TLB_SHIFT)); + io_tlb_start =3D alloc_bootmem_low_pages(bytes); if (!io_tlb_start) panic("Cannot allocate SWIOTLB buffer"); - io_tlb_end =3D io_tlb_start + io_tlb_nslabs * (1 << IO_TLB_SHIFT); + io_tlb_end =3D io_tlb_start + bytes; =20 /* * Allocate and initialize the free list array. This array is = used @@ -166,8 +168,8 @@ swiotlb_init_with_default_size (size_t d virt_to_bus(io_tlb_start), virt_to_bus(io_tlb_end)); } =20 -void -swiotlb_init (void) +void __init +swiotlb_init(void) { swiotlb_init_with_default_size(64 * (1<<20)); /* default to 64MB = */ } @@ -178,9 +180,9 @@ swiotlb_init (void) * This should be just like above, but with some error catching. */ int -swiotlb_late_init_with_default_size (size_t default_size) +swiotlb_late_init_with_default_size(size_t default_size) { - unsigned long i, req_nslabs =3D io_tlb_nslabs; + unsigned long i, bytes, req_nslabs =3D io_tlb_nslabs; unsigned int order; =20 if (!io_tlb_nslabs) { @@ -191,8 +193,9 @@ swiotlb_late_init_with_default_size (siz /* * Get IO TLB memory from the low pages */ - order =3D get_order(io_tlb_nslabs * (1 << IO_TLB_SHIFT)); + order =3D get_order(io_tlb_nslabs << IO_TLB_SHIFT); io_tlb_nslabs =3D SLABS_PER_PAGE << order; + bytes =3D io_tlb_nslabs << IO_TLB_SHIFT; =20 while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) { io_tlb_start =3D (char *)__get_free_pages(GFP_DMA | = __GFP_NOWARN, @@ -205,13 +208,14 @@ swiotlb_late_init_with_default_size (siz if (!io_tlb_start) goto cleanup1; =20 - if (order !=3D get_order(io_tlb_nslabs * (1 << IO_TLB_SHIFT))) { + if (order !=3D get_order(bytes)) { printk(KERN_WARNING "Warning: only able to allocate %ld MB = " "for software IO TLB\n", (PAGE_SIZE << order) >> = 20); io_tlb_nslabs =3D SLABS_PER_PAGE << order; + bytes =3D io_tlb_nslabs << IO_TLB_SHIFT; } - io_tlb_end =3D io_tlb_start + io_tlb_nslabs * (1 << IO_TLB_SHIFT); - memset(io_tlb_start, 0, io_tlb_nslabs * (1 << IO_TLB_SHIFT)); + io_tlb_end =3D io_tlb_start + bytes; + memset(io_tlb_start, 0, bytes); =20 /* * Allocate and initialize the free list array. This array is = used @@ -242,8 +246,8 @@ swiotlb_late_init_with_default_size (siz if (!io_tlb_overflow_buffer) goto cleanup4; =20 - printk(KERN_INFO "Placing %ldMB software IO TLB between 0x%lx - " - "0x%lx\n", (io_tlb_nslabs * (1 << IO_TLB_SHIFT)) >> 20, + printk(KERN_INFO "Placing %luMB software IO TLB between 0x%lx - " + "0x%lx\n", bytes >> 20, virt_to_bus(io_tlb_start), virt_to_bus(io_tlb_end)); =20 return 0; @@ -256,8 +260,8 @@ cleanup3: free_pages((unsigned long)io_tlb_list, get_order(io_tlb_nslabs * sizeof(int))); io_tlb_list =3D NULL; - io_tlb_end =3D NULL; cleanup2: + io_tlb_end =3D NULL; free_pages((unsigned long)io_tlb_start, order); io_tlb_start =3D NULL; cleanup1: @@ -434,7 +438,7 @@ void * swiotlb_alloc_coherent(struct device *hwdev, size_t size, dma_addr_t *dma_handle, gfp_t flags) { - unsigned long dev_addr; + dma_addr_t dev_addr; void *ret; int order =3D get_order(size); =20 @@ -474,8 +478,9 @@ swiotlb_alloc_coherent(struct device *hw =20 /* Confirm address can be DMA'd by device */ if (address_needs_mapping(hwdev, dev_addr)) { - printk("hwdev DMA mask =3D 0x%016Lx, dev_addr =3D = 0x%016lx\n", - (unsigned long long)*hwdev->dma_mask, dev_addr); + printk("hwdev DMA mask =3D 0x%016Lx, dev_addr =3D = 0x%016Lx\n", + (unsigned long long)*hwdev->dma_mask, + (unsigned long long)dev_addr); panic("swiotlb_alloc_coherent: allocated memory is out of = " "range for device"); } @@ -505,7 +510,7 @@ swiotlb_full(struct device *dev, size_t=20 * When the mapping is small enough return a static buffer to = limit * the damage, or panic when the transfer is too big. */ - printk(KERN_ERR "DMA: Out of SW-IOMMU space for %lu bytes at " + printk(KERN_ERR "DMA: Out of SW-IOMMU space for %zu bytes at " "device %s\n", size, dev ? dev->bus_id : "?"); =20 if (size > io_tlb_overflow && do_panic) { @@ -526,7 +531,7 @@ swiotlb_full(struct device *dev, size_t=20 dma_addr_t swiotlb_map_single(struct device *hwdev, void *ptr, size_t size, int dir) { - unsigned long dev_addr =3D virt_to_bus(ptr); + dma_addr_t dev_addr =3D virt_to_bus(ptr); void *map; =20 if (dir =3D=3D DMA_NONE) @@ -674,7 +679,7 @@ swiotlb_map_sg(struct device *hwdev, str int dir) { void *addr; - unsigned long dev_addr; + dma_addr_t dev_addr; int i; =20 if (dir =3D=3D DMA_NONE) @@ -773,7 +778,7 @@ swiotlb_dma_mapping_error(dma_addr_t dma * this function. */ int -swiotlb_dma_supported (struct device *hwdev, u64 mask) +swiotlb_dma_supported(struct device *hwdev, u64 mask) { return (virt_to_bus(io_tlb_end) - 1) <=3D mask; } --=__PartD7F31481.1__= Content-Type: text/plain; name="swiotlb-bugs.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="swiotlb-bugs.patch" This patch fixes - marking I-cache clean of pages DMAed to now only done for IA64 - broken multiple inclusion in include/asm-x86_64/swiotlb.h - missing phys-to-virt translation in swiotlb_sync_sg() - missing call to mark_clean in swiotlb_sync_sg() Signed-off-by: Jan Beulich Index: sle10-sp1-2006-12-18/arch/ia64/mm/init.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-18.orig/arch/ia64/mm/init.c 2006-03-20 = 06:53:29.000000000 +0100 +++ sle10-sp1-2006-12-18/arch/ia64/mm/init.c 2006-12-20 12:02:01.0000000= 00 +0100 @@ -123,6 +123,25 @@ lazy_mmu_prot_update (pte_t pte) set_bit(PG_arch_1, &page->flags); /* mark page as clean */ } =20 +/* + * Since DMA is i-cache coherent, any (complete) pages that were written = via + * DMA can be marked as "clean" so that lazy_mmu_prot_update() doesn't = have to + * flush them when they get mapped into an executable vm-area. + */ +void +dma_mark_clean(void *addr, size_t size) +{ + unsigned long pg_addr, end; + + pg_addr =3D PAGE_ALIGN((unsigned long) addr); + end =3D (unsigned long) addr + size; + while (pg_addr + PAGE_SIZE <=3D end) { + struct page *page =3D virt_to_page(pg_addr); + set_bit(PG_arch_1, &page->flags); + pg_addr +=3D PAGE_SIZE; + } +} + inline void ia64_set_rbs_bot (void) { Index: sle10-sp1-2006-12-18/include/asm-ia64/dma.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-18.orig/include/asm-ia64/dma.h 2006-03-20 = 06:53:29.000000000 +0100 +++ sle10-sp1-2006-12-18/include/asm-ia64/dma.h 2006-12-20 12:02:01.0000000= 00 +0100 @@ -20,4 +20,6 @@ extern unsigned long MAX_DMA_ADDRESS; =20 #define free_dma(x) =20 +void dma_mark_clean(void *addr, size_t size); + #endif /* _ASM_IA64_DMA_H */ Index: sle10-sp1-2006-12-18/include/asm-x86_64/mach-xen/asm/dma-mapping.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-18.orig/include/asm-x86_64/mach-xen/asm/dma-mapping.h= 2006-12-20 15:53:41.000000000 +0100 +++ sle10-sp1-2006-12-18/include/asm-x86_64/mach-xen/asm/dma-mapping.h = 2006-12-20 12:02:01.000000000 +0100 @@ -10,7 +10,6 @@ =20 #include #include -#include =20 struct dma_mapping_ops { int (*mapping_error)(dma_addr_t dma_addr); Index: sle10-sp1-2006-12-18/include/asm-x86_64/swiotlb.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-18.orig/include/asm-x86_64/swiotlb.h 2006-12-20 = 15:53:41.000000000 +0100 +++ sle10-sp1-2006-12-18/include/asm-x86_64/swiotlb.h 2006-12-20 = 15:53:57.000000000 +0100 @@ -1,5 +1,5 @@ #ifndef _ASM_SWIOTLB_H -#define _ASM_SWTIOLB_H 1 +#define _ASM_SWIOTLB_H 1 =20 #include =20 @@ -59,4 +59,6 @@ extern int swiotlb; =20 extern void pci_swiotlb_init(void); =20 -#endif /* _ASM_SWTIOLB_H */ +static inline void dma_mark_clean(void *addr, size_t size) {} + +#endif /* _ASM_SWIOTLB_H */ Index: sle10-sp1-2006-12-18/lib/swiotlb.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-18.orig/lib/swiotlb.c 2006-03-20 06:53:29.0000000= 00 +0100 +++ sle10-sp1-2006-12-18/lib/swiotlb.c 2006-12-20 12:02:01.000000000 = +0100 @@ -560,25 +560,6 @@ swiotlb_map_single(struct device *hwdev, } =20 /* - * Since DMA is i-cache coherent, any (complete) pages that were written = via - * DMA can be marked as "clean" so that lazy_mmu_prot_update() doesn't = have to - * flush them when they get mapped into an executable vm-area. - */ -static void -mark_clean(void *addr, size_t size) -{ - unsigned long pg_addr, end; - - pg_addr =3D PAGE_ALIGN((unsigned long) addr); - end =3D (unsigned long) addr + size; - while (pg_addr + PAGE_SIZE <=3D end) { - struct page *page =3D virt_to_page(pg_addr); - set_bit(PG_arch_1, &page->flags); - pg_addr +=3D PAGE_SIZE; - } -} - -/* * Unmap a single streaming mode DMA translation. The dma_addr and size = must * match what was provided for in a previous swiotlb_map_single call. = All * other usages are undefined. @@ -597,7 +578,7 @@ swiotlb_unmap_single(struct device *hwde if (dma_addr >=3D io_tlb_start && dma_addr < io_tlb_end) unmap_single(hwdev, dma_addr, size, dir); else if (dir =3D=3D DMA_FROM_DEVICE) - mark_clean(dma_addr, size); + dma_mark_clean(dma_addr, size); } =20 /* @@ -621,7 +602,7 @@ swiotlb_sync_single(struct device *hwdev if (dma_addr >=3D io_tlb_start && dma_addr < io_tlb_end) sync_single(hwdev, dma_addr, size, dir, target); else if (dir =3D=3D DMA_FROM_DEVICE) - mark_clean(dma_addr, size); + dma_mark_clean(dma_addr, size); } =20 void @@ -653,7 +634,7 @@ swiotlb_sync_single_range(struct device=20 if (dma_addr >=3D io_tlb_start && dma_addr < io_tlb_end) sync_single(hwdev, dma_addr, size, dir, target); else if (dir =3D=3D DMA_FROM_DEVICE) - mark_clean(dma_addr, size); + dma_mark_clean(dma_addr, size); } =20 void @@ -704,7 +685,6 @@ swiotlb_map_sg(struct device *hwdev, str dev_addr =3D virt_to_phys(addr); if (swiotlb_force || address_needs_mapping(hwdev, = dev_addr)) { void *map =3D map_single(hwdev, addr, sg->length, = dir); - sg->dma_address =3D virt_to_bus(map); if (!map) { /* Don't panic here, we expect map_sg = users to do proper error handling. */ @@ -713,6 +693,7 @@ swiotlb_map_sg(struct device *hwdev, str sg[0].dma_length =3D 0; return 0; } + sg->dma_address =3D virt_to_bus(map); } else sg->dma_address =3D dev_addr; sg->dma_length =3D sg->length; @@ -737,7 +718,7 @@ swiotlb_unmap_sg(struct device *hwdev, s if (sg->dma_address !=3D SG_ENT_PHYS_ADDRESS(sg)) unmap_single(hwdev, (void *) phys_to_virt(sg->dma_a= ddress), sg->dma_length, dir); else if (dir =3D=3D DMA_FROM_DEVICE) - mark_clean(SG_ENT_VIRT_ADDRESS(sg), sg->dma_length)= ; + dma_mark_clean(SG_ENT_VIRT_ADDRESS(sg), sg->dma_len= gth); } =20 /* @@ -758,8 +739,10 @@ swiotlb_sync_sg(struct device *hwdev, st =20 for (i =3D 0; i < nelems; i++, sg++) if (sg->dma_address !=3D SG_ENT_PHYS_ADDRESS(sg)) - sync_single(hwdev, (void *) sg->dma_address, + sync_single(hwdev, phys_to_virt(sg->dma_address), sg->dma_length, dir, target); + else if (dir =3D=3D DMA_FROM_DEVICE) + dma_mark_clean(SG_ENT_VIRT_ADDRESS(sg), sg->dma_len= gth); } =20 void --=__PartD7F31481.1__= Content-Type: text/plain; name="xen-swiotlb.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="xen-swiotlb.patch" This patch eliminates Xen's special version of the swiotlb code. Along = with that it adds trivial forwarding of dma_{,un}map_page when not using = highmem. Signed-off-by: Jan Beulich Index: sle10-sp1-2006-12-21/arch/i386/kernel/swiotlb.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/arch/i386/kernel/swiotlb.c 2006-12-21 = 16:13:03.000000000 +0100 +++ /dev/null 1970-01-01 00:00:00.000000000 +0000 @@ -1,683 +0,0 @@ -/* - * Dynamic DMA mapping support. - * - * This implementation is a fallback for platforms that do not support - * I/O TLBs (aka DMA address translation hardware). - * Copyright (C) 2000 Asit Mallick - * Copyright (C) 2000 Goutham Rao - * Copyright (C) 2000, 2003 Hewlett-Packard Co - * David Mosberger-Tang - * Copyright (C) 2005 Keir Fraser - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -int swiotlb; -EXPORT_SYMBOL(swiotlb); - -#define OFFSET(val,align) ((unsigned long)((val) & ( (align) - 1))) - -#define SG_ENT_PHYS_ADDRESS(sg) (page_to_bus((sg)->page) + = (sg)->offset) - -/* - * Maximum allowable number of contiguous slabs to map, - * must be a power of 2. What is the appropriate value ? - * The complexity of {map,unmap}_single is linearly dependent on this = value. - */ -#define IO_TLB_SEGSIZE 128 - -/* - * log of the size of each IO TLB slab. The number of slabs is command = line - * controllable. - */ -#define IO_TLB_SHIFT 11 - -/* Width of DMA addresses. 30 bits is a b44 limitation. */ -#define DEFAULT_DMA_BITS 30 - -static int swiotlb_force; -static char *iotlb_virt_start; -static unsigned long iotlb_nslabs; - -/* - * Used to do a quick range check in swiotlb_unmap_single and - * swiotlb_sync_single_*, to see if the memory was in fact allocated by = this - * API. - */ -static unsigned long iotlb_pfn_start, iotlb_pfn_end; - -/* Does the given dma address reside within the swiotlb aperture? */ -static inline int in_swiotlb_aperture(dma_addr_t dev_addr) -{ - unsigned long pfn =3D mfn_to_local_pfn(dev_addr >> PAGE_SHIFT); - return (pfn_valid(pfn) - && (pfn >=3D iotlb_pfn_start) - && (pfn < iotlb_pfn_end)); -} - -/* - * When the IOMMU overflows we return a fallback buffer. This sets the = size. - */ -static unsigned long io_tlb_overflow =3D 32*1024; - -void *io_tlb_overflow_buffer; - -/* - * This is a free list describing the number of free entries available = from - * each index - */ -static unsigned int *io_tlb_list; -static unsigned int io_tlb_index; - -/* - * We need to save away the original address corresponding to a mapped = entry - * for the sync operations. - */ -static struct phys_addr { - struct page *page; - unsigned int offset; -} *io_tlb_orig_addr; - -/* - * Protect the above data structures in the map and unmap calls - */ -static DEFINE_SPINLOCK(io_tlb_lock); - -unsigned int dma_bits =3D DEFAULT_DMA_BITS; -static int __init -setup_dma_bits(char *str) -{ - dma_bits =3D simple_strtoul(str, NULL, 0); - return 0; -} -__setup("dma_bits=3D", setup_dma_bits); - -static int __init -setup_io_tlb_npages(char *str) -{ - /* Unlike ia64, the size is aperture in megabytes, not 'slabs'! */ - if (isdigit(*str)) { - iotlb_nslabs =3D simple_strtoul(str, &str, 0) << - (20 - IO_TLB_SHIFT); - iotlb_nslabs =3D ALIGN(iotlb_nslabs, IO_TLB_SEGSIZE); - /* Round up to power of two (xen_create_contiguous_region).= */ - while (iotlb_nslabs & (iotlb_nslabs-1)) - iotlb_nslabs +=3D iotlb_nslabs & ~(iotlb_nslabs-1);= - } - if (*str =3D=3D ',') - ++str; - /* - * NB. 'force' enables the swiotlb, but doesn't force its use for - * every DMA like it does on native Linux. 'off' forcibly = disables - * use of the swiotlb. - */ - if (!strcmp(str, "force")) - swiotlb_force =3D 1; - else if (!strcmp(str, "off")) - swiotlb_force =3D -1; - return 1; -} -__setup("swiotlb=3D", setup_io_tlb_npages); -/* make io_tlb_overflow tunable too? */ - -/* - * Statically reserve bounce buffer space and initialize bounce buffer = data - * structures for the software IO TLB used to implement the PCI DMA API. - */ -void -swiotlb_init_with_default_size (size_t default_size) -{ - unsigned long i, bytes; - - if (!iotlb_nslabs) { - iotlb_nslabs =3D (default_size >> IO_TLB_SHIFT); - iotlb_nslabs =3D ALIGN(iotlb_nslabs, IO_TLB_SEGSIZE); - /* Round up to power of two (xen_create_contiguous_region).= */ - while (iotlb_nslabs & (iotlb_nslabs-1)) - iotlb_nslabs +=3D iotlb_nslabs & ~(iotlb_nslabs-1);= - } - - bytes =3D iotlb_nslabs * (1UL << IO_TLB_SHIFT); - - /* - * Get IO TLB memory from the low pages - */ - iotlb_virt_start =3D alloc_bootmem_low_pages(bytes); - if (!iotlb_virt_start) - panic("Cannot allocate SWIOTLB buffer!\n" - "Use dom0_mem Xen boot parameter to reserve\n" - "some DMA memory (e.g., dom0_mem=3D-128M).\n"); - - for (i =3D 0; i < iotlb_nslabs; i +=3D IO_TLB_SEGSIZE) { - int rc =3D xen_create_contiguous_region( - (unsigned long)iotlb_virt_start + (i << IO_TLB_SHIF= T), - get_order(IO_TLB_SEGSIZE << IO_TLB_SHIFT), - dma_bits); - BUG_ON(rc); - } - - /* - * Allocate and initialize the free list array. This array is = used - * to find contiguous free memory regions of size up to IO_TLB_SEGS= IZE. - */ - io_tlb_list =3D alloc_bootmem(iotlb_nslabs * sizeof(int)); - for (i =3D 0; i < iotlb_nslabs; i++) - io_tlb_list[i] =3D IO_TLB_SEGSIZE - OFFSET(i, IO_TLB_SEGSIZ= E); - io_tlb_index =3D 0; - io_tlb_orig_addr =3D alloc_bootmem( - iotlb_nslabs * sizeof(*io_tlb_orig_addr)); - - /* - * Get the overflow emergency buffer - */ - io_tlb_overflow_buffer =3D alloc_bootmem_low(io_tlb_overflow); - - iotlb_pfn_start =3D __pa(iotlb_virt_start) >> PAGE_SHIFT; - iotlb_pfn_end =3D iotlb_pfn_start + (bytes >> PAGE_SHIFT); - - printk(KERN_INFO "Software IO TLB enabled: \n" - " Aperture: %lu megabytes\n" - " Kernel range: 0x%016lx - 0x%016lx\n" - " Address size: %u bits\n", - bytes >> 20, - (unsigned long)iotlb_virt_start, - (unsigned long)iotlb_virt_start + bytes, - dma_bits); -} - -void -swiotlb_init(void) -{ - long ram_end; - size_t defsz =3D 64 * (1 << 20); /* 64MB default size */ - - if (swiotlb_force =3D=3D 1) { - swiotlb =3D 1; - } else if ((swiotlb_force !=3D -1) && - is_running_on_xen() && - is_initial_xendomain()) { - /* Domain 0 always has a swiotlb. */ - ram_end =3D HYPERVISOR_memory_op(XENMEM_maximum_ram_page, = NULL); - if (ram_end <=3D 0x7ffff) - defsz =3D 2 * (1 << 20); /* 2MB on <2GB on = systems. */ - swiotlb =3D 1; - } - - if (swiotlb) - swiotlb_init_with_default_size(defsz); - else - printk(KERN_INFO "Software IO TLB disabled\n"); -} - -/* - * We use __copy_to_user_inatomic to transfer to the host buffer because = the - * buffer may be mapped read-only (e.g, in blkback driver) but lower-level= - * drivers map the buffer for DMA_BIDIRECTIONAL access. This causes an - * unnecessary copy from the aperture to the host buffer, and a page = fault. - */ -static void -__sync_single(struct phys_addr buffer, char *dma_addr, size_t size, int = dir) -{ - if (PageHighMem(buffer.page)) { - size_t len, bytes; - char *dev, *host, *kmp; - len =3D size; - while (len !=3D 0) { - if (((bytes =3D len) + buffer.offset) > PAGE_SIZE) - bytes =3D PAGE_SIZE - buffer.offset; - kmp =3D kmap_atomic(buffer.page, KM_SWIOTLB); - dev =3D dma_addr + size - len; - host =3D kmp + buffer.offset; - if (dir =3D=3D DMA_FROM_DEVICE) { - if (__copy_to_user_inatomic(host, dev, = bytes)) - /* inaccessible */; - } else - memcpy(dev, host, bytes); - kunmap_atomic(kmp, KM_SWIOTLB); - len -=3D bytes; - buffer.page++; - buffer.offset =3D 0; - } - } else { - char *host =3D (char *)phys_to_virt( - page_to_pseudophys(buffer.page)) + buffer.offset; - if (dir =3D=3D DMA_FROM_DEVICE) { - if (__copy_to_user_inatomic(host, dma_addr, size)) - /* inaccessible */; - } else if (dir =3D=3D DMA_TO_DEVICE) - memcpy(dma_addr, host, size); - } -} - -/* - * Allocates bounce buffer and returns its kernel virtual address. - */ -static void * -map_single(struct device *hwdev, struct phys_addr buffer, size_t size, = int dir) -{ - unsigned long flags; - char *dma_addr; - unsigned int nslots, stride, index, wrap; - int i; - - /* - * For mappings greater than a page, we limit the stride (and - * hence alignment) to a page size. - */ - nslots =3D ALIGN(size, 1 << IO_TLB_SHIFT) >> IO_TLB_SHIFT; - if (size > PAGE_SIZE) - stride =3D (1 << (PAGE_SHIFT - IO_TLB_SHIFT)); - else - stride =3D 1; - - BUG_ON(!nslots); - - /* - * Find suitable number of IO TLB entries size that will fit this - * request and allocate a buffer from that IO TLB pool. - */ - spin_lock_irqsave(&io_tlb_lock, flags); - { - wrap =3D index =3D ALIGN(io_tlb_index, stride); - - if (index >=3D iotlb_nslabs) - wrap =3D index =3D 0; - - do { - /* - * If we find a slot that indicates we have = 'nslots' - * number of contiguous buffers, we allocate the - * buffers from that slot and mark the entries as = '0' - * indicating unavailable. - */ - if (io_tlb_list[index] >=3D nslots) { - int count =3D 0; - - for (i =3D index; i < (int)(index + = nslots); i++) - io_tlb_list[i] =3D 0; - for (i =3D index - 1; - (OFFSET(i, IO_TLB_SEGSIZE) !=3D - IO_TLB_SEGSIZE -1) && io_tlb_list[i];= - i--) - io_tlb_list[i] =3D ++count; - dma_addr =3D iotlb_virt_start + - (index << IO_TLB_SHIFT); - - /* - * Update the indices to avoid searching = in - * the next round. - */ - io_tlb_index =3D=20 - ((index + nslots) < iotlb_nslabs - ? (index + nslots) : 0); - - goto found; - } - index +=3D stride; - if (index >=3D iotlb_nslabs) - index =3D 0; - } while (index !=3D wrap); - - spin_unlock_irqrestore(&io_tlb_lock, flags); - return NULL; - } - found: - spin_unlock_irqrestore(&io_tlb_lock, flags); - - /* - * Save away the mapping from the original address to the DMA = address. - * This is needed when we sync the memory. Then we sync the = buffer if - * needed. - */ - io_tlb_orig_addr[index] =3D buffer; - if ((dir =3D=3D DMA_TO_DEVICE) || (dir =3D=3D DMA_BIDIRECTIONAL)) - __sync_single(buffer, dma_addr, size, DMA_TO_DEVICE); - - return dma_addr; -} - -/* - * dma_addr is the kernel virtual address of the bounce buffer to unmap. - */ -static void -unmap_single(struct device *hwdev, char *dma_addr, size_t size, int dir) -{ - unsigned long flags; - int i, count, nslots =3D ALIGN(size, 1 << IO_TLB_SHIFT) >> = IO_TLB_SHIFT; - int index =3D (dma_addr - iotlb_virt_start) >> IO_TLB_SHIFT; - struct phys_addr buffer =3D io_tlb_orig_addr[index]; - - /* - * First, sync the memory before unmapping the entry - */ - if ((dir =3D=3D DMA_FROM_DEVICE) || (dir =3D=3D DMA_BIDIRECTIONAL))= - __sync_single(buffer, dma_addr, size, DMA_FROM_DEVICE); - - /* - * Return the buffer to the free list by setting the corresponding - * entries to indicate the number of contigous entries available. - * While returning the entries to the free list, we merge the = entries - * with slots below and above the pool being returned. - */ - spin_lock_irqsave(&io_tlb_lock, flags); - { - count =3D ((index + nslots) < ALIGN(index + 1, IO_TLB_SEGSI= ZE) ? - io_tlb_list[index + nslots] : 0); - /* - * Step 1: return the slots to the free list, merging the - * slots with superceeding slots - */ - for (i =3D index + nslots - 1; i >=3D index; i--) - io_tlb_list[i] =3D ++count; - /* - * Step 2: merge the returned slots with the preceding = slots, - * if available (non zero) - */ - for (i =3D index - 1; - (OFFSET(i, IO_TLB_SEGSIZE) !=3D - IO_TLB_SEGSIZE -1) && io_tlb_list[i]; - i--) - io_tlb_list[i] =3D ++count; - } - spin_unlock_irqrestore(&io_tlb_lock, flags); -} - -static void -sync_single(struct device *hwdev, char *dma_addr, size_t size, int dir) -{ - int index =3D (dma_addr - iotlb_virt_start) >> IO_TLB_SHIFT; - struct phys_addr buffer =3D io_tlb_orig_addr[index]; - BUG_ON((dir !=3D DMA_FROM_DEVICE) && (dir !=3D DMA_TO_DEVICE)); - __sync_single(buffer, dma_addr, size, dir); -} - -static void -swiotlb_full(struct device *dev, size_t size, int dir, int do_panic) -{ - /* - * Ran out of IOMMU space for this operation. This is very bad. - * Unfortunately the drivers cannot handle this operation = properly. - * unless they check for pci_dma_mapping_error (most don't) - * When the mapping is small enough return a static buffer to = limit - * the damage, or panic when the transfer is too big. - */ - printk(KERN_ERR "PCI-DMA: Out of SW-IOMMU space for %lu bytes at " - "device %s\n", (unsigned long)size, dev ? dev->bus_id : = "?"); - - if (size > io_tlb_overflow && do_panic) { - if (dir =3D=3D PCI_DMA_FROMDEVICE || dir =3D=3D PCI_DMA_BID= IRECTIONAL) - panic("PCI-DMA: Memory would be corrupted\n"); - if (dir =3D=3D PCI_DMA_TODEVICE || dir =3D=3D PCI_DMA_BIDIR= ECTIONAL) - panic("PCI-DMA: Random memory would be DMAed\n"); - } -} - -/* - * Map a single buffer of the indicated size for DMA in streaming mode. = The - * PCI address to use is returned. - * - * Once the device is given the dma address, the device owns this memory = until - * either swiotlb_unmap_single or swiotlb_dma_sync_single is performed. - */ -dma_addr_t -swiotlb_map_single(struct device *hwdev, void *ptr, size_t size, int dir) -{ - dma_addr_t dev_addr =3D virt_to_bus(ptr); - void *map; - struct phys_addr buffer; - - BUG_ON(dir =3D=3D DMA_NONE); - - /* - * If the pointer passed in happens to be in the device's DMA = window, - * we can safely return the device addr and not worry about bounce - * buffering it. - */ - if (!range_straddles_page_boundary(ptr, size) && - !address_needs_mapping(hwdev, dev_addr)) - return dev_addr; - - /* - * Oh well, have to allocate and map a bounce buffer. - */ - buffer.page =3D virt_to_page(ptr); - buffer.offset =3D (unsigned long)ptr & ~PAGE_MASK; - map =3D map_single(hwdev, buffer, size, dir); - if (!map) { - swiotlb_full(hwdev, size, dir, 1); - map =3D io_tlb_overflow_buffer; - } - - dev_addr =3D virt_to_bus(map); - return dev_addr; -} - -/* - * Unmap a single streaming mode DMA translation. The dma_addr and size = must - * match what was provided for in a previous swiotlb_map_single call. = All - * other usages are undefined. - * - * After this call, reads by the cpu to the buffer are guaranteed to see - * whatever the device wrote there. - */ -void -swiotlb_unmap_single(struct device *hwdev, dma_addr_t dev_addr, size_t = size, - int dir) -{ - BUG_ON(dir =3D=3D DMA_NONE); - if (in_swiotlb_aperture(dev_addr)) - unmap_single(hwdev, bus_to_virt(dev_addr), size, dir); -} - -/* - * Make physical memory consistent for a single streaming mode DMA = translation - * after a transfer. - * - * If you perform a swiotlb_map_single() but wish to interrogate the = buffer - * using the cpu, yet do not wish to teardown the PCI dma mapping, you = must - * call this function before doing so. At the next point you give the = PCI dma - * address back to the card, you must first perform a - * swiotlb_dma_sync_for_device, and then the device again owns the buffer - */ -void -swiotlb_sync_single_for_cpu(struct device *hwdev, dma_addr_t dev_addr, - size_t size, int dir) -{ - BUG_ON(dir =3D=3D DMA_NONE); - if (in_swiotlb_aperture(dev_addr)) - sync_single(hwdev, bus_to_virt(dev_addr), size, dir); -} - -void -swiotlb_sync_single_for_device(struct device *hwdev, dma_addr_t dev_addr, - size_t size, int dir) -{ - BUG_ON(dir =3D=3D DMA_NONE); - if (in_swiotlb_aperture(dev_addr)) - sync_single(hwdev, bus_to_virt(dev_addr), size, dir); -} - -/* - * Map a set of buffers described by scatterlist in streaming mode for = DMA. - * This is the scatter-gather version of the above swiotlb_map_single - * interface. Here the scatter gather list elements are each tagged with = the - * appropriate dma address and length. They are obtained via - * sg_dma_{address,length}(SG). - * - * NOTE: An implementation may be able to use a smaller number of - * DMA address/length pairs than there are SG table elements. - * (for example via virtual mapping capabilities) - * The routine returns the number of addr/length pairs actually - * used, at most nents. - * - * Device ownership issues as mentioned above for swiotlb_map_single are = the - * same here. - */ -int -swiotlb_map_sg(struct device *hwdev, struct scatterlist *sg, int nelems, - int dir) -{ - struct phys_addr buffer; - dma_addr_t dev_addr; - char *map; - int i; - - BUG_ON(dir =3D=3D DMA_NONE); - - for (i =3D 0; i < nelems; i++, sg++) { - dev_addr =3D SG_ENT_PHYS_ADDRESS(sg); - if (address_needs_mapping(hwdev, dev_addr)) { - buffer.page =3D sg->page; - buffer.offset =3D sg->offset; - map =3D map_single(hwdev, buffer, sg->length, = dir); - if (!map) { - /* Don't panic here, we expect map_sg = users - to do proper error handling. */ - swiotlb_full(hwdev, sg->length, dir, 0); - swiotlb_unmap_sg(hwdev, sg - i, i, dir); - sg[0].dma_length =3D 0; - return 0; - } - sg->dma_address =3D (dma_addr_t)virt_to_bus(map); - } else - sg->dma_address =3D dev_addr; - sg->dma_length =3D sg->length; - } - return nelems; -} - -/* - * Unmap a set of streaming mode DMA translations. Again, cpu read rules - * concerning calls here are the same as for swiotlb_unmap_single() = above. - */ -void -swiotlb_unmap_sg(struct device *hwdev, struct scatterlist *sg, int = nelems, - int dir) -{ - int i; - - BUG_ON(dir =3D=3D DMA_NONE); - - for (i =3D 0; i < nelems; i++, sg++) - if (sg->dma_address !=3D SG_ENT_PHYS_ADDRESS(sg)) - unmap_single(hwdev,=20 - (void *)bus_to_virt(sg->dma_address), - sg->dma_length, dir); -} - -/* - * Make physical memory consistent for a set of streaming mode DMA = translations - * after a transfer. - * - * The same as swiotlb_sync_single_* but for a scatter-gather list, same = rules - * and usage. - */ -void -swiotlb_sync_sg_for_cpu(struct device *hwdev, struct scatterlist *sg, - int nelems, int dir) -{ - int i; - - BUG_ON(dir =3D=3D DMA_NONE); - - for (i =3D 0; i < nelems; i++, sg++) - if (sg->dma_address !=3D SG_ENT_PHYS_ADDRESS(sg)) - sync_single(hwdev, - (void *)bus_to_virt(sg->dma_address), - sg->dma_length, dir); -} - -void -swiotlb_sync_sg_for_device(struct device *hwdev, struct scatterlist *sg, - int nelems, int dir) -{ - int i; - - BUG_ON(dir =3D=3D DMA_NONE); - - for (i =3D 0; i < nelems; i++, sg++) - if (sg->dma_address !=3D SG_ENT_PHYS_ADDRESS(sg)) - sync_single(hwdev, - (void *)bus_to_virt(sg->dma_address), - sg->dma_length, dir); -} - -dma_addr_t -swiotlb_map_page(struct device *hwdev, struct page *page, - unsigned long offset, size_t size, - enum dma_data_direction direction) -{ - struct phys_addr buffer; - dma_addr_t dev_addr; - char *map; - - dev_addr =3D page_to_bus(page) + offset; - if (address_needs_mapping(hwdev, dev_addr)) { - buffer.page =3D page; - buffer.offset =3D offset; - map =3D map_single(hwdev, buffer, size, direction); - if (!map) { - swiotlb_full(hwdev, size, direction, 1); - map =3D io_tlb_overflow_buffer; - } - dev_addr =3D (dma_addr_t)virt_to_bus(map); - } - - return dev_addr; -} - -void -swiotlb_unmap_page(struct device *hwdev, dma_addr_t dma_address, - size_t size, enum dma_data_direction direction) -{ - BUG_ON(direction =3D=3D DMA_NONE); - if (in_swiotlb_aperture(dma_address)) - unmap_single(hwdev, bus_to_virt(dma_address), size, = direction); -} - -int -swiotlb_dma_mapping_error(dma_addr_t dma_addr) -{ - return (dma_addr =3D=3D virt_to_bus(io_tlb_overflow_buffer)); -} - -/* - * Return whether the given PCI device DMA address mask can be supported - * properly. For example, if your device can only drive the low 24-bits - * during PCI bus mastering, then you would pass 0x00ffffff as the mask = to - * this function. - */ -int -swiotlb_dma_supported (struct device *hwdev, u64 mask) -{ - return (mask >=3D ((1UL << dma_bits) - 1)); -} - -EXPORT_SYMBOL(swiotlb_init); -EXPORT_SYMBOL(swiotlb_map_single); -EXPORT_SYMBOL(swiotlb_unmap_single); -EXPORT_SYMBOL(swiotlb_map_sg); -EXPORT_SYMBOL(swiotlb_unmap_sg); -EXPORT_SYMBOL(swiotlb_sync_single_for_cpu); -EXPORT_SYMBOL(swiotlb_sync_single_for_device); -EXPORT_SYMBOL(swiotlb_sync_sg_for_cpu); -EXPORT_SYMBOL(swiotlb_sync_sg_for_device); -EXPORT_SYMBOL(swiotlb_map_page); -EXPORT_SYMBOL(swiotlb_unmap_page); -EXPORT_SYMBOL(swiotlb_dma_mapping_error); -EXPORT_SYMBOL(swiotlb_dma_supported); Index: sle10-sp1-2006-12-21/arch/i386/kernel/pci-dma-xen.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/arch/i386/kernel/pci-dma-xen.c 2006-12-21 = 16:13:03.000000000 +0100 +++ sle10-sp1-2006-12-21/arch/i386/kernel/pci-dma-xen.c 2006-12-21 = 15:51:29.000000000 +0100 @@ -17,7 +17,7 @@ #include #include #include -#include +#include #include =20 #ifdef __x86_64__ @@ -38,6 +38,15 @@ __init int iommu_setup(char *p) } #endif =20 +unsigned int dma_bits =3D DEFAULT_DMA_BITS; +static int __init +setup_dma_bits(char *str) +{ + dma_bits =3D simple_strtoul(str, NULL, 0); + return 0; +} +__setup("dma_bits=3D", setup_dma_bits); + struct dma_coherent_mem { void *virt_base; u32 device_base; @@ -94,13 +103,7 @@ dma_unmap_sg(struct device *hwdev, struc } EXPORT_SYMBOL(dma_unmap_sg); =20 -/* - * XXX This file is also used by xenLinux/ia64.=20 - * "defined(__i386__) || defined (__x86_64__)" means "!defined(__ia64__)".= - * This #if work around should be removed once this file is merbed back = into - * i386' pci-dma or is moved to drivers/xen/core. - */ -#if defined(__i386__) || defined(__x86_64__) +#ifdef CONFIG_HIGHMEM dma_addr_t dma_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction direction) @@ -130,7 +133,7 @@ dma_unmap_page(struct device *dev, dma_a swiotlb_unmap_page(dev, dma_address, size, direction); } EXPORT_SYMBOL(dma_unmap_page); -#endif /* defined(__i386__) || defined(__x86_64__) */ +#endif /* CONFIG_HIGHMEM */ =20 int dma_mapping_error(dma_addr_t dma_addr) Index: sle10-sp1-2006-12-21/include/asm-i386/mach-xen/asm/dma-mapping.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/include/asm-i386/mach-xen/asm/dma-mapping.h = 2006-12-21 16:13:03.000000000 +0100 +++ sle10-sp1-2006-12-21/include/asm-i386/mach-xen/asm/dma-mapping.h = 2006-12-21 11:01:11.000000000 +0100 @@ -24,7 +24,7 @@ address_needs_mapping(struct device *hwd } =20 static inline int -range_straddles_page_boundary(void *p, size_t size) +range_straddles_page_boundary(const void *p, size_t size) { extern unsigned long *contiguous_bitmap; return (((((unsigned long)p & ~PAGE_MASK) + size) > PAGE_SIZE) && @@ -53,6 +53,7 @@ extern int dma_map_sg(struct device *hwd extern void dma_unmap_sg(struct device *hwdev, struct scatterlist *sg, int nents, enum dma_data_direction direction); =20 +#ifdef CONFIG_HIGHMEM extern dma_addr_t dma_map_page(struct device *dev, struct page *page, unsigned long offset, size_t size, enum dma_data_direction direction); @@ -60,6 +61,11 @@ dma_map_page(struct device *dev, struct=20 extern void dma_unmap_page(struct device *dev, dma_addr_t dma_address, size_t size, enum dma_data_direction direction); +#else +#define dma_map_page(dev, page, offset, size, dir) \ + dma_map_single(dev, page_address(page) + (offset), (size), (dir)) +#define dma_unmap_page dma_unmap_single +#endif =20 extern void dma_sync_single_for_cpu(struct device *dev, dma_addr_t dma_handle, size_t = size, Index: sle10-sp1-2006-12-21/include/asm-i386/mach-xen/asm/swiotlb.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/include/asm-i386/mach-xen/asm/swiotlb.h = 2006-12-21 16:13:03.000000000 +0100 +++ sle10-sp1-2006-12-21/include/asm-i386/mach-xen/asm/swiotlb.h = 2006-12-21 15:56:11.000000000 +0100 @@ -1,45 +1,26 @@ -#ifndef _ASM_SWIOTLB_H -#define _ASM_SWIOTLB_H 1 +#ifndef _ASM_XEN_SWIOTLB_H +#define _ASM_XEN_SWIOTLB_H 1 =20 #include +#include =20 -/* SWIOTLB interface */ +#ifdef CONFIG_HIGHMEM + +/* + * Intentionally leaving off the base address of the page here - it may = not + * be valid at all (for highmem pages), and the macro is needed only for + * passing as argument to range_needs_mapping(), which doesn't care about + * the base address, and dma_mark_clean(), which is a no-op. + */ +#define SG_ENT_VIRT_ADDRESS(sg) ((void *)(sg)->offset) +#define SG_ENT_PHYS_ADDRESS(sg) (page_to_bus((sg)->page) + = (sg)->offset) =20 -extern dma_addr_t swiotlb_map_single(struct device *hwdev, void *ptr, = size_t size, - int dir); -extern void swiotlb_unmap_single(struct device *hwdev, dma_addr_t = dev_addr, - size_t size, int dir); -extern void swiotlb_sync_single_for_cpu(struct device *hwdev, - dma_addr_t dev_addr, - size_t size, int dir); -extern void swiotlb_sync_single_for_device(struct device *hwdev, - dma_addr_t dev_addr, - size_t size, int dir); -extern void swiotlb_sync_sg_for_cpu(struct device *hwdev, - struct scatterlist *sg, int nelems, - int dir); -extern void swiotlb_sync_sg_for_device(struct device *hwdev, - struct scatterlist *sg, int = nelems, - int dir); -extern int swiotlb_map_sg(struct device *hwdev, struct scatterlist *sg, - int nents, int direction); -extern void swiotlb_unmap_sg(struct device *hwdev, struct scatterlist = *sg, - int nents, int direction); -extern int swiotlb_dma_mapping_error(dma_addr_t dma_addr); extern dma_addr_t swiotlb_map_page(struct device *hwdev, struct page = *page, unsigned long offset, size_t size, enum dma_data_direction direction); extern void swiotlb_unmap_page(struct device *hwdev, dma_addr_t dma_addres= s, size_t size, enum dma_data_direction = direction); -extern int swiotlb_dma_supported(struct device *hwdev, u64 mask); -extern void swiotlb_init(void); - -extern unsigned int dma_bits; =20 -#ifdef CONFIG_SWIOTLB -extern int swiotlb; -#else -#define swiotlb 0 #endif =20 #endif Index: sle10-sp1-2006-12-21/include/asm-ia64/swiotlb.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/include/asm-ia64/swiotlb.h 2006-12-21 = 16:13:18.000000000 +0100 +++ sle10-sp1-2006-12-21/include/asm-ia64/swiotlb.h 2006-12-21 = 16:13:45.000000000 +0100 @@ -1,9 +1,17 @@ #ifndef _ASM_SWIOTLB_H #define _ASM_SWIOTLB_H 1 =20 +#include + +#ifndef CONFIG_XEN + #include =20 #define SWIOTLB_ARCH_NEED_LATE_INIT #define SWIOTLB_ARCH_NEED_ALLOC =20 +#else +#include +#endif + #endif /* _ASM_SWIOTLB_H */ Index: sle10-sp1-2006-12-21/include/asm-x86_64/mach-xen/asm/swiotlb.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ sle10-sp1-2006-12-21/include/asm-x86_64/mach-xen/asm/swiotlb.h = 2006-12-21 11:01:12.000000000 +0100 @@ -0,0 +1 @@ +#include Index: sle10-sp1-2006-12-21/include/asm-x86_64/swiotlb.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/include/asm-x86_64/swiotlb.h 2006-12-21 = 16:13:03.000000000 +0100 +++ sle10-sp1-2006-12-21/include/asm-x86_64/swiotlb.h 2006-12-21 = 11:01:12.000000000 +0100 @@ -43,14 +43,6 @@ extern void swiotlb_free_coherent (struc extern int swiotlb_dma_supported(struct device *hwdev, u64 mask); extern void swiotlb_init(void); =20 -#ifdef CONFIG_XEN -extern dma_addr_t swiotlb_map_page(struct device *hwdev, struct page = *page, - unsigned long offset, size_t size, - enum dma_data_direction direction); -extern void swiotlb_unmap_page(struct device *hwdev, dma_addr_t dma_addres= s, - size_t size, enum dma_data_direction = direction); -#endif - #ifdef CONFIG_SWIOTLB #define SWIOTLB_ARCH_NEED_ALLOC extern int swiotlb; Index: sle10-sp1-2006-12-21/include/xen/swiotlb.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /dev/null 1970-01-01 00:00:00.000000000 +0000 +++ sle10-sp1-2006-12-21/include/xen/swiotlb.h 2006-12-21 16:14:27.0000000= 00 +0100 @@ -0,0 +1,166 @@ +/* + * Copyright (C) 2005 Keir Fraser + */ + +#ifndef _XEN_SWIOTLB_H +#define _XEN_SWIOTLB_H 1 + +#include +#include +#include +#include + +/* Width of DMA addresses. 30 bits is a b44 limitation. */ +#define DEFAULT_DMA_BITS 30 +extern unsigned int dma_bits; + +#undef SWIOTLB_ARCH_NEED_ALLOC + +#define SWIOTLB_EXTRA_VARIABLES \ +static int __init setup_io_tlb_npages(char *str) \ +{ \ + if (isdigit(*str)) { \ + io_tlb_nslabs =3D simple_strtoul(str, &str, 0); \ + /* Unlike ia64, the size is aperture in megabytes, not = 'slabs'! */ \ + io_tlb_nslabs <<=3D (20 - IO_TLB_SHIFT); \ + /* avoid tail segment of size < IO_TLB_SEGSIZE */ \ + io_tlb_nslabs =3D ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE); \ + } \ + if (*str =3D=3D ',') \ + ++str; \ + /* \ + * NB. 'force' enables the swiotlb, but doesn't force its use for = \ + * every DMA like it does on native Linux. 'off' forcibly disables = \ + * use of the swiotlb. \ + */ \ + if (!strcmp(str, "force")) \ + swiotlb_force =3D 1; \ + else if (!strcmp(str, "off")) \ + swiotlb_force =3D -1; \ + return 1; \ +} \ +int swiotlb; \ +EXPORT_SYMBOL(swiotlb) +#define SWIOTLB_ARCH_HAS_SETUP_IO_TLB_NPAGES + +#define swiotlb_adjust_size(size) do { \ + /* Round up to power of two (xen_create_contiguous_region). */ \ + while (size & (size - 1)) \ + size +=3D size & ~(size - 1); \ +} while (0) + +#define swiotlb_adjust_seg(start, size) do { \ + int rc =3D xen_create_contiguous_region( \ + (unsigned long)(start), \ + get_order(size), \ + dma_bits); \ + BUG_ON(rc); \ +} while (0) + +#define swiotlb_print_info(bytes) \ + printk(KERN_INFO "Software IO TLB enabled: \n" \ + " Aperture: %lu megabytes\n" \ + " Kernel range: 0x%016lx - 0x%016lx\n" \ + " Address size: %u bits\n", \ + bytes >> 20, \ + (unsigned long)io_tlb_start, \ + (unsigned long)io_tlb_end, \ + dma_bits) + +#define __swiotlb_init_with_default_size(defsz) do { \ + size_t size =3D (defsz); \ + if (swiotlb_force =3D=3D 1) { \ + swiotlb =3D 1; \ + } else if ((swiotlb_force !=3D -1) && \ + is_running_on_xen() && \ + is_initial_xendomain()) { \ + /* Domain 0 always has a swiotlb. */ \ + long ram_end =3D HYPERVISOR_memory_op(XENMEM_maximum_ram_pa= ge, NULL); \ + if (ram_end <=3D 0x7ffff) \ + size =3D 2 * (1 << 20); /* 2MB on <2GB on systems. = */ \ + swiotlb =3D 1; \ + } \ + if (swiotlb) \ + swiotlb_init_with_default_size(size); \ + else \ + printk(KERN_INFO "Software IO TLB disabled\n"); \ +} while(0) + +#define range_needs_mapping range_straddles_page_boundary +#define order_needs_mapping(order) ((order) !=3D 0) +#define SWIOTLB_ARCH_HAS_NEEDS_MAPPING + +typedef struct { + struct page *page; + unsigned int offset; +} io_tlb_addr_t; +#define SWIOTLB_ARCH_HAS_IO_TLB_ADDR_T +#define swiotlb_orig_addr_null(buffer) (!(buffer).page) +#define ptr_to_io_tlb_addr(ptr) ({ \ + io_tlb_addr_t __buf; \ + if (ptr) \ + __buf.page =3D virt_to_page(ptr); \ + else \ + __buf.page =3D NULL; \ + __buf.offset =3D (unsigned long)(ptr) & ~PAGE_MASK; \ + __buf; \ +}) +#define page_to_io_tlb_addr(pg, off) ({ \ + io_tlb_addr_t __buf; \ + __buf.page =3D pg; \ + __buf.offset =3D off; \ + __buf; \ +}) +#define sg_to_io_tlb_addr(sg) ({ \ + io_tlb_addr_t __buf; \ + __buf.page =3D (sg)->page; \ + __buf.offset =3D (sg)->offset; \ + __buf; \ +}) + +/* + * We use __copy_to_user_inatomic to transfer to the host buffer because = the + * buffer may be mapped read-only (e.g, in blkback driver) but lower-level= + * drivers map the buffer for DMA_BIDIRECTIONAL access. This causes an + * unnecessary copy from the aperture to the host buffer, and a page = fault. + */ +static inline void +__swiotlb_arch_sync_single(io_tlb_addr_t buffer, char *dma_addr, size_t = size, int dir) +{ + if (PageHighMem(buffer.page)) { + size_t len, bytes; + char *dev, *host, *kmp; + len =3D size; + while (len !=3D 0) { + if (((bytes =3D len) + buffer.offset) > PAGE_SIZE) + bytes =3D PAGE_SIZE - buffer.offset; + kmp =3D kmap_atomic(buffer.page, KM_SWIOTLB); + dev =3D dma_addr + size - len; + host =3D kmp + buffer.offset; + if (dir =3D=3D DMA_FROM_DEVICE) { + if (__copy_to_user_inatomic(host, dev, = bytes)) + /* inaccessible */; + } else + memcpy(dev, host, bytes); + kunmap_atomic(kmp, KM_SWIOTLB); + len -=3D bytes; + buffer.page++; + buffer.offset =3D 0; + } + } else { + char *host =3D (char *)phys_to_virt( + page_to_pseudophys(buffer.page)) + buffer.offset; + if (dir =3D=3D DMA_FROM_DEVICE) { + if (__copy_to_user_inatomic(host, dma_addr, size)) + /* inaccessible */; + } else if (dir =3D=3D DMA_TO_DEVICE) + memcpy(dma_addr, host, size); + } +} +#define SWIOTLB_ARCH_HAS_SYNC_SINGLE + +#define SWIOTLB_ARCH_NEED_MAP_PAGE + +#define __swiotlb_dma_supported(hwdev, mask) ((((u64)1 << dma_bits) - 1) = <=3D (mask)) + +#endif /* _XEN_SWTIOLB_H */ Index: sle10-sp1-2006-12-21/lib/Makefile =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- sle10-sp1-2006-12-21.orig/lib/Makefile 2006-12-21 16:13:03.0000000= 00 +0100 +++ sle10-sp1-2006-12-21/lib/Makefile 2006-12-21 11:01:12.000000000 = +0100 @@ -49,7 +49,6 @@ obj-$(CONFIG_SGRB) +=3D sgrb.o obj-$(CONFIG_STATISTICS) +=3D statistic.o =20 obj-$(CONFIG_SWIOTLB) +=3D swiotlb.o -swiotlb-$(CONFIG_XEN) :=3D ../arch/i386/kernel/swiotlb.o =20 hostprogs-y :=3D gen_crc32table clean-files :=3D crc32table.h --=__PartD7F31481.1__= Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --=__PartD7F31481.1__=--