Re: Buffers not reachable by PCI

["Kalev, Leonid" <Leonid.Kalev@ca.com>]

On 12/14/2011 06:42 PM, Taylor, Neal E wrote:
> The last quoted printout is calculated the same way as the test that fails which leads me to question the computation's validity.

The computation validity seems OK to me (the 'phys' address is, as correctly stated 
in the comments, not suitable for DMA, but it is first translated via 
xen_phys_to_bus, which gives the machine address - and on x86 that is also the bus 
address)

I have a STUPID question, though: why are the start and end addresses of the swiotlb 
memory area not page-aligned???

The io_tlb_end address is one byte PAST the valid area, which is why the 
xen_swiotlb_dma_supported function uses (xen_io_tlb_end-1). However, this will work 
properly only if the value is page-aligned. If it isn't, then decrementing it by one 
will keep the value in the same page (which is one page past the last valid one).

The function that allocates the memory uses alloc_bootmem(), which provides just 
cache-aligned memory (not page-aligned).

If it is OK for the swiotlb area not to be page-aligned, then the 
xen_swiotlb_dma_supported should use (xen_io_tlb_end - (PAGE_SIZE-1))

If the memory should be in fact aligned, then the allocation must be changed to 
alloc_bootmem_pages() (which is the same as alloc_bootmem, but page-aligned).


>
> The test that fails is (from drivers/xen/swiotlb-xen.c):
> xen_swiotlb_dma_supported(struct device *hwdev, u64 mask)
> {
>          return xen_virt_to_bus(xen_io_tlb_end - 1)<= mask;
> }
>
>
> "xen_virt_to_bus" in turn:
> static dma_addr_t xen_virt_to_bus(void *address)
> {
>          return xen_phys_to_bus(virt_to_phys(address));
> }
>
>
> Now, "virt_to_phys" (out of arch/x86/include/asm/io.h) is defined as follows but carries a comment bothersome to this usage of it as we're, basically, dealing with a dma "transfer" and calling from a device driver:
> /**
>   *      virt_to_phys    -       map virtual addresses to physical
>   *      @address: address to remap
>   *
>   *      The returned physical address is the physical (CPU) mapping for
>   *      the memory address given. It is only valid to use this function on
>   *      addresses directly mapped or allocated via kmalloc.
>   *
>   *      This function does not give bus mappings for DMA transfers. In
>   *      almost all conceivable cases a device driver should not be using
>   *      this function
>   */
> static inline phys_addr_t virt_to_phys(volatile void *address)
> {
>          return __pa(address);
> }
>
>
> Going a couple of steps further, "__pa" is defined (in arch/x86/include/asm/page.h) as:
> #define __pa(x)         __phys_addr((unsigned long)(x))
>
>
> and "__phys_addr" (in arch/x86/mm/physaddr.c) as:
> unsigned long __phys_addr(unsigned long x)
> {
>          if (x>= __START_KERNEL_map) {
>                  x -= __START_KERNEL_map;
>                  VIRTUAL_BUG_ON(x>= KERNEL_IMAGE_SIZE);
>                  x += phys_base;
>          } else {
>                  VIRTUAL_BUG_ON(x<  PAGE_OFFSET);
>                  x -= PAGE_OFFSET;
>                  VIRTUAL_BUG_ON(!phys_addr_valid(x));
>          }
>          return x;
> }
> EXPORT_SYMBOL(__phys_addr);
>
>
> So, if "virt_to_phys" isn't yielding a valid test of whether addresses will be reachable from a PCI device, what's the correct way to test it and should xen_swiotlb_dma_supported be updated to the correct way (I think so) or a new function be created?
>
> Neal
>
>
> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Wednesday, December 14, 2011 1:20 AM
> To: Taylor, Neal E
> Cc: Kalev, Leonid; Konrad Rzeszutek Wilk; Tushar N Dave; xen-devel
> Subject: Re: [Xen-devel] Buffers not reachable by PCI
>
>>>> On 14.12.11 at 01:38, "Taylor, Neal E"<Neal.Taylor@ca.com>  wrote:
>> [    0.000000] MFN 0x7f7f->0x7f00
>
> This is clearly indicating the last chunk ends well below the 4G boundary.
>
>> [    0.000000] Placing 64MB software IO TLB between d832cf00 - dc32cf00
>> [    0.000000] software IO TLB at phys 0x1832cf00 - 0x1c32cf00
>> [    0.000000] software IO TLB at bus 0x1c0f00 - 0x120fdff00
>
> Consequently, the question is how you got to this value, or what
> changed between the first and last quoted printouts.
>
> Jan
>


-- 
Leonid Kalev
CA Technologies
Principal Software Engineer
Tel:     +972 4 825 3952
Mobile:  +972 54 4631508
Leonid.Kalev@ca.com