From mboxrd@z Thu Jan 1 00:00:00 1970 Message-ID: <4D2E27C4.7040106@domain.hid> Date: Wed, 12 Jan 2011 23:14:28 +0100 From: Gilles Chanteperdrix MIME-Version: 1.0 References: <6FCCA913376DD7488F4139A4D11B8F48016311A5@domain.hid> <4D1FAE66.8080005@domain.hid> <6FCCA913376DD7488F4139A4D11B8F4801673CAB@troe2k1.cs.myharris.net> <4D24E3AC.2000904@domain.hid> <4D24EB92.3050104@domain.hid> <4D24F0B7.7010604@domain.hid> <6FCCA913376DD7488F4139A4D11B8F4801673CD4@domain.hid> <4D25C9D6.4030707@domain.hid> <4D2CE77C.7000109@domain.hid> <6FCCA913376DD7488F4139A4D11B8F4801674163@domain.hid> <4D2DAE16.2060805@domain.hid> <6FCCA913376DD7488F4139A4D11B8F48016741A9@domain.hid> <4D2DD61E.7000006@domain.hid> <6FCCA913376DD7488F4139A4D11B8F48016741F6@domain.hid> <4D2E10E4.7040608@domain.hid> <6FCCA913376DD7488F4139A4D11B8F4801674280@domain.hid> <1294870292.1828.258.camel@domain.hid> In-Reply-To: <1294870292.1828.258.camel@domain.hid> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Subject: Re: [Xenomai-help] [Xenomai -help] User space access to DMA memory List-Id: Help regarding installation and common use of Xenomai List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Philippe Gerum Cc: xenomai@xenomai.org Philippe Gerum wrote: > On Wed, 2011-01-12 at 16:57 -0500, Herrera-Bendezu, Luis wrote: >>> -----Original Message----- >>> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org] >>> Sent: Wednesday, January 12, 2011 3:37 PM >>> To: Herrera-Bendezu, Luis >>> Cc: xenomai@xenomai.org >>> Subject: Re: [Xenomai-help] [Xenomai -help] User space access to DMA memory >>> >>> Herrera-Bendezu, Luis wrote: >>>>> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org] >>>>> Sent: Wednesday, January 12, 2011 11:26 AM >>>>> To: Herrera-Bendezu, Luis >>>>> Cc: xenomai@xenomai.org >>>>> Subject: Re: [Xenomai-help] [Xenomai -help] User space access to DMA memory >>>>> >>>>> Herrera-Bendezu, Luis wrote: >>>>>>> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org] >>>>>>> Sent: Wednesday, January 12, 2011 8:35 AM >>>>>>> To: Herrera-Bendezu, Luis >>>>>>> Cc: xenomai@xenomai.org >>>>>>> Subject: Re: [Xenomai-help] [Xenomai -help] User space access to DMA memory >>>>>>> >>>>>>> Herrera-Bendezu, Luis wrote: >>>>>>>>> -----Original Message----- >>>>>>>>> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org] >>>>>>>>> Sent: Tuesday, January 11, 2011 6:28 PM >>>>>>>>> To: Herrera-Bendezu, Luis >>>>>>>>> Cc: xenomai@xenomai.org >>>>>>>>> Subject: Re: [Xenomai-help] [Xenomai -help] User space access to DMA memory >>>>>>>>> >>>>>>>>> Gilles Chanteperdrix wrote: >>>>>>>>>> Herrera-Bendezu, Luis wrote: >>>>>>>>>>> On 01/05/2011 5:29 PM Gilles Chanteperdrix wrote: >>>>>>>>>>>> Steven A. Falco wrote: >>>>>>>>>>>>> On 01/05/2011 04:33 PM, Gilles Chanteperdrix wrote: >>>>>>>>>>>> Ok. Could you try to do the same operation with the native API? You just >>>>>>>>>>>> have to pass H_SHARED | H_DMA | H_NONCACHED as flags to rt_heap_create >>>>>>>>>>>> to get the same effect as pci_dma_alloc_coherent. >>>>>>>>>>>> >>>>>>>>>>>> Just to see if the error lies in RTDM implementation or in Xenomai >>>>>>>>>>>> generic code. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> (...) >>>>>>>>>> What about the other thing I asked you to test? >>>>>>>>>> >>>>>>>>> Ping? Any news about this test? >>>>>>>> rt_heap_create() with flags H_SHARED | H_DMA | H_NONCACHED report -EINVAL. >>>>>>>> Documentation indicates that H_NONCACHE is not compatible with H_DMA. >>>>>>> Right, supporting H_NONCACHED | H_DMA would mean that we would have to >>>>>>> use kmalloc/get_free_pages then establish a non-cached mapping in >>>>>>> kernel-space with vm_map_ram. >>>>>>> >>>>>>> Could you try with H_NONCACHED, but without H_DMA? Of course, the >>>>>>> mapping can not be used for DMA, as it will probably not be physically >>>>>>> contiguous, but at least you can try whether accessing in user-space a >>>>>>> non-cached mapping works. >>>>>> It does work but unless an external unit writes to heap memory it would >>>>>> be difficult to verify that the non-cached memory actually works, i.e. >>>>>> access from user space gets expected value(s). >>>>> I am quite confident this works. >>>>> >>>>>>> In any case, we see a defect in the RTDM interface here: we can not ask >>>>>>> the mapping to be mapped non-cacheable, which somewhat defeats the >>>>>>> purpose of pci_alloc_consistent. I do not know enough the powerpc >>>>>>> architecture to know whether this could be the cause of your issue (the >>>>>>> same physical area mapped twice, once cached, once non-cached). However, >>>>>>> on the ARM architecture, for instance, it is bad. >>>>>>> >>>>>> Let me summarize the ideas discussed so far to allocate consistent memory >>>>>> suitable for DMA and corresponding mapping to user space: >>>>>> * rt_heap cannot be created with both H_NONCACHED and H_DMA. But it is >>>>>> automatically mapped to user space with rt_heap_bind(). >>>>>> >>>>>> A solution can be to allocate heap with H_SHARED | H_DMA, somehow >>>>>> get bus address of single block of memory (rt_heap { sba } ?) to >>>>>> used for DMA operations. >>>>> The test with rt_heap was just a test to have a comparison between >>>>> xnheap code and rtdm_mmap. But we may indeed want to fix mmappable >>>>> xnheaps later on. >>>>> >>>>>> * RTDM interface rtdm_mmap/unmap cannot handle non-cacheable memory. >>>>>> >>>>>> A solution can be to allocate GFP_DMA memory with kmalloc(), use >>>>>> rtdm_mmap to map to user space. Programmatically make memory >>>>>> consistent before/after DMA transfer to/from external unit, e.g. >>>>>> dma_sync_single_for_device()/dma_sync_single_for_cpu(). This is >>>>>> similar to what streaming DMA mappings do. >>>>>> >>>>>> * Ideally, it should be possible to take memory allocated from >>>>>> dma_alloc_coherent()/pci_alloc_consistent() and mapped it to >>>>>> user space using rtdm_mmap(). >>>>> To check that the non-cacheable mapping is indeed the problem, here is a >>>>> patch which adds the possibility to add non-cacheable mappings: >>>>> >>>>> diff --git a/ksrc/skins/rtdm/drvlib.c b/ksrc/skins/rtdm/drvlib.c >>>>> index 3495e63..ec0ddae 100644 >>>>> --- a/ksrc/skins/rtdm/drvlib.c >>>>> +++ b/ksrc/skins/rtdm/drvlib.c >>>>> @@ -1791,6 +1791,7 @@ void rtdm_nrtsig_pend(rtdm_nrtsig_t *nrt_sig); >>>>> >>>>> #if defined(CONFIG_XENO_OPT_PERVASIVE) || defined(DOXYGEN_CPP) >>>>> struct rtdm_mmap_data { >>>>> + int noncached; >>>>> void *src_vaddr; >>>>> phys_addr_t src_paddr; >>>>> struct vm_operations_struct *vm_ops; >>>>> @@ -1815,6 +1816,9 @@ static int rtdm_mmap_buffer(struct file *filp, >>>>> struct vm_area_struct *vma) >>>>> maddr = vma->vm_start; >>>>> size = vma->vm_end - vma->vm_start; >>>>> >>>>> + if (mmap_data->noncached) >>>>> + vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); >>>>> + >>>>> #ifdef CONFIG_MMU >>>>> /* Catch vmalloc memory (vaddr is 0 for I/O mapping) */ >>>>> if ((vaddr >= VMALLOC_START) && (vaddr < VMALLOC_END)) { >>>>> @@ -1975,6 +1979,24 @@ int rtdm_mmap_to_user(rtdm_user_info_t *user_info, >>>>> void *vm_private_data) >>>>> { >>>>> struct rtdm_mmap_data mmap_data = { >>>>> + .noncached = 0, >>>>> + .src_vaddr = src_addr, >>>>> + .src_paddr = 0, >>>>> + .vm_ops = vm_ops, >>>>> + .vm_private_data = vm_private_data >>>>> + }; >>>>> + >>>>> + return rtdm_do_mmap(user_info, &mmap_data, len, prot, pptr); >>>>> +} >>>>> + >>>>> +int rtdm_mmap_noncached_to_user(rtdm_user_info_t *user_info, >>>>> + void *src_addr, size_t len, >>>>> + int prot, void **pptr, >>>>> + struct vm_operations_struct *vm_ops, >>>>> + void *vm_private_data) >>>>> +{ >>>>> + struct rtdm_mmap_data mmap_data = { >>>>> + .noncached = 1, >>>>> .src_vaddr = src_addr, >>>>> .src_paddr = 0, >>>>> .vm_ops = vm_ops, >>>>> @@ -2043,6 +2065,7 @@ int rtdm_iomap_to_user(rtdm_user_info_t *user_info, >>>>> void *vm_private_data) >>>>> { >>>>> struct rtdm_mmap_data mmap_data = { >>>>> + .noncached = 0, >>>>> .src_vaddr = NULL, >>>>> .src_paddr = src_addr, >>>>> .vm_ops = vm_ops, >>>>> >>>>> Simply call rtdm_mmap_noncached_to_user instead of rtdm_mmap_to_user. >>>>> >>>> Get same kernel panic error as before with this patch. I need to look closer >>>> at what are the memory allocation functions used by pci_ and dma_. >>> Could you try rtdm_iomap_to_user (but beware, pass the physical address >>> returned by pointer by pci_alloc_consistent)? virt_to_page, or __pa will >>> not work with vmalloc/ioremap addresses. And pci_alloc_consistent >>> probably returns a vmalloc mapping due to the fact that the mapping >>> needs to be non-cacheable, which will happen if your powerpc does not >>> support cache snooping, and define CONFIG_NOT_COHERENT_CACHE. >>> >> Your suggestion works. Can you highlight why it fails when using >> rtdm_mmap_to_user()? > > Because unlike the former, it does not get the right platform-dependent > protection bits for the vma from phys_mem_access_prot() for this kind of > memory. There is another issue: the memory returned by pci_alloc_consistent should be handled as vmalloc memory, but it is allocated in a separated area, the CONSISTENT_BASE, CONSISTENT_END area. So the test: if ((vaddr >= VMALLOC_START) && (vaddr < VMALLOC_END)) returns false, and the memory is handled as if it were kmalloced memory. -- Gilles.