How to make mmap'ed kernel buffer non-cacheable

How to make mmap'ed kernel buffer non-cacheable

[Bhuvan Kumar MITTAL]

Hi,
  I am working on an audio device driver development on Linux. I have a kernel buffer which I have mapped to user space using mmap call from user space. My problem is that the data which comes to the kernel buffer is getting dropped in user space and I get only 50-60% of the data which is randomly ordered. The user to kernel level buffer address translation code is fine and I suspect this data dropping is occurring coz the kernel buffer is cacheable. Please suggest me some way of making the entire buffer non cacheable. I am stuck on this for quite a while now. 

Regards,

Bhuvan

Re: How to make mmap'ed kernel buffer non-cacheable

[Alan Cox]

On Fri, 20 Apr 2007 18:30:25 +0530
Bhuvan Kumar MITTAL <bhuvan.mittal@st.com> wrote:

> Hi,
>   I am working on an audio device driver development on Linux. I have a kernel buffer which I have mapped to user space using mmap call from user space. My problem is that the data which comes to the kernel buffer is getting dropped in user space and I get only 50-60% of the data which is randomly ordered. The user to kernel level buffer address translation code is fine and I suspect this data dropping is occurring coz the kernel buffer is cacheable. Please suggest me some way of making the entire buffer non cacheable. I am stuck on this for quite a while now. 

The dma mapping API (or the PCI equivalent) provide the neccessary
behaviours for DMA receive, DMA send and consistent memory space in a
portable fashion.

That may not be done using uncachable memory in all cases as not all
processors even support uncacheable memory spaces.

If you are using the ALSA core routines (snd_dma_alloc_coherent) then
ALSA already uses dma_alloc_coherent to ensure the memory is allocated
for the appropriate use and will be kernel marked uncached.

RE: How to make mmap'ed kernel buffer non-cacheable

[Bhuvan Kumar MITTAL]

Hi Alan,

I believe that dma_alloc_coherent will mark the kernel buffer as uncached at alocation time. 
But that is not my intention. I have mapped some user space memory to the kernel buffer and I wish to ensure that the contents of both are coherent and correctly ordered. 

In other words I wish to flush the contents of the kernel buffer to user space as soon as new data is available in my kernel buffer. How to do that? Will doing mysnc from the user space help?

Rather than flushing everytime (or msyncing) I intend to make my user-to-kernel mapping as non cacheable so that multiple flushing can be avoided.

Bhuvan 

> Hi,
>   I am working on an audio device driver development on Linux. I have a kernel buffer which I have mapped to user space using mmap call from user space. My problem is that the data which comes to the kernel buffer is getting dropped in user space and I get only 50-60% of the data which is randomly ordered. The user to kernel level buffer address translation code is fine and I suspect this data dropping is occurring coz the kernel buffer is cacheable. Please suggest me some way of making the entire buffer non cacheable. I am stuck on this for quite a while now. 

The dma mapping API (or the PCI equivalent) provide the neccessary
behaviours for DMA receive, DMA send and consistent memory space in a
portable fashion.

That may not be done using uncachable memory in all cases as not all
processors even support uncacheable memory spaces.

If you are using the ALSA core routines (snd_dma_alloc_coherent) then
ALSA already uses dma_alloc_coherent to ensure the memory is allocated
for the appropriate use and will be kernel marked uncached.

Re: How to make mmap'ed kernel buffer non-cacheable

[Nick Piggin]

Bhuvan Kumar MITTAL wrote:
> Hi Alan,
> 
> I believe that dma_alloc_coherent will mark the kernel buffer as uncached at alocation time. 
> But that is not my intention. I have mapped some user space memory to the kernel buffer and I wish to ensure that the contents of both are coherent and correctly ordered. 
> 
> In other words I wish to flush the contents of the kernel buffer to user space as soon as new data is available in my kernel buffer. How to do that? Will doing mysnc from the user space help?

msync is only for pagecache. If you modify user mapped RAM from the kernel, or wish
to read user modified RAM from the kernel, you should issue a flush_dcache_page after
and before, respectively. See Documentation/cachetlb.h.

Does that fix it? What are the details of your platform?

-- 
SUSE Labs, Novell Inc.

RE: How to make mmap'ed kernel buffer non-cacheable

[Bhuvan Kumar MITTAL]

Hi Alan,Nick

Thanks for your inputs. I was able to solve the problem of mapping kernel buffer to user space. Just FYI, I am working on a sh4 based architecture proprietary board with Linux kernel 2.6.17.3

Initially I was adopting the following approach: 
1.The kernel thread was copying some data and waiting on a semaphore for the user task to make an ioctl into the kernel space and relieve the wait.
2. Then the user task was getting spawned and taking the data from kernel.

---> This was resulting in some initial data drop coz the kernel task was overwhelming and outspeeding the user task (Maybe someone can explain how)

How I solved the problem?

Instead of making the kernel wait for the user task, I reversed the sequence and did not let the kernel task be activated till the user task was spawned and waiting in kernel space(after making ioctl) for the kernel task. Then I started the kernel task and the problem of data dropping was resolved.

In continuation to this, I have another query:
As mentioned above, If there is a kernel task waiting on a semaphore (maybe even in hung state) and I wish to kill the kernel task from user space by making an ioctl call into the kernel, then how is it possible? Is it feasible at all?


Regards,

Bhuvan

On 20 Apr, 14:20, Bhuvan Kumar MITTAL <bhuvan.mit...@st.com> wrote:
> Hi,
>   I am working on an audio device driver development on Linux. I have 
> a kernel buffer which I have mapped to user space using mmap call from 
> user space. My problem is that the data which comes to the kernel 
> buffer is getting dropped in user space and I get only 50-60% of the 
> data which is randomly ordered. The user to kernel level buffer 
> address translation code is fine and I suspect this data dropping is 
> occurring coz the kernel buffer is cacheable. Please suggest me some 
> way of making the entire buffer non cacheable. I am stuck on this for quite a while now.

You can use rdmsr() in your driver to check if the page attribute table MSR is available, then find and/or add the right entry in the table to set in each page's flags. This is documented in the Intel
IA-32 Intel Architecture Software Developer's Manuals at intel.com.
-----Original Message-----
From: Nick Piggin [mailto:nickpiggin@yahoo.com.au] 
Sent: Monday, April 23, 2007 9:56 AM
To: Bhuvan Kumar MITTAL
Cc: 'Alan Cox'; linux-kernel@vger.kernel.org
Subject: Re: How to make mmap'ed kernel buffer non-cacheable

Bhuvan Kumar MITTAL wrote:
> Hi Alan,
> 
> I believe that dma_alloc_coherent will mark the kernel buffer as uncached at alocation time. 
> But that is not my intention. I have mapped some user space memory to the kernel buffer and I wish to ensure that the contents of both are coherent and correctly ordered. 
> 
> In other words I wish to flush the contents of the kernel buffer to user space as soon as new data is available in my kernel buffer. How to do that? Will doing mysnc from the user space help?

msync is only for pagecache. If you modify user mapped RAM from the kernel, or wish
to read user modified RAM from the kernel, you should issue a flush_dcache_page after
and before, respectively. See Documentation/cachetlb.h.

Does that fix it? What are the details of your platform?

-- 
SUSE Labs, Novell Inc.

Re: How to make mmap'ed kernel buffer non-cacheable

[Nick Piggin]

Bhuvan Kumar MITTAL wrote:
> Hi Alan,Nick
> 
> Thanks for your inputs. I was able to solve the problem of mapping kernel buffer to user space. Just FYI, I am working on a sh4 based architecture proprietary board with Linux kernel 2.6.17.3

So... you used flush_dcache_page? Or what?


> Initially I was adopting the following approach: 
> 1.The kernel thread was copying some data and waiting on a semaphore for the user task to make an ioctl into the kernel space and relieve the wait.
> 2. Then the user task was getting spawned and taking the data from kernel.
> 
> ---> This was resulting in some initial data drop coz the kernel task was overwhelming and outspeeding the user task (Maybe someone can explain how)
> 
> How I solved the problem?
> 
> Instead of making the kernel wait for the user task, I reversed the sequence and did not let the kernel task be activated till the user task was spawned and waiting in kernel space(after making ioctl) for the kernel task. Then I started the kernel task and the problem of data dropping was resolved.

OK. Seems like a straightfoward producer-consumer problem.

I don't know why you should be dropping data if your kernel task blocks when
its output buffer is full and your user task blocks when it is empty. But anyway,
it seems you solved it. Good :)


> In continuation to this, I have another query:
> As mentioned above, If there is a kernel task waiting on a semaphore (maybe even in hung state) and I wish to kill the kernel task from user space by making an ioctl call into the kernel, then how is it possible? Is it feasible at all?

If the kernel task is hung because of some bug in your code, then it isn't
really possible in general. You could always just "up" the semaphore and hope,
but things might be corrupted.

If you just want to tell the kernel task to quit, I guess you'd just introduce
some quit ioctl message, and have the kernel thread respond to that.

-- 
SUSE Labs, Novell Inc.

RE: How to make mmap'ed kernel buffer non-cacheable

[Bhuvan Kumar MITTAL]

I'll rephrase the problem as follow:

I have a userthread which makes ioctl calls to the kernel and once it reaches inside the kernel it waits on a semaphore. It then does some work inside the kernel and continuously keeps looping between the kernel and user space in an endless while loop. 

I want to kill that thread from another user thread while the former is waiting on a semaphore in kernel (which it will never get) coz I want a clean exit of my driver. Please suggest me a way. I did try the use of pthread_cancel, but that is possible only if the target thread is executing in user space and making use of a pthread_testcancel call (with cancelability enabled ofcourse). 

Kindly suggest something. 

> In continuation to this, I have another query:
> As mentioned above, If there is a kernel task waiting on a semaphore (maybe even in hung state) and I wish to kill the kernel task from user space by making an ioctl call into the kernel, then how is it possible? Is it feasible at all?

If the kernel task is hung because of some bug in your code, then it isn't
really possible in general. You could always just "up" the semaphore and hope,
but things might be corrupted.

If you just want to tell the kernel task to quit, I guess you'd just introduce
some quit ioctl message, and have the kernel thread respond to that.

Re: How to make mmap'ed kernel buffer non-cacheable

[Nick Piggin]

You can do down_interruptible to make the down interruptible.

Bhuvan Kumar MITTAL wrote:
> I'll rephrase the problem as follow:
> 
> I have a userthread which makes ioctl calls to the kernel and once it reaches inside the kernel it waits on a semaphore. It then does some work inside the kernel and continuously keeps looping between the kernel and user space in an endless while loop. 
> 
> I want to kill that thread from another user thread while the former is waiting on a semaphore in kernel (which it will never get) coz I want a clean exit of my driver. Please suggest me a way. I did try the use of pthread_cancel, but that is possible only if the target thread is executing in user space and making use of a pthread_testcancel call (with cancelability enabled ofcourse). 
> 
> Kindly suggest something. 
> 
> 
>>In continuation to this, I have another query:
>>As mentioned above, If there is a kernel task waiting on a semaphore (maybe even in hung state) and I wish to kill the kernel task from user space by making an ioctl call into the kernel, then how is it possible? Is it feasible at all?
> 
> 
> If the kernel task is hung because of some bug in your code, then it isn't
> really possible in general. You could always just "up" the semaphore and hope,
> but things might be corrupted.
> 
> If you just want to tell the kernel task to quit, I guess you'd just introduce
> some quit ioctl message, and have the kernel thread respond to that.
> 
> 
> 


-- 
SUSE Labs, Novell Inc.

Re: How to make mmap'ed kernel buffer non-cacheable

[Daniel J Blueman]

On 20 Apr, 14:20, Bhuvan Kumar MITTAL <bhuvan.mit...@st.com> wrote:
> Hi,
>   I am working on an audio device driver development on Linux. I have a kernel
> buffer which I have mapped to user space using mmap call from user space. My
> problem is that the data which comes to the kernel buffer is getting dropped in
> user space and I get only 50-60% of the data which is randomly ordered. The user
> to kernel level buffer address translation code is fine and I suspect this data
> dropping is occurring coz the kernel buffer is cacheable. Please suggest me some
> way of making the entire buffer non cacheable. I am stuck on this for quite a while
> now.

You can use rdmsr() in your driver to check if the page attribute
table MSR is available, then find and/or add the right entry in the
table to set in each page's flags. This is documented in the Intel
IA-32 Intel Architecture Software Developer's Manuals at intel.com.

Dan
-- 
Daniel J Blueman