[BUG] pgprot_noncached() is -NOT- safe for mapping vmalloc buffers into userspace

[BUG] pgprot_noncached() is -NOT- safe for mapping vmalloc buffers into userspace

[Benjamin Herrenschmidt]

Hi Takashi !

While working on endian-fixing xHCI with Matt (CC), we discovered the
source of our problems with usb-audio on a board we were working on.

c32d977b8157bf67cdf47729ce7dd054a26eb534
"ALSA: pcm - Call pgprot_noncached() for vmalloc'ed buffers"

I'm afraid that this is totally bogus :-)

I don't know on what arch it is safe to have the same memory be mapped
cachable in the kernel (and accessed via this cached mapping) and
non-cachable in userspace, but I can confidently say that wherever it
works it does so by accident.

In the case of usb-audio, what we observed is that the user application
was writing samples using an uncached mapping, so directly to memory,
which does -not- invalidate conflicting cache lines on the way, an the
kernel would then memcpy those data to the USB buffers using a cached
mapping (vmalloc) and essentially get stale stuff from the cache instead
of the real samples.

Worse, on some processors, it's actually -illegal- to create (and even
more to -access-) a conflicting mapping of a page of memory, ie, have it
mapped cached somewhere and uncached somewhere else. It will lockup some
processors and afaik, some x86 as well.

In fact, cache coherent architectures often don't support mapping memory
uncached -at-all- so something like snd_pcm_lib_mmap_noncached()
shouldn't exist, or at least be under arch control. There's no case
where it's "always safe". There will almost always be a cache alias in
the linear mapping unless special arch specific sauce has been applied.

Now, there's another problem on top of that, which is that
snd_pcm_default_mmap() will not work properly the "other way around" on
powerpc, where the mapping -needs- to be uncached bcs you are running on
a non cache coherent embedded CPU and trying to mmap DMA memory, but
that's something that needs fixing inside powerpc by properly defining
dma_mmap_coherent() & ARCH_HAS_DMA_MMAP_COHERENT (I thought we had added
it a while back but it's not upstream, patch must have got lost). We
must also make sure we don't go down that path for vmalloc memory
though.

Cheers,
Ben.

Re: [BUG] pgprot_noncached() is -NOT- safe for mapping vmalloc buffers into userspace

[Takashi Iwai]

At Fri, 25 Mar 2011 09:16:48 +1100,
Benjamin Herrenschmidt wrote:
> 
> Hi Takashi !
> 
> While working on endian-fixing xHCI with Matt (CC), we discovered the
> source of our problems with usb-audio on a board we were working on.
> 
> c32d977b8157bf67cdf47729ce7dd054a26eb534
> "ALSA: pcm - Call pgprot_noncached() for vmalloc'ed buffers"
> 
> I'm afraid that this is totally bogus :-)
> 
> I don't know on what arch it is safe to have the same memory be mapped
> cachable in the kernel (and accessed via this cached mapping) and
> non-cachable in userspace, but I can confidently say that wherever it
> works it does so by accident.
> 
> In the case of usb-audio, what we observed is that the user application
> was writing samples using an uncached mapping, so directly to memory,
> which does -not- invalidate conflicting cache lines on the way, an the
> kernel would then memcpy those data to the USB buffers using a cached
> mapping (vmalloc) and essentially get stale stuff from the cache instead
> of the real samples.
> 
> Worse, on some processors, it's actually -illegal- to create (and even
> more to -access-) a conflicting mapping of a page of memory, ie, have it
> mapped cached somewhere and uncached somewhere else. It will lockup some
> processors and afaik, some x86 as well.
> 
> In fact, cache coherent architectures often don't support mapping memory
> uncached -at-all- so something like snd_pcm_lib_mmap_noncached()
> shouldn't exist, or at least be under arch control. There's no case
> where it's "always safe". There will almost always be a cache alias in
> the linear mapping unless special arch specific sauce has been applied.

I see.  I'll take your removal patch.
It should be applied to stable kernel, too, right?

> Now, there's another problem on top of that, which is that
> snd_pcm_default_mmap() will not work properly the "other way around" on
> powerpc, where the mapping -needs- to be uncached bcs you are running on
> a non cache coherent embedded CPU and trying to mmap DMA memory, but
> that's something that needs fixing inside powerpc by properly defining
> dma_mmap_coherent() & ARCH_HAS_DMA_MMAP_COHERENT (I thought we had added
> it a while back but it's not upstream, patch must have got lost).

Mea culpa.  I moved to a different team and have had little time for
the upstream development since then...

> We
> must also make sure we don't go down that path for vmalloc memory
> though.

Yes.

Your patch looks good.  Thanks for taking care of this!


Takashi

Re: [BUG] pgprot_noncached() is -NOT- safe for mapping vmalloc buffers into userspace

[Benjamin Herrenschmidt]

On Fri, 2011-03-25 at 09:01 +0100, Takashi Iwai wrote:
> > In fact, cache coherent architectures often don't support mapping memory
> > uncached -at-all- so something like snd_pcm_lib_mmap_noncached()
> > shouldn't exist, or at least be under arch control. There's no case
> > where it's "always safe". There will almost always be a cache alias in
> > the linear mapping unless special arch specific sauce has been applied.
> 
> I see.  I'll take your removal patch.
> It should be applied to stable kernel, too, right?

I suppose so :-)

> > Now, there's another problem on top of that, which is that
> > snd_pcm_default_mmap() will not work properly the "other way around" on
> > powerpc, where the mapping -needs- to be uncached bcs you are running on
> > a non cache coherent embedded CPU and trying to mmap DMA memory, but
> > that's something that needs fixing inside powerpc by properly defining
> > dma_mmap_coherent() & ARCH_HAS_DMA_MMAP_COHERENT (I thought we had added
> > it a while back but it's not upstream, patch must have got lost).
> 
> Mea culpa.  I moved to a different team and have had little time for
> the upstream development since then...

That's ok, I did a new one, which you have noticed already anyways :-)

> > We
> > must also make sure we don't go down that path for vmalloc memory
> > though.
>
> Yes.

I haven't actually checked, but I assume that the test

substream->dma_buffer.dev.type == SNDRV_DMA_TYPE_DEV

In snd_pcm_default_mmap() takes care of that, please correct me if
I'm wrong in which case we'll need something else there.

> Your patch looks good.  Thanks for taking care of this! 

Are you taking care of sending it upstream ?

Cheers,
Ben.

Re: [BUG] pgprot_noncached() is -NOT- safe for mapping vmalloc buffers into userspace

[Takashi Iwai]

At Fri, 25 Mar 2011 20:15:33 +1100,
Benjamin Herrenschmidt wrote:
> 
> > > We
> > > must also make sure we don't go down that path for vmalloc memory
> > > though.
> >
> > Yes.
> 
> I haven't actually checked, but I assume that the test
> 
> substream->dma_buffer.dev.type == SNDRV_DMA_TYPE_DEV
> 
> In snd_pcm_default_mmap() takes care of that, please correct me if
> I'm wrong in which case we'll need something else there.

Well, in the case of usb-audio, it's not handled via
dma_mmap_coherent(), as the page isn't allocated via
dma_alloc_coherent() but vmalloc().

The bad commit was to overcome some problems on SH platform, IIRC,
when it's used with dmix -- i.e. concurrent accesses on the mmapped
buffer from multiple processes.  But, this looks obviously like a
wrong approach.

Actually, the buffer allocated there in usb-audio is an intermediate
buffer, that isn't directly transferred to hardware.  We may need a
bit more consideration what is the best way to solve that issue (if
it's still really present).

> > Your patch looks good.  Thanks for taking care of this! 
> 
> Are you taking care of sending it upstream ?

I'll apply the patch to remove vmalloc pgprot thingy surely to sound
tree and include in the next pull request.

Others should be sent through arch maintainers (PPC and ARM), right?


thanks,

Takashi

Re: [BUG] pgprot_noncached() is -NOT- safe for mapping vmalloc buffers into userspace

[Benjamin Herrenschmidt]

On Fri, 2011-03-25 at 11:12 +0100, Takashi Iwai wrote:
> At Fri, 25 Mar 2011 20:15:33 +1100,
> Benjamin Herrenschmidt wrote:
> > 
> > > > We
> > > > must also make sure we don't go down that path for vmalloc memory
> > > > though.
> > >
> > > Yes.
> > 
> > I haven't actually checked, but I assume that the test
> > 
> > substream->dma_buffer.dev.type == SNDRV_DMA_TYPE_DEV
> > 
> > In snd_pcm_default_mmap() takes care of that, please correct me if
> > I'm wrong in which case we'll need something else there.
> 
> Well, in the case of usb-audio, it's not handled via
> dma_mmap_coherent(), as the page isn't allocated via
> dma_alloc_coherent() but vmalloc().

Right, I just wanted to make sure I was right to assume that a page
allocated by vmalloc() was going to fail the above test in 
snd_pcm_default_mmap() and thus -not- get into dma_mmap_coherent()...
just double checking as I'm not totally familiar with the intricacies of
the pcm code :-)

> The bad commit was to overcome some problems on SH platform, IIRC,
> when it's used with dmix -- i.e. concurrent accesses on the mmapped
> buffer from multiple processes.  But, this looks obviously like a
> wrong approach.

Is this a vivt architecture ? Maybe enforcing some restrictions on the
virtual addresses so they hit the same cache congruence classes ?

> Actually, the buffer allocated there in usb-audio is an intermediate
> buffer, that isn't directly transferred to hardware.  We may need a
> bit more consideration what is the best way to solve that issue (if
> it's still really present).

Right. I wouldn't expect vmalloc stuff to hit HW in most cases anyways,
though I do wonder why you don't pass the buffer directly to the HCD and
avoid that intermediate step but that's a completely different
question :-)

> > > Your patch looks good.  Thanks for taking care of this! 
> > 
> > Are you taking care of sending it upstream ?
> 
> I'll apply the patch to remove vmalloc pgprot thingy surely to sound
> tree and include in the next pull request.
> 
> Others should be sent through arch maintainers (PPC and ARM), right?

Well, I am ppc maintainer so that's sorted :-) I've CCed Russell for the
other, it's up to him, I have no specific dependency there, it's just an
easy cleanup I stumbled upon.

Cheers,
Ben.

> 
> thanks,
> 
> Takashi
> --
> To unsubscribe from this list: send the line "unsubscribe linux-arch" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [BUG] pgprot_noncached() is -NOT- safe for mapping vmalloc buffers into userspace

[Takashi Iwai]

At Fri, 25 Mar 2011 21:23:26 +1100,
Benjamin Herrenschmidt wrote:
> 
> On Fri, 2011-03-25 at 11:12 +0100, Takashi Iwai wrote:
> > At Fri, 25 Mar 2011 20:15:33 +1100,
> > Benjamin Herrenschmidt wrote:
> > > 
> > > > > We
> > > > > must also make sure we don't go down that path for vmalloc memory
> > > > > though.
> > > >
> > > > Yes.
> > > 
> > > I haven't actually checked, but I assume that the test
> > > 
> > > substream->dma_buffer.dev.type == SNDRV_DMA_TYPE_DEV
> > > 
> > > In snd_pcm_default_mmap() takes care of that, please correct me if
> > > I'm wrong in which case we'll need something else there.
> > 
> > Well, in the case of usb-audio, it's not handled via
> > dma_mmap_coherent(), as the page isn't allocated via
> > dma_alloc_coherent() but vmalloc().
> 
> Right, I just wanted to make sure I was right to assume that a page
> allocated by vmalloc() was going to fail the above test in 
> snd_pcm_default_mmap() and thus -not- get into dma_mmap_coherent()...
> just double checking as I'm not totally familiar with the intricacies of
> the pcm code :-)
> 
> > The bad commit was to overcome some problems on SH platform, IIRC,
> > when it's used with dmix -- i.e. concurrent accesses on the mmapped
> > buffer from multiple processes.  But, this looks obviously like a
> > wrong approach.
> 
> Is this a vivt architecture ? Maybe enforcing some restrictions on the
> virtual addresses so they hit the same cache congruence classes ?

Possibly, yes.  But we need to re-test the problem first with the
recent kernel.

> > Actually, the buffer allocated there in usb-audio is an intermediate
> > buffer, that isn't directly transferred to hardware.  We may need a
> > bit more consideration what is the best way to solve that issue (if
> > it's still really present).
> 
> Right. I wouldn't expect vmalloc stuff to hit HW in most cases anyways,
> though I do wonder why you don't pass the buffer directly to the HCD and
> avoid that intermediate step but that's a completely different
> question :-)

The direct buffer was used partly in the older versions.  But there
was a problem in the accuracy of the buffer playback position, so this
was switched to full double-buffering.


thanks,

Takashi