Re: [RFC] Heterogeneous memory management (mirror process address space on a device mmu).

[Jerome Glisse <j.glisse@gmail.com>]

On Tue, May 06, 2014 at 08:47:48AM -0700, Linus Torvalds wrote:
> On Tue, May 6, 2014 at 8:33 AM, Jerome Glisse <j.glisse@gmail.com> wrote:
> >
> > So how can i solve the issue at hand. A device that has its own page
> > table and can not mirror the cpu page table, nor can the device page
> > table be updated atomicly from the cpu.
> 
> So? Just model it as a TLB.
> 
> Sure, the TLB is slow and crappy and is in external memory rather than
> on-die, but it's still a TLB.
> 
> We have CPU's that do that kind of crazy thing (powerpc and sparc both
> have these kinds of "in-memory TLB extensions" in addition to the
> on-die TLB, they just call them "inverse page tables" to try to fool
> people about what they are).
> 
> > I understand that we do not want to sleep when updating process cpu
> > page table but note that only process that use the gpu would have to
> > sleep. So only process that can actually benefit from the using GPU
> > will suffer the consequences.
> 
> NO!
> 
> You don't get it. If a callback can sleep, then we cannot protect it
> with a spinlock.
> 
> It doesn't matter if it only sleeps once in a millennium. It still
> forces its crap on the rest of the system.

I do understand that i was pointing out that if i move to, tlb which i
am fine with, i will still need to sleep there. That's all i wanted to
stress, i did not wanted force using mmu_notifier, i am fine with them
becoming atomic as long as i have a place where i can intercept cpu
page table update and propagate them to device mmu.

> 
> So there is no way in hell that we will allow that VM notifier crap. None.
> 
> And as I've mentioned, there is a correct place to slot this in, and
> that correct way is the _only_ way to ever support future GPU's that
> _do_ share direct access to the page tables.

This work was done in cooperation with NVidia and we discussed with AMD too
so i am very much aware of what is coming next on hardware front and being
able to have GPU have their own GPU page table ie not walking the CPU one is
something of interest to the people who design those future generation of
GPU.

> 
> So trying to do it any other way is broken _anyway_.
> 
>            Linus

I will respin without using mmu_notifier and by hooking it as tlb shootdown.
But it will still need to sleep during the device tlb shootdown and that's
the point i want to make sure is clear.

Cheers,
Jérôme

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>