[Qemu-devel] Address translation - virt->phys->ram

[Qemu-devel] Address translation - virt->phys->ram

[Ian Molton]

Hi folks,

I've been updating some old patches which make use of a function to
translate guest virtual addresses into pointers into the guest RAM.

As I understand it qemu has guest virtual and physical addresses, the
latter of which map somehow to host ram addresses.

The function which the code had been using appears not to work under
kvm, which leads me to think that qemu doesnt emulate the MMU (or at
least not in the same manner) when it is using kvm as opposed to pure
emulation.

If I turn off kvm, the patch works, albeit slowly. If I enable it, the
code takes the path which looks for the magic value (below).

Is there a 'proper' way to translate guest virtual addresses into host
RAM addresses?

Here is the code:-

static /*inline*/ void *get_phys_mem_addr(CPUState *env, target_ulong addr)
{
    int mmu_idx;
    int index;
        int i;

    index = (addr >> TARGET_PAGE_BITS) & (CPU_TLB_SIZE - 1);
    mmu_idx = cpu_mmu_index(env);

    if (__builtin_expect
        (env->tlb_table[mmu_idx][index].addr_code !=
         (addr & TARGET_PAGE_MASK), 0)) {
        target_ulong ret = cpu_get_phys_page_debug((CPUState *) env, addr);
        if (ret == -1) {
            fprintf(stderr,
                    "not in phys mem " TARGET_FMT_lx "(" TARGET_FMT_lx " "
                    TARGET_FMT_lx ")\n", addr,
                    env->tlb_table[mmu_idx][index].addr_code,
                    addr & TARGET_PAGE_MASK);
            fprintf(stderr, "cpu_x86_handle_mmu_fault = %d\n",
                    cpu_x86_handle_mmu_fault((CPUState *) env, addr, 0,
mmu_idx, 1));
            return NULL;
        } else {
            if (ret + TARGET_PAGE_SIZE <= ram_size) {
                return qemu_get_ram_ptr((ret + (((target_ulong) addr) &
(TARGET_PAGE_SIZE - 1))));
            } else {
                fprintf(stderr,
                        "cpu_get_phys_page_debug(env, " TARGET_FMT_lx ")
== "
                        TARGET_FMT_lx "\n", addr, ret);
                fprintf(stderr,
                        "ram_size= " TARGET_FMT_lx "\n", ret,
(target_ulong) ram_size);

        for(i = 0 ; i < ram_size-10 ; i++) {
                char *ptr = qemu_get_ram_ptr(i);
                if(!strncmp("magic_string", ptr, 10)) {
                        fprintf(stderr, "found magic_string at: %lx
%lx\n", i, ptr);
                        break;
                }
        }

                return qemu_get_ram_ptr(i-128); //Evil horrible hack
            }
        }
    } else
        return (void *) addr + env->tlb_table[mmu_idx][index].addend;
}

Re: [Qemu-devel] Address translation - virt->phys->ram

[Anthony Liguori]

On 02/22/2010 07:59 AM, Ian Molton wrote:
> Hi folks,
>
> I've been updating some old patches which make use of a function to
> translate guest virtual addresses into pointers into the guest RAM.
>
> As I understand it qemu has guest virtual and physical addresses, the
> latter of which map somehow to host ram addresses.
>
> The function which the code had been using appears not to work under
> kvm, which leads me to think that qemu doesnt emulate the MMU (or at
> least not in the same manner) when it is using kvm as opposed to pure
> emulation.
>
> If I turn off kvm, the patch works, albeit slowly. If I enable it, the
> code takes the path which looks for the magic value (below).
>
> Is there a 'proper' way to translate guest virtual addresses into host
> RAM addresses?
>    

cpu_physical_memory_map().

But this function has some subtle characteristics.  It may return a 
bounce buffer if you attempt to map MMIO memory.  There is a limited 
pool of bounce buffers available so it may return NULL in the event that 
it cannot allocate a bounce buffer.

It may also return a partial result if you're attempting to map a region 
that straddles multiple memory slots.

Regards,

Anthony Liguori

Re: [Qemu-devel] Address translation - virt->phys->ram

[Ian Molton]

Anthony Liguori wrote:

> cpu_physical_memory_map().
> 
> But this function has some subtle characteristics.  It may return a
> bounce buffer if you attempt to map MMIO memory.  There is a limited
> pool of bounce buffers available so it may return NULL in the event that
> it cannot allocate a bounce buffer.
> 
> It may also return a partial result if you're attempting to map a region
> that straddles multiple memory slots.

Thanks. I had found this, but was unsure as to wether it was quite what
I wanted. (also is it possible to tell when it has (eg.) allocated a
bounce buffer?)

Basically, I need to get buffer(s) from guest userspace into the hosts
address space. The buffers are virtually contiguous but likely
physically discontiguous. They are allocated with malloc() and theres
nothing I can do about that.

The obvious but slow solution would be to copy all the buffers into nice
virtio-based scatter/gather buffers and feed them to the host that way,
however its not fast enough.

Right now I have a little driver I have written that allows a buffer to
be mmap()ed by the guest userspace, and this is pushed to the host via
virtio s/g io when the guest calls fsync(). This buffer contains the
data that must be passed to the host, however this data may often
contain pointers to (that is, userspace virtual addresses of) buffers of
unknown sizes which the host also needs to access. These buffers are
what I need to read from the guests RAM.

The buffers will likely remain active across multiple different calls to
the host so their pages will need to be available. As the calls always
happen when that specific process is active, I'd guess the worst we need
to do is generate a page fault to unswap the page(s). Can that be caused
by qemu (under kvm)?

It seems that cpu_physical_memory_map() deals with physically contiguous
areas of guest address space. I need to get a host-side mapping of a
*virtually* contiguous (possibly physically discontiguous) set of guest
pages. If this can be done, it'd mean direct transfer of data from guest
application to host shared library, which would be a major win.

-Ian

Re: [Qemu-devel] Address translation - virt->phys->ram

[Anthony Liguori]

On 02/22/2010 10:46 AM, Ian Molton wrote:
> Anthony Liguori wrote:
>
>    
>> cpu_physical_memory_map().
>>
>> But this function has some subtle characteristics.  It may return a
>> bounce buffer if you attempt to map MMIO memory.  There is a limited
>> pool of bounce buffers available so it may return NULL in the event that
>> it cannot allocate a bounce buffer.
>>
>> It may also return a partial result if you're attempting to map a region
>> that straddles multiple memory slots.
>>      
> Thanks. I had found this, but was unsure as to wether it was quite what
> I wanted. (also is it possible to tell when it has (eg.) allocated a
> bounce buffer?)
>
> Basically, I need to get buffer(s) from guest userspace into the hosts
> address space. The buffers are virtually contiguous but likely
> physically discontiguous. They are allocated with malloc() and theres
> nothing I can do about that.
>
> The obvious but slow solution would be to copy all the buffers into nice
> virtio-based scatter/gather buffers and feed them to the host that way,
> however its not fast enough.
>    

Why is this slow?

Regards,

Anthony Liguori

> Right now I have a little driver I have written that allows a buffer to
> be mmap()ed by the guest userspace, and this is pushed to the host via
> virtio s/g io when the guest calls fsync(). This buffer contains the
> data that must be passed to the host, however this data may often
> contain pointers to (that is, userspace virtual addresses of) buffers of
> unknown sizes which the host also needs to access. These buffers are
> what I need to read from the guests RAM.
>    



> The buffers will likely remain active across multiple different calls to
> the host so their pages will need to be available. As the calls always
> happen when that specific process is active, I'd guess the worst we need
> to do is generate a page fault to unswap the page(s). Can that be caused
> by qemu (under kvm)?
>
> It seems that cpu_physical_memory_map() deals with physically contiguous
> areas of guest address space. I need to get a host-side mapping of a
> *virtually* contiguous (possibly physically discontiguous) set of guest
> pages. If this can be done, it'd mean direct transfer of data from guest
> application to host shared library, which would be a major win.
>
> -Ian
>    

Re: [Qemu-devel] Address translation - virt->phys->ram

[Ian Molton]

Anthony Liguori wrote:
> On 02/22/2010 10:46 AM, Ian Molton wrote:
>> Anthony Liguori wrote:
>>
>>   
>>> cpu_physical_memory_map().
>>>
>>> But this function has some subtle characteristics.  It may return a
>>> bounce buffer if you attempt to map MMIO memory.  There is a limited
>>> pool of bounce buffers available so it may return NULL in the event that
>>> it cannot allocate a bounce buffer.
>>>
>>> It may also return a partial result if you're attempting to map a region
>>> that straddles multiple memory slots.
>>>      
>> Thanks. I had found this, but was unsure as to wether it was quite what
>> I wanted. (also is it possible to tell when it has (eg.) allocated a
>> bounce buffer?)
>>
>> Basically, I need to get buffer(s) from guest userspace into the hosts
>> address space. The buffers are virtually contiguous but likely
>> physically discontiguous. They are allocated with malloc() and theres
>> nothing I can do about that.
>>
>> The obvious but slow solution would be to copy all the buffers into nice
>> virtio-based scatter/gather buffers and feed them to the host that way,
>> however its not fast enough.
>>    
> 
> Why is this slow?

Because the buffers will all have to be copied. So far, switching from
abusing an instruction to interrupt qemu to using virtio has incurred a
roughly 5x slowdown. I'd guess much of this is down to the fact we have
to switch to kernel-mode on the guest and back again for every single GL
call...

If I can establish some kind of stable guest_virt->phys->host_virt
mapping, many of the problems will just 'go away'. a way to interrupt
qemu from user-mode on the guest without involving the guest kernel
would be quite awesome also (theres really nothing we want the kernel to
actually /do/ here, it just adds overhead).

-Ian

Re: [Qemu-devel] Address translation - virt->phys->ram

[Alexander Graf]

Ian Molton wrote:
> Anthony Liguori wrote:
>   
>> On 02/22/2010 10:46 AM, Ian Molton wrote:
>>     
>>> Anthony Liguori wrote:
>>>
>>>   
>>>       
>>>> cpu_physical_memory_map().
>>>>
>>>> But this function has some subtle characteristics.  It may return a
>>>> bounce buffer if you attempt to map MMIO memory.  There is a limited
>>>> pool of bounce buffers available so it may return NULL in the event that
>>>> it cannot allocate a bounce buffer.
>>>>
>>>> It may also return a partial result if you're attempting to map a region
>>>> that straddles multiple memory slots.
>>>>      
>>>>         
>>> Thanks. I had found this, but was unsure as to wether it was quite what
>>> I wanted. (also is it possible to tell when it has (eg.) allocated a
>>> bounce buffer?)
>>>
>>> Basically, I need to get buffer(s) from guest userspace into the hosts
>>> address space. The buffers are virtually contiguous but likely
>>> physically discontiguous. They are allocated with malloc() and theres
>>> nothing I can do about that.
>>>
>>> The obvious but slow solution would be to copy all the buffers into nice
>>> virtio-based scatter/gather buffers and feed them to the host that way,
>>> however its not fast enough.
>>>    
>>>       
>> Why is this slow?
>>     
>
> Because the buffers will all have to be copied. So far, switching from
> abusing an instruction to interrupt qemu to using virtio has incurred a
> roughly 5x slowdown. I'd guess much of this is down to the fact we have
> to switch to kernel-mode on the guest and back again for every single GL
> call...
>
> If I can establish some kind of stable guest_virt->phys->host_virt
> mapping, many of the problems will just 'go away'. a way to interrupt
> qemu from user-mode on the guest without involving the guest kernel
> would be quite awesome also (theres really nothing we want the kernel to
> actually /do/ here, it just adds overhead).
>   

I guess what you really want is some shm region between host and guess
that you can use as ring buffer. Then you could run a timer on the host
side to flush it or have some sort of callback when you urgently need to
flush it manually.

The benefit here is that you can actually make use of multiple threads.
There's no need to intercept the guest at all just because it wants to
issue some GL operations.


Alex

Re: [Qemu-devel] Address translation - virt->phys->ram

[Ian Molton]

Alexander Graf wrote:

> I guess what you really want is some shm region between host and guess
> that you can use as ring buffer. Then you could run a timer on the host
> side to flush it or have some sort of callback when you urgently need to
> flush it manually.
> 
> The benefit here is that you can actually make use of multiple threads.
> There's no need to intercept the guest at all just because it wants to
> issue some GL operations.

Something like that should work. The problem right now is mostly the
'some sort of callback'. Im not sure there exists any mechanism for the
guests userspace to interrupt qemu directly when running under kvm...

Re: [Qemu-devel] Address translation - virt->phys->ram

[Alexander Graf]

On 23.02.2010, at 16:46, Ian Molton wrote:

> Alexander Graf wrote:
> 
>> I guess what you really want is some shm region between host and guess
>> that you can use as ring buffer. Then you could run a timer on the host
>> side to flush it or have some sort of callback when you urgently need to
>> flush it manually.
>> 
>> The benefit here is that you can actually make use of multiple threads.
>> There's no need to intercept the guest at all just because it wants to
>> issue some GL operations.
> 
> Something like that should work. The problem right now is mostly the
> 'some sort of callback'. Im not sure there exists any mechanism for the
> guests userspace to interrupt qemu directly when running under kvm...

I'm not aware of any mechanism, but you could easily write a simple UIO driver that takes over this exact task.
Or you build on top of cam's shm patches and create a device node that exposes a poke ioctl.


Alex

Re: [Qemu-devel] Address translation - virt->phys->ram

[Anthony Liguori]

On 02/22/2010 11:47 AM, Ian Molton wrote:
> Anthony Liguori wrote:
>    
>> On 02/22/2010 10:46 AM, Ian Molton wrote:
>>      
>>> Anthony Liguori wrote:
>>>
>>>
>>>        
>>>> cpu_physical_memory_map().
>>>>
>>>> But this function has some subtle characteristics.  It may return a
>>>> bounce buffer if you attempt to map MMIO memory.  There is a limited
>>>> pool of bounce buffers available so it may return NULL in the event that
>>>> it cannot allocate a bounce buffer.
>>>>
>>>> It may also return a partial result if you're attempting to map a region
>>>> that straddles multiple memory slots.
>>>>
>>>>          
>>> Thanks. I had found this, but was unsure as to wether it was quite what
>>> I wanted. (also is it possible to tell when it has (eg.) allocated a
>>> bounce buffer?)
>>>
>>> Basically, I need to get buffer(s) from guest userspace into the hosts
>>> address space. The buffers are virtually contiguous but likely
>>> physically discontiguous. They are allocated with malloc() and theres
>>> nothing I can do about that.
>>>
>>> The obvious but slow solution would be to copy all the buffers into nice
>>> virtio-based scatter/gather buffers and feed them to the host that way,
>>> however its not fast enough.
>>>
>>>        
>> Why is this slow?
>>      
> Because the buffers will all have to be copied.

Why?

It sounds like your kernel driver is doing the wrong thing if you can't 
preserve zero-copy from userspace.

>   So far, switching from
> abusing an instruction to interrupt qemu to using virtio has incurred a
> roughly 5x slowdown.

If you post patches, we can help determine what you're doing that's 
causing such a slow down.

Regards,

Anthony Liguori

>   I'd guess much of this is down to the fact we have
> to switch to kernel-mode on the guest and back again for every single GL
> call...
>
> If I can establish some kind of stable guest_virt->phys->host_virt
> mapping, many of the problems will just 'go away'. a way to interrupt
> qemu from user-mode on the guest without involving the guest kernel
> would be quite awesome also (theres really nothing we want the kernel to
> actually /do/ here, it just adds overhead).
>
> -Ian
>
>