[Qemu-devel] passing translated address out in QEMU

[Qemu-devel] passing translated address out in QEMU

[Xin Tong]

Hello

In tcg_gen_qemu_ld8s(tmp, addr, index); a TCGv addr is passed to the
INDEX_op_qemu_ld8s as param and the loaded value is passed back in
tmp. i want to get the translated value as well.  how can i do that ?

one way i can think of right now is to modify the INDEX_op_qemu_ld8s
and make it take one more TCGv and pass the translated address back
there. but this may need non-trial modifications to the target-* and
tcg.

another way is to pass out the translated address in the TCGv addr.
but it seems the register  TCGv addr is assigned to is saved (edx on
i386 linux) before entering the translation. how can i make addr
clobberable in the tcg_out_qemu_ld/tcg_out_qemu_st ?


Thanks

Xin

Re: [Qemu-devel] passing translated address out in QEMU

[陳韋任 (Wei-Ren Chen)]

> In tcg_gen_qemu_ld8s(tmp, addr, index); a TCGv addr is passed to the
> INDEX_op_qemu_ld8s as param and the loaded value is passed back in
> tmp. i want to get the translated value as well.  how can i do that ?

  IIUC, qemu_ld takes addr as guest virtual address, then loads the
value in that address into tmp. So, what "translated value" you mean
here? The guest physical address, host virtual address, or the value
of the guest virtual address? 

Regards,
chenwj

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667
Homepage: http://people.cs.nctu.edu.tw/~chenwj

Re: [Qemu-devel] passing translated address out in QEMU

[Xin Tong]

On Wed, Aug 22, 2012 at 8:14 PM, 陳韋任 (Wei-Ren Chen)
<chenwj@iis.sinica.edu.tw> wrote:
>> In tcg_gen_qemu_ld8s(tmp, addr, index); a TCGv addr is passed to the
>> INDEX_op_qemu_ld8s as param and the loaded value is passed back in
>> tmp. i want to get the translated value as well.  how can i do that ?
>
>   IIUC, qemu_ld takes addr as guest virtual address, then loads the
> value in that address into tmp. So, what "translated value" you mean
> here? The guest physical address, host virtual address, or the value
> of the guest virtual address?

either the guest physical or the host virtual is fine with me. but
better to be guest physical. i think guest physical can be obtained by
subtracting host virtual with a fixed offset for ram. but the thing i
do not know how to do is how to pass the guest physical back, can i
store the guest physical in the TCGv from the guest virtual.

Xin

> Regards,
> chenwj
>
> --
> Wei-Ren Chen (陳韋任)
> Computer Systems Lab, Institute of Information Science,
> Academia Sinica, Taiwan (R.O.C.)
> Tel:886-2-2788-3799 #1667
> Homepage: http://people.cs.nctu.edu.tw/~chenwj

Re: [Qemu-devel] passing translated address out in QEMU

[陳韋任 (Wei-Ren Chen)]

On Wed, Aug 22, 2012 at 09:38:17PM -0700, Xin Tong wrote:
> On Wed, Aug 22, 2012 at 8:14 PM, 陳韋任 (Wei-Ren Chen)
> <chenwj@iis.sinica.edu.tw> wrote:
> >> In tcg_gen_qemu_ld8s(tmp, addr, index); a TCGv addr is passed to the
> >> INDEX_op_qemu_ld8s as param and the loaded value is passed back in
> >> tmp. i want to get the translated value as well.  how can i do that ?
> >
> >   IIUC, qemu_ld takes addr as guest virtual address, then loads the
> > value in that address into tmp. So, what "translated value" you mean
> > here? The guest physical address, host virtual address, or the value
> > of the guest virtual address?
> 
> either the guest physical or the host virtual is fine with me. but
> better to be guest physical. i think guest physical can be obtained by
> subtracting host virtual with a fixed offset for ram. but the thing i
> do not know how to do is how to pass the guest physical back, can i
> store the guest physical in the TCGv from the guest virtual.

  qemu_ld will go through software tlb first, you will get host virtual
address there. It will go ld{b,w,l,q}_mmu if software tlb miss, call
tlb_fill -> cpu_x86_handle_mmu_fault, you can get guest physical address
there (target_phys_addr_t paddr). If you want to get guest physical
address for each guest memory access, I guess you need to disable
software tlb lookup, you can refer to Max's patch [1]. Or as you said,
maybe you can get guest physical addr from host virtual addr. You can
refer to cpu_physical_memory_map (exec.c) which map guest physical addr
to host virtual addr, then see how you can do the reverse.

HTH,
chenwj

[1] http://lists.gnu.org/archive/html/qemu-devel/2012-08/msg03226.html

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667
Homepage: http://people.cs.nctu.edu.tw/~chenwj

Re: [Qemu-devel] passing translated address out in QEMU

[陳韋任 (Wei-Ren Chen)]

> subtracting host virtual with a fixed offset for ram. but the thing i
> do not know how to do is how to pass the guest physical back, can i
> store the guest physical in the TCGv from the guest virtual.

  Maybe you can store it on a host unused register, perhaps xmm0? ;)

Regards,
chenwj

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667
Homepage: http://people.cs.nctu.edu.tw/~chenwj

Re: [Qemu-devel] passing translated address out in QEMU

[Xin Tong]

I am emulating arm on x86. i want to track the virt and physical
address of last memory operation. so i put 2 fields in the CPUState
and make tcg_global_mem_new_i32 on them Therefore, before every
translation i generate code to save the virtual address as follow:

static inline void gen_st32(TCGv val, TCGv addr, int index)
{
    tcg_gen_mov_i32(cpu_last_vaddr, addr);
    tcg_gen_qemu_st32(val, addr, index);
   // tcg_gen_mov_i32(cpu_last_paddr, addr);
    tcg_temp_free_i32(val);
}

But i do not know how to save the physical, as the physical address is
never passed out of tcg_gen_qemu_st32. what would be the best way to
get the physical address here ? i want to pass it out by the "TCGv
addr here" but it did not work ...

Xin




   0xf3753166:  mov    eax,DWORD PTR [ebp+0x18]
   0xf3753169:  mov    edx,eax
   0xf375316b:  mov    ecx,eax
   0xf375316d:  mov    DWORD PTR [ebp+0x6ccc],eax
   0xf3753173:  mov    DWORD PTR [ebp+0x3d4],edx
   0xf3753179:  mov    eax,0x4
   0xf375317e:  mov    DWORD PTR [ebp+0x3dc],eax
   0xf3753184:  xor    eax,eax
   0xf3753186:  mov    DWORD PTR [ebp+0x3e0],eax
   0xf375318c:  mov    edx,ecx
   0xf375318e:  mov    eax,ecx
   0xf3753190:  shr    edx,0x6
   0xf3753193:  and    eax,0xfffffc03
   0xf3753199:  and    edx,0xff0
   0xf375319f:  lea    edx,[ebp+edx*1+0x4c4]
   0xf37531a6:  cmp    eax,DWORD PTR [edx]
   0xf37531a8:  mov    eax,ecx
   0xf37531aa:  jne    0xf37531b3
   0xf37531ac:  add    eax,DWORD PTR [edx+0xc]
   0xf37531af:  mov    eax,DWORD PTR [eax]
   0xf37531b1:  jmp    0xf37531bc
   0xf37531b3:  xor    edx,edx
   0xf37531b5:  call   0x8184cb0 <__ldl_mmu>
   0xf37531ba:  mov    edx,eax
   0xf37531bc:  mov    edx,DWORD PTR [ebp+0x6ccc]
   0xf37531c2:  mov    DWORD PTR [esp],ebp
   0xf37531c5:  mov    ecx,0xf7d034ac
   0xf37531ca:  mov    DWORD PTR [esp+0x4],ecx
   0xf37531ce:  xor    ecx,ecx
   0xf37531d0:  mov    DWORD PTR [esp+0x8],ecx
   0xf37531d4:  mov    ecx,DWORD PTR [ebp+0x3d4]
   0xf37531da:  mov    DWORD PTR [esp+0xc],ecx
   0xf37531de:  mov    DWORD PTR [esp+0x10],edx
   0xf37531e2:  mov    DWORD PTR [ebp+0x4],eax
   0xf37531e5:  mov    DWORD PTR [ebp+0x3d8],edx



On Thu, Aug 23, 2012 at 3:11 AM, 陳韋任 (Wei-Ren Chen)
<chenwj@iis.sinica.edu.tw> wrote:
>> subtracting host virtual with a fixed offset for ram. but the thing i
>> do not know how to do is how to pass the guest physical back, can i
>> store the guest physical in the TCGv from the guest virtual.
>
>   Maybe you can store it on a host unused register, perhaps xmm0? ;)
>
> Regards,
> chenwj
>
> --
> Wei-Ren Chen (陳韋任)
> Computer Systems Lab, Institute of Information Science,
> Academia Sinica, Taiwan (R.O.C.)
> Tel:886-2-2788-3799 #1667
> Homepage: http://people.cs.nctu.edu.tw/~chenwj

Re: [Qemu-devel] passing translated address out in QEMU

[Peter Maydell]

On 23 August 2012 16:34, Xin Tong <xerox.time.tech@gmail.com> wrote:
> I am emulating arm on x86. i want to track the virt and physical
> address of last memory operation. so i put 2 fields in the CPUState
> and make tcg_global_mem_new_i32 on them Therefore, before every
> translation i generate code to save the virtual address as follow:
>
> static inline void gen_st32(TCGv val, TCGv addr, int index)
> {
>     tcg_gen_mov_i32(cpu_last_vaddr, addr);
>     tcg_gen_qemu_st32(val, addr, index);
>    // tcg_gen_mov_i32(cpu_last_paddr, addr);
>     tcg_temp_free_i32(val);
> }
>
> But i do not know how to save the physical, as the physical address is
> never passed out of tcg_gen_qemu_st32. what would be the best way to
> get the physical address here ? i want to pass it out by the "TCGv
> addr here" but it did not work ...

This is quite difficult because our fast-path code doesn't actually
deal with the guest physical address at all: we create a TLB which
maps directly from guest virtual address to host virtual address
and use that most of the time.

In general you are running into the problem that QEMU is designed
to run code fast, not to be easy to instrument.

-- PMM

Re: [Qemu-devel] passing translated address out in QEMU

[Xin Tong]

On Thu, Aug 23, 2012 at 8:52 AM, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 23 August 2012 16:34, Xin Tong <xerox.time.tech@gmail.com> wrote:
>> I am emulating arm on x86. i want to track the virt and physical
>> address of last memory operation. so i put 2 fields in the CPUState
>> and make tcg_global_mem_new_i32 on them Therefore, before every
>> translation i generate code to save the virtual address as follow:
>>
>> static inline void gen_st32(TCGv val, TCGv addr, int index)
>> {
>>     tcg_gen_mov_i32(cpu_last_vaddr, addr);
>>     tcg_gen_qemu_st32(val, addr, index);
>>    // tcg_gen_mov_i32(cpu_last_paddr, addr);
>>     tcg_temp_free_i32(val);
>> }
>>
>> But i do not know how to save the physical, as the physical address is
>> never passed out of tcg_gen_qemu_st32. what would be the best way to
>> get the physical address here ? i want to pass it out by the "TCGv
>> addr here" but it did not work ...
>
> This is quite difficult because our fast-path code doesn't actually
> deal with the guest physical address at all: we create a TLB which
> maps directly from guest virtual address to host virtual address
> and use that most of the time.
>
> In general you are running into the problem that QEMU is designed
> to run code fast, not to be easy to instrument.

even that. is it possible to pass host virtual out. the fast path add
the addend to get host virtual ? so it must be in a register, most
likely eax in i386. what do you think would be the best way to get
that out ?

Xin

>
> -- PMM

Re: [Qemu-devel] passing translated address out in QEMU

[Steven]

Hi, Xin,
Try Max's patch
http://lists.gnu.org/archive/html/qemu-devel/2012-08/msg03226.html

I used it to get all the guest virtual address because this patch
disable the fast mmu path.

Steven

On Thu, Aug 23, 2012 at 12:18 PM, Xin Tong <xerox.time.tech@gmail.com> wrote:
> On Thu, Aug 23, 2012 at 8:52 AM, Peter Maydell <peter.maydell@linaro.org> wrote:
>> On 23 August 2012 16:34, Xin Tong <xerox.time.tech@gmail.com> wrote:
>>> I am emulating arm on x86. i want to track the virt and physical
>>> address of last memory operation. so i put 2 fields in the CPUState
>>> and make tcg_global_mem_new_i32 on them Therefore, before every
>>> translation i generate code to save the virtual address as follow:
>>>
>>> static inline void gen_st32(TCGv val, TCGv addr, int index)
>>> {
>>>     tcg_gen_mov_i32(cpu_last_vaddr, addr);
>>>     tcg_gen_qemu_st32(val, addr, index);
>>>    // tcg_gen_mov_i32(cpu_last_paddr, addr);
>>>     tcg_temp_free_i32(val);
>>> }
>>>
>>> But i do not know how to save the physical, as the physical address is
>>> never passed out of tcg_gen_qemu_st32. what would be the best way to
>>> get the physical address here ? i want to pass it out by the "TCGv
>>> addr here" but it did not work ...
>>
>> This is quite difficult because our fast-path code doesn't actually
>> deal with the guest physical address at all: we create a TLB which
>> maps directly from guest virtual address to host virtual address
>> and use that most of the time.
>>
>> In general you are running into the problem that QEMU is designed
>> to run code fast, not to be easy to instrument.
>
> even that. is it possible to pass host virtual out. the fast path add
> the addend to get host virtual ? so it must be in a register, most
> likely eax in i386. what do you think would be the best way to get
> that out ?
>
> Xin
>
>>
>> -- PMM
>

Re: [Qemu-devel] passing translated address out in QEMU

[陳韋任 (Wei-Ren Chen)]

> even that. is it possible to pass host virtual out. the fast path add
> the addend to get host virtual ? so it must be in a register, most
> likely eax in i386. what do you think would be the best way to get
> that out ?

  Take a look on comment on tcg_out_tlb_load (tcg/i386/tcg-target.c).

   First argument register is loaded with the low part of the address.
   In the TLB hit case, it has been adjusted as indicated by the TLB
   and so is a host address.  In the TLB miss case, it continues to
   hold a guest address.

So I guess you can take "tcg_target_call_iarg_regs[0]" as host virtual
address if software tlb hit.

    /* TLB Hit.  */
    tcg_out_qemu_ld_direct(s, data_reg, data_reg2,
                           tcg_target_call_iarg_regs[0], 0, opc);

As for how do you get host virtual address out, I think you can write
a helper function logging those host virtual address into a file or
something else, then insert a helper function call at the point of
TLB hit.

HTH,
chenwj

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667
Homepage: http://people.cs.nctu.edu.tw/~chenwj

Re: [Qemu-devel] passing translated address out in QEMU

[Xin Tong]

On Thu, Aug 23, 2012 at 7:57 PM, 陳韋任 (Wei-Ren Chen)
<chenwj@iis.sinica.edu.tw> wrote:
>> even that. is it possible to pass host virtual out. the fast path add
>> the addend to get host virtual ? so it must be in a register, most
>> likely eax in i386. what do you think would be the best way to get
>> that out ?
>
>   Take a look on comment on tcg_out_tlb_load (tcg/i386/tcg-target.c).
>
>    First argument register is loaded with the low part of the address.
>    In the TLB hit case, it has been adjusted as indicated by the TLB
>    and so is a host address.  In the TLB miss case, it continues to
>    hold a guest address.
>
> So I guess you can take "tcg_target_call_iarg_regs[0]" as host virtual
> address if software tlb hit.
>
>     /* TLB Hit.  */
>     tcg_out_qemu_ld_direct(s, data_reg, data_reg2,
>                            tcg_target_call_iarg_regs[0], 0, opc);
>
> As for how do you get host virtual address out, I think you can write
> a helper function logging those host virtual address into a file or
> something else, then insert a helper function call at the point of
> TLB hit.
>
> HTH,
> chenwj
>
> --
> Wei-Ren Chen (陳韋任)
> Computer Systems Lab, Institute of Information Science,
> Academia Sinica, Taiwan (R.O.C.)
> Tel:886-2-2788-3799 #1667
> Homepage: http://people.cs.nctu.edu.tw/~chenwj

That might be difficult. what i did was that i disabled inlined
translated and push the virt/phys address into 2 new fields in the cpu
structure in the call out lookup. because in the callout lookup we
have a handle to the cpu env.

not too sure how much impact inlined lookup has on the performance.
since i disabled it, next step i would just get rid of that piece of
generated assembly, as it is no good for icache ( generated for every
memory operation).

Xin

Re: [Qemu-devel] passing translated address out in QEMU

[陳韋任 (Wei-Ren Chen)]

> That might be difficult. what i did was that i disabled inlined
> translated and push the virt/phys address into 2 new fields in the cpu
> structure in the call out lookup. because in the callout lookup we
> have a handle to the cpu env.

  What you mean by "disabled inlined translated"? You mean apply Max's
patch so that all guest memory access go through the slow path without
looking software tlb? Since you said you're running arm on x86 host,
I guess what you did might be,

int cpu_arm_handle_mmu_fault (CPUARMState *env, target_ulong address,
                              int access_type, int mmu_idx)
{
    ...

    ret = get_phys_addr(env, address, access_type, is_user, &phys_addr, &prot,
                        &page_size);

    // store phys_addr into env->cpu_last_paddr

    ...
}
 
> not too sure how much impact inlined lookup has on the performance.
> since i disabled it, next step i would just get rid of that piece of
> generated assembly, as it is no good for icache ( generated for every
> memory operation).

  You can run a benchmark inside your guest. I guess if you run a
long-running benchmark, you can see performance degradation. If software
tlb hit, you can get the value of guest memory in the code cache
with a few host instructions. Disabling software tlb lookup, every guest
memory access will call a helper function which takes a lot of time.
What you mean by "get rid of that piece of generated assembly"?

Regards,
chenwj

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667
Homepage: http://people.cs.nctu.edu.tw/~chenwj

Re: [Qemu-devel] passing translated address out in QEMU

[Xin Tong]

On Thu, Aug 23, 2012 at 9:06 PM, 陳韋任 (Wei-Ren Chen)
<chenwj@iis.sinica.edu.tw> wrote:
>> That might be difficult. what i did was that i disabled inlined
>> translated and push the virt/phys address into 2 new fields in the cpu
>> structure in the call out lookup. because in the callout lookup we
>> have a handle to the cpu env.
>
>   What you mean by "disabled inlined translated"? You mean apply Max's
> patch so that all guest memory access go through the slow path without
> looking software tlb? Since you said you're running arm on x86 host,
> I guess what you did might be,
>
> int cpu_arm_handle_mmu_fault (CPUARMState *env, target_ulong address,
>                               int access_type, int mmu_idx)
> {
>     ...
>
>     ret = get_phys_addr(env, address, access_type, is_user, &phys_addr, &prot,
>                         &page_size);
>
>     // store phys_addr into env->cpu_last_paddr
>
>     ...
> }
>
>> not too sure how much impact inlined lookup has on the performance.
>> since i disabled it, next step i would just get rid of that piece of
>> generated assembly, as it is no good for icache ( generated for every
>> memory operation).
>
>   You can run a benchmark inside your guest. I guess if you run a
> long-running benchmark, you can see performance degradation. If software
> tlb hit, you can get the value of guest memory in the code cache
> with a few host instructions. Disabling software tlb lookup, every guest
> memory access will call a helper function which takes a lot of time.
> What you mean by "get rid of that piece of generated assembly"?

every inlined TLB lookup has ~10 instructions .

Xin

>
> Regards,
> chenwj
>
> --
> Wei-Ren Chen (陳韋任)
> Computer Systems Lab, Institute of Information Science,
> Academia Sinica, Taiwan (R.O.C.)
> Tel:886-2-2788-3799 #1667
> Homepage: http://people.cs.nctu.edu.tw/~chenwj

Re: [Qemu-devel] passing translated address out in QEMU

[陳韋任 (Wei-Ren Chen)]

> >> not too sure how much impact inlined lookup has on the performance.
> >> since i disabled it, next step i would just get rid of that piece of
> >> generated assembly, as it is no good for icache ( generated for every
> >> memory operation).
> >
> >   You can run a benchmark inside your guest. I guess if you run a
> > long-running benchmark, you can see performance degradation. If software
> > tlb hit, you can get the value of guest memory in the code cache
> > with a few host instructions. Disabling software tlb lookup, every guest
> > memory access will call a helper function which takes a lot of time.
> > What you mean by "get rid of that piece of generated assembly"?
> 
> every inlined TLB lookup has ~10 instructions .

  I still don't think remove inline tlb lookup will improve anything.
True, the inline tlb lookup will take some space on icache, but I
believe it's not a big deal since you're on a x86 host.

Regards,
chenwj

-- 
Wei-Ren Chen (陳韋任)
Computer Systems Lab, Institute of Information Science,
Academia Sinica, Taiwan (R.O.C.)
Tel:886-2-2788-3799 #1667
Homepage: http://people.cs.nctu.edu.tw/~chenwj