[Qemu-devel] On x86 MMU modes

[Qemu-devel] On x86 MMU modes

[Sandhya Kumar]

[-- Attachment #1: Type: text/plain, Size: 748 bytes --]

[Query on intended logic]

I am trying to learn qemu's MMU emulation logic for x86 and came across H.
Peter Anvin's SMAP commit (link
<http://lists.gnu.org/archive/html/qemu-devel/2012-09/msg04622.html>). I
have the following doubt on the intended logic (apologies if it is trivial)

As per my understanding (which matches versions prior to this commit), we
generally maintain only two TLBs [one for kernel and one for user] in x86
ISA for caching address translations. With this commit we seem to have
three modes of MMU, although only two will be actually used (either KSMAP
or KNOSMAP). Is my claim valid ? Why cannot those two original modes serve
the purpose and why is the separation (of KNOMAP and KSMAP) needed?

Thanks in advance,
Sandhya

[-- Attachment #2: Type: text/html, Size: 883 bytes --]

Re: [Qemu-devel] On x86 MMU modes

[Paolo Bonzini]

On 03/06/2015 08:51, Sandhya Kumar wrote:
> As per my understanding (which matches versions prior to this commit),
> we generally maintain only two TLBs [one for kernel and one for user] in
> x86 ISA for caching address translations. With this commit we seem to
> have three modes of MMU, although only two will be actually used (either
> KSMAP or KNOSMAP).

This is not accurate.  If AC=0, data accesses from the kernel use
KNOSMAP, but implicit accesses (e.g. reads of the IDT) use KSMAP.

> Is my claim valid ? Why cannot those two original
> modes serve the purpose and why is the separation (of KNOMAP and KSMAP)
> needed?

Because the QEMU TLB just has a single bit for "is this page readable".
 In supervisor mode and with SMAP enabled, this changes depending on the
value of the AC bit.  Without separate TLBs for KNOSMAP/KSMAP, you would
have to flush the TLB on every CLAC or STAC instruction.

Paolo

Re: [Qemu-devel] On x86 MMU modes

[Sandhya Kumar]

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

Thanks for your mail.  Are these TLB modes logic specific to QEMU
implementation for x86?
Asking this as I am not able to get any information about seperate TLBs
from Intel developer manuals

On Wed, Jun 3, 2015 at 3:01 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:

>
>
> On 03/06/2015 08:51, Sandhya Kumar wrote:
> > As per my understanding (which matches versions prior to this commit),
> > we generally maintain only two TLBs [one for kernel and one for user] in
> > x86 ISA for caching address translations. With this commit we seem to
> > have three modes of MMU, although only two will be actually used (either
> > KSMAP or KNOSMAP).
>
> This is not accurate.  If AC=0, data accesses from the kernel use
> KNOSMAP, but implicit accesses (e.g. reads of the IDT) use KSMAP.
>
> > Is my claim valid ? Why cannot those two original
> > modes serve the purpose and why is the separation (of KNOMAP and KSMAP)
> > needed?
>
> Because the QEMU TLB just has a single bit for "is this page readable".
>  In supervisor mode and with SMAP enabled, this changes depending on the
> value of the AC bit.  Without separate TLBs for KNOSMAP/KSMAP, you would
> have to flush the TLB on every CLAC or STAC instruction.
>
> Paolo
>

[-- Attachment #2: Type: text/html, Size: 1736 bytes --]

Re: [Qemu-devel] On x86 MMU modes

[Paolo Bonzini]

On 03/06/2015 09:41, Sandhya Kumar wrote:
> Thanks for your mail.  Are these TLB modes logic specific to QEMU
> implementation for x86? 

Yes, they are specific to QEMU.

> Asking this as I am not able to get any information about seperate TLBs
> from Intel developer manuals

Real hardware TLBs probably work in a completely different (and
undocumented) way.

Paolo

Re: [Qemu-devel] On x86 MMU modes

[Sandhya Kumar]

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]

Thanks again. One more question.

On versions prior to the mentioned commit, is there any specific reason (in
x86 source code of QEMU) to choose separate modes for address translations
(of kernel and user mode)? Or was that done just for performance
improvement?



On Wed, Jun 3, 2015 at 3:58 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:

>
>
> On 03/06/2015 09:41, Sandhya Kumar wrote:
> > Thanks for your mail.  Are these TLB modes logic specific to QEMU
> > implementation for x86?
>
> Yes, they are specific to QEMU.
>
> > Asking this as I am not able to get any information about seperate TLBs
> > from Intel developer manuals
>
> Real hardware TLBs probably work in a completely different (and
> undocumented) way.
>
> Paolo
>

[-- Attachment #2: Type: text/html, Size: 1285 bytes --]

Re: [Qemu-devel] On x86 MMU modes

[Paolo Bonzini]

On 03/06/2015 10:07, Sandhya Kumar wrote:
> Thanks again. One more question.
> 
> On versions prior to the mentioned commit, is there any specific reason
> (in x86 source code of QEMU) to choose separate modes for address
> translations (of kernel and user mode)? Or was that done just for
> performance improvement?

It's because pages can be accessible only from kernel space, so the
protection bits for the pages can be different in the two TLBs.

Paolo

Re: [Qemu-devel] On x86 MMU modes

[Sandhya Kumar]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

Well, I think we can also achieve this like adding a flag in the structure
of CPUTLBEntry.
Am I missing something?


On Wed, Jun 3, 2015 at 4:22 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:

>
>
> On 03/06/2015 10:07, Sandhya Kumar wrote:
> > Thanks again. One more question.
> >
> > On versions prior to the mentioned commit, is there any specific reason
> > (in x86 source code of QEMU) to choose separate modes for address
> > translations (of kernel and user mode)? Or was that done just for
> > performance improvement?
>
> It's because pages can be accessible only from kernel space, so the
> protection bits for the pages can be different in the two TLBs.
>
> Paolo
>

[-- Attachment #2: Type: text/html, Size: 1201 bytes --]

Re: [Qemu-devel] On x86 MMU modes

[Peter Maydell]

On 3 June 2015 at 10:24, Sandhya Kumar <insatiablecuriousity07@gmail.com> wrote:
> Well, I think we can also achieve this like adding a flag in the structure
> of CPUTLBEntry.
> Am I missing something?

The point of the TLB data structure is to allow very fast access
in the common case of "TLB hit to guest RAM". If we had extra
flags to check in this code path it would slow it down. At the
moment the code only needs to look up the entry in the TLB
for the mmu_index it wants, and if it finds a hit it knows that
it is valid.

-- PMM

Re: [Qemu-devel] On x86 MMU modes

[Sandhya Kumar]

[-- Attachment #1: Type: text/plain, Size: 2057 bytes --]

Thanks Peter for your explanation.

[The following question on TLB working could be a deviation from the first
mail here, but asking here instead of starting new thread.]

I picked up a simple 'Hello world' ELF executable (shown at the end) and
tried to experiment with QEMU's address translations (i.e. guest VA -> host
VA in *softmmu_template.h*) occurring in userland for that process. This is
the sequence of guest VA (in hexadecimal) being translated:

*401bee*
*401c07*
*401c0e*
*401c13*
401d23
401d39
402009
...... and so on

The *italized* ones (first four) belong to *_start* of my executable and
the next few can be traced to *__libc_start_main *in my executable*. *Can
anyone please help me understand why the order is appearing like this?
Intuitively, I would guess to be as in every instruction of executable
(401bee, 401bf0 etc). Also there was no context switch during the above
which rules out TLB getting flushed at random time points. Let me know if
you need more information.


./hello:     file format elf64-x86-64

Disassembly of section .text:

0000000000401bee <_start>:
  401bee:       31 ed                   xor    %ebp,%ebp
  401bf0:       49 89 d1                mov    %rdx,%r9
  401bf3:       5e                      pop    %rsi
  401bf4:       48 89 e2                mov    %rsp,%rdx
  401bf7:       48 83 e4 f0             and    $0xfffffffffffffff0,%rsp

...... and so on



On Wed, Jun 3, 2015 at 5:36 PM, Peter Maydell <peter.maydell@linaro.org>
wrote:

> On 3 June 2015 at 10:24, Sandhya Kumar <insatiablecuriousity07@gmail.com>
> wrote:
> > Well, I think we can also achieve this like adding a flag in the
> structure
> > of CPUTLBEntry.
> > Am I missing something?
>
> The point of the TLB data structure is to allow very fast access
> in the common case of "TLB hit to guest RAM". If we had extra
> flags to check in this code path it would slow it down. At the
> moment the code only needs to look up the entry in the TLB
> for the mmu_index it wants, and if it finds a hit it knows that
> it is valid.
>
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 3027 bytes --]

Re: [Qemu-devel] On x86 MMU modes

[Peter Maydell]

On 6 June 2015 at 08:36, Sandhya Kumar <insatiablecuriousity07@gmail.com> wrote:
> Thanks Peter for your explanation.
>
> [The following question on TLB working could be a deviation from the first
> mail here, but asking here instead of starting new thread.]
>
> I picked up a simple 'Hello world' ELF executable (shown at the end) and
> tried to experiment with QEMU's address translations (i.e. guest VA -> host
> VA in softmmu_template.h) occurring in userland for that process. This is
> the sequence of guest VA (in hexadecimal) being translated:
>
> 401bee
> 401c07
> 401c0e
> 401c13
> 401d23
> 401d39
> 402009
> ...... and so on
>
> The italized ones (first four) belong to _start of my executable and the
> next few can be traced to __libc_start_main in my executable. Can anyone
> please help me understand why the order is appearing like this?

Most code loads don't go through the softmmu_template.h code. The
frontend (target-*/translate.c) calls cpu_ld*_code functions, which
are implemented by macros in include/exec/cpu_ldst_template.h. Those
functions will try to do a direct lookup in QEMU's TLB first, and will
only call the helper functions in softmmu_template.h if they miss.
So you're not going to see a call for every instruction. (My guess is
you're seeing one call every basic block, but it's not possible to tell
from the detail you give.)

-- PMM

Re: [Qemu-devel] On x86 MMU modes

[Sandhya Kumar]

[-- Attachment #1: Type: text/plain, Size: 3211 bytes --]

Thanks Peter for your response. I notice that *tlb_fill()*  is happening
only in *softmmu_template.h *and not anywhere else in code base. This means
I should expect the TLB itself to be populated here for other code loads to
have successful look up later. Am I wrong with my understanding?

Even I guessed TLB to be fetching basic block (i.e. chunk of code with
single entry and exit), but realized its not and hence I posted the
question here. If we look at the sequence mentioned earlier in thread (i.e.
401bee , 401c07, 401c0e, 401c13)
401c07 and 401c0e forms a pattern - it is from the immediate value after
"mov" opcode, modRM bytes in lines 10, 11. I also checked other "mov"
lines. This pattern appears to match everywhere *expect* for the "mov" in
line 9 .

Let me know if you need more information.

[My executable]

0000000000401bee <_start>:
  401bee:       31 ed                   xor    %ebp,%ebp
  401bf0:       49 89 d1                mov    %rdx,%r9
  401bf3:       5e                      pop    %rsi
  401bf4:       48 89 e2                mov    %rsp,%rdx
  401bf7:       48 83 e4 f0             and    $0xfffffffffffffff0,%rsp
  401bfb:       50                      push   %rax
  401bfc:       54                      push   %rsp
  401bfd:       49 c7 c0 20 24 40 00    mov    $0x402420,%r8
  // [Line 9]
  401c04:       48 c7 c1 90 23 40 00    mov    $0x402390,%rcx
// [Line 10]
  401c0b:       48 c7 c7 fe 1c 40 00    mov    $0x401cfe,%rdi
   // [Line 11]
  401c12:       e8 09 01 00 00          callq  401d20 <__libc_start_main>
  401c17:       f4                      hlt
  401c18:       0f 1f 84 00 00 00 00    nopl   0x0(%rax,%rax,1)
  401c1f:       00



On Sun, Jun 7, 2015 at 6:34 AM, Peter Maydell <peter.maydell@linaro.org>
wrote:

> On 6 June 2015 at 08:36, Sandhya Kumar <insatiablecuriousity07@gmail.com>
> wrote:
> > Thanks Peter for your explanation.
> >
> > [The following question on TLB working could be a deviation from the
> first
> > mail here, but asking here instead of starting new thread.]
> >
> > I picked up a simple 'Hello world' ELF executable (shown at the end) and
> > tried to experiment with QEMU's address translations (i.e. guest VA ->
> host
> > VA in softmmu_template.h) occurring in userland for that process. This is
> > the sequence of guest VA (in hexadecimal) being translated:
> >
> > 401bee
> > 401c07
> > 401c0e
> > 401c13
> > 401d23
> > 401d39
> > 402009
> > ...... and so on
> >
> > The italized ones (first four) belong to _start of my executable and the
> > next few can be traced to __libc_start_main in my executable. Can anyone
> > please help me understand why the order is appearing like this?
>
> Most code loads don't go through the softmmu_template.h code. The
> frontend (target-*/translate.c) calls cpu_ld*_code functions, which
> are implemented by macros in include/exec/cpu_ldst_template.h. Those
> functions will try to do a direct lookup in QEMU's TLB first, and will
> only call the helper functions in softmmu_template.h if they miss.
> So you're not going to see a call for every instruction. (My guess is
> you're seeing one call every basic block, but it's not possible to tell
> from the detail you give.)
>
> -- PMM
>

[-- Attachment #2: Type: text/html, Size: 4395 bytes --]