Re: [Qemu-devel] [Qemu-discuss] ASan'ed binaries start up very slow under qemu-aarch64.

Re: [Qemu-devel] [Qemu-discuss] ASan'ed binaries start up very slow under qemu-aarch64.

[Peter Maydell]

(CCing qemu-devel, which is more likely to get developer attention)

On 18 July 2016 at 15:45, Maxim Ostapenko <m.ostapenko@samsung.com> wrote:
> 1) AddressSanitizer mmaps quite large regions of memory for redzones and
> shadow gap. In particular, for 39-bit AS it mmapes:
>
> || `[0x1400000000, 0x1fffffffff]` || HighShadow || - 48 Gb
> || `[0x1200000000, 0x13ffffffff]` || ShadowGap  || - 8 Gb
> || `[0x1000000000, 0x11ffffffff]` || LowShadow  || - 4 Gb
>
> 2) In QEMU, page_set_flags is called for these ranges. It cuts given range
> to individual pages and sets flags for them.  Given the page size is 4 Kb,
> for 8 Gb range we have 2097152 iterations and for 48 Gb 12582912 iterations
> in inner loop. This is obviously a performance bottleneck.

Mmm, the algorithm here is pretty simple and basically assumes the
guest isn't going to be doing enormous allocations like that.
(If the host process doesn't happen to have a suitable big lump of its
VA space free then the mmap will fail anyway.)

> 3) Same issue may happen when ASan tries to read /proc/self/map later in
> page_check_range function, after it already mmaped HighShadow, ShadowGap and
> LowShadow regions.
>
> Could someone help me, how can I mitigate this performance issue? Do we
> really need to set flags to each page on entire (quite big) memory region?

Well, we do need to do some things:
 * we're populating the PageDesc data structure which we later use
   to cache generated code
 * if we're marking the range as writeable and it wasn't previously
   writeable, we need to check whether there's already generated code
   anywhere in this memory range and invalidate those translations

This could probably be done in a way that doesn't iterate naively
through every page, though.

thanks
-- PMM

Re: [Qemu-devel] [Qemu-discuss] ASan'ed binaries start up very slow under qemu-aarch64.

[Maxim Ostapenko]

On 18/07/16 18:51, Peter Maydell wrote:
> (CCing qemu-devel, which is more likely to get developer attention)

Peter, thank you for your answer.

>
> On 18 July 2016 at 15:45, Maxim Ostapenko <m.ostapenko@samsung.com> wrote:
>> 1) AddressSanitizer mmaps quite large regions of memory for redzones and
>> shadow gap. In particular, for 39-bit AS it mmapes:
>>
>> || `[0x1400000000, 0x1fffffffff]` || HighShadow || - 48 Gb
>> || `[0x1200000000, 0x13ffffffff]` || ShadowGap  || - 8 Gb
>> || `[0x1000000000, 0x11ffffffff]` || LowShadow  || - 4 Gb
>>
>> 2) In QEMU, page_set_flags is called for these ranges. It cuts given range
>> to individual pages and sets flags for them.  Given the page size is 4 Kb,
>> for 8 Gb range we have 2097152 iterations and for 48 Gb 12582912 iterations
>> in inner loop. This is obviously a performance bottleneck.
> Mmm, the algorithm here is pretty simple and basically assumes the
> guest isn't going to be doing enormous allocations like that.
> (If the host process doesn't happen to have a suitable big lump of its
> VA space free then the mmap will fail anyway.)

Hm, it seems that ASan is really special here. Actually, I think that 
this slowdown is not critical for individual runs, but it certainly 
critical for people who rely on QEMU in their builds (e.g. in Aarch64 
chroot). Not sure it's a common case, though.

>
>> 3) Same issue may happen when ASan tries to read /proc/self/map later in
>> page_check_range function, after it already mmaped HighShadow, ShadowGap and
>> LowShadow regions.
>>
>> Could someone help me, how can I mitigate this performance issue? Do we
>> really need to set flags to each page on entire (quite big) memory region?
> Well, we do need to do some things:
>   * we're populating the PageDesc data structure which we later use
>     to cache generated code
>   * if we're marking the range as writeable and it wasn't previously
>     writeable, we need to check whether there's already generated code
>     anywhere in this memory range and invalidate those translations
>
> This could probably be done in a way that doesn't iterate naively
> through every page, though.

Oh, I see. Perhaps we can restrict QEMU to use some well defined pages 
for generated code?

Thanks,
-Maxim

>
> thanks
> -- PMM
>
>

Re: [Qemu-devel] [Qemu-discuss] ASan'ed binaries start up very slow under qemu-aarch64.

[Peter Maydell]

On 19 July 2016 at 10:22, Maxim Ostapenko <m.ostapenko@samsung.com> wrote:
> On 18/07/16 18:51, Peter Maydell wrote:
>> Well, we do need to do some things:
>>   * we're populating the PageDesc data structure which we later use
>>     to cache generated code
>>   * if we're marking the range as writeable and it wasn't previously
>>     writeable, we need to check whether there's already generated code
>>     anywhere in this memory range and invalidate those translations
>>
>> This could probably be done in a way that doesn't iterate naively
>> through every page, though.
>
>
> Oh, I see. Perhaps we can restrict QEMU to use some well defined pages for
> generated code?

I mean *guest* pages. We need to detect self-modifying guest code,
and the guest could do that for any page in its virtual address
space.

thanks
-- PMM