Re: [PATCH v5 6/7] tcg: implement JIT for iOS and Apple Silicon

[Alexander Graf <agraf@csgraf.de>]

On 20.11.20 16:58, Joelle van Dyne wrote:
> On Fri, Nov 20, 2020 at 3:08 AM Alexander Graf <agraf@csgraf.de> wrote:
>>
>> On 09.11.20 00:24, Joelle van Dyne wrote:
>>> When entitlements are available (macOS or jailbroken iOS), a hardware
>>> feature called APRR exists on newer Apple Silicon that can cheaply mark JIT
>>> pages as either RX or RW. Reverse engineered functions from
>>> libsystem_pthread.dylib are implemented to handle this.
>>>
>>> The following rules apply for JIT write protect:
>>>     * JIT write-protect is enabled before tcg_qemu_tb_exec()
>>>     * JIT write-protect is disabled after tcg_qemu_tb_exec() returns
>>>     * JIT write-protect is disabled inside do_tb_phys_invalidate() but if it
>>>       is called inside of tcg_qemu_tb_exec() then write-protect will be
>>>       enabled again before returning.
>>>     * JIT write-protect is disabled by cpu_loop_exit() for interrupt handling.
>>>     * JIT write-protect is disabled everywhere else.
>>>
>>> See https://developer.apple.com/documentation/apple_silicon/porting_just-in-time_compilers_to_apple_silicon
>>>
>>> Signed-off-by: Joelle van Dyne <j@getutm.app>
>>> ---
>>>    include/exec/exec-all.h     |  2 +
>>>    include/tcg/tcg-apple-jit.h | 86 +++++++++++++++++++++++++++++++++++++
>>>    include/tcg/tcg.h           |  3 ++
>>>    accel/tcg/cpu-exec-common.c |  2 +
>>>    accel/tcg/cpu-exec.c        |  2 +
>>>    accel/tcg/translate-all.c   | 46 ++++++++++++++++++++
>>>    tcg/tcg.c                   |  4 ++
>>>    7 files changed, 145 insertions(+)
>>>    create mode 100644 include/tcg/tcg-apple-jit.h
>>>
>>> diff --git a/include/exec/exec-all.h b/include/exec/exec-all.h
>>> index aa65103702..3829f3d470 100644
>>> --- a/include/exec/exec-all.h
>>> +++ b/include/exec/exec-all.h
>>> @@ -549,6 +549,8 @@ TranslationBlock *tb_htable_lookup(CPUState *cpu, target_ulong pc,
>>>                                       target_ulong cs_base, uint32_t flags,
>>>                                       uint32_t cf_mask);
>>>    void tb_set_jmp_target(TranslationBlock *tb, int n, uintptr_t addr);
>>> +void tb_exec_lock(void);
>>> +void tb_exec_unlock(void);
>>>
>>>    /* GETPC is the true target of the return instruction that we'll execute.  */
>>>    #if defined(CONFIG_TCG_INTERPRETER)
>>> diff --git a/include/tcg/tcg-apple-jit.h b/include/tcg/tcg-apple-jit.h
>>> new file mode 100644
>>> index 0000000000..9efdb2000d
>>> --- /dev/null
>>> +++ b/include/tcg/tcg-apple-jit.h
>>> @@ -0,0 +1,86 @@
>>> +/*
>>> + * Apple Silicon functions for JIT handling
>>> + *
>>> + * Copyright (c) 2020 osy
>>> + *
>>> + * This library is free software; you can redistribute it and/or
>>> + * modify it under the terms of the GNU Lesser General Public
>>> + * License as published by the Free Software Foundation; either
>>> + * version 2.1 of the License, or (at your option) any later version.
>>> + *
>>> + * This library is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>>> + * Lesser General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU Lesser General Public
>>> + * License along with this library; if not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#ifndef TCG_APPLE_JIT_H
>>> +#define TCG_APPLE_JIT_H
>>> +
>>> +/*
>>> + * APRR handling
>>> + * Credits to: https://siguza.github.io/APRR/
>>> + * Reversed from /usr/lib/system/libsystem_pthread.dylib
>>> + */
>>> +
>>> +#if defined(__aarch64__) && defined(CONFIG_DARWIN)
>>> +
>>> +#define _COMM_PAGE_START_ADDRESS        (0x0000000FFFFFC000ULL) /* In TTBR0 */
>>> +#define _COMM_PAGE_APRR_SUPPORT         (_COMM_PAGE_START_ADDRESS + 0x10C)
>>> +#define _COMM_PAGE_APPR_WRITE_ENABLE    (_COMM_PAGE_START_ADDRESS + 0x110)
>>> +#define _COMM_PAGE_APRR_WRITE_DISABLE   (_COMM_PAGE_START_ADDRESS + 0x118)
>>> +
>>> +static __attribute__((__always_inline__)) bool jit_write_protect_supported(void)
>>> +{
>>> +    /* Access shared kernel page at fixed memory location. */
>>> +    uint8_t aprr_support = *(volatile uint8_t *)_COMM_PAGE_APRR_SUPPORT;
>>> +    return aprr_support > 0;
>>> +}
>>> +
>>> +/* write protect enable = write disable */
>>> +static __attribute__((__always_inline__)) void jit_write_protect(int enabled)
>>> +{
>>> +    /* Access shared kernel page at fixed memory location. */
>>> +    uint8_t aprr_support = *(volatile uint8_t *)_COMM_PAGE_APRR_SUPPORT;
>>> +    if (aprr_support == 0 || aprr_support > 3) {
>>> +        return;
>>> +    } else if (aprr_support == 1) {
>>> +        __asm__ __volatile__ (
>>> +            "mov x0, %0\n"
>>> +            "ldr x0, [x0]\n"
>>> +            "msr S3_4_c15_c2_7, x0\n"
>>> +            "isb sy\n"
>>> +            :: "r" (enabled ? _COMM_PAGE_APRR_WRITE_DISABLE
>>> +                            : _COMM_PAGE_APPR_WRITE_ENABLE)
>>> +            : "memory", "x0"
>>> +        );
>>> +    } else {
>>> +        __asm__ __volatile__ (
>>> +            "mov x0, %0\n"
>>> +            "ldr x0, [x0]\n"
>>> +            "msr S3_6_c15_c1_5, x0\n"
>>> +            "isb sy\n"
>>> +            :: "r" (enabled ? _COMM_PAGE_APRR_WRITE_DISABLE
>>> +                            : _COMM_PAGE_APPR_WRITE_ENABLE)
>>> +            : "memory", "x0"
>>> +        );
>>> +    }
>>> +}
>>
>> Is there a particular reason you're not just calling
>> pthread_jit_write_protect_np()? That would remove the dependency on
>> anything reverse engineered.
> Those APIs are not available on iOS 13 or below, which has the same
> APRR requirements. If for legal reasons we cannot include this code,
> then it is fine to remove this file and replace the calls with the
> APIs, but we would lose support on lower iOS versions.


I don't think we realistically care about running QEMU on iOS13, do we? 
Let's just focus on making the code maintainable for anything going 
forward from now :).


Alex