Re: [RFC v2] xSplice design

[Lars Kurth <lars.kurth@citrix.com>]

Adding Don Slutz as he requested to be added
Lars

On 15/05/2015 20:44, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com>
wrote:

>Hey!
>
>During the Xen Hacka^H^H^H^HProject Summit? we chatted about live-patching
>the hypervisor. We sketched out how it could be done, and brainstormed
>some of the problems.
>
>I took that and wrote an design - which is very much RFC. The design is
>laid out in two sections - the format of the ELF payload - and then the
>hypercalls to act on it.
>
>Hypercall preemption has caused a couple of XSAs so I've baked the need
>for that in the design so we hopefully won't have an XSA for this code.
>
>There are two big *TODO* in the design which I had hoped to get done
>before sending this out - however I am going on vacation for two weeks
>so I figured it would be better to send this off for folks to mull now
>then to have it languish.
>
>Please feel free to add more folks on the CC list.
>
>Enjoy!
>
>
># xSplice Design v1 (EXTERNAL RFC v2)
>
>## Rationale
>
>A mechanism is required to binarily patch the running hypervisor with new
>opcodes that have come about due to primarily security updates.
>
>This document describes the design of the API that would allow us to
>upload to the hypervisor binary patches.
>
>## Glossary
>
> * splice - patch in the binary code with new opcodes
> * trampoline - a jump to a new instruction.
> * payload - telemetries of the old code along with binary blob of the new
>   function (if needed).
> * reloc - telemetries contained in the payload to construct proper
>trampoline.
>
>## Multiple ways to patch
>
>The mechanism needs to be flexible to patch the hypervisor in multiple
>ways
>and be as simple as possible. The compiled code is contiguous in memory
>with
>no gaps - so we have no luxury of 'moving' existing code and must either
>insert a trampoline to the new code to be executed - or only modify
>in-place
>the code if there is sufficient space. The placement of new code has to
>be done
>by hypervisor and the virtual address for the new code is allocated
>dynamically.
>i
>This implies that the hypervisor must compute the new offsets when
>splicing
>in the new trampoline code. Where the trampoline is added (inside
>the function we are patching or just the callers?) is also important.
>
>To lessen the amount of code in hypervisor, the consumer of the API
>is responsible for identifying which mechanism to employ and how many
>locations
>to patch. Combinations of modifying in-place code, adding trampoline, etc
>has to be supported. The API should allow read/write any memory within
>the hypervisor virtual address space.
>
>We must also have a mechanism to query what has been applied and a
>mechanism
>to revert it if needed.
>
>We must also have a mechanism to: provide an copy of the old code - so
>that
>the hypervisor can verify it against the code in memory; the new code;
>the symbol name of the function to be patched; or offset from the symbol;
>or virtual address.
>
>The complications that this design will encounter are explained later
>in this document.
>
>## Patching code
>
>The first mechanism to patch that comes in mind is in-place replacement.
>That is replace the affected code with new code. Unfortunately the x86
>ISA is variable size which places limits on how much space we have
>available
>to replace the instructions.
>
>The second mechanism is by replacing the call or jump to the
>old function with the address of the new function.
>
>A third mechanism is to add a jump to the new function at the
>start of the old function.
>
>### Example of trampoline and in-place splicing
>
>As example we will assume the hypervisor does not have XSA-132 (see
>*domctl/sysctl: don't leak hypervisor stack to toolstacks*
>4ff3449f0e9d175ceb9551d3f2aecb59273f639d) and we would like to binary
>patch
>the hypervisor with it. The original code looks as so:
>
><pre>
>   48 89 e0                  mov    %rsp,%rax
>   48 25 00 80 ff ff         and    $0xffffffffffff8000,%rax
></pre>
>
>while the new patched hypervisor would be:
>
><pre>
>   48 c7 45 b8 00 00 00 00   movq   $0x0,-0x48(%rbp)
>   48 c7 45 c0 00 00 00 00   movq   $0x0,-0x40(%rbp)
>   48 c7 45 c8 00 00 00 00   movq   $0x0,-0x38(%rbp)
>   48 89 e0                  mov    %rsp,%rax
>   48 25 00 80 ff ff         and    $0xffffffffffff8000,%rax
></pre>
>
>This is inside the arch_do_domctl. This new change adds 21 extra
>bytes of code which alters all the offsets inside the function. To alter
>these offsets and add the extra 21 bytes of code we might not have enough
>space in .text to squeze this in.
>
>As such we could simplify this problem by only patching the site
>which calls arch_do_domctl:
>
><pre>
><do_domctl>:  
> e8 4b b1 05 00          callq  ffff82d08015fbb9 <arch_do_domctl>
></pre>
>
>with a new address for where the new `arch_do_domctl` would be (this
>area would be allocated dynamically).
>
>Astute readers will wonder what we need to do if we were to patch
>`do_domctl`
>- which is not called directly by hypervisor but on behalf of the guests
>via
>the `compat_hypercall_table` and `hypercall_table`.
>Patching the offset in `hypercall_table` for `do_domctl:
>(ffff82d080103079 <do_domctl>:)
><pre>
>
> ffff82d08024d490:   79 30
> ffff82d08024d492:   10 80 d0 82 ff ff
>
></pre>
>with the new address where the new `do_domctl` is possible. The other
>place where it is used is in `hvm_hypercall64_table` which would need
>to be patched in a similar way. This would require an in-place splicing
>of the new virtual address of `arch_do_domctl`.
>
>In summary this example patched the callee of the affected function by
> * allocating memory for the new code to live in,
> * changing the virtual address of all the functions which called the old
>   code (computing the new offset, patching the callq with a new callq).
> * changing the function pointer tables with the new virtual address of
>   the function (splicing in the new virtual address). Since this table
>   resides in the .rodata section we would need to temporarily change the
>   page table permissions during this part.
>
>
>However it has severe drawbacks - the safety checks which have to make
>sure
>the function is not on the stack - must also check every caller. For some
>patches this could if there were an sufficient large amount of callers
>that we would never be able to apply the update.
>
>### Example of different trampoline patching.
>
>An alternative mechanism exists where we can insert an trampoline in the
>existing function to be patched to jump directly to the new code. This
>lessens the locations to be patched to one but it puts pressure on the
>CPU branching logic (I-cache, but it is just one unconditional jump).
>
>For this example we will assume that the hypervisor has not been compiled
>with fe2e079f642effb3d24a6e1a7096ef26e691d93e (XSA-125: *pre-fill
>structures
>for certain HYPERVISOR_xen_version sub-ops*) which mem-sets an structure
>in `xen_version` hypercall. This function is not called **anywhere** in
>the hypervisor (it is called by the guest) but referenced in the
>`compat_hypercall_table` and `hypercall_table` (and indirectly called
>from that). Patching the offset in `hypercall_table` for the old
>`do_xen_version` (ffff82d080112f9e <do_xen_version>)
>
></pre>
> ffff82d08024b270 <hypercall_table>
> ...  
> ffff82d08024b2f8:   9e 2f 11 80 d0 82 ff ff
>
></pre>
>with the new address where the new `do_xen_version` is possible. The other
>place where it is used is in `hvm_hypercall64_table` which would need
>to be patched in a similar way. This would require an in-place splicing
>of the new virtual address of `do_xen_version`.
>
>An alternative solution would be to patch insert an trampoline in the
>old `do_xen_version' function to directly jump to the new
>`do_xen_version`.
>
><pre>
> ffff82d080112f9e <do_xen_version>:
> ffff82d080112f9e:       48 c7 c0 da ff ff ff    mov
>$0xffffffffffffffda,%rax
> ffff82d080112fa5:       83 ff 09                cmp    $0x9,%edi
> ffff82d080112fa8:       0f 87 24 05 00 00       ja     ffff82d0801134d2
><do_xen_version+0x534>
></pre>
>
>with:
>
><pre>
> ffff82d080112f9e <do_xen_version>:
> ffff82d080112f9e:       e9 XX YY ZZ QQ          jmpq   [new
>do_xen_version]  
></pre>
>
>which would lessen the amount of patching to just one location.
>
>In summary this example patched the affected function to jump to the
>new replacement function which required:
> * allocating memory for the new code to live in,
> * inserting trampoline with new offset in the old function to point to
>the
>   new function.
> * Optionally we can insert in the old function an trampoline jump to an
>function
>   providing an BUG_ON to catch errant code.
>
>The disadvantage of this are that the unconditional jump will consume a
>small
>I-cache penalty. However the simplicity of the patching of safety checks
>make this a worthwhile option.
>
>### Security
>
>With this method we can re-write the hypervisor - and as such we **MUST**
>be
>diligent in only allowing certain guests to perform this operation.
>
>Furthermore with SecureBoot or tboot, we **MUST** also verify the
>signature
>of the payload to be certain it came from a trusted source.
>
>As such the hypercall **MUST** support an XSM policy to limit the what
>guest is allowed. If the system is booted with signature checking the
>signature checking will be enforced.
>
>## Payload format
>
>The payload **MUST** contain enough data to allow us to apply the update
>and also safely reverse it. As such we **MUST** know:
>
> * What the old code is expected to be. We **MUST** verify it against the
>   runtime code.
> * The locations in memory to be patched. This can be determined
>dynamically
>   via symbols or via virtual addresses.
> * The new code to be used.
> * Signature to verify the payload.
>
>This binary format can be constructed using an custom binary format but
>there are severe disadvantages of it:
>
> * The format might need to be change and we need an mechanism to
>accommodate
>   that.
> * It has to be platform agnostic.
> * Easily constructed using existing tools.
>
>As such having the payload in an ELF file is the sensible way. We would be
>carrying the various set of structures (and data) in the ELF sections
>under
>different names and with definitions. The prefix for the ELF section name
>would always be: *.xsplice_*
>
>Note that every structure has padding. This is added so that the
>hypervisor
>can re-use those fields as it sees fit.
>
>There are five sections *.xsplice_* sections:
>
> * `.xsplice_symbols` and `.xsplice_str`. The array of symbols to be
>referenced
>   during the update. This can contain the symbols (functions) that will
>be
>   patched, or the list of symbols (functions) to be checked pre-patching
>which
>   may not be on the stack.
>
>* `.xsplice_reloc` and `.xsplice_reloc_howto`. The howto properly
>construct
>   trampolines for an patch. We can have multiple locations for which we
>   need to insert an trampoline for a payload and each location might
>require
>   a different way of handling it. This would naturally reference the
>`.text`
>   section and its proper offset. The `.xsplice_reloc` is not directly
>concerned
>   with patches but rather is an ELF relocation - describing the target
>   of a relocation and how that is performed.  They're also used for where
>   the new code references the run code too.
>
> * `.xsplice_sections`. The safety data for the old code and new code.
>   This contains an array of symbols (pointing to `.xsplice_symbols` to
>   and `.text`) which are to be used during safety and dependency
>checking.
>
>
> * `.xsplice_patches`: The description of the new functions to be patched
>   in (size, type, pointer to code, etc.).
>
> * `.xsplice_change`. The structure that ties all of this together and
>defines
>   the payload.
>
>Additionally the ELF file would contain:
>
> * `.text` section for the new and old code (function).
> * `.rela.text` relocation data for the `.text` (both new and old).
> * `.rela.xsplice_patches` relocation data for `.xsplice_patches` (such
>as offset
>   to the `.text` ,`.xsplice_symbols`, or `.xsplice_reloc` section).
> * `.bss` section for the new code (function)
> * `.data` and `.data.read_mostly` section for the new and old code
>(function)
> * `.rodata` section for the new and old code (function).
>
>In short the *.xsplice_* sections represent various structures and the
>ELF provides the mechanism to glue it all together when loaded in memory.
>
>Note that a lot of these ideas are borrowed from kSplice which is
>available at: https://github.com/jirislaby/ksplice
>
>For ELF understanding the best starting point is the OSDev Wiki
>(http://wiki.osdev.org/ELF). Furthermore the ELF specification is
>at http://www.skyfree.org/linux/references/ELF_Format.pdf and
>at Oracle's web site:
>http://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-46512.html#scro
>lltoc
>
>### ASCII art of the ELF structures
>
>*TODO*: Include an ASCII art of how the sections are tied together.
>
>### xsplice_symbols
>
>The section contains an array of an structure that outlines the name
>of the symbol to be patched (or checked against). The structure is
>as follow:
>
><pre>
>struct xsplice_symbol {
>    const char *name; /* The ELF name of the symbol. */
>    const char *label; /* A unique xSplice name for the symbol. */
>    uint8_t pad[16]; /* Must be zero. */
>};  
></pre>
>The structures may be in the section in any order and in any amount
>(duplicate entries are permitted).
>
>Both `name` and `label` would be pointing to entries in `.xsplice_str`.
>
>The `label` is used for diagnostic purposes - such as including the
>name and the offset.
>
>### xsplice_reloc and xsplice_reloc_howto
>
>The section contains an array of a structure that outlines the different
>locations (and howto) for which an trampoline is to be inserted.
>
>The howto defines in the detail the change. It contains the type,
>whether the relocation is relative, the size of the relocation,
>bitmask for which parts of the instruction or data are to be replaced,
>amount of final relocation is shifted by (to drop unwanted data), and
>whether the replacement should be interpreted as signed value.
>
>The structure is as follow:
>
><pre>
>#define XSPLICE_HOWTO_RELOC_INLINE  0 /* Inline replacement. */
>#define XSPLICE_HOWTO_RELOC_PATCH   1 /* Add trampoline. */
>#define XSPLICE_HOWTO_RELOC_DATA    2 /*  __DATE__ type change. */
>#define XSPLICE_HOWTO_RELOC_TIME    3 /* __TIME__ type chnage. */
>#define XSPLICE_HOWTO_BUG           4 /* BUG_ON being replaced.*/
>#define XSPLICE_HOWTO_EXTABLE       5 /* exception_table change. */
>#define XSPLICE_HOWTO_SYMBOL        6 /* change in symbol table. */
>
>#define XSPLICE_HOWTO_FLAG_PC_REL    0x00000001 /* Is PC relative. */
>#define XSPLICE_HOWOT_FLAG_SIGN      0x00000002 /* Should the new value
>be treated as signed value. */
>
>struct xsplice_reloc_howto {
>    uint32_t    type; /* XSPLICE_HOWTO_* */
>    uint32_t    flag; /* XSPLICE_HOWTO_FLAG_* */
>    uint32_t    size; /* Size, in bytes, of the item to be relocated. */
>    uint32_t    r_shift; /* The value the final relocation is shifted
>right by; used to drop unwanted data from the relocation. */
>    uint64_t    mask; /* Bitmask for which parts of the instruction or
>data are replaced with the relocated value. */
>    uint8_t     pad[8]; /* Must be zero. */
>};  
>
></pre>
>
>This structure is used in:
>
><pre>
>struct xsplice_reloc {
>    uint64_t addr; /* The address of the relocation (if known). */
>    struct xsplice_symbol *symbol; /* Symbol for this relocation. */
>    struct xsplice_reloc_howto  *howto; /* Pointer to the above
>structure. */  
>    uint64_t isns_added; /* ELF addend resulting from quirks of
>instruction one of whose operands is the relocation. For example, this is
>-4 on x86 pc-relative jumps. */
>    uint64_t isns_target; /* rest of the ELF addend.  This is equal to
>the offset against the symbol that the relocation refers to. */
>    uint8_t pad[8];  /* Must be zero. */
>};  
></pre>
>
>### xsplice_sections
>
>The structure defined in this section is used to verify that it is safe
>to update with the new changes. It can contain safety data on the old code
>and what kind of matching we are to expect.
>
>It also can contain safety date of what to check when about to patch.
>That is whether any of the addresses (either provided or resolved
>when payload is loaded by referencing the symbols) are in memory
>with what we expect it to be.
>
>As such the flags can be or-ed together:
>
><pre>
>#define XSPLICE_SECTION_TEXT   0x00000001 /* Section is in .text */
>#define XSPLICE_SECTION_RODATA 0x00000002 /* Section is in .ro */
>#define XSPLICE_SECTION_DATA   0x00000004 /* Section is in .rodata */
>#define XSPLICE_SECTION_STRING 0x00000008 /* Section is in .str */
>#define XSPLICE_SECTION_ALTINSTRUCTIONS 0x00000010 /* Section has
>.altinstructions. */
>#define XSPLICE_SECTION_TEXT_INPLACE 0x00000200 /* Change is in place. */
>  
>#dekine XSPLICE_SECTION_MATCH_EXACT 0x00000400 /* Must match exactly. */
>#define XSPLICE_SECTION_NO_STACKCHECK 0x00000800 /* Do not check the
>stack. */  
>
>struct xsplice_section {
>    struct xsplice_symbol *symbol; /* The symbol associated with this
>change. */  
>    uint64_t address; /* The address of the section (if known). */
>    uint64_t size; /* The size of the section. */
>    uint64_t flags; /* Various XSPLICE_SECTION_* flags. */
>    uint8_t pad[16]; /* To be zero. */
>};
>
></pre>
>
>### xsplice_patches
>
>Within this section we have an array of a structure defining the new code
>(patch).
>
>This structure consist of an pointer to the new code (which in ELF ends up
>pointing to an offset in `.text` or `.data` section); the type of patch:
>inline - either text or data, or requesting an trampoline; and size of
>patch.
>
>The structure is as follow:
>
><pre>
>#define XSPLICE_PATCH_INLINE_TEXT   0
>#define XSPLICE_PATCH_INLINE_DATA   1
>#define XSPLICE_PATCH_RELOC_TEXT    2
>
>struct xsplice_patch {
>    uint32_t type; /* XSPLICE_PATCH_* .*/
>    uint32_t size; /* Size of patch. */
>    uint64_t addr; /* The address of the new code (or data). */
>    void *content; /* The bytes to be installed. */
>    uint8_t pad[16]; /* Must be zero. */
>};
>
></pre>
>
>### xsplice_code
>
>The structure embedded within this section ties it all together.
>It has the name of the patch, and pointers to all the above
>mentioned structures (the start and end addresses).
>
>The structure is as follow:
>
><pre>
>struct xsplice_code {
>    const char *name; /* A sensible name for the patch. Up to 40
>characters. */  
>    struct xsplice_reloc *relocs, *relocs_end; /* How to patch it */
>    struct xsplice_section *sections, *sections_end; /* Safety data */
>    struct xsplice_patch *patches, *patches_end; /* Patch code & data */
>    uint8_t pad[32]; /* Must be zero. */
>};
></pre>
>
>There should only be one such structure in the section.
>
>### Example
>
>*TODO*: Include an objdump of how the ELF would look like for the XSA
>mentioned earlier.
>
>## Signature checking requirements.
>
>The signature checking requires that the layout of the data in memory
>**MUST** be same for signature to be verified. This means that the payload
>data layout in ELF format **MUST** match what the hypervisor would be
>expecting such that it can properly do signature verification.
>
>The signature is based on the all of the payloads continuously laid out
>in memory. The signature is to be appended at the end of the ELF payload
>prefixed with the string '~Module signature appended~\n", followed by
>an signature header then followed by the signature, key identifier, and
>signers
>name.
>
>Specifically the signature header would be:
>
><pre>
>#define PKEY_ALGO_DSA       0
>#define PKEY_ALGO_RSA       1
>
>#define PKEY_ID_PGP         0 /* OpenPGP generated key ID */
>#define PKEY_ID_X509        1 /* X.509 arbitrary subjectKeyIdentifier */
>
>#define HASH_ALGO_MD4          0
>#define HASH_ALGO_MD5          1
>#define HASH_ALGO_SHA1         2
>#define HASH_ALGO_RIPE_MD_160  3
>#define HASH_ALGO_SHA256       4
>#define HASH_ALGO_SHA384       5
>#define HASH_ALGO_SHA512       6
>#define HASH_ALGO_SHA224       7
>#define HASH_ALGO_RIPE_MD_128  8
>#define HASH_ALGO_RIPE_MD_256  9
>#define HASH_ALGO_RIPE_MD_320 10
>#define HASH_ALGO_WP_256      11
>#define HASH_ALGO_WP_384      12
>#define HASH_ALGO_WP_512      13
>#define HASH_ALGO_TGR_128     14
>#define HASH_ALGO_TGR_160     15
>#define HASH_ALGO_TGR_192     16
>
>
>struct elf_payload_signature {
>	u8	algo;		/* Public-key crypto algorithm PKEY_ALGO_*. */
>	u8	hash;		/* Digest algorithm: HASH_ALGO_*. */
>	u8	id_type;	/* Key identifier type PKEY_ID*. */
>	u8	signer_len;	/* Length of signer's name */
>	u8	key_id_len;	/* Length of key identifier */
>	u8	__pad[3];  
>	__be32	sig_len;	/* Length of signature data */
>};
>
></pre>
>(Note that this has been borrowed from Linux module signature code.).
>
>
>## Hypercalls
>
>We will employ the sub operations of the system management hypercall
>(sysctl).
>There are to be four sub-operations:
>
> * upload the payloads.
> * listing of payloads summary uploaded and their state.
> * getting an particular payload summary and its state.
> * command to apply, delete, or revert the payload.
>
>The patching is asynchronous therefore the caller is responsible
>to verify that it has been applied properly by retrieving the summary of
>it
>and verifying that there are no error codes associated with the payload.
>
>We **MUST** make it asynchronous due to the nature of patching: it
>requires
>every physical CPU to be lock-step with each other. The patching mechanism
>while an implementation detail, is not an short operation and as such
>the design **MUST** assume it will be an long-running operation.
>
>Furthermore it is possible to have multiple different payloads for the
>same
>function. As such an unique id has to be visible to allow proper
>manipulation.
>
>The hypercall is part of the `xen_sysctl`. The top level structure
>contains
>one uint32_t to determine the sub-operations:
>
><pre>
>struct xen_sysctl_xsplice_op {
>    uint32_t cmd; 
>	union {  
>          ... see below ...
>        } u;  
>};  
>
></pre>
>while the rest of hypercall specific structures are part of the this
>structure.
>
>
>### XEN_SYSCTL_XSPLICE_UPLOAD (0)
>
>Upload a payload to the hypervisor. The payload is verified and if there
>are any issues the proper return code will be returned. The payload is
>not applied at this time - that is controlled by
>*XEN_SYSCTL_XSPLICE_ACTION*.
>
>The caller provides:
>
> * `id` unique id.
> * `payload` the virtual address of where the ELF payload is.
>
>The return value is zero if the payload was succesfully uploaded and the
>signature was verified. Otherwise an EXX return value is provided.
>Duplicate `id` are not supported.
>
>The `payload` is the ELF payload as mentioned in the `Payload format`
>section.
>
>The structure is as follow:
>
><pre>
>struct xen_sysctl_xsplice_upload {
>    char id[40];  /* IN, name of the patch. */
>    uint64_t size; /* IN, size of the ELF file. */
>    XEN_GUEST_HANDLE_64(uint8) payload; /* ELF file. */
>}; 
></pre>
>
>### XEN_SYSCTL_XSPLICE_GET (1)
>
>Retrieve an summary of an specific payload. This caller provides:
>
> * `id` the unique id.
> * `status` *MUST* be set to zero.
> * `rc` *MUST* be set to zero.
>
>The `summary` structure contains an summary of payload which includes:
>
> * `id` the unique id.
> * `status` - whether it has been:
> 1. *XSPLICE_STATUS_LOADED* (0) has been loaded.
> 2. *XSPLICE_STATUS_PROGRESS* (1) acting on the
>**XEN_SYSCTL_XSPLICE_ACTION** command.
> 3. *XSPLICE_STATUS_CHECKED*  (2) the ELF payload safety checks passed.
> 4. *XSPLICE_STATUS_APPLIED* (3) loaded, checked, and applied.
> 5. *XSPLICE_STATUS_REVERTED* (4) loaded, checked, applied and then also
>reverted.
> 6. *XSPLICE_STATUS_IN_ERROR* (5) loaded and in a failed state. Consult
>`rc` for details.
> * `rc` - its error state if any.
>
>The structure is as follow:
>
><pre>
>#define XSPLICE_STATUS_LOADED    0
>#define XSPLICE_STATUS_PROGRESS  1
>#define XSPLICE_STATUS_CHECKED   2
>#define XSPLICE_STATUS_APPLIED   3
>#define XSPLICE_STATUS_REVERTED  4
>#define XSPLICE_STATUS_IN_ERROR  5
>
>struct xen_sysctl_xsplice_summary {
>    char id[40];  /* IN/OUT, name of the patch. */
>    uint32_t status;   /* OUT */
>    int32_t rc;  /* OUT */
>}; 
></pre>
>
>### XEN_SYSCTL_XSPLICE_LIST (2)
>
>Retrieve an array of abbreviated summary of payloads that are loaded in
>the
>hypervisor.
>
>The caller provides:
>
> * `idx` index iterator. Initially it *MUST* be zero.
> * `count` the max number of entries to populate.
> * `summary` virtual address of where to write payload summaries.
>
>The hypercall returns zero on success and updates the `idx` (index)
>iterator
>with the number of payloads returned, `count` to the number of remaining
>payloads, and `summary` with an number of payload summaries.
>
>If the hypercall returns E2BIG the `count` is too big and should be
>lowered.
>
>Note that due to the asynchronous nature of hypercalls the domain might
>have
>added or removed the number of payloads making this information stale. It
>is
>the responsibility of the domain to provide proper accounting.
>
>The `summary` structure contains an summary of payload which includes:
>
> * `id` unique id.
> * `status` - whether it has been:
> 1. *XSPLICE_STATUS_LOADED* (0) has been loaded.
> 2. *XSPLICE_STATUS_PROGRESS* (1) acting on the
>**XEN_SYSCTL_XSPLICE_ACTION** command.
> 3. *XSPLICE_STATUS_CHECKED*  (2) the payload `old` and `addr` match with
>the hypervisor.
> 4. *XSPLICE_STATUS_APPLIED* (3) loaded, checked, and applied.
> 5. *XSPLICE_STATUS_REVERTED* (4) loaded, checked, applied and then also
>reverted.
> 6. *XSPLICE_STATUS_IN_ERROR* (5) loaded and in a failed state. Consult
>`rc` for details.
> * `rc` - its error state if any.
>
>The structure is as follow:
>
><pre>
>struct xen_sysctl_xsplice_list {
>    uint32_t idx;  /* IN/OUT */
>    uint32_t count;  /* IN/OUT */
>    XEN_GUEST_HANDLE_64(xen_sysctl_xsplice_summary) summary;  /* OUT */
>};  
>
>struct xen_sysctl_xsplice_summary {
>    char id[40];  /* OUT, name of the patch. */
>    uint32_t status;   /* OUT */
>    int32_t rc;  /* OUT */
>};  
>
></pre>
>### XEN_SYSCTL_XSPLICE_ACTION (3)
>
>Perform an operation on the payload structure referenced by the `id`
>field.
>The operation request is asynchronous and the status should be retrieved
>by using either **XEN_SYSCTL_XSPLICE_GET** or **XEN_SYSCTL_XSPLICE_LIST**
>hypercall.
>
>The caller provides:
>
> * `id` the unique id.
> * `cmd` the command requested:
>  1. *XSPLICE_ACTION_CHECK* (0) check that the payload will apply
>properly.
>  2. *XSPLICE_ACTION_UNLOAD* (1) unload the payload.
>  3. *XSPLICE_ACTION_REVERT* (2) revert the payload.
>  4. *XSPLICE_ACTION_APPLY* (3) apply the payload.
>
>
>The return value will be zero unless the provided fields are incorrect.
>
>The structure is as follow:
>
><pre>
>#define XSPLICE_ACTION_CHECK  0
>#define XSPLICE_ACTION_UNLOAD 1
>#define XSPLICE_ACTION_REVERT 2
>#define XSPLICE_ACTION_APPLY  3
>
>struct xen_sysctl_xsplice_action {
>    char id[40];  /* IN, name of the patch. */
>    uint32_t cmd; /* IN */
>};  
>
></pre>
>
>## Sequence of events.
>
>The normal sequence of events is to:
>
> 1. *XEN_SYSCTL_XSPLICE_UPLOAD* to upload the payload. If there are
>errors *STOP* here.
> 2. *XEN_SYSCTL_XSPLICE_GET* to check the `->status`. If in
>*XSPLICE_STATUS_PROGRESS* spin. If in *XSPLICE_STATUS_LOADED* go to next
>step.
> 3. *XEN_SYSCTL_XSPLICE_ACTION* with *XSPLICE_ACTION_CHECK* command to
>verify that the payload can be succesfully applied.
> 4. *XEN_SYSCTL_XSPLICE_GET* to check the `->status`. If in
>*XSPLICE_STATUS_PROGRESS* spin. If in *XSPLICE_STATUS_CHECKED* go to next
>step.
> 5. *XEN_SYSCTL_XSPLICE_ACTION* with *XSPLICE_ACTION_APPLY* to apply the
>patch.
> 6. *XEN_SYSCTL_XSPLICE_GET* to check the `->status`. If in
>*XSPLICE_STATUS_PROGRESS* spin. If in *XSPLICE_STATUS_APPLIED* exit with
>success.
>
> 
>## Addendum
>
>Implementation quirks should not be discussed in a design document.
>
>However these observations can provide aid when developing against this
>document.
>
>
>### Alternative assembler
>
>Alternative assembler is a mechanism to use different instructions
>depending
>on what the CPU supports. This is done by providing multiple streams of
>code
>that can be patched in - or if the CPU does not support it - padded with
>`nop` operations. The alternative assembler macros cause the compiler to
>expand the code to place a most generic code in place - emit a special
>ELF .section header to tag this location. During run-time the hypervisor
>can leave the areas alone or patch them with an better suited opcodes.
>
>As we might be patching the alternative assembler sections as well - by
>providing a new better suited op-codes or perhaps with nops - we need to
>also re-run the alternative assembler patching after we have done our
>patching.
>
>Also when we are doing safety checks the code we are checking might be
>utilizing alternative assembler. As such we should relax out checks to
>accomodate that.
>
>### .rodata sections
>
>The patching might require strings to be updated as well. As such we must
>be
>also able to patch the strings as needed. This sounds simple - but the
>compiler
>has a habit of coalescing strings that are the same - which means if we
>in-place
>alter the strings - other users will be inadvertently affected as well.
>
>This is also where pointers to functions live - and we may need to patch
>this
>as well.
>
>To guard against that we must be prepared to do patching similar to
>trampoline patching or in-line depending on the flavour. If we can
>do in-line patching we would need to:
>
> * alter `.rodata` to be writeable.
> * inline patch.
> * alter `.rodata` to be read-only.
>
>If are doing trampoline patching we would need to:
>
> * allocate a new memory location for the string.
> * all locations which use this string will have to be updated to use the
>   offset to the string.
> * mark the region RO when we are done.
>
>### .bss sections
>
>Patching writable data is not suitable as it is unclear what should be
>done
>depending on the current state of data. As such it should not be
>attempted.
>
>
>### Patching code which is in the stack.
>
>We should not patch the code which is on the stack. That can lead
>to corruption.
>
>### Trampoline (e9 opcode)
>
>The e9 opcode used for jmpq uses a 32-bit signed displacement. That means
>we are limited to up to 2GB of virtual address to place the new code
>from the old code. That should not be a problem since Xen hypervisor has
>a very small footprint.
>
>However if we need - we can always add two trampolines. One at the 2GB
>limit that calls the next trampoline.
>
>### Time rendezvous code instead of stop_machine for patching
>
>The hypervisor's time rendezvous code runs synchronously across all CPUs
>every second. Using the stop_machine to patch can stall the time
>rendezvous
>code and result in NMI. As such having the patching be done at the tail
>of rendezvous code should avoid this problem.
>
>### Security
>
>Only the privileged domain should be allowed to do this operation.