RFC: another way to make livepatch

RFC: another way to make livepatch

[laokz]

Hi,

Is it off-topic talking about livepatch making tool? I tried another way and
want to get your expert opinion if there any fatal pitfall.

Thanks.

laokz

Re: RFC: another way to make livepatch

[Marcos Paulo de Souza]

On Thu, 2023-12-21 at 22:06 +0800, laokz wrote:
> Hi,
> 
> Is it off-topic talking about livepatch making tool? I tried another
> way and
> want to get your expert opinion if there any fatal pitfall.

I don't think it's out of scope, but what exactly you mean by "another
way to make livepatch"? You would like to know about different
approaches like source-based livepatch compared to kpatch, or you mean
something different?

Thanks!

> 
> Thanks.
> 
> laokz
> 
> 

Re: RFC: another way to make livepatch

[pizza]

On Wed, 2024-01-10 at 08:52 -0300, Marcos Paulo de Souza wrote:
> On Thu, 2023-12-21 at 22:06 +0800, laokz wrote:
> > Hi,
> > 
> > Is it off-topic talking about livepatch making tool? I tried another
> > way and
> > want to get your expert opinion if there any fatal pitfall.
> 
> I don't think it's out of scope, but what exactly you mean by "another

Thanks.

> way to make livepatch"? You would like to know about different
> approaches like source-based livepatch compared to kpatch, or you mean

Yes, exactly. Inspired by kpatch, I tried to make livepatch on source level
to avoid difficult binary hacking, just like what we did normal module
development. The main idea is similar to some topics of this mail list,

1. Write livepatch-sample.c stylish source code, put all needed non-
export/non-include symbols(I call them KLPSYMs) declarations in file.
2. Generate KLPSYMs position information through kallsyms etc.
3. `KBUILD_MODPOST_WARN=1 make` to build a "partial linked" .ko.
4. Use a klp-convert like tool to transform the KLPSYMs.

For simple patch, hand-write source might be easy though a little time
consuming. I used libclang to auto abbreviate livepatch source[1]. 

The main obstacle, IMO, is "how to determine non-included local symbols",
because they might be inlined, optimized out, duplicate names, mangled
names, and because kallsyms, vmlinux .symtab less some verbose information.
In my toy, I used DWARF along with kallsyms to try to verify all of them. 

But now I realized that DWARF might be the fatal pitfall. That's a pity. I
still get hope that's not too bad:)

[1] https://gitee.com/laokz/klpmake
[2] mirror: https://github.com/laokz/klpmake

> something different?
> 
> Thanks!
> 
> > 
> > Thanks.
> > 
> > laokz
> > 
> > 
> 
> 

Re: RFC: another way to make livepatch

[Marcos Paulo de Souza]

On Wed, 2024-01-10 at 22:06 +0800, pizza wrote:
> On Wed, 2024-01-10 at 08:52 -0300, Marcos Paulo de Souza wrote:
> > On Thu, 2023-12-21 at 22:06 +0800, laokz wrote:
> > > Hi,
> > > 
> > > Is it off-topic talking about livepatch making tool? I tried
> > > another
> > > way and
> > > want to get your expert opinion if there any fatal pitfall.
> > 
> > I don't think it's out of scope, but what exactly you mean by
> > "another
> 
> Thanks.
> 
> > way to make livepatch"? You would like to know about different
> > approaches like source-based livepatch compared to kpatch, or you
> > mean
> 
> Yes, exactly. Inspired by kpatch, I tried to make livepatch on source
> level
> to avoid difficult binary hacking, just like what we did normal
> module
> development.

This is how we create livepatches at SUSE.

> The main idea is similar to some topics of this mail list,
> 
> 1. Write livepatch-sample.c stylish source code, put all needed non-
> export/non-include symbols(I call them KLPSYMs) declarations in file.
> 2. Generate KLPSYMs position information through kallsyms etc.
> 3. `KBUILD_MODPOST_WARN=1 make` to build a "partial linked" .ko.
> 4. Use a klp-convert like tool to transform the KLPSYMs.

For now we are using kallsyms to get the address of the symbols, but it
will be changed soon due to IBT. The plan here is to also use klp-
convert.

> 
> For simple patch, hand-write source might be easy though a little
> time
> consuming. I used libclang to auto abbreviate livepatch source[1].
> 
> The main obstacle, IMO, is "how to determine non-included local
> symbols",
> because they might be inlined, optimized out, duplicate names,
> mangled
> names, and because kallsyms, vmlinux .symtab less some verbose
> information.
> In my toy, I used DWARF along with kallsyms to try to verify all of
> them. 
> 

That's is the tricky part, indeed.

> But now I realized that DWARF might be the fatal pitfall. That's a
> pity. I
> still get hope that's not too bad:)

Interesting. I didn't checked your code, but I can explain how we are
doing things now, and how we plan to do in the future.

Currently we are using klp-ccp[3] to extract the functions for us. The
tool also requires the ELF object that we are patching, plus Symvers
file and the ipa-clones generated when the kernel was compiled. This
way we can detect the symbols that are public (exported by vmlinux),
public from modules (exported by other modules), and private (exported
by the patched module).

For the symbols from vmlinux, we depend on them and call them directly.
For symbols from modules and private symbols, we use kallsyms to get
their address in order to call them.

For optimized symbols, we usually check the callers of the symbol
manually.

We also use a tool called klp-build (we plan to open source it later
this year) to detect duplicated symbols, check if the same symbol is
available and not optimized in different architectures (since we create
livepatches for three different architectures), and to check later if
the code generated/patches doesn't rely on symbols from other modules.

What we are doing right now is a new tool that uses LLVM/libclang to
extract the functions. The new tool also relies on the same
dependencies: module to be patched (to read the symbol table), Symvers
and ipa-clones. The main advantage of it is to rely on LLVM's AST. We
also plan to release this tool later this year. Another interesting
aspect of the new tool is that our userspace livepatching team was who
created the tool, and are using it to create their livepatches. Kernel
supported was added to it afterwards.

We plan to advertise both tools in this ML when they are released,
since we use both to create LPs today (klp-build is used currently with
klp-ccp already, but supported for the new tool is already done, but
not production ready).

Stay tuned :)

> 
> [1] https://gitee.com/laokz/klpmake
> [2] mirror: https://github.com/laokz/klpmake
[3]: https://github.com/SUSE/klp-ccp
> 
> > something different?
> > 
> > Thanks!
> > 
> > > 
> > > Thanks.
> > > 
> > > laokz
> > > 
> > > 
> > 
> > 
> 

Re: RFC: another way to make livepatch

[laokz]

On Wed, 2024-01-10 at 13:28 -0300, Marcos Paulo de Souza wrote:
> On Wed, 2024-01-10 at 22:06 +0800, pizza wrote:
> > On Wed, 2024-01-10 at 08:52 -0300, Marcos Paulo de Souza wrote:
> > > On Thu, 2023-12-21 at 22:06 +0800, laokz wrote:
> > > > Hi,
> > > > 
> > > > Is it off-topic talking about livepatch making tool? I tried
> > > > another
> > > > way and
> > > > want to get your expert opinion if there any fatal pitfall.
> > > 
> > > I don't think it's out of scope, but what exactly you mean by
> > > "another
> > 
> > Thanks.
> > 
> > > way to make livepatch"? You would like to know about different
> > > approaches like source-based livepatch compared to kpatch, or you
> > > mean
> > 
> > Yes, exactly. Inspired by kpatch, I tried to make livepatch on source
> > level
> > to avoid difficult binary hacking, just like what we did normal
> > module
> > development.
> 
> This is how we create livepatches at SUSE.
> 
> > The main idea is similar to some topics of this mail list,
> > 
> > 1. Write livepatch-sample.c stylish source code, put all needed non-
> > export/non-include symbols(I call them KLPSYMs) declarations in file.
> > 2. Generate KLPSYMs position information through kallsyms etc.
> > 3. `KBUILD_MODPOST_WARN=1 make` to build a "partial linked" .ko.
> > 4. Use a klp-convert like tool to transform the KLPSYMs.
> 
> For now we are using kallsyms to get the address of the symbols, but it
> will be changed soon due to IBT. The plan here is to also use klp-
> convert.
> 
> > 
> > For simple patch, hand-write source might be easy though a little
> > time
> > consuming. I used libclang to auto abbreviate livepatch source[1].
> > 
> > The main obstacle, IMO, is "how to determine non-included local
> > symbols",
> > because they might be inlined, optimized out, duplicate names,
> > mangled
> > names, and because kallsyms, vmlinux .symtab less some verbose
> > information.
> > In my toy, I used DWARF along with kallsyms to try to verify all of
> > them. 
> > 
> 
> That's is the tricky part, indeed.
> 
> > But now I realized that DWARF might be the fatal pitfall. That's a
> > pity. I
> > still get hope that's not too bad:)
> 
> Interesting. I didn't checked your code, but I can explain how we are
> doing things now, and how we plan to do in the future.
> 
> Currently we are using klp-ccp[3] to extract the functions for us. The
> tool also requires the ELF object that we are patching, plus Symvers
> file and the ipa-clones generated when the kernel was compiled. This
> way we can detect the symbols that are public (exported by vmlinux),
> public from modules (exported by other modules), and private (exported
> by the patched module).
> 
> For the symbols from vmlinux, we depend on them and call them directly.
> For symbols from modules and private symbols, we use kallsyms to get
> their address in order to call them.
> 
> For optimized symbols, we usually check the callers of the symbol
> manually.
> 
> We also use a tool called klp-build (we plan to open source it later
> this year) to detect duplicated symbols, check if the same symbol is
> available and not optimized in different architectures (since we create
> livepatches for three different architectures), and to check later if
> the code generated/patches doesn't rely on symbols from other modules.
> 
> What we are doing right now is a new tool that uses LLVM/libclang to
> extract the functions. The new tool also relies on the same
> dependencies: module to be patched (to read the symbol table), Symvers
> and ipa-clones. The main advantage of it is to rely on LLVM's AST. We
> also plan to release this tool later this year. Another interesting
> aspect of the new tool is that our userspace livepatching team was who
> created the tool, and are using it to create their livepatches. Kernel
> supported was added to it afterwards.
> 
> We plan to advertise both tools in this ML when they are released,
> since we use both to create LPs today (klp-build is used currently with
> klp-ccp already, but supported for the new tool is already done, but
> not production ready).
> 
> Stay tuned :)
> 

Thanks for sharing the information. Coping with compiler optimization is
another big pitfall,, Right, I'll stay tuned with your good news)


> > 
> > [1] https://gitee.com/laokz/klpmake
> > [2] mirror: https://github.com/laokz/klpmake
> [3]: https://github.com/SUSE/klp-ccp
> > 
> > > something different?
> > > 
> > > Thanks!
> > > 
> > > > 
> > > > Thanks.
> > > > 
> > > > laokz
> > > > 
> > > > 
> > > 
> > > 
> > 
> 
>