Re: ppc64le STRICT_MODULE_RWX and livepatch apply_relocate_add() crashes

[Christophe Leroy <christophe.leroy@csgroup.eu>]

[-- Attachment #1: Type: text/plain, Size: 5223 bytes --]



Le 14/12/2021 à 14:00, Joe Lawrence a écrit :
> On 12/14/21 7:44 AM, Christophe Leroy wrote:
>>
>>
>> Le 13/12/2021 à 18:26, Joe Lawrence a écrit :
>>> On 12/13/21 11:36 AM, Christophe Leroy wrote:
>>>>
>>>>
>>>> Le 13/12/2021 à 15:47, Joe Lawrence a écrit :
>>>>> On 12/13/21 2:42 AM, Christophe Leroy wrote:
>>>>>>
>>>>>> Hello Joe,
>>>>>>
>>>>>> I'm implementing LIVEPATCH on PPC32 and I wanted to test with
>>>>>> STRICT_MODULE_RWX enabled so I took your branch as suggested, but I'm
>>>>>> getting the following errors on build. What shall I do ?
>>>>>>
>>>>>>       CALL    scripts/checksyscalls.sh
>>>>>>       CALL    scripts/atomic/check-atomics.sh
>>>>>>       CHK     include/generated/compile.h
>>>>>>       KLP     lib/livepatch/test_klp_convert1.ko
>>>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>>>> beyond nr_entries
>>>>>>
>>>>>> klp-convert: Unable to load user-provided sympos
>>>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>>>> lib/livepatch/test_klp_convert1.ko] Error 255
>>>>>>       KLP     lib/livepatch/test_klp_convert2.ko
>>>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>>>> beyond nr_entries
>>>>>>
>>>>>> klp-convert: Unable to load user-provided sympos
>>>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>>>> lib/livepatch/test_klp_convert2.ko] Error 255
>>>>>>       KLP     lib/livepatch/test_klp_convert_sections.ko
>>>>>> klp-convert: section .rela.klp.module_relocs.test_klp_convert_mod length
>>>>>> beyond nr_entries
>>>>>>
>>>>>> klp-convert: Unable to load user-provided sympos
>>>>>> make[2]: *** [scripts/Makefile.modfinal:79:
>>>>>> lib/livepatch/test_klp_convert_sections.ko] Error 255
>>>>>> make[2]: Target '__modfinal' not remade because of errors.
>>>>>> make[1]: *** [scripts/Makefile.modpost:145: __modpost] Error 2
>>>>>> make: *** [Makefile:1770: modules] Error 2
>>>>>>
>>>>>
>>>>> Hi Christophe,
>>>>>
>>>>> Interesting failure mode.  That's klp-convert complaining that it found
>>>>> more relocations in a .klp.module_relocs.<objname> section than
>>>>> expected, i.e. nr_entries = sec->size / sizeof(struct klp_module_reloc).
>>>>>
>>>>> A few possibilities: the ELF sec->size was incorrectly set/read by
>>>>> build/libelf (I doubt that).  Or maybe the layout/size of struct
>>>>> klp_module_reloc is not consistent between kernel and userspace (I'm
>>>>> more suspicious of this).
>>>>>
>>>>> Can you post a copy of the build's symbols.klp and
>>>>> lib/livepatch/test_klp_convert1.tmp.ko somewhere?  I should be able to
>>>>> start debug with those files.
>>>>>
>>>>
>>>> I sent you both files off list.
>>>>
>>>> It looks like klp-convert doesn't use the correct size. It finds a
>>>> struct of size 12 hence 3 entries for a section of size 40.
>>>>
>>>> On PPC32 the struct has size 8 (void * is 4 and int is 4).
>>>>
>>>> But I'm cross-building from x86_64 where the struct is 8 + 4 = 12.
>>>>
>>>> Can it be the reason ?
>>>>
>>>
>>> I'm pretty sure that is it.  I haven't had much runtime with klp-convert
>>> and cross-building (I've only found one big/little endian bug with
>>> x86_64->s390x) and was going to ask you how you were testing :)
>>>
>>> Do you know if there are other kernel build tools that deal with similar
>>> situations?  This seems like a tricky job for the userspace build tool
>>> to determine non-native target struct layout.
>>>
>>> In the meantime, hacking in:
>>>
>>>    struct klp_module_reloc {
>>> -       void *sym;
>>> +       uint32_t sym;
>>>           unsigned int sympos;
>>>    } __packed;
>>>
>>> gets me generating an output .ko file, but the readelf output doesn't
>>> look right.
>>>
>>> I'll add this to the patchset TODO list, but may not get to it for a
>>> while -- is there any chance the above hack works or could you test a
>>> local non-cross build?
>>>
>>
>> No I have no way to do a non-cross build. My target is an embedded board
>> with slow CPU and little memory.
>>
>> I tested with your hack, I get:
>>
>> root@vgoip:~# insmod /lib/modules/test_klp_convert1.ko
>> insmod: can't insert '/lib/modules/test_klp_convert1.ko': unknown symbol
>> in module, or unknown parameter
>> root@vgoip:~# insmod /lib/modules/test_klp_livepatch.ko
>> insmod: can't insert '/lib/modules/test_klp_livepatch.ko': unknown
>> symbol in module, or unknown parameter
>>
>>
>> I agree with you readelf shows something went wrong with relocations.
>>
> 
> Thanks for trying that.  Can you point me to the cross-compiler suite
> that you are using for build and readelf?  Kernel .config would be handy
> too and I can try to reproduce locally for debugging.
> 

Livepatch on PPC32 is WIP, branch 'livepatch' on 
https://github.com/chleroy/linux.git.

You can use branch 'joe' on https://github.com/chleroy/linux.git which 
is a merge of your branch klp-convert-v5-expanded-v5.16-rebase1 with my 
livepatch branch.

Crosstool: 
https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/x86_64/11.1.0/x86_64-gcc-11.1.0-nolibc-powerpc64-linux.tar.xz

defconfig is attached

Thanks for your help
Christophe

[-- Attachment #2: defconfig --]
[-- Type: text/plain, Size: 1974 bytes --]

# CONFIG_SWAP is not set
CONFIG_SYSVIPC=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_BPF_JIT=y
CONFIG_VIRT_CPU_ACCOUNTING_NATIVE=y
CONFIG_LOG_BUF_SHIFT=14
CONFIG_EXPERT=y
# CONFIG_ELF_CORE is not set
# CONFIG_BASE_FULL is not set
# CONFIG_FUTEX is not set
CONFIG_KALLSYMS_ALL=y
CONFIG_PERF_EVENTS=y
# CONFIG_VM_EVENT_COUNTERS is not set
CONFIG_PPC_8xx=y
CONFIG_8xx_GPIO=y
CONFIG_SMC_UCODE_PATCH=y
CONFIG_PIN_TLB=y
CONFIG_GEN_RTC=y
CONFIG_HZ_100=y
CONFIG_MATH_EMULATION=y
CONFIG_PPC_16K_PAGES=y
CONFIG_ADVANCED_OPTIONS=y
CONFIG_LIVEPATCH=y
# CONFIG_SECCOMP is not set
CONFIG_MODULES=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_PNP=y
CONFIG_SYN_COOKIES=y
# CONFIG_FW_LOADER is not set
CONFIG_MTD=y
CONFIG_MTD_BLOCK=y
CONFIG_MTD_JEDECPROBE=y
CONFIG_MTD_CFI_ADV_OPTIONS=y
CONFIG_MTD_CFI_GEOMETRY=y
# CONFIG_MTD_MAP_BANK_WIDTH_1 is not set
# CONFIG_MTD_MAP_BANK_WIDTH_2 is not set
# CONFIG_MTD_CFI_I1 is not set
# CONFIG_MTD_CFI_I2 is not set
CONFIG_MTD_CFI_I4=y
CONFIG_MTD_CFI_AMDSTD=y
CONFIG_MTD_PHYSMAP=y
CONFIG_MTD_PHYSMAP_OF=y
# CONFIG_BLK_DEV is not set
CONFIG_NETDEVICES=y
CONFIG_FS_ENET=y
# CONFIG_FS_ENET_HAS_SCC is not set
CONFIG_DAVICOM_PHY=y
# CONFIG_INPUT is not set
# CONFIG_SERIO is not set
# CONFIG_VT is not set
# CONFIG_LEGACY_PTYS is not set
CONFIG_SERIAL_CPM=y
CONFIG_SERIAL_CPM_CONSOLE=y
CONFIG_SPI=y
CONFIG_SPI_FSL_SPI=y
# CONFIG_HWMON is not set
CONFIG_WATCHDOG=y
CONFIG_8xxx_WDT=y
# CONFIG_USB_SUPPORT is not set
# CONFIG_DNOTIFY is not set
CONFIG_TMPFS=y
CONFIG_CRAMFS=y
CONFIG_NFS_FS=y
CONFIG_ROOT_NFS=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_DEV_TALITOS=y
CONFIG_CRC32_SLICEBY4=y
CONFIG_DYNAMIC_DEBUG=y
CONFIG_DEBUG_INFO=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_FS=y
CONFIG_DEBUG_VM_PGTABLE=y
CONFIG_DETECT_HUNG_TASK=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FTRACE_STARTUP_TEST=y
# CONFIG_EVENT_TRACE_STARTUP_TEST is not set
CONFIG_BDI_SWITCH=y
CONFIG_PPC_EARLY_DEBUG=y
CONFIG_TEST_LIVEPATCH=m