Problems with livepatching

Problems with livepatching

[M A Young]

I have been experimenting with live patching with the recent batch of 
security updates on Fedora xen with very limited success. I had most 
attempts with a test build of xen-4.8.0-rc6, and of the updates I have 
tried only xsa192.patch uploads successfully. For example with 
xsa191.patch the upload fails with the output
Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes)
Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such 
file or directory)!
and in xl dmesg a long debugging output ends with the line
(XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0
with a similar line (mostly with .LC0 but with .LC3 in one case) for the 
other failed attempts. Am I doing something wrong or is there a problem 
with live patching in this case?

	Michael Young

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Problems with livepatching

[Andrew Cooper]

On 23/11/2016 23:06, M A Young wrote:
> I have been experimenting with live patching with the recent batch of 
> security updates on Fedora xen with very limited success. I had most 
> attempts with a test build of xen-4.8.0-rc6, and of the updates I have 
> tried only xsa192.patch uploads successfully. For example with 
> xsa191.patch the upload fails with the output
> Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes)
> Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such 
> file or directory)!
> and in xl dmesg a long debugging output ends with the line
> (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0
> with a similar line (mostly with .LC0 but with .LC3 in one case) for the 
> other failed attempts. Am I doing something wrong or is there a problem 
> with live patching in this case?

This looks like a problem generating the livepatch itself, not the
livepatching mechanism.

Make sure you are completely up to date with the livepatch tools
userspace.  There was one bug in livepatch generation which was
discovered due to XSA-191 and fixed (actually a preexisting bug even in
Linux), but its symptoms were innocuous until you patched, at which vcpu
context switch blew up.

With that one bug fixed, I can confirm that all of the recent XSAs are
properly livepatchable with the available tools, and confirmed to work.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Problems with livepatching

[M A Young]

On Wed, 23 Nov 2016, Andrew Cooper wrote:

> On 23/11/2016 23:06, M A Young wrote:
> > I have been experimenting with live patching with the recent batch of 
> > security updates on Fedora xen with very limited success. I had most 
> > attempts with a test build of xen-4.8.0-rc6, and of the updates I have 
> > tried only xsa192.patch uploads successfully. For example with 
> > xsa191.patch the upload fails with the output
> > Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes)
> > Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such 
> > file or directory)!
> > and in xl dmesg a long debugging output ends with the line
> > (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0
> > with a similar line (mostly with .LC0 but with .LC3 in one case) for the 
> > other failed attempts. Am I doing something wrong or is there a problem 
> > with live patching in this case?
> 
> This looks like a problem generating the livepatch itself, not the
> livepatching mechanism.
> 
> Make sure you are completely up to date with the livepatch tools
> userspace.  There was one bug in livepatch generation which was
> discovered due to XSA-191 and fixed (actually a preexisting bug even in
> Linux), but its symptoms were innocuous until you patched, at which vcpu
> context switch blew up.

Where is this latest version available from? I had checked the git repo on 
xenbits 
http://xenbits.xenproject.org/gitweb/?p=livepatch-build-tools.git;a=summary 
but the last commit was 4 months ago.

	Michael Young

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Problems with livepatching

[Konrad Rzeszutek Wilk]

On Thu, Nov 24, 2016 at 12:13:43AM +0000, M A Young wrote:
> On Wed, 23 Nov 2016, Andrew Cooper wrote:
> 
> > On 23/11/2016 23:06, M A Young wrote:
> > > I have been experimenting with live patching with the recent batch of 
> > > security updates on Fedora xen with very limited success. I had most 
> > > attempts with a test build of xen-4.8.0-rc6, and of the updates I have 
> > > tried only xsa192.patch uploads successfully. For example with 
> > > xsa191.patch the upload fails with the output
> > > Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes)
> > > Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such 
> > > file or directory)!
> > > and in xl dmesg a long debugging output ends with the line
> > > (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0
> > > with a similar line (mostly with .LC0 but with .LC3 in one case) for the 
> > > other failed attempts. Am I doing something wrong or is there a problem 
> > > with live patching in this case?
> > 
> > This looks like a problem generating the livepatch itself, not the
> > livepatching mechanism.
> > 
> > Make sure you are completely up to date with the livepatch tools
> > userspace.  There was one bug in livepatch generation which was
> > discovered due to XSA-191 and fixed (actually a preexisting bug even in
> > Linux), but its symptoms were innocuous until you patched, at which vcpu
> > context switch blew up.
> 
> Where is this latest version available from? I had checked the git repo on 
> xenbits 
> http://xenbits.xenproject.org/gitweb/?p=livepatch-build-tools.git;a=summary 
> but the last commit was 4 months ago.

Here is the patch that Ross has been working on - it is being first reviewed
for merge in the upstream version of kpatch. Hence hasn't been posted. But
posting it here.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH LIVEPATCH-BUILD-TOOLS] create-diff-object: Update fixup offsets in .rela.ex_table

[Konrad Rzeszutek Wilk]

From: Ross Lagerwall <ross.lagerwall@citrix.com>

When pruning entries from the fixup table, update the offsets in
.rela.ex_table otherwise the relas might point to the wrong fixup entry
or even out of the .fixup section.

Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
Signed-off-by: Konrad Rzeszutek Wilk <konrad@kernel.org>
---
v2: Squashed fix from kpatch upstream
   [see https://github.com/dynup/kpatch/pull/616]
---
 create-diff-object.c | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/create-diff-object.c b/create-diff-object.c
index 69bcd88..58f7c4c 100644
--- a/create-diff-object.c
+++ b/create-diff-object.c
@@ -1057,6 +1057,34 @@ static int should_keep_rela_group(struct section *sec, int start, int size)
 	return found;
 }
 
+/*
+ * When updating .fixup, the corresponding addends in .ex_table need to be
+ * updated too. Stash the result in rela.r_addend so that the calculation in
+ * fixup_group_size() is not affected.
+ */
+void kpatch_update_ex_table_addend(struct kpatch_elf *kelf,
+				   struct special_section *special,
+				   int src_offset, int dest_offset,
+				   int group_size)
+{
+	struct rela *rela;
+	struct section *sec;
+
+	if (strcmp(special->name, ".fixup"))
+		return;
+
+	sec = find_section_by_name(&kelf->sections, ".rela.ex_table");
+	if (!sec)
+		ERROR("missing .rela.ex_table section");
+
+	list_for_each_entry(rela, &sec->relas, list) {
+		if (!strcmp(rela->sym->name, ".fixup") &&
+		    rela->addend >= src_offset &&
+		    rela->addend < src_offset + group_size)
+			rela->rela.r_addend = rela->addend - (src_offset - dest_offset);
+	}
+}
+
 static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
 				              struct special_section *special,
 				              struct section *sec)
@@ -1073,6 +1101,14 @@ static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
 	if (!dest)
 		ERROR("malloc");
 
+	/* Restore the stashed r_addend from kpatch_update_ex_table_addend. */
+	if (!strcmp(special->name, ".ex_table")) {
+		list_for_each_entry(rela, &sec->relas, list) {
+			if (!strcmp(rela->sym->name, ".fixup"))
+				rela->addend = rela->rela.r_addend;
+		}
+	}
+
 	group_size = 0;
 	src_offset = 0;
 	dest_offset = 0;
@@ -1100,6 +1136,11 @@ static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
 				rela->rela.r_offset = rela->offset;
 
 				rela->sym->include = 1;
+
+				kpatch_update_ex_table_addend(kelf, special,
+							      src_offset,
+							      dest_offset,
+							      group_size);
 			}
 		}
 
-- 
2.9.3


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH LIVEPATCH-BUILD-TOOLS] create-diff-object: Update fixup offsets in .rela.ex_table

[Ross Lagerwall]

On 11/24/2016 01:08 AM, Konrad Rzeszutek Wilk wrote:
> From: Ross Lagerwall <ross.lagerwall@citrix.com>
>
> When pruning entries from the fixup table, update the offsets in
> .rela.ex_table otherwise the relas might point to the wrong fixup entry
> or even out of the .fixup section.
>
> Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
> Signed-off-by: Konrad Rzeszutek Wilk <konrad@kernel.org>
> ---
> v2: Squashed fix from kpatch upstream
>    [see https://github.com/dynup/kpatch/pull/616]
> ---
>  create-diff-object.c | 41 +++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 41 insertions(+)
>
> diff --git a/create-diff-object.c b/create-diff-object.c
> index 69bcd88..58f7c4c 100644
> --- a/create-diff-object.c
> +++ b/create-diff-object.c
> @@ -1057,6 +1057,34 @@ static int should_keep_rela_group(struct section *sec, int start, int size)
>  	return found;
>  }
>
> +/*
> + * When updating .fixup, the corresponding addends in .ex_table need to be
> + * updated too. Stash the result in rela.r_addend so that the calculation in
> + * fixup_group_size() is not affected.
> + */
> +void kpatch_update_ex_table_addend(struct kpatch_elf *kelf,
> +				   struct special_section *special,
> +				   int src_offset, int dest_offset,
> +				   int group_size)
> +{
> +	struct rela *rela;
> +	struct section *sec;
> +
> +	if (strcmp(special->name, ".fixup"))
> +		return;

As suggested by kpatch upstream, the code would be clearer if the above 
check is in kpatch_regenerate_special_section() rather than this function.

I'll update and resend.

> +
> +	sec = find_section_by_name(&kelf->sections, ".rela.ex_table");
> +	if (!sec)
> +		ERROR("missing .rela.ex_table section");
> +
> +	list_for_each_entry(rela, &sec->relas, list) {
> +		if (!strcmp(rela->sym->name, ".fixup") &&
> +		    rela->addend >= src_offset &&
> +		    rela->addend < src_offset + group_size)
> +			rela->rela.r_addend = rela->addend - (src_offset - dest_offset);
> +	}
> +}
> +
>  static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
>  				              struct special_section *special,
>  				              struct section *sec)
> @@ -1073,6 +1101,14 @@ static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
>  	if (!dest)
>  		ERROR("malloc");
>
> +	/* Restore the stashed r_addend from kpatch_update_ex_table_addend. */
> +	if (!strcmp(special->name, ".ex_table")) {
> +		list_for_each_entry(rela, &sec->relas, list) {
> +			if (!strcmp(rela->sym->name, ".fixup"))
> +				rela->addend = rela->rela.r_addend;
> +		}
> +	}
> +
>  	group_size = 0;
>  	src_offset = 0;
>  	dest_offset = 0;
> @@ -1100,6 +1136,11 @@ static void kpatch_regenerate_special_section(struct kpatch_elf *kelf,
>  				rela->rela.r_offset = rela->offset;
>
>  				rela->sym->include = 1;
> +
> +				kpatch_update_ex_table_addend(kelf, special,
> +							      src_offset,
> +							      dest_offset,
> +							      group_size);
>  			}
>  		}
>
>


-- 
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Problems with livepatching

[Ross Lagerwall]

On 11/24/2016 01:08 AM, Konrad Rzeszutek Wilk wrote:
> On Thu, Nov 24, 2016 at 12:13:43AM +0000, M A Young wrote:
>> On Wed, 23 Nov 2016, Andrew Cooper wrote:
>>
>>> On 23/11/2016 23:06, M A Young wrote:
>>>> I have been experimenting with live patching with the recent batch of
>>>> security updates on Fedora xen with very limited success. I had most
>>>> attempts with a test build of xen-4.8.0-rc6, and of the updates I have
>>>> tried only xsa192.patch uploads successfully. For example with
>>>> xsa191.patch the upload fails with the output
>>>> Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes)
>>>> Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such
>>>> file or directory)!
>>>> and in xl dmesg a long debugging output ends with the line
>>>> (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0
>>>> with a similar line (mostly with .LC0 but with .LC3 in one case) for the
>>>> other failed attempts. Am I doing something wrong or is there a problem
>>>> with live patching in this case?
>>>
>>> This looks like a problem generating the livepatch itself, not the
>>> livepatching mechanism.
>>>
>>> Make sure you are completely up to date with the livepatch tools
>>> userspace.  There was one bug in livepatch generation which was
>>> discovered due to XSA-191 and fixed (actually a preexisting bug even in
>>> Linux), but its symptoms were innocuous until you patched, at which vcpu
>>> context switch blew up.
>>
>> Where is this latest version available from? I had checked the git repo on
>> xenbits
>> http://xenbits.xenproject.org/gitweb/?p=livepatch-build-tools.git;a=summary
>> but the last commit was 4 months ago.
>
> Here is the patch that Ross has been working on - it is being first reviewed
> for merge in the upstream version of kpatch. Hence hasn't been posted. But
> posting it here.
>

I've also got a patch to fix the build with GCC 6 which I shall send out 
shortly. This should fix the 'Unknown symbol' issue.

-- 
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Problems with livepatching

[Lars Kurth]

> On 24 Nov 2016, at 01:08, Konrad Rzeszutek Wilk <konrad@kernel.org> wrote:
> 
> On Thu, Nov 24, 2016 at 12:13:43AM +0000, M A Young wrote:
>> On Wed, 23 Nov 2016, Andrew Cooper wrote:
>> 
>>> On 23/11/2016 23:06, M A Young wrote:
>>>> I have been experimenting with live patching with the recent batch of 
>>>> security updates on Fedora xen with very limited success. I had most 
>>>> attempts with a test build of xen-4.8.0-rc6, and of the updates I have 
>>>> tried only xsa192.patch uploads successfully. For example with 
>>>> xsa191.patch the upload fails with the output
>>>> Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes)
>>>> Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such 
>>>> file or directory)!
>>>> and in xl dmesg a long debugging output ends with the line
>>>> (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0
>>>> with a similar line (mostly with .LC0 but with .LC3 in one case) for the 
>>>> other failed attempts. Am I doing something wrong or is there a problem 
>>>> with live patching in this case?
>>> 
>>> This looks like a problem generating the livepatch itself, not the
>>> livepatching mechanism.
>>> 
>>> Make sure you are completely up to date with the livepatch tools
>>> userspace.  There was one bug in livepatch generation which was
>>> discovered due to XSA-191 and fixed (actually a preexisting bug even in
>>> Linux), but its symptoms were innocuous until you patched, at which vcpu
>>> context switch blew up.
>> 
>> Where is this latest version available from? I had checked the git repo on 
>> xenbits 
>> http://xenbits.xenproject.org/gitweb/?p=livepatch-build-tools.git;a=summary 
>> but the last commit was 4 months ago.
> 
> Here is the patch that Ross has been working on - it is being first reviewed
> for merge in the upstream version of kpatch. Hence hasn't been posted. But
> posting it here.

I think the link is missing.
Lars
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Problems with livepatching

[Ross Lagerwall]

On 11/24/2016 09:24 AM, Lars Kurth wrote:
>
>> On 24 Nov 2016, at 01:08, Konrad Rzeszutek Wilk <konrad@kernel.org> wrote:
>>
>> On Thu, Nov 24, 2016 at 12:13:43AM +0000, M A Young wrote:
>>> On Wed, 23 Nov 2016, Andrew Cooper wrote:
>>>
>>>> On 23/11/2016 23:06, M A Young wrote:
>>>>> I have been experimenting with live patching with the recent batch of
>>>>> security updates on Fedora xen with very limited success. I had most
>>>>> attempts with a test build of xen-4.8.0-rc6, and of the updates I have
>>>>> tried only xsa192.patch uploads successfully. For example with
>>>>> xsa191.patch the upload fails with the output
>>>>> Uploading /tmp/xen-debuginfo/out1/xsa191.livepatch (1561400 bytes)
>>>>> Upload failed: /tmp/xen-debuginfo/out1/xsa191.livepatch, error: 2(No such
>>>>> file or directory)!
>>>>> and in xl dmesg a long debugging output ends with the line
>>>>> (XEN) livepatch_elf.c:295: livepatch: xsa191: Unknown symbol: .LC0
>>>>> with a similar line (mostly with .LC0 but with .LC3 in one case) for the
>>>>> other failed attempts. Am I doing something wrong or is there a problem
>>>>> with live patching in this case?
>>>>
>>>> This looks like a problem generating the livepatch itself, not the
>>>> livepatching mechanism.
>>>>
>>>> Make sure you are completely up to date with the livepatch tools
>>>> userspace.  There was one bug in livepatch generation which was
>>>> discovered due to XSA-191 and fixed (actually a preexisting bug even in
>>>> Linux), but its symptoms were innocuous until you patched, at which vcpu
>>>> context switch blew up.
>>>
>>> Where is this latest version available from? I had checked the git repo on
>>> xenbits
>>> http://xenbits.xenproject.org/gitweb/?p=livepatch-build-tools.git;a=summary
>>> but the last commit was 4 months ago.
>>
>> Here is the patch that Ross has been working on - it is being first reviewed
>> for merge in the upstream version of kpatch. Hence hasn't been posted. But
>> posting it here.
>
> I think the link is missing.
> Lars
>

Konrad posted the patch to xen-devel.

For what it's worth, I sometimes push changes that I'm working on to a 
branch at https://github.com/rosslagerwall/livepatch-build-tools

-- 
Ross Lagerwall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel