100% reliable oops on Xen 4.1.3 (initially reported on 4.0.1)

100% reliable oops on Xen 4.1.3 (initially reported on 4.0.1)

[Peter Moody]

[-- Attachment #1: Type: text/plain, Size: 2004 bytes --]

TL;DR, the domU crash I reported over the summer on Xen 4.0.1 can be
reproduced on 4.1.3 and on more processor families and with out the
special memory/cpu configurations I previously reported.

Longer version:
apropos of this thread [1] from last summer, I've managed to test for
this bug on a more recent version of Xen and I can confirm that it
exists in at least 4.1.3. Also, based on the release notes for 4.0.1
[2] (the original version of Xen where I encountered this issue), I
reproduced the bug on an AMD Athlon processor in case the interrupts
issue mentioned had an effect.

The patch I posted to the audit list didn't actually fix the problem.

Steps I used to reproduce:
1) installed Xen from ubuntu packages and boot into Xen enabled system.
2) installed ubuntu 12.10 domU using 20G flat file as disk (the
previous system used drbd).
3) installed auditd and inserted any syscall rule (audit on chmod's
for example).
4) compiled the attached sample program as a 32 bit binary.
5) ran it (works as a normal user).

The result is an immediate crash (if KILLDIR doesn't exist or isn't
writable, you just get a segfault).

Interestingly, it also seems to leave dom0 in a funky state where dom0
is unable to reboot (I think it has to do with the disk file not being
unmounted). I can only recover from this cleanly by running xm destroy
on the crashed domain.

This is my xen configuration for this particular domain:

memory = "1024"
disk = [ 'file:/home/pmoody/virt/xen/xen-bug/disk1.img,xvda,w', ]
vif = [ 'bridge=xenbr0', ]
vcpus=4
on_reboot = "restart"
on_crash = "restart"

(the number of vcpus doesn't appear to be important). From my recent
testing, it seems like it should be very easy for someone else to
reproduce this issue.

So, does anyone have any idea of what might be going on?

Cheers,
peter

[1] http://lists.xen.org/archives/html/xen-devel/2012-08/msg01052.html
[2] http://wiki.xen.org/wiki/Xen_4.0_Release_Notes#Known_issues

-- 
[ Peter Moody | Security Engineer | Google ]

[-- Attachment #2: crasher.c --]
[-- Type: text/x-csrc, Size: 3441 bytes --]

/*
 * steps:
 *  1) compile with gcc -m32
 *  2) start auditd, install any rule (I've only tested syscall auditing, but any syscall seems to work).
 *     /etc/init.d/auditd start ; auditctl -D ; auditctl -a exit,always -F arch=b64 -S chmod
 *  3) run'n wait (this only loops twice for me before dying)
 *     ./a.out
 *  4) instantaneous kernel oops.
 *
 * kernel BUG at /build/buildd/linux-3.5.0/fs/buffer.c:1218!
 * invalid opcode: 0000 [#3] SMP 
 * CPU 3 
 * Modules linked in:[  499.132393]  lp microcode parport
 * 
 * Pid: 6249, comm: a.out Tainted: G      D      3.5.0-22-generic #34-Ubuntu  
 * RIP: e030:[<ffffffff81679962>]  [<ffffffff81679962>] check_irqs_on.part.9+0x4/0x6
 * RSP: e02b:ffff8800f36e1c08  EFLAGS: 00010046
 * RAX: ffff880003570000 RBX: 0000000000001000 RCX: ffff880003573000
 * RDX: 0000000000001000 RSI: 000000000020021d RDI: ffff8800045dc340
 * RBP: ffff8800f36e1c08 R08: 0000000000000000 R09: 0000000000000000
 * R10: ffff880003573000 R11: ffff8800f39de860 R12: ffff8800f36e1dd0
 * R13: 0000000000001fd6 R14: ffff880003573400 R15: ffff8800045dc340
 * FS:  00007f12fe20e700(0000) GS:ffff8800ff980000(0063) knlGS:0000000000000000
 * CS:  e033 DS: 002b ES: 002b CR0: 000000008005003b
 * CR2: 00000000f7652ad0 CR3: 000000000330f000 CR4: 0000000000000660
 * DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 * DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
 * Process a.out (pid: 6249, threadinfo ffff8800f36e0000, task ffff8800f2cb5c00)
 * Stack:
 *  ffff8800f36e1c78 ffffffff811b35d9 ffff8800f36e1c28 ffffffff8107e6c1
 *  ffff8800f36e1c78 ffffffff8118d5c6 ffff8800f36e1d20 ffff8800f2cb5c00
 *  ffff8800f36e1e78 ffff8800e0117c30 0000000000001000 ffff8800f36e1dd0
 * Call Trace:
 *  [<ffffffff811b35d9>] __find_get_block+0x1f9/0x200
 *  [<ffffffff8107e6c1>] ? in_group_p+0x31/0x40
 *  [<ffffffff8118d5c6>] ? generic_permission+0x176/0x260
 *  [<ffffffff811b3604>] __getblk+0x24/0x2e0
 *  [<ffffffff8125bb84>] ? ext4_xattr_get+0xe4/0x2e0
 *  [<ffffffff8121c00d>] __ext4_get_inode_loc+0xed/0x410
 *  [<ffffffff8121d70d>] ext4_get_inode_loc+0x1d/0x20
 *  [<ffffffff812200dd>] ext4_reserve_inode_write+0x2d/0xa0
 *  [<ffffffff812223bf>] ? ext4_evict_inode+0x1bf/0x490
 *  [<ffffffff8122019f>] ext4_mark_inode_dirty+0x4f/0x1f0
 *  [<ffffffff812223bf>] ext4_evict_inode+0x1bf/0x490
 *  [<ffffffff8119c806>] evict+0xb6/0x1b0
 *  [<ffffffff8119d019>] iput+0x109/0x190
 *  [<ffffffff81197be8>] d_kill+0x108/0x150
 *  [<ffffffff81198332>] dput+0xe2/0x1d0
 *  [<ffffffff8118c8da>] path_put+0x1a/0x30
 *  [<ffffffff810cebab>] __audit_syscall_exit+0x34b/0x480
 *  [<ffffffff8168d622>] sysretl_audit+0x29/0x5b
 * Code: 80 0e a2 81 e8 9c cc ff ff 4d 85 e4 74 14 49 8d 7c 24 5c e8 91 82 99 ff 66 90 4c 89 
 * e7 e8 f7 e8 b1 ff 5b 41 5c 5d c3 55 48 89 e5 <0f> 0b 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 
 * 55 48 89 e5 0f 0b 55 
 * RIP  [<ffffffff81679962>] check_irqs_on.part.9+0x4/0x6
 *  RSP <ffff8800f36e1c08>
 */

#include <stdio.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

#define KILLDIR "/usr/local/tmp/crasher/kill_dir"

int main(void) {
  FILE *f;
  char fullpath[512];
  int i = 0;

  while (1) {
    fprintf(stderr, "%d ", i++);
    mkdir(KILLDIR, 0777);
    chdir(KILLDIR);
    sprintf(fullpath, "%s/file", KILLDIR);
    f = fopen(fullpath, "w+");
    fprintf(f, "nothing to see here");
    fclose(f);
    unlink(fullpath);
    rmdir(KILLDIR);
  }
  return 0;
}

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: 100% reliable oops on Xen 4.1.3 (initially reported on 4.0.1)

[Jan Beulich]

>>> On 28.01.13 at 20:17, Peter Moody <pmoody@google.com> wrote:
> TL;DR, the domU crash I reported over the summer on Xen 4.0.1 can be
> reproduced on 4.1.3 and on more processor families and with out the
> special memory/cpu configurations I previously reported.
> 
> Longer version:
> apropos of this thread [1] from last summer, I've managed to test for
> this bug on a more recent version of Xen and I can confirm that it
> exists in at least 4.1.3. Also, based on the release notes for 4.0.1
> [2] (the original version of Xen where I encountered this issue), I
> reproduced the bug on an AMD Athlon processor in case the interrupts
> issue mentioned had an effect.

I'm surprised this is still unresolved, but part of the problem may be
that you tag your problem (in the subject) to a particular Xen
version, thus implying it is a hypervisor issue. From the data you
provide I would think it's a kernel issue though.

> The patch I posted to the audit list didn't actually fix the problem.
> 
> Steps I used to reproduce:
> 1) installed Xen from ubuntu packages and boot into Xen enabled system.
> 2) installed ubuntu 12.10 domU using 20G flat file as disk (the
> previous system used drbd).
> 3) installed auditd and inserted any syscall rule (audit on chmod's
> for example).
> 4) compiled the attached sample program as a 32 bit binary.
> 5) ran it (works as a normal user).
> 
> The result is an immediate crash (if KILLDIR doesn't exist or isn't
> writable, you just get a segfault).
> 
> So, does anyone have any idea of what might be going on?

Sure - the code in question wants to run with interrupts enabled,
but they aren't for (I think) quite obvious a reason:
arch/x86/ia32/ia32entry.S:auditsys_exit has hard STI/CLI in it,
when those really should be ENABLE_INTERRUPTS() and
DISABLE_INTERRUPTS() respectively. Does the below help?

Jan

--- a/arch/x86/ia32/ia32entry.S
+++ b/arch/x86/ia32/ia32entry.S
@@ -207,7 +207,7 @@ sysexit_from_sys_call:
 	testl $(_TIF_ALLWORK_MASK & ~_TIF_SYSCALL_AUDIT),TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
 	jnz ia32_ret_from_sys_call
 	TRACE_IRQS_ON
-	sti
+	ENABLE_INTERRUPTS(CLBR_NONE)
 	movl %eax,%esi		/* second arg, syscall return value */
 	cmpl $-MAX_ERRNO,%eax	/* is it an error ? */
 	jbe 1f
@@ -217,7 +217,7 @@ sysexit_from_sys_call:
 	call __audit_syscall_exit
 	movq RAX-ARGOFFSET(%rsp),%rax	/* reload syscall return value */
 	movl $(_TIF_ALLWORK_MASK & ~_TIF_SYSCALL_AUDIT),%edi
-	cli
+	DISABLE_INTERRUPTS(CLBR_NONE)
 	TRACE_IRQS_OFF
 	testl %edi,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
 	jz \exit

Re: 100% reliable oops on Xen 4.1.3 (initially reported on 4.0.1)

[Peter Moody]

On Tue, Jan 29, 2013 at 3:38 AM, Jan Beulich <JBeulich@suse.com> wrote:

> Sure - the code in question wants to run with interrupts enabled,
> but they aren't for (I think) quite obvious a reason:
> arch/x86/ia32/ia32entry.S:auditsys_exit has hard STI/CLI in it,
> when those really should be ENABLE_INTERRUPTS() and
> DISABLE_INTERRUPTS() respectively. Does the below help?

I'll test this when I get home tonight. This is for the guest kernel, right?

Thanks!

Cheers,
peter


-- 
[ Peter Moody | Security Engineer | Google ]

Re: 100% reliable oops on Xen 4.1.3 (initially reported on 4.0.1)

[Peter Moody]

On Tue, Jan 29, 2013 at 1:05 PM, Peter Moody <pmoody@google.com> wrote:
> On Tue, Jan 29, 2013 at 3:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
>
>> Sure - the code in question wants to run with interrupts enabled,
>> but they aren't for (I think) quite obvious a reason:
>> arch/x86/ia32/ia32entry.S:auditsys_exit has hard STI/CLI in it,
>> when those really should be ENABLE_INTERRUPTS() and
>> DISABLE_INTERRUPTS() respectively. Does the below help?
>
> I'll test this when I get home tonight. This is for the guest kernel, right?

I just tested on one of my previously crashing work machines and this
looks good. Thanks again!

Cheers,
peter

-- 
[ Peter Moody | Security Engineer | Google ]

Re: 100% reliable oops on Xen 4.1.3 (initially reported on 4.0.1)

[Peter Moody]

On Tue, Jan 29, 2013 at 1:44 PM, Peter Moody <pmoody@google.com> wrote:
> On Tue, Jan 29, 2013 at 1:05 PM, Peter Moody <pmoody@google.com> wrote:
>> On Tue, Jan 29, 2013 at 3:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>
>>> Sure - the code in question wants to run with interrupts enabled,
>>> but they aren't for (I think) quite obvious a reason:
>>> arch/x86/ia32/ia32entry.S:auditsys_exit has hard STI/CLI in it,
>>> when those really should be ENABLE_INTERRUPTS() and
>>> DISABLE_INTERRUPTS() respectively. Does the below help?
>>
>> I'll test this when I get home tonight. This is for the guest kernel, right?
>
> I just tested on one of my previously crashing work machines and this
> looks good. Thanks again!

poor form to keep replying to myself here but, is this patch something
that would make it back into linus' tree from here or is there a more
appropriate forum where I should push this?

Cheers,
peter

-- 
[ Peter Moody | Security Engineer | Google ]

Re: 100% reliable oops on Xen 4.1.3 (initially reported on 4.0.1)

[Jan Beulich]

>>> On 29.01.13 at 23:21, Peter Moody <pmoody@google.com> wrote:
> On Tue, Jan 29, 2013 at 1:44 PM, Peter Moody <pmoody@google.com> wrote:
>> On Tue, Jan 29, 2013 at 1:05 PM, Peter Moody <pmoody@google.com> wrote:
>>> On Tue, Jan 29, 2013 at 3:38 AM, Jan Beulich <JBeulich@suse.com> wrote:
>>>
>>>> Sure - the code in question wants to run with interrupts enabled,
>>>> but they aren't for (I think) quite obvious a reason:
>>>> arch/x86/ia32/ia32entry.S:auditsys_exit has hard STI/CLI in it,
>>>> when those really should be ENABLE_INTERRUPTS() and
>>>> DISABLE_INTERRUPTS() respectively. Does the below help?
>>>
>>> I'll test this when I get home tonight. This is for the guest kernel, right?
>>
>> I just tested on one of my previously crashing work machines and this
>> looks good. Thanks again!
> 
> poor form to keep replying to myself here but, is this patch something
> that would make it back into linus' tree from here or is there a more
> appropriate forum where I should push this?

Just sent this upstream, with you Cc-ed so you can see eventual
progress. Hopefully older stable trees will also pick this up sooner
or later.

Jan

Re: 100% reliable oops on Xen 4.1.3 (initially reported on 4.0.1)

[David Vrabel]

On 28/01/13 19:17, Peter Moody wrote:
> TL;DR, the domU crash I reported over the summer on Xen 4.0.1 can be
> reproduced on 4.1.3 and on more processor families and with out the
> special memory/cpu configurations I previously reported.
> 
> Longer version:
> apropos of this thread [1] from last summer, I've managed to test for
> this bug on a more recent version of Xen and I can confirm that it
> exists in at least 4.1.3. Also, based on the release notes for 4.0.1
> [2] (the original version of Xen where I encountered this issue), I
> reproduced the bug on an AMD Athlon processor in case the interrupts
> issue mentioned had an effect.
> 
> The patch I posted to the audit list didn't actually fix the problem.
> 
> Steps I used to reproduce:
> 1) installed Xen from ubuntu packages and boot into Xen enabled system.
> 2) installed ubuntu 12.10 domU using 20G flat file as disk (the
> previous system used drbd).
> 3) installed auditd and inserted any syscall rule (audit on chmod's
> for example).
> 4) compiled the attached sample program as a 32 bit binary.
> 5) ran it (works as a normal user).

The BUG is because irqs_disabled().

The call to __audit_syscall_exit is from ia32_sysenter_target in
arch/x86/ia32/ia32entry.S which attempts to enable interrupts prior to
the call with an sti instruction.

I don't think this works as expected with a PV kernel and I'm surprised
that this doesn't cause a #GP fault.

Jan (Cc'd) is more familar with these low-level bits but does (untested)
this patch help?

---8<-------------
>From 8a3ebe942a8e6f930ee1636e8fe54a357144b007 Mon Sep 17 00:00:00 2001
From: David Vrabel <david.vrabel@citrix.com>
Date: Tue, 29 Jan 2013 11:48:14 +0000
Subject: [PATCH] x86/ia32: correctly enable irqs before calling
__audit_syscall_exit

Before calling __audit_syscall_exit, local interrupt were being
enabled with sti (and the disabled with cli).  This does not work in
paravirtualized guests so use the correct ENABLE_INTERRUPTS() and
DISABLE_INTERRUPTS() macros instead.

This fixes a BUG when auditing systems calls from a 32-bit userspace
process inside a 64-bit Xen PV guest.

Signed-off-by: David Vrabel <david.vrabel@citrix.com>
---
 arch/x86/ia32/ia32entry.S |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/ia32/ia32entry.S b/arch/x86/ia32/ia32entry.S
index 102ff7c..142c4ce 100644
--- a/arch/x86/ia32/ia32entry.S
+++ b/arch/x86/ia32/ia32entry.S
@@ -207,7 +207,7 @@ sysexit_from_sys_call:
 	testl $(_TIF_ALLWORK_MASK &
~_TIF_SYSCALL_AUDIT),TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
 	jnz ia32_ret_from_sys_call
 	TRACE_IRQS_ON
-	sti
+	ENABLE_INTERRUPTS(CLBR_NONE)
 	movl %eax,%esi		/* second arg, syscall return value */
 	cmpl $-MAX_ERRNO,%eax	/* is it an error ? */
 	jbe 1f
@@ -217,7 +217,7 @@ sysexit_from_sys_call:
 	call __audit_syscall_exit
 	movq RAX-ARGOFFSET(%rsp),%rax	/* reload syscall return value */
 	movl $(_TIF_ALLWORK_MASK & ~_TIF_SYSCALL_AUDIT),%edi
-	cli
+	DISABLE_INTERRUPTS(CLBR_NONE)
 	TRACE_IRQS_OFF
 	testl %edi,TI_flags+THREAD_INFO(%rsp,RIP-ARGOFFSET)
 	jz \exit

Re: 100% reliable oops on Xen 4.1.3 (initially reported on 4.0.1)

[Jan Beulich]

>>> On 29.01.13 at 12:56, David Vrabel <david.vrabel@citrix.com> wrote:
> On 28/01/13 19:17, Peter Moody wrote:
>> TL;DR, the domU crash I reported over the summer on Xen 4.0.1 can be
>> reproduced on 4.1.3 and on more processor families and with out the
>> special memory/cpu configurations I previously reported.
>> 
>> Longer version:
>> apropos of this thread [1] from last summer, I've managed to test for
>> this bug on a more recent version of Xen and I can confirm that it
>> exists in at least 4.1.3. Also, based on the release notes for 4.0.1
>> [2] (the original version of Xen where I encountered this issue), I
>> reproduced the bug on an AMD Athlon processor in case the interrupts
>> issue mentioned had an effect.
>> 
>> The patch I posted to the audit list didn't actually fix the problem.
>> 
>> Steps I used to reproduce:
>> 1) installed Xen from ubuntu packages and boot into Xen enabled system.
>> 2) installed ubuntu 12.10 domU using 20G flat file as disk (the
>> previous system used drbd).
>> 3) installed auditd and inserted any syscall rule (audit on chmod's
>> for example).
>> 4) compiled the attached sample program as a 32 bit binary.
>> 5) ran it (works as a normal user).
> 
> The BUG is because irqs_disabled().
> 
> The call to __audit_syscall_exit is from ia32_sysenter_target in
> arch/x86/ia32/ia32entry.S which attempts to enable interrupts prior to
> the call with an sti instruction.
> 
> I don't think this works as expected with a PV kernel and I'm surprised
> that this doesn't cause a #GP fault.

It does, but it gets dealt with by the hypervisor. Just that the
code handling this is commented out (i.e. both STI and CLI are
effectively NOPs), because of the inconsistency their emulation
would cause with PUSHF/POPF. See the respective cases in
arch/x86/traps.c:emulate_privileged_op().

> Jan (Cc'd) is more familar with these low-level bits but does (untested)
> this patch help?

Apart from the title and description, the one I had sent a couple
of minutes earlier than you sent yours, is identical, so I guess
both of us having come to the same conclusion is a good sign...

Jan