From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.182]) by ozlabs.org (Postfix) with ESMTP id B8FEBDEC77 for ; Thu, 21 Aug 2008 00:55:47 +1000 (EST) Received: by py-out-1112.google.com with SMTP id a29so295930pyi.27 for ; Wed, 20 Aug 2008 07:55:45 -0700 (PDT) Message-ID: <9e4733910808200755y1128ae56p6a1235684bfbb3ec@mail.gmail.com> Date: Wed, 20 Aug 2008 10:55:45 -0400 From: "Jon Smirl" To: "Eran Liberty" Subject: Re: ftrace introduces instability into kernel 2.6.27(-rc2,-rc3) In-Reply-To: <48AC23F4.80900@extricom.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 References: <48591941.4070408@extricom.com> <1219114600.8062.15.camel@pasglop> <1219119431.8062.35.camel@pasglop> <1219216705.21386.46.camel@pasglop> <48AC1DD8.9080702@extricom.com> <48AC23F4.80900@extricom.com> Cc: Mathieu Desnoyers , linux-kernel@vger.kernel.org, Steven Rostedt , linuxppc-dev@ozlabs.org, Steven Rostedt , Alan Modra , Scott Wood , "Paul E. McKenney" List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On 8/20/08, Eran Liberty wrote: > Steven Rostedt wrote: > > > On Wed, 20 Aug 2008, Eran Liberty wrote: > > > > > > > > > Steven Rostedt wrote: > > > > > > > > > > On Wed, 20 Aug 2008, Steven Rostedt wrote: > > > > > > > > > > > > > > > > > On Wed, 20 Aug 2008, Benjamin Herrenschmidt wrote: > > > > > > > > > > > > > > > > > > > > > Found the problem (or at least -a- problem), it's a gcc bug. > > > > > > > > > > > > Well, first I must say the code generated by -pg is just plain > > > > > > horrible :-) > > > > > > > > > > > > Appart from that, look at the exit of, for example, __d_lookup, as > > > > > > generated by gcc when ftrace is enabled: > > > > > > > > > > > > c00c0498: 38 60 00 00 li r3,0 > > > > > > c00c049c: 81 61 00 00 lwz r11,0(r1) > > > > > > c00c04a0: 80 0b 00 04 lwz r0,4(r11) > > > > > > c00c04a4: 7d 61 5b 78 mr r1,r11 > > > > > > c00c04a8: bb 0b ff e0 lmw r24,-32(r11) > > > > > > c00c04ac: 7c 08 03 a6 mtlr r0 > > > > > > c00c04b0: 4e 80 00 20 blr > > > > > > > > > > > > As you can see, it restores r1 -before- it pops r24..r31 off > > > > > > the stack ! I let you imagine what happens if an interrupt happens > > > > > > just in between those two instructions (mr and lmw). We don't do > > > > > > redzones on our ABI, so basically, the registers end up corrupted > > > > > > by the interrupt. > > > > > > > > > > > > > > > > > Ouch! You've disassembled this without -pg too, and it does not > have this > > > > > bug? What version of gcc do you have? > > > > > > > > > > > > > > > > > > > I have: > > > > gcc (Debian 4.3.1-2) 4.3.1 > > > > > > > > c00c64c8: 81 61 00 00 lwz r11,0(r1) > > > > c00c64cc: 7f 83 e3 78 mr r3,r28 > > > > c00c64d0: 80 0b 00 04 lwz r0,4(r11) > > > > c00c64d4: ba eb ff dc lmw r23,-36(r11) > > > > c00c64d8: 7d 61 5b 78 mr r1,r11 > > > > c00c64dc: 7c 08 03 a6 mtlr r0 > > > > c00c64e0: 4e 80 00 20 blr > > > > > > > > > > > > My version looks fine. I'm thinking that this is a separate issue > than what > > > > Eran is seeing. > > > > > > > > Eran, can you do an "objdump -dr vmlinux" and search for __d_lookup, > and > > > > print out the end of the function dump. > > > > > > > > Thanks, > > > > > > > > -- Steve > > > > > > > > > > > > > > > > > > > > > > > powerpc-linux-gnu-objdump -dr --start-address=0xc00bb584 vmlinux | head > -n 100 > > > > > > vmlinux: file format elf32-powerpc > > > > > > Disassembly of section .text: > > > > > > c00bb584 <__d_lookup>: > > > > > > > > > > [...] > > > > > > > > > c00bb670: 41 9e 00 50 beq- cr7,c00bb6c0 <__d_lookup+0x13c> > > > c00bb674: 83 de 00 00 lwz r30,0(r30) > > > c00bb678: 2f 9e 00 00 cmpwi cr7,r30,0 > > > c00bb67c: 40 9e ff 98 bne+ cr7,c00bb614 <__d_lookup+0x90> > > > c00bb680: 38 60 00 00 li r3,0 > > > c00bb684: 81 61 00 00 lwz r11,0(r1) > > > c00bb688: 80 0b 00 04 lwz r0,4(r11) > > > c00bb68c: 7d 61 5b 78 mr r1,r11 > > > > > > > > > > [ BUG HERE IF INTERRUPT HAPPENS ] > > > > > > > > > c00bb690: bb 0b ff e0 lmw r24,-32(r11) > > > c00bb694: 7c 08 03 a6 mtlr r0 > > > c00bb698: 4e 80 00 20 blr > > > > > > > > > > Yep, you have the same bug in your compiler. > > > > -- Steve > > > > > Hmm... so whats now? > > Is there a way to prove this scenario is indeed the one that caused the > opps? Manually edit the broken binary to change the order of the restore and see if the problem disappears. That will keep everything else constant. > > -- Liberty > > _______________________________________________ > Linuxppc-dev mailing list > Linuxppc-dev@ozlabs.org > https://ozlabs.org/mailman/listinfo/linuxppc-dev > -- Jon Smirl jonsmirl@gmail.com