Re: reverse-{debugging,continue} not working on v7.2.0, i386 guest

[Hyeonggon Yoo <42.hyeyoo@gmail.com>]

On Wed, Jan 18, 2023 at 12:39:16PM +0300, Pavel Dovgalyuk wrote:
> Sometimes replay (or reverse debugging) have problems due to incomplete or
> incorrect virtual device save/load implementation.
> 
> Can you try removing -cpu from your command line?
> 
> Or you can provide the files you load and I'll debug this case.

Ah, sorry to bother. I installed breakpoint _after_ kernel panic,
and installing breakpoint before boot worked fine. Every seems great!

Just a side question, is there a reason QEMU record/replay
does not support -smp N (> 1)? is this feature planed, or should I use
other tools to debug SMP bugs?

> On 18.01.2023 11:47, Hyeonggon Yoo wrote:
> > On Wed, Jan 18, 2023 at 10:12:48AM +0300, Pavel Dovgalyuk wrote:
> > > As replay works well, the reverse debugging should be ok too.
> > > But for "going back" it needs a VM snapshot that can be used for reload.
> > > 
> > > Snapshots are saved on qcow2 images connected to QEMU.
> > > Therefore you need to add an empty qcow2 to your command line with the
> > > following option: -drive file=empty.qcow2,if=none,id=rr
> > 
> > Oh, I guessed it's possible to reverse-debug without snapshot,
> > and your comments definitely helped! adding empty disk and snapshotting solved it.
> > 
> > But I faced another problem:
> > 
> > (gdb) b __list_del_entry_valid
> > (gdb) reverse-continue
> > 
> > (it stuck forever)
> > ^C
> > (gdb) info registers
> > eax            0xefe19f74          -270426252
> > ecx            0x0                 0
> > edx            0xefe19f74          -270426252
> > ebx            0xf6ff4620          -151042528
> > esp            0xc02e9a34          0xc02e9a34
> > ebp            0xc02e9a6c          0xc02e9a6c
> > esi            0xc4fffb20          -989856992
> > edi            0xefe19f70          -270426256
> > eip            0xc1f38400          0xc1f38400 <__list_del_entry_valid>
> > eflags         0x6                 [ IOPL=0 PF ]
> > cs             0x60                96
> > ss             0x68                104
> > ds             0x7b                123
> > es             0x7b                123
> > fs             0xd8                216
> > gs             0x0                 0
> > fs_base        0x31cb4000          835403776
> > gs_base        0x0                 0
> > k_gs_base      0x0                 0
> > cr0            0x80050033          [ PG AM WP NE ET MP PE ]
> > cr2            0xffcb1000          -3469312
> > cr3            0x534e000           [ PDBR=0 PCID=0 ]
> > cr4            0x406d0             [ PSE MCE PGE OSFXSR OSXMMEXCPT OSXSAVE ]
> > cr8            0x1                 1
> > efer           0x0                 [ ]
> > 
> > it stuck here and it's not 'last breakpoint hit' from the panic
> > (it's early in boot), and stepi, nexti, continue commands do not work and
> > there's no forward progress. (eip doesn't change)
> > 
> > Did I miss something or did something wrong?
> > 
> > thank you so much with your help.
> > 
> > --
> > Best regards,
> > Hyeonggon
> > 
> > > 
> > > And you also need to add rrsnapshot to icount for creating the snapshot at
> > > the start of VM execution:
> > > -icount shift=auto,rr=record,rrfile=$REPLAY_FILE,rrsnapshot=start
> > > 
> > > 
> > > On 18.01.2023 09:14, Hyeonggon Yoo wrote:
> > > > Hello QEMU folks.
> > > > I was struggling to fix a recent heisenbug in the Linux kernel,
> > > > and fortunately the bug was reproducible with TCG and -smp 1.
> > > > 
> > > > I'm using qemu version 7.2.0, and guest architecture is i386.
> > > > I tried to inspect the bug using record/replay and reverse-debugging
> > > > feature in the QEMU.
> > > > 
> > > > 
> > > > recorded with:
> > > > 
> > > > qemu-system-i386 \
> > > >           -icount shift=auto,rr=record,rrfile=$REPLAY_FILE \
> > > >           -kernel arch/x86/boot/bzImage \
> > > >           -cpu SandyBridge \
> > > >           -initrd debian-i386.cgz \
> > > >           -smp 1 \
> > > >           -m 1024 \
> > > >           -nographic \
> > > >           -net none \
> > > >           -append "page_owner=on console=ttyS0"
> > > > 
> > > > and replayed with:
> > > > 
> > > > qemu-system-i386 \
> > > >           -icount shift=auto,rr=replay,rrfile=$REPLAY_FILE \
> > > >           -kernel arch/x86/boot/bzImage \
> > > >           -cpu SandyBridge \
> > > >           -initrd debian-i386.cgz \
> > > >           -smp 1 \
> > > >           -m 1024 \
> > > >           -nographic \
> > > >           -net none \
> > > >           -s \
> > > >           -append "page_owner=on console=ttyS0"
> > > > 
> > > > (I'm using a initrd image instead of a disk file.)
> > > > 
> > > > The record and replay works well. The bug is reliably reproduced
> > > > when relaying. but when I try to reverse-continue or reverse-stepi after
> > > > kernel panic, the gdb only says:
> > > > 
> > > > 	"remote failure reply 'E14'"
> > > > 
> > > > Is there something I'm missing, or record/replay do not work with
> > > > QEMU v7.2.0 or i386?
> > > > 
> > > > --
> > > > Best regards,
> > > > Hyeonggon
> > > 
> 

-- 
Thanks,
Hyeonggon