Re: [Qemu-devel] RFH: difference in read-only mapped bios.bin - memory corruption?

[Laszlo Ersek <lersek@redhat.com>]

On 08/14/17 20:39, Dr. David Alan Gilbert wrote:
> * Philipp Hahn (hahn@univention.de) wrote:
>> Hello,
>>
>> I'm currently investigating a problem, were a Linux VM does not reboot
>> and gets stuck in the SeaBIOS reboot code:
>>
>> I'm using SeaBIOS-1.7 from Debian with a more modern qemu-2.8
>>
>>> virsh # qemu-monitor-command --hmp ucs41-414 info roms
>>> fw=genroms/kvmvapic.bin size=0x002400 name="kvmvapic.bin"
>>> addr=00000000fffe0000 size=0x020000 mem=rom name="bios.bin"
>>
>> which (to my understanding) is mapped at two physical locations:
>>> virsh # qemu-monitor-command --hmp ucs41-414 info mtree
>> ...> memory-region: system
>> ...>       00000000000e0000-00000000000fffff (prio 1, R-): alias
>> isa-bios @pc.bios 0000000000000000-000000000001ffff
>>>       00000000fffe0000-00000000ffffffff (prio 0, R-): pc.bios
>>
>> If I dump both regions and compare them, I get a difference:
>>> virsh # qemu-monitor-command --pretty --domain ucs41-414 '{"execute":"pmemsave","arguments":{"val":917504,"size":131072,"filename":"/tmp/bios-low.dump"}}'
>>> virsh # qemu-monitor-command --pretty --domain ucs41-414 '{"execute":"pmemsave","arguments":{"val":4294836224,"size":131072,"filename":"/tmp/bios-high.dump"}}'
>>> # diff --suppress-common-lines -y <(od -Ax -tx1 -w1 -v /tmp/bios-low.dump) <(od -Ax -tx1 -w1 -v /tmp/bios-high.dump)
>>> 00f798 fa                                                     | 00f798 80
>>> 00f799 7a                                                     | 00f799 89
>>> 00f79a f4                                                     | 00f79a f2
>>> 016d40 00                                                     | 016d40 ff
>>> 016d41 00                                                     | 016d41 ff
>>> 016d42 00                                                     | 016d42 ff
>>> 016d43 00                                                     | 016d43 ff
>>
>> The high address dump is the same as the original:
> 
> You might want seabios commit c68aff5 and b837e6 that got fixed after
> I tracked down some reboot hangs - although they were rare, not every
> time.  c68aff5 did certainly cause a corruption, and the address of that
> corruption was determined at link time and could overlay random useful
> bits of code if you were unlucky.
> 
>>> # cmp -l /tmp/bios-high.dump /usr/share/seabios/bios.bin
>>
>>> virsh # qemu-monitor-command --hmp ucs41-414 x/6i 0x00000000000ef78f
>>> 0x00000000000ef78f:  mov    $0xcf8,%esi
>>> 0x00000000000ef794:  mov    $0xfa000000,%eax
>>> 0x00000000000ef799:  jp     0xef78f
>>                        ^^^^^^^^^^^^^^ BUG: endless loop
>>> 0x00000000000ef79b:  out    %eax,(%dx)
>>> 0x00000000000ef79c:  mov    $0xfe,%dl
>>> 0x00000000000ef79e:  in     (%dx),%ax
>>
>>> virsh # qemu-monitor-command --hmp ucs41-414 xp/6i 0x00000000fffef78f
>>> 0x00000000fffef78f:  mov    $0xcf8,%esi
>>> 0x00000000fffef794:  mov    $0x80000000,%eax
>>> 0x00000000fffef799:  mov    %esi,%edx
>>                        ^^^^^^^^^^^^^^^^ CORRECT original code
>>> 0x00000000fffef79b:  out    %eax,(%dx)
>>> 0x00000000fffef79c:  mov    $0xfe,%dl
>>> 0x00000000fffef79e:  in     (%dx),%ax
>>
>> (That's some code from seabios-1.7.0/src/pci.c)
>>
>> I had exactly the same run some weeks ago, but I also get different
>> patterns:
>>> # diff --suppress-common-lines -y <(od -Ax -tx1 -w1 -v /tmp/bios2.dump) <(od -Ax -tx1 -w1 -v bios.bin)
>>> 00f798 f0                                                     | 00f798 80
>>> 00f799 8d                                                     | 00f799 89
>>> 00f79a f3                                                     | 00f79a f2
>>> 016d40 00                                                     | 016d40 ff
>>> 016d41 00                                                     | 016d41 ff
>>> 016d42 00                                                     | 016d42 ff
>>> 016d43 00                                                     | 016d43 ff
>>
>> Not all runs lead to reboot problems, but I don't know if any other
>> corruption happened there.
>>
>> I had a similar problem with OVMF back in June
>> <https://lists.nongnu.org/archive/html/qemu-devel/2017-06/msg03940.html>,
>> which I "solved" by upgrading the OVMF version: I have not seen the
>> problem there since than, but this problems looks very similar.
>>
>>
>> 1. How can it be, that the low-mem ROM mapping is modified?
> 
> I can't remember all the details, but PC ROM is shadowed and mapped over
> with RAM at various times,

Right. I don't remember for sure, but I believe the state of the PAM
registers doesn't only affect what the VCPUs see in that address range,
but also what your monitor commands will dump. (This would be the
logical choice -- make the monitor output what the VCPUs see anyway, at
the moment, dependent on the PAM settings.)

Thanks
Laszlo

> and the bioses play lots of silly tricks of
> being copied and then reusing bits of the copied space as temporaries
> and..... oh it's just a mess.
> 
> Dave
> 
>> 2. Can I tell QEMU or gdb to trap any modification of that 128 KiB area?
>>
>> I'll try to get http://rr-project.org/ running, but any help is appreciated.
>>
>> Philipp
>> -- 
>> Philipp Hahn
>> Open Source Software Engineer
>>
>> Univention GmbH
>> be open.
>> Mary-Somerville-Str. 1
>> D-28359 Bremen
>> Tel.: +49 421 22232-0
>> Fax : +49 421 22232-99
>> hahn@univention.de
>>
>> http://www.univention.de/
>> Geschäftsführer: Peter H. Ganten
>> HRB 20755 Amtsgericht Bremen
>> Steuer-Nr.: 71-597-02876
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>