Re: Data corruption in guest using KVM

[Aurelien Jarno <aurelien-rXXEIb44qovR7s880joybQ@public.gmane.org>]

[-- Attachment #1: Type: text/plain, Size: 2936 bytes --]

On Sun, Jul 22, 2007 at 10:52:03AM +0300, Avi Kivity wrote:
> Aurelien Jarno wrote:
> > Hi all,
> >
> > For a long time I am seeing data corruption in guests when using KVM,
> > but I am convinced only since today that the problem comes from KVM.
> >
> > The symptoms are a few bytes that are mangled to 0x00 in a file that has
> > been written. For now I have only seen 2 or 4 consecutive bytes mangled,
> > but that may due to statistics given the limited samples.
> >
> > The problem appears very rarely. I am only seeing it when doing huge 
> > compilations (for example gcc or glibc), and not for every build. Note
> > that I am only detecting build failures, so I can miss some corruptions.
> >
> > Note that I have observed the problem on GNU/Linux, GNU/kFreeBSD and
> > plain FreeBSD, for both 32 and 64-bit guests. I always used 64-bit 
> > hosts, and I have seen the problem on both Core 2 and Athlon 64 CPU
> > (always multi-core).
> >
> > I have never seen such corruptions using QEMU, so I would say the
> > problem does not comes from the disk emulation, though it may be due to
> > statistics. Note that I have made a lot of compilation in a MIPS QEMU
> > guest (a few hundred of hours), without any problem. This platform uses
> > the same IDE controller as the one in KVM.
> >
> > Does anybody have seen the same kind of problem? Without a way to 
> > reproduce the corruption, I think it will be very difficult to debug 
> > the problem.
> 
> Did you observe anything about the corruption?  For example, are the 
> offsets at page boundary?  Can you provide a corrupted file and the 
> same, non-corrupted file as a reference?

For now I am still trying to find an easy way to reproduce it. You will
find below a sample of a bad and a good file. I have gzipped them to
make sure they will not be mangled once more by a MUA or a MTA.

What is strange with this sample is that the size of the file is not the
same. I will try to get more corrupted file.

I have been able to reproduce the bug with one or multiple guests
running, so it is not dependent on the number of guests running.


> For the 32-bit case, were the guests pae, nonpae, or both?

I am using nonpae guests (I only give 1GB of memory to the guests).


> How would I go about reproducing this?   Is a single ./configure; make 
> clean; make in a loop compiling gcc sufficient?

Yes basically that's what I am doing but on the glibc sources as I get 
more "success" to reproduce the bug. Note that you should run the 
configure in a different directory from the sources.

I generally observed the bug every 10 to 15 builds. One build takes
about 45 minutes here.

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org         | aurelien-rXXEIb44qovR7s880joybQ@public.gmane.org
   `-    people.debian.org/~aurel32 | www.aurel32.net

[-- Attachment #2: sem_close.o.d.bad.gz --]
[-- Type: application/octet-stream, Size: 1494 bytes --]

[-- Attachment #3: sem_close.o.d.good.gz --]
[-- Type: application/octet-stream, Size: 1471 bytes --]

[-- Attachment #4: Type: text/plain, Size: 315 bytes --]

-------------------------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc.
Still grepping through log files to find problems?  Stop.
Now Search log events and configuration files using AJAX and a browser.
Download your FREE copy of Splunk now >>  http://get.splunk.com/

[-- Attachment #5: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel