From mboxrd@z Thu Jan 1 00:00:00 1970 From: Aurelien Jarno Subject: Re: Data corruption in guest using KVM Date: Sun, 22 Jul 2007 15:38:18 +0200 Message-ID: <20070722133818.GG16993@hall.aurel32.net> References: <20070721172248.GA1555@hall.aurel32.net> <46A30CA3.3090100@qumranet.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="dDRMvlgZJXvWKvBx" Cc: kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org To: Avi Kivity Return-path: Content-Disposition: inline In-Reply-To: <46A30CA3.3090100-atKUWr5tajBWk0Htik3J/w@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org Errors-To: kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org List-Id: kvm.vger.kernel.org --dDRMvlgZJXvWKvBx Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline On Sun, Jul 22, 2007 at 10:52:03AM +0300, Avi Kivity wrote: > Aurelien Jarno wrote: > > Hi all, > > > > For a long time I am seeing data corruption in guests when using KVM, > > but I am convinced only since today that the problem comes from KVM. > > > > The symptoms are a few bytes that are mangled to 0x00 in a file that has > > been written. For now I have only seen 2 or 4 consecutive bytes mangled, > > but that may due to statistics given the limited samples. > > > > The problem appears very rarely. I am only seeing it when doing huge > > compilations (for example gcc or glibc), and not for every build. Note > > that I am only detecting build failures, so I can miss some corruptions. > > > > Note that I have observed the problem on GNU/Linux, GNU/kFreeBSD and > > plain FreeBSD, for both 32 and 64-bit guests. I always used 64-bit > > hosts, and I have seen the problem on both Core 2 and Athlon 64 CPU > > (always multi-core). > > > > I have never seen such corruptions using QEMU, so I would say the > > problem does not comes from the disk emulation, though it may be due to > > statistics. Note that I have made a lot of compilation in a MIPS QEMU > > guest (a few hundred of hours), without any problem. This platform uses > > the same IDE controller as the one in KVM. > > > > Does anybody have seen the same kind of problem? Without a way to > > reproduce the corruption, I think it will be very difficult to debug > > the problem. > > Did you observe anything about the corruption? For example, are the > offsets at page boundary? Can you provide a corrupted file and the > same, non-corrupted file as a reference? For now I am still trying to find an easy way to reproduce it. You will find below a sample of a bad and a good file. I have gzipped them to make sure they will not be mangled once more by a MUA or a MTA. What is strange with this sample is that the size of the file is not the same. I will try to get more corrupted file. I have been able to reproduce the bug with one or multiple guests running, so it is not dependent on the number of guests running. > For the 32-bit case, were the guests pae, nonpae, or both? I am using nonpae guests (I only give 1GB of memory to the guests). > How would I go about reproducing this? Is a single ./configure; make > clean; make in a loop compiling gcc sufficient? Yes basically that's what I am doing but on the glibc sources as I get more "success" to reproduce the bug. Note that you should run the configure in a different directory from the sources. I generally observed the bug every 10 to 15 builds. One build takes about 45 minutes here. -- .''`. Aurelien Jarno | GPG: 1024D/F1BCDB73 : :' : Debian developer | Electrical Engineer `. `' aurel32-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org | aurelien-rXXEIb44qovR7s880joybQ@public.gmane.org `- people.debian.org/~aurel32 | www.aurel32.net --dDRMvlgZJXvWKvBx Content-Type: application/octet-stream Content-Disposition: attachment; filename="sem_close.o.d.bad.gz" Content-Transfer-Encoding: base64 H4sICK8eo0YCA3NlbV9jbG9zZS5vLmQuYmFkAKVY246jOBBdaaV9yFe0tPuw+0DQZFqtUb5i PmCkFhgn8bTBCJtOer5+jS9QLgyYnodWh7q57Lq4fP76+88/8itnJclOx5e87BmvMtVRmhd1 9fKcsa/fXvKmVTyXtH4lXEh6FE8/Dk9P//xLRF2LJhPlz/by+A8LnZ+mD2I0jsecNYT3Fc3N gvKjLgWXx9vMFhHNhV01w6nJD1nRVuZ30VWS/aLZ11NutbNbQd4GE8A47bpGAGVV6eVGKhC8 0EL1HUXqerGcVPQiJxM1kySgL/hUMjURJnVv+dr02pu+RBb6hj2Gj3d9LE3/sFbgJnJVtyBI FS1Z0YCzHJTS5QtZ75O2SbBP5Uob2jHyO1pZWUgahkZxEBSbcO4MjYvLbHzERrzimeV7/3rZ DZmZXwnJH99eXnX+G/FsCNzz8XT8MiWJqkoh+Oc0dRLNc9vvXfNZo+b86BZswpFb0aEcNjk/ q4GRvJV994KpCy+u4DxHulS6anDJNZWOpTOsOtZcJxLax+Q3VPKGHlyQgruw29+AhuppqEj1 0foCboW0W4FEs5L/hlsxNF2maCeK1W714Zf/jCwsKadEWdGxP4zExU1DtdUQSHaVVGGnR++W dI0XH7IuSCd2VYOx36pbR4sKnddYonyIhI+yyaiABDers4S8seYiFnxwKzkDnNVhGDRB+/OJ 8goUgUG7vSFDvryOq9mMmbNSDs1oGd+BUkqn3vbwtOzhKfQw0H2IljaAj29Vm1a6vYiVbfq4 cCY3stSGGTZQLBxKRExFVcdGuKKIsl0HgSfffdqGPoR9l1+gM0v1iKN+E2U7jDlLDaHN9ApW 0QShopLgTk5uFCzsepwjbvaQUHm0SdXP2jtsfoekSNNy3Ngubm2m25Ju+fETgOzkZsTFndN3 OvSGt+2rcMiCtuiKPVWYLj/kzC5pmzJQJTqtuEqzw4CXcP9fqzK3bP1rHkCnCvRw+U4SOPbs 2hTchcr8nkgpE4c5606QS2RqDi9Lc38CcorlXtJuuSP0zZ01VbiuHt8Ff7c0+3sioa03VDH9 p7+9Cf0xJ4IrXmcfvILtd8hCKj0TaCSwlI2jNVWWLglXHwk6lbrkhA62lpjR+3SMOBtnkB1r MDi4rJ6DD9q8Y5Ufisp70caaX9GRGwwSIMBQ1jUcX8d4OnLSZA5M6Mdw0d5ER7/DqvXE3YMa 1HS1/t0ZAZfIurFG0U4XPhr21tpJdukbophoUPEXStSMLNwP7Nk7vSSGPXvTflGeTc/zw2ER OzgfDkvowdmobWMH58C8e7E65QA7CAUn90I6wAksA6MHy34h/CC0C9CD0MIafqAld6AHm9IA O0iRDZCDJAWEG+zXcahBeHSKj4e2gBpEmFuYweDbZxCD/XpmRAhjjtGC9YyYYQUoYx0sEOQ8 IiaiBVZ6hhWgEnMPf79egBeE683RgtDUiA1YMkYM5pXpup1lzBAD4P4CxSIGKL3MgGNpADWY L+4f/6gvhORlzCAhFB41gG4DZxJRg9RqiGAG4aY9PBBkFiIuoAYRHyKYwRldDfZpvbPEgNoa YgBzZoG5CzFI7M1b3p3WvAuYS2hB5A7FWMFKNCxSkNCAQONcRQripiIsjBQk5bjFCpJuOv/q T7/qgMYSShDfgscJYi3AoQQgBG7EQw3GPvaDzgZJKUgBsuie/U4/RAqWm9UiH0AB8VOYCexF CrYzwD3LE2svVRpgBOnJMils4gNAJo4QhIEbX/8rZYtkMD7gTjJECNKKfMQHIrcvvikxMREh iKePxwfCdT0cYKkIIwhFIRjgOQgjiFzq7lnsfJpjBHMV89JHQwCgbaMEKRUdrI5QgqSEBhZ2 IgTJ5pmbNvaiAwn7Zyuz5IQO4IZnoQAYmoA0xwdQFCExFR/Q0hAdgDU7vfJ3jWSh3oQP6I/g 6tiDD2y3EogOoDHQPfwj8ZijA+uHN8MGzof/ASbL5lDEIAAA --dDRMvlgZJXvWKvBx Content-Type: application/octet-stream Content-Disposition: attachment; filename="sem_close.o.d.good.gz" Content-Transfer-Encoding: base64 H4sICHVUo0YCA3NlbV9jbG9zZS5vLmQuZ29vZAClWM1y4ygQvvspcpjD7kFWTZJKTfkp5gG2 KiUjbDNBQiVQ7OzTL+JHalogoewhFav/aOgfmu/HX0Q0jWgLcf7TXR5/t53ipaTNO+FC0qN4 +ufw9PRjQ+j0NH8Qo3E8lqwlfKhpydmZFPKrOQsuj7eFLSLaC7tqhlOTX7KmnSzvoq8l+5cW L8+l1S5uFfkYTQDjtO9bAZRVrZebqEDwQis19BSp68VKUtOLnE00TJKAnvDpzNRMmNW95Ws7 aG+GM7IwtOwxfnzqY2mHh7UCN1Gqpiuv5syej29lTc+sasFZjkr58pVs9kkX7OXX206VK21p z8j/0SrOlaRhaBQHQbEJ587QuJhm4yM24jUvLN/7N8h+zMzySkj5+PX2/vZaGPFiDNzr8efx ZU4SVZ+F4N/T1Em0zG2/d81nrVryo1uwCUduVY9y2OT8ogYm8lb23SumLry6gvOc6FLpqsEl 19Y6ls6w6ll7nUloH7PfUMkbenBBKu7Cbn8DGqqnsSLVV+cLuBPSbgUSzUr+G27F0HSZop0o 1rjVx1/+M7KwpJwSZUWn/jARk5uGaqshkOwqqcJOT96ldI0XX7KpSC92VYOx36lbT6sanddU onyMhI+yyaiABDers4R8sPYiEj64lZwBzpowDJqg/flGeQWKwKDd3pghP9+n1WzGLFk5h2a0 jO9AKadTb3v4nPbwOfQw0H2IjraAj29Vm1a6vYiVbfq4cCY3stSGGTZQLBxKRExFVadGuKKI sl0HgWfffdqGPoR9l1+gs0j1iKN+E+duHHNSDaEr9ApW0QShppLgTk5uFCzsepwjbvaQUHmy SdWfxjtsfoekSNNy3Ngubl2h25Ju+fETgOzsZsTFndNPOvaGj+2rcMyCruqrPVWYLz/mzC5p mzJQJTqtuEqzw4CXcP/f63Np2frXMoBOFejh8p0lcOzZta24C5X5PZNyJg5z1r0gl8jUHF6W 5v4E5BzLg6R9uiMM7Z21dbiuHt8F/7Q0+3smoa23VDH9p7+9Cf2xJIIrXmcfvILtd8hCKgMT aCSwlI2jNVWWLwlXnwg6lfrshA62lpnR+3SMOJtmkB1rMDi4rJ6DD9qyY52/FJX3qos1v6on NxgkQIChbBo4vk7xdOSsyRyY0I/hqruJnv6GVeuJuwc1qOlq/bczAi6RdWOtor0ufDTsrbWT 4jK0RDHRouKvlGgYSdwP7NU7nRLDnn1ovygv5uf54ZDEDk6HQwo9OBm1bezgFJh3L1anHGAH oeDsXkgHOIFlYPQg7RfCD0K7AD0ILazhB1pyB3qwKQ2wgxzZADnIUkC4wX4dhxqER6f4dGgJ 1CDC3MIMRt++gxjs1zMjQhhzjBasZ8QCK0AZ62CBIOcRMRMtsNILrACVmHv4+/UCvCBcb4kW hKYmbMCSMWKwrEzX7SxjgRgA9xMUixig9DIDjqUB1GC5uH/8o74QktOYQUYoPGoA3QbOZKIG udUQwQzCTXt4IMgsREygBhEfIpjBCV0N9mm9s8SA2hpiAHMmwdyFGGT25i3vnte8C5gptCBy h2KsYCUaFinIaECgca4iBXFTERZGCrJy3GIFWTedf/XnX3VAI4USxLfgcYJYC3AoAQiBG/FQ g7GP/aCzQVIOUoAsume/0w+RgnSzSvIBFBA/hYXAXqRgOwPcszyz9nKlAUaQnyyzwiY+AGTi CEEYuOn1v1K2SAbjA+4kQ4Qgr8gnfCBy++KbEhMzEYJ4+nh8IFzXwwGWijCCUBSCAZ6DMILI pe6exc6nJUawVDEvfTQEANo2SpBT0cHqCCXISmhgYSdCkG2euWljLzqQsX+2MkvO6ABueBYK gKEJSEt8AEUREnPxAS0N0QFYs/Mrf9dIFurN+ID+CK6OPfjAdiuB6AAaA93DPxKPJTqwfngL bOB0+A87hU3zryAAAA== --dDRMvlgZJXvWKvBx Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ --dDRMvlgZJXvWKvBx Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ kvm-devel mailing list kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org https://lists.sourceforge.net/lists/listinfo/kvm-devel --dDRMvlgZJXvWKvBx--