From mboxrd@z Thu Jan 1 00:00:00 1970 From: Marek Marczykowski Subject: Re: xen-4.1: PV domain hanging at startup, jiffies stopped Date: Wed, 31 Aug 2011 18:27:54 +0200 Message-ID: <4E5E610A.1010501@mimuw.edu.pl> References: <4E5A3F0A.8060700@mimuw.edu.pl> <20110829200749.GA17265@dumpdata.com> <4E5BF4C3.2050108@mimuw.edu.pl> <20110829205938.GB18697@dumpdata.com> <4E5D1B57.7040106@mimuw.edu.pl> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0543069950==" Return-path: In-Reply-To: <4E5D1B57.7040106@mimuw.edu.pl> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Konrad Rzeszutek Wilk Cc: "xen-devel@lists.xensource.com" , Joanna Rutkowska , Rafal Wojtczuk List-Id: xen-devel@lists.xenproject.org This is a cryptographically signed message in MIME format. --===============0543069950== Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms010309000505010907010502" This is a cryptographically signed message in MIME format. --------------ms010309000505010907010502 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On 30.08.2011 19:18, Marek Marczykowski wrote: > On 29.08.2011 22:59, Konrad Rzeszutek Wilk wrote: >> Ok, but I am still unsure where it is hanging in DomU. Can you run wit= h >> 'console=3Dhvc0 debug initcall_debug loglevel=3D8 earlyprintk=3Dxen' t= o get an idea >> of what is stuck in the guest?=20 >=20 > With "initcall_debug" parameter problem does not appear (at least for > 200 domU starts)... It looks like race condition which doesn't happens > on slowed down kernel (by printing lots of debug info). This also > explains why this bug appears only on fast hardware. >=20 >> You might also have better luck using >> 'xenctx' to get a stack trace of what is hangning in the guest. >> (you will need the System.map file from the guest's kernel.. but that = should >> be fairly easy to extract). >=20 > xenctx didn't provide any useful data :/ It always shows following trac= e > for hanged domU: > ----------------- > rip: ffffffff810013aa hypercall_page+0x3aa > flags: 00001246 i z p > rsp: ffffffff81801ee0 > rax: 0000000000000000 rcx: ffffffff810013aa rdx: 0000000000000000 > rbx: ffffffff81800010 rsi: 00000000deadbeef rdi: 00000000deadbeef > rbp: ffffffff81801ef8 r8: 0000000000000000 r9: 0000000000000000 > r10: 0000000000000000 r11: 0000000000000246 r12: 0000000000000000 > r13: 0000000000000000 r14: ffffffffffffffff r15: 0000000000000000 > cs: e033 ss: e02b ds: 0000 es: 0000 > fs: 0000 @ 0000000000000000 > gs: 0000 @ ffff880018ee7000/0000000000000000 > Code (instr addr ffffffff810013aa) > cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 1d 00 00 00 0f 05 <41> 5b > 59 c3 cc cc cc cc cc cc cc >=20 >=20 > Stack: > 0000000000000000 0000000000000000 ffffffff810072a0 ffffffff81801f18 > ffffffff81012528 ffffffff81800010 ffffffff8185a2a0 ffffffff81801f38 > ffffffff81009faf 0000000000000000 6db6db6db6db6db7 ffffffff81801f48 > ffffffff813fb388 ffffffff81801f88 ffffffff81875c79 ffffffff81801f88 >=20 > Call Trace: > [] hypercall_page+0x3aa <-- > [] xen_safe_halt+0x10 > [] default_idle+0x58 > [] cpu_idle+0x5f > [] rest_init+0x68 > [] start_kernel+0x36f > [] x86_64_start_reservations+0x131 > [] xen_start_kernel+0x5f1 > ------------------ >=20 > I've collected few more messages from successful and failed domU starts= =2E > The only difference is the place where "Switched to NOHz mode on CPU #0= " > appears and existence of "CE: xen increased min_delta_ns to ..." and > "CE: Reprogramming failure. Giving up" messages. >=20 > I think it can be related to: > http://lists.xensource.com/archives/html/xen-devel/2010-07/msg00649.htm= l > (this was on HVM not PV, but looks similar) >=20 > I've tried also xenpm set-max-cstate 0 and tsc_mode=3D1 in domU config,= > but it doesn't help. Also pinning vcpu doesn't help (this domUs have > only 1 vcpu). Is 'xenpm set-max-cstate 0' the same as booting xen with > max_cstate=3D0? Looks like tsc_mode=3D2 solves the problem. --=20 Pozdrawiam / Best Regards, Marek Marczykowski | RLU #390519 marmarek at mimuw edu pl | xmpp:marmarek at staszic waw pl --------------ms010309000505010907010502 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIRqDCC BVcwggQ/oAMCAQICEGdLwnOg7SU/xzieX7FaFewwDQYJKoZIhvcNAQEFBQAwgd0xCzAJBgNV BAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1 c3QgTmV0d29yazE7MDkGA1UECxMyVGVybXMgb2YgdXNlIGF0IGh0dHBzOi8vd3d3LnZlcmlz aWduLmNvbS9ycGEgKGMpMDkxHjAcBgNVBAsTFVBlcnNvbmEgTm90IFZhbGlkYXRlZDE3MDUG A1UEAxMuVmVyaVNpZ24gQ2xhc3MgMSBJbmRpdmlkdWFsIFN1YnNjcmliZXIgQ0EgLSBHMzAe Fw0xMDEyMDgwMDAwMDBaFw0xMTEyMDgyMzU5NTlaMIIBGjEXMBUGA1UEChMOVmVyaVNpZ24s IEluYy4xHzAdBgNVBAsTFlZlcmlTaWduIFRydXN0IE5ldHdvcmsxRjBEBgNVBAsTPXd3dy52 ZXJpc2lnbi5jb20vcmVwb3NpdG9yeS9SUEEgSW5jb3JwLiBieSBSZWYuLExJQUIuTFREKGMp OTgxHjAcBgNVBAsTFVBlcnNvbmEgTm90IFZhbGlkYXRlZDEzMDEGA1UECxMqRGlnaXRhbCBJ RCBDbGFzcyAxIC0gTmV0c2NhcGUgRnVsbCBTZXJ2aWNlMRswGQYDVQQDFBJNQVJFSyBNYXJj enlrb3dza2kxJDAiBgkqhkiG9w0BCQEWFW1hcm1hcmVrQG1pbXV3LmVkdS5wbDCCASIwDQYJ KoZIhvcNAQEBBQADggEPADCCAQoCggEBAMGGGDj1KJXXy13nNqo0NP7EJNnzjWHih+Z88UA/ c/CZOIu3/SUnA9gly922moQ+r4AM0k6EQaVkAdldSOy1hO/fTcJQBwDemcnACWstMDMvIbWn dkxyxNgh8bQD/xKXx1SyfKTixcx0QZXosY8HnHLWeF0Te5U9J8yUPxigCFJFe6rtY2iARrWc qWXDr4DRQc5J43aohBXGRHw+OakREqLZds6HEE6+YvJH+udcfqlrfkLKHJeiyA4UPtZwBvam EGOKrm2rqyps/JeivaTs6zKh0+4eNxSKlOGTUmj4eHVb/K3gkbX6bihmWS+9nnN3I4HL+KPZ j88xQCHX07Gh4nECAwEAAaOB0jCBzzAJBgNVHRMEAjAAMEQGA1UdIAQ9MDswOQYLYIZIAYb4 RQEHFwEwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYTALBgNV HQ8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMFAGA1UdHwRJMEcwRaBD oEGGP2h0dHA6Ly9pbmRjMWRpZ2l0YWxpZC1nMy1jcmwudmVyaXNpZ24uY29tL0luZEMxRGln aXRhbElELUczLmNybDANBgkqhkiG9w0BAQUFAAOCAQEAl7oehxokYa9omPurM44GROaRxtIQ acWWpCgeMNDqJXyEgm30u0hY1szXiSyVnkIx/sh4HuBqvXJnZZvLjksEIzdALTgCAskojzeT oeX+340loS87OO82/XUDT5YY3+R5G+4oQIxKVqF7kDmOuJ2opbjU4dFxHhIypiEcWjG2sjWd 127poxMQeY4lCk3qKw94pK5RpVPJK8hzaf+pWoL2D7i3L2xE+DMzZJ1OB3AxcGxhvupUq+zz 3Z9IIHo6DNOwUyCNR+ShVnuCaoJcya35mQzhs1s5+LcP/ysrUzhPo+iYNXGv+q5iNQEXZ7Dw kPgAm17fnxmih6D4KRbmrc3xpjCCBVcwggQ/oAMCAQICEGdLwnOg7SU/xzieX7FaFewwDQYJ KoZIhvcNAQEFBQAwgd0xCzAJBgNVBAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEf MB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazE7MDkGA1UECxMyVGVybXMgb2YgdXNl IGF0IGh0dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9ycGEgKGMpMDkxHjAcBgNVBAsTFVBlcnNv bmEgTm90IFZhbGlkYXRlZDE3MDUGA1UEAxMuVmVyaVNpZ24gQ2xhc3MgMSBJbmRpdmlkdWFs IFN1YnNjcmliZXIgQ0EgLSBHMzAeFw0xMDEyMDgwMDAwMDBaFw0xMTEyMDgyMzU5NTlaMIIB GjEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlTaWduIFRydXN0IE5l dHdvcmsxRjBEBgNVBAsTPXd3dy52ZXJpc2lnbi5jb20vcmVwb3NpdG9yeS9SUEEgSW5jb3Jw LiBieSBSZWYuLExJQUIuTFREKGMpOTgxHjAcBgNVBAsTFVBlcnNvbmEgTm90IFZhbGlkYXRl ZDEzMDEGA1UECxMqRGlnaXRhbCBJRCBDbGFzcyAxIC0gTmV0c2NhcGUgRnVsbCBTZXJ2aWNl MRswGQYDVQQDFBJNQVJFSyBNYXJjenlrb3dza2kxJDAiBgkqhkiG9w0BCQEWFW1hcm1hcmVr QG1pbXV3LmVkdS5wbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMGGGDj1KJXX y13nNqo0NP7EJNnzjWHih+Z88UA/c/CZOIu3/SUnA9gly922moQ+r4AM0k6EQaVkAdldSOy1 hO/fTcJQBwDemcnACWstMDMvIbWndkxyxNgh8bQD/xKXx1SyfKTixcx0QZXosY8HnHLWeF0T e5U9J8yUPxigCFJFe6rtY2iARrWcqWXDr4DRQc5J43aohBXGRHw+OakREqLZds6HEE6+YvJH +udcfqlrfkLKHJeiyA4UPtZwBvamEGOKrm2rqyps/JeivaTs6zKh0+4eNxSKlOGTUmj4eHVb /K3gkbX6bihmWS+9nnN3I4HL+KPZj88xQCHX07Gh4nECAwEAAaOB0jCBzzAJBgNVHRMEAjAA MEQGA1UdIAQ9MDswOQYLYIZIAYb4RQEHFwEwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cu dmVyaXNpZ24uY29tL3JwYTALBgNVHQ8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsG AQUFBwMCMFAGA1UdHwRJMEcwRaBDoEGGP2h0dHA6Ly9pbmRjMWRpZ2l0YWxpZC1nMy1jcmwu dmVyaXNpZ24uY29tL0luZEMxRGlnaXRhbElELUczLmNybDANBgkqhkiG9w0BAQUFAAOCAQEA l7oehxokYa9omPurM44GROaRxtIQacWWpCgeMNDqJXyEgm30u0hY1szXiSyVnkIx/sh4HuBq vXJnZZvLjksEIzdALTgCAskojzeToeX+340loS87OO82/XUDT5YY3+R5G+4oQIxKVqF7kDmO uJ2opbjU4dFxHhIypiEcWjG2sjWd127poxMQeY4lCk3qKw94pK5RpVPJK8hzaf+pWoL2D7i3 L2xE+DMzZJ1OB3AxcGxhvupUq+zz3Z9IIHo6DNOwUyCNR+ShVnuCaoJcya35mQzhs1s5+LcP /ysrUzhPo+iYNXGv+q5iNQEXZ7DwkPgAm17fnxmih6D4KRbmrc3xpjCCBu4wggXWoAMCAQIC EHEVZgVK5JEhTem8RPms09wwDQYJKoZIhvcNAQEFBQAwgcoxCzAJBgNVBAYTAlVTMRcwFQYD VQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazE6 MDgGA1UECxMxKGMpIDE5OTkgVmVyaVNpZ24sIEluYy4gLSBGb3IgYXV0aG9yaXplZCB1c2Ug b25seTFFMEMGA1UEAxM8VmVyaVNpZ24gQ2xhc3MgMSBQdWJsaWMgUHJpbWFyeSBDZXJ0aWZp Y2F0aW9uIEF1dGhvcml0eSAtIEczMB4XDTA5MDUwMTAwMDAwMFoXDTE5MDQzMDIzNTk1OVow gd0xCzAJBgNVBAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVy aVNpZ24gVHJ1c3QgTmV0d29yazE7MDkGA1UECxMyVGVybXMgb2YgdXNlIGF0IGh0dHBzOi8v d3d3LnZlcmlzaWduLmNvbS9ycGEgKGMpMDkxHjAcBgNVBAsTFVBlcnNvbmEgTm90IFZhbGlk YXRlZDE3MDUGA1UEAxMuVmVyaVNpZ24gQ2xhc3MgMSBJbmRpdmlkdWFsIFN1YnNjcmliZXIg Q0EgLSBHMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAO3ER98qKB18Bmu71yEy yWwTj+mxjUFONPfaC+Nq+mWIIAsRE+mb4ElOi2/VAdBfDUeRilpMdD4/xpEJu0w0no1uoYJR YvdpdliWB6+eFBgHT1q9n9IxslQZc0ZqGUIR7BJzIY313DDN5dlWCjHFNm0pFJe9LdqJRxmI 2EsEPeu2PGcedAATDdCG2pNn+DMDrho8a2l49sAsjuGDP3f5mf/+n1JawrSHCthsqUfBVCll Qz5KwJYfwa33d69ssQRevsG2lC2XkC0n0rse6YNqhPbEsq4jBmUmpSdYKwcitG+mYkgad/LV UCeaKdOW+yj1uiR2YuOMWev7btVCxL5Bx/UCAwEAAaOCArkwggK1MDQGCCsGAQUFBwEBBCgw JjAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AudmVyaXNpZ24uY29tMBIGA1UdEwEB/wQIMAYB Af8CAQAwcAYDVR0gBGkwZzBlBgtghkgBhvhFAQcXATBWMCgGCCsGAQUFBwIBFhxodHRwczov L3d3dy52ZXJpc2lnbi5jb20vY3BzMCoGCCsGAQUFBwICMB4aHGh0dHBzOi8vd3d3LnZlcmlz aWduLmNvbS9ycGEwNAYDVR0fBC0wKzApoCegJYYjaHR0cDovL2NybC52ZXJpc2lnbi5jb20v cGNhMS1nMy5jcmwwDgYDVR0PAQH/BAQDAgEGMG4GCCsGAQUFBwEMBGIwYKFeoFwwWjBYMFYW CWltYWdlL2dpZjAhMB8wBwYFKw4DAhoEFEtruSiWBgy70FI4mymsSweLIQUYMCYWJGh0dHA6 Ly9sb2dvLnZlcmlzaWduLmNvbS92c2xvZ28xLmdpZjAuBgNVHREEJzAlpCMwITEfMB0GA1UE AxMWUHJpdmF0ZUxhYmVsNC0yMDQ4LTExODAdBgNVHQ4EFgQUeUdhCEH9OASiS+e1zPVD9kkr EfgwgfEGA1UdIwSB6TCB5qGB0KSBzTCByjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlT aWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEo YykgMTk5OSBWZXJpU2lnbiwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYD VQQDEzxWZXJpU2lnbiBDbGFzcyAxIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0 aG9yaXR5IC0gRzOCEQCLW3VWhFSFCwDPrzhIzrGkMA0GCSqGSIb3DQEBBQUAA4IBAQA5Tc9B mYG1qQW1UjjpOYSJbOQ0qFrn2GwJTCQaulmkhztzIfGTgc+/aGNaZ/41hSuhw12jSsI6Gd0w 1sxN7/HSgZfKVFpDvzeLeo4ZjQ9DqIzyr2CzFYqzlZw84J6zJ5ikNXIX5fwqXYfTig3C0UUq +MD0rCqTOtWuEnAI6/s74nfs6CtkNXbNutrg0csU1nFYm77VPn222egkxSRmTF2RH3azFz5/ DcYhiS+zN7ih/1yybUneZVJC+w6I0u1KHb9L4/jMcvpIDmWOScjW+JmYO7eUPjFxBof6bFlT LtffK+1fYwCsFe0DuFUWjMZoA+ciqHMLsbyg2lJY3QoOf8GCMYIE7DCCBOgCAQEwgfIwgd0x CzAJBgNVBAYTAlVTMRcwFQYDVQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNp Z24gVHJ1c3QgTmV0d29yazE7MDkGA1UECxMyVGVybXMgb2YgdXNlIGF0IGh0dHBzOi8vd3d3 LnZlcmlzaWduLmNvbS9ycGEgKGMpMDkxHjAcBgNVBAsTFVBlcnNvbmEgTm90IFZhbGlkYXRl ZDE3MDUGA1UEAxMuVmVyaVNpZ24gQ2xhc3MgMSBJbmRpdmlkdWFsIFN1YnNjcmliZXIgQ0Eg LSBHMwIQZ0vCc6DtJT/HOJ5fsVoV7DAJBgUrDgMCGgUAoIICzjAYBgkqhkiG9w0BCQMxCwYJ KoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xMTA4MzExNjI3NTRaMCMGCSqGSIb3DQEJBDEW BBS8v+8KVqN1qhc/Iy2wsl+AGzuA4zBfBgkqhkiG9w0BCQ8xUjBQMAsGCWCGSAFlAwQBAjAK BggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYI KoZIhvcNAwICASgwggEDBgkrBgEEAYI3EAQxgfUwgfIwgd0xCzAJBgNVBAYTAlVTMRcwFQYD VQQKEw5WZXJpU2lnbiwgSW5jLjEfMB0GA1UECxMWVmVyaVNpZ24gVHJ1c3QgTmV0d29yazE7 MDkGA1UECxMyVGVybXMgb2YgdXNlIGF0IGh0dHBzOi8vd3d3LnZlcmlzaWduLmNvbS9ycGEg KGMpMDkxHjAcBgNVBAsTFVBlcnNvbmEgTm90IFZhbGlkYXRlZDE3MDUGA1UEAxMuVmVyaVNp Z24gQ2xhc3MgMSBJbmRpdmlkdWFsIFN1YnNjcmliZXIgQ0EgLSBHMwIQZ0vCc6DtJT/HOJ5f sVoV7DCCAQUGCyqGSIb3DQEJEAILMYH1oIHyMIHdMQswCQYDVQQGEwJVUzEXMBUGA1UEChMO VmVyaVNpZ24sIEluYy4xHzAdBgNVBAsTFlZlcmlTaWduIFRydXN0IE5ldHdvcmsxOzA5BgNV BAsTMlRlcm1zIG9mIHVzZSBhdCBodHRwczovL3d3dy52ZXJpc2lnbi5jb20vcnBhIChjKTA5 MR4wHAYDVQQLExVQZXJzb25hIE5vdCBWYWxpZGF0ZWQxNzA1BgNVBAMTLlZlcmlTaWduIENs YXNzIDEgSW5kaXZpZHVhbCBTdWJzY3JpYmVyIENBIC0gRzMCEGdLwnOg7SU/xzieX7FaFeww DQYJKoZIhvcNAQEBBQAEggEANE30ODMJ//NMMjwujTSUbJ21CBc/q2yQfVx7xKLUVyHP4qg7 SQd3c11WHxlcF0tzs99ISz5mbk75HJJpo0h66Odfd3hIGCKg2Cgsw1GHEOB+YutAk2Y5Y0m0 IzvYN2p1X9SQQMorPNu/+W68nP6qeN6rcnQrkyrJqifsx2r0GVfUXfGqXkJZCYPrfe+i6SDY rxHzy5SYhJG+Xx9iyZshsJbWQ2O7I8KJ6ex5dKCpf5hEh8LQUMzQOHjO/aTVwNEaKZeGhh/j Z8mXZv+Xk3Q60FymKAcCPovgfI8Nn2LEYsvi7J7H7XioJv8zkO1M9ld+dfmESRe0t220moWL XskE5QAAAAAAAA== --------------ms010309000505010907010502-- --===============0543069950== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --===============0543069950==--