Testing status of fully virtualized guests (Intel VT) on 64bit XEN unstable

[Ed Smith <esmith@virtualiron.com>]

Summary: 64 bit guests are booting again on 2 CPU Hypervisors
          32 bit guests are NOT booting on 2 CPU Hypervisors

Test Configuration:
Dell Precision WorkStation 380, Dual Core, 2GB, 3 SATA (Intel VT)
64bit XEN Hypervisor on a RHEL4U2 64bit root (/dev/sda)
32bit fully virtualized (HVM) guest RHEL4U2 256MB (/dev/sdb)
       pae=1, acpi=1, apic=1
       kernargs clock=pit
64bit fully virtualized (HVM) guest RHEL4U2 256MB (/dev/sdc)
       pae=1, acpi=1, apic=1
       kernargs clock=pit noapic

Boot Tests:
Boot a fully virtualized (HVM) guest to the login prompt
Results are marked Pass|Fail where (n) points to a failure description

Regression Tests:
851 tests (850 ltp tests and one 30 minute user load test)
Tests are marked #Pass/#Fail where (n) points to a failure description

XEN 64bit 1 CPU Hypervisor (booted nosmp):
  ----------------------------------------------------------------------
| XEN      | Guest Kernel (SMP kernels booted with 1 CPU)              |
| Changeset|-----------------------------------------------------------|
|          | 32bit UP     | 32bit SMP    | 64bit UP     | 64bit SMP    |
|          |--------------|--------------|--------------|--------------|
|          | Boot | Test  | Boot | Test  | Boot | Test  | Boot | Test  |
|----------|------|-------|------|-------|------|-------|------|-------|
| 10031    | Pass | 850/1 | Fail |       | Pass |       | Pass | 851/0 |
|          | (8)  | (1)   | (3,8)|       | (8)  |       | (8)  |       |
|----------|------|-------|------|-------|------|-------|------|-------|
| 10005    | Pass | 849/2 | Fail |       | Pass |       | Pass | 851/0 |
|          | (8)  | (1,10)| (3,8)|       | (8)  |       | (8)  |       |
|----------|------|-------|------|-------|------|-------|------|-------|
| 9993     | Pass | 850/1 | Fail |       | Pass |       | Pass | 851/0 |
|          | (2)  | (1)   | (3,8)|       | (2)  |       | (2)  |       |
|----------|------|-------|------|-------|------|-------|------|-------|
| 9988     | Pass | 850/1 | Fail |       | Pass |       | Pass | 851/0 |
|          | (2)  | (1)   | (7)  |       | (2)  |       | (2)  |       |
|----------|------|-------|------|-------|------|-------|------|-------|
| 9983     | Pass | 850/1 | Fail |       | Pass |       | Pass | 851/0 |
|          | (2)  | (1)   | (7)  |       | (2)  |       | (2)  |       |
  ----------------------------------------------------------------------

XEN 64bit 2 CPU Hypervisor (booted smp):
  ----------------------------------------------------------------------
| XEN      | Guest Kernel (SMP kernels booted with 2 CPUs)             |
| Changeset|-----------------------------------------------------------|
|          | 32bit UP     | 32bit SMP    | 64bit UP     | 64bit SMP    |
|          |--------------|--------------|--------------|--------------|
|          | Boot | Test  | Boot | Test  | Boot | Test  | Boot | Test  |
|----------|------|-------|------|-------|------|-------|------|-------|
| 10031    | Fail |       | Fail |       | Pass |       | Pass | Fail  |
|          | (8)  |       | (8)  |       | (8)  |       | (8)  | (4)   |
|----------|------|-------|------|-------|------|-------|------|-------|
| 10005    | Fail |       | Fail |       | Pass |       | Pass |       |
|          | (8)  |       | (8)  |       | (2)  |       | (2)  |       |
|----------|------|-------|------|-------|------|-------|------|-------|
| 9993     | Fail |       | Fail |       | Fail |       | Fail |       |
|          | (8)  |       | (8)  |       | (9)  |       | (9)  |       |
|----------|------|-------|------|-------|------|-------|------|-------|
| 9988     | Fail |       | Fail |       | Fail |       | Fail |       |
|          | (8)  |       | (8)  |       | (9)  |       | (9)  |       |
|----------|------|-------|------|-------|------|-------|------|-------|
| 9983     | Pass |       | Fail |       | Pass |       | Pass | Fail  |
|          | (2)  |       | (7)  |       | (2)  |       | (5,6)| (4)   |
  ----------------------------------------------------------------------

Failures:
  1. 32bit UP guest fail ltp gettimeofday02:
     "Time is going backwards"
  2. 32bit and 64bit guests fail first boot attempt after booting XEN:
     "(XEN) Failed vm entry, domain_crash_sync called from vmx.c"
  3. 32bit SMP guest hangs on boot:
     "Uncompressing Linux... Ok, booting the kernel."
  4. 64bit SMP guest fails testing:
     Guest hangs under load.
  5. 64bit SMP guest report these messages the XEN console:
     "(XEN) spurious IRQ irq got=-1"
  6. 64bit SMP guest report these messages the XEN console during testing:
     "(XEN) <vlapic_accept_irq>level trig mode repeatedly for vector 252"
  7. 32bit SMP guest crashes on boot:
     "(XEN) __hvm_bug at vmx.c:2286"
  8. 32bit guests crash on boot:
     "(XEN) domain_crash_sync called from shadow.c:4117"
  9. 64bit guests crash XEN SMP Hypervisor on boot:
     "(XEN) CPU1 GENERAL PROTECTION FAULT"
10. 32bit UP guest failed ltp aio-stress012, guest hung
     "ata1: command 0x25 timeout, stat 0x50 host_stat 0x64"