Re: performace issue when turn on apicv

["Liuqiming (John)" <john.liuqiming@huawei.com>]

Hi,

We have test on upstream Xen, it seem the upstream version also has the same problem.

When using FIO  to test the performance of SSD passthroughed in vm the result show that: When the apicv is on, each EXIT_REASON_MSR_WRITE event spent more time than apicv is off.

Following is the xentrace result:

apicv on:

   VMExitCode                            VMExitReason      VMExitCnt         VMExitTicks        VMExitTicks/VMExitCnt

0x0000000001          EXIT_REASON_EXTERNAL_INTERRUPT         270334          2730912532      10101.99432

0x0000000012                      EXIT_REASON_VMCALL             20              438736      21936.8

0x000000001c                   EXIT_REASON_CR_ACCESS         381340          1096174160      2874.532333

0x000000001e              EXIT_REASON_IO_INSTRUCTION            413            32958356      79802.31477

0x0000000020                  EXIT_REASON_MSR_WRITE         111830           818317724      7317.515193

0x000000002d                 EXIT_REASON_EOI_INDUCED         58944           234914864      3985.390608

0x0000000030               EXIT_REASON_EPT_VIOLATION             10              298368      29836.8

Total                                                          822891          4914014740

apicv off:

   VMExitCode                            VMExitReason      VMExitCnt         VMExitTicks VMExitTicks/VMExitCnt

0x0000000001          EXIT_REASON_EXTERNAL_INTERRUPT         237100          2419717824     10205.47374

0x0000000007           EXIT_REASON_PENDING_VIRT_INTR            792             2324824     2935.383838

0x0000000012                      EXIT_REASON_VMCALL             19              415168     21850.94737

0x000000001c                   EXIT_REASON_CR_ACCESS         406848          1075393292     2643.231113

0x000000001e              EXIT_REASON_IO_INSTRUCTION            413            39433068     95479.58354

0x000000001f                    EXIT_REASON_MSR_READ             48              150528    3136

0x0000000020                   EXIT_REASON_MSR_WRITE         229609          1004000084    4372.651264

0x0000000030               EXIT_REASON_EPT_VIOLATION             10              249172     24917.2

Total                                                        874839             4541683960


On 2015/6/11 16:34, Zhang, Yang Z wrote:
> Liuqiming (John) wrote on 2015-06-11:
> > I will setup a test environment for xen upstream version but need some time.
> > Meanwhile, can you give me some idea about what MAY cause this problem?
>
> How you set the irq affinity of assigned SSD device? IIRC, the irq is migrated with vcpu by default in current Xen and you may not benefit from posted interrupt with pass-through device. So it's better to set the interrupt affinity of the pass-through device in a different core manually.
>
> >
> > I have been using xentrace to trace the problem, from what I see, the
> > apicv feature itself works
> >
> > apicv=1
> >                2583096 VMEXIT          19027420184 TSC WRMSR
> >                 459708 VMEXIT           6924749392 TSC External interrupt
> >                 293818 VMEXIT            451974088 TSC Virtualized
> > EOI
> >                    843 VMEXIT             54729244 TSC I/O
> > instruction
> >                   3260 VMEXIT             15979024 TSC Control-register
> >                   accesses 1345 VMEXIT              2199736 TSC Exception
> >                   or
> > non-maskable interrupt (NMI)
> >                     39 VMEXIT              1516768 TSC EPT violation
> >                     54 VMEXIT               891712 TSC VMCALL
> >                    205 VMEXIT               370864 TSC CPUID
> > apicv=0
> >                3416159 VMEXIT          20929093044 TSC WRMSR
> >                1098428 VMEXIT          11029334704 TSC External
> > interrupt
> >                  41128 VMEXIT             64360924 TSC Interrupt
> > window
> >                    664 VMEXIT             49245372 TSC I/O
> > instruction
> >                   3221 VMEXIT             20116036 TSC Control-register
> >                   accesses 1401 VMEXIT              2280412 TSC Exception
> >                   or
> > non-maskable interrupt (NMI)
> >                     39 VMEXIT              1581428 TSC EPT violation
> >                     53 VMEXIT               749588 TSC VMCALL
> >                    205 VMEXIT               355500 TSC CPUID
> >                    113 VMEXIT               298568 TSC RDMSR
> > RDMSR gone,so "APIC Register Virtualization" works
> >
> > apicv=1
> > IRQ                  IRQ_MOVE_CLEANUP_VECTOR(  32):
> > 21
> > IRQ                       LOCAL_TIMER_VECTOR( 249):
> > 423804
> > IRQ                     CALL_FUNCTION_VECTOR( 251):
> > 1171
> > IRQ                       EVENT_CHECK_VECTOR( 252):
> > 1130
> > IRQ                    INVALIDATE_TLB_VECTOR( 253):
> > 1
> > apicv=0
> > IRQ                  IRQ_MOVE_CLEANUP_VECTOR(  32):
> > 22
> > IRQ                      LAST_DYNAMIC_VECTOR( 223):
> > 27
> > IRQ                       LOCAL_TIMER_VECTOR( 249):
> > 448057
> > IRQ                     CALL_FUNCTION_VECTOR( 251):
> > 1173
> > IRQ                       EVENT_CHECK_VECTOR( 252):
> > 608024
> >
> > vmexit caused by External interrupt: EVENT_CHECK_VECTOR reduced a lot,
> > so "Virtual Interrupt Delivery" and "Posted Interrupt Processing" works, I guess.
>
> Maybe you can turn on the three features one by one(need some changes in code). Then it will help us to know which feature really causes the problem
>
> >
> > I think the problem is not caused by apicv itself, maybe some other
> > logic has conflict with apicv.
> >
> > On 2015/6/11 15:35, Zhang, Yang Z wrote:
> >> Liuqiming (John) wrote on 2015-06-11:
> >>> Hi,
> >>>
> >>> Recently I encounter a strange performance problem with APIC
> >>> virtualization.
> >>>
> >>> My host has Intel(R) Xeon(R) CPU E7-4890 v2 CPU installed which
> >>> support APIC virtualization and x2apic, and there are 4 socket * 15
> >>> cores_per_socket = 60 core available for VM. There is also a SSD disk
> >>> on host and the host support vt-d, so I can passsthrough this SSD to
> >>> VM.
> >>>
> >>> A VM created with 60 vcpus,  400G memory and SSD device assigned.
> >>> I pin these vcpus 1:1 to phisical cpu, and in this VM only keep 15 vcpus
> >>>     online.
> >>> The problem is: when apicv turn on, a significant performace
> >>> decrease can be observed and it seems related to cpu topology.
> >>>
> >>> I had test follow cases
> >>> apicv=1:
> >>> 1)  ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> >>>            PIN TO
> >>>               PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> >>> 2)  ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> >>>            PIN TO
> >>>               PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> >>> apicv=0:
> >>> 3)  ONLINE VCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> >>>            PIN TO
> >>>               PCPU {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14}
> >>> 4)  ONLINE VCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> >>>            PIN TO
> >>>               PCPU {0,5,9,13,17,21,25,29,33,37,41,45,49,53,57}
> >>> the result is (the lower the better):
> >>> 1) 989
> >>> 2) 2209
> >>> 3) 978
> >>> 4) 1130
> >>>
> >>> It is a database testcase running on suse11sp3 system in the VM,
> >>> and I had traced that "read" and "write" syscall get much slower in 2) case.
> >>>
> >>> I have disabled NUMA in BIOS, so it seems apicv cause this bad
> >>> performance when using cpus in different nodes.
> >>>
> >>> Can any one shed some light on this?
> >>>
> >>> Btw, I am using xen 4.1.5 version with apicv backported, so I am
> >>> not sure whether something broken when backporting or just apicv
> >>> behaves this way.
> >>
> >> Can you retest it based on upstream Xen? Just as you suspected, your
> >> backporting may be the culprit.
> >>
> >>>
> >>>
> >>>
> >>
> >>
> >> Best regards,
> >> Yang
> >>
> >>
> >
>
>
> Best regards,
> Yang
>
>