* Re: PCI capabilities support for assigned devices
2010-03-10 13:52 PCI capabilities support for assigned devices Sebastian Hetze
@ 2010-03-12 14:19 ` Sebastian Hetze
0 siblings, 0 replies; 2+ messages in thread
From: Sebastian Hetze @ 2010-03-12 14:19 UTC (permalink / raw)
To: kvm
On Wed, Mar 10, 2010 at 02:52:30PM +0100, Sebastian Hetze wrote:
> Hi *,
>
> in qemu-kvm/hw/device-assignment.c assigned_device_pci_cap_init()
> appearently only PCI_CAP_ID_MSI and PCI_CAP_ID_MSIX are exposed
> to the guest.
>
> Linux Broadcom bnx2 and tg3 drivers expect PCI_CAP_ID_PM to be present.
>
> Are there any plans to implement this and possibly other PCI capability
> features for assigned devices?
>
> If not, is there a list of network cards known to work with PCI
> assignment in KVM?
>
Answering my own mail, I can contribute experience with direct
assignment of two network cards.
I was able to test Intel 82571EB (e1000e) and 82575EB (igb). Both drivers work
with IOMMU assigned devices in a SMP guest running 2.6.31-16-generic-pae
ubuntu kernel with 4 CPUs.
However, the e1000e driver hangs reproduceable on NFS load after 10 to 20 minutes
[ 1312.989127] 0000:00:05.0: eth0: Detected Tx Unit Hang:
[ 1312.989130] TDH <b8>
[ 1312.989131] TDT <f4>
[ 1312.989132] next_to_use <f4>
[ 1312.989133] next_to_clean <b5>
[ 1312.989134] buffer_info[next_to_clean]:
[ 1312.989135] time_stamp <3d3dc>
[ 1312.989136] next_to_watch <b8>
[ 1312.989137] jiffies <3dd3f>
[ 1312.989138] next_to_watch.status <0>
[ 1313.988199] ------------[ cut here ]------------
[ 1313.988237] WARNING: at /build/buildd/linux-2.6.31/net/sched/sch_generic.c:246 dev_watchdog+0x1f6/0x210()
[ 1313.988247] Hardware name:
[ 1313.988249] NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
[ 1313.988251] Modules linked in: ppdev lp parport autofs4 video output nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc e100 via_rhine 3c59x 8139too mii snd_ens1370 gameport snd_rawmidi snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr psmouse virtio_net i2c_piix4 serio_raw joydev virtio_pci virtio_ring virtio usbhid floppy fbcon tileblit font bitblit softcursor e1000e
[ 1313.988294] Pid: 0, comm: swapper Not tainted 2.6.31-16-generic-pae #53-Ubuntu
[ 1313.988297] Call Trace:
[ 1313.988323] [<c0146a1d>] warn_slowpath_common+0x6d/0xa0
[ 1313.988327] [<c04b6746>] ? dev_watchdog+0x1f6/0x210
[ 1313.988331] [<c04b6746>] ? dev_watchdog+0x1f6/0x210
[ 1313.988334] [<c0146a96>] warn_slowpath_fmt+0x26/0x30
[ 1313.988337] [<c04b6746>] dev_watchdog+0x1f6/0x210
[ 1313.988350] [<c0159efb>] ? insert_work+0x5b/0xa0
[ 1313.988361] [<c0128098>] ? default_spin_lock_flags+0x8/0x10
[ 1313.988375] [<c0576bda>] ? _spin_lock_irqsave+0x2a/0x40
[ 1313.988379] [<c015a291>] ? __queue_work+0x31/0x40
[ 1313.988383] [<c01520d7>] run_timer_softirq+0x117/0x200
[ 1313.988390] [<c016c045>] ? tick_dev_program_event+0x45/0xe0
[ 1313.988393] [<c04b6550>] ? dev_watchdog+0x0/0x210
[ 1313.988399] [<c014cc40>] __do_softirq+0x90/0x1a0
[ 1313.988402] [<c0161e83>] ? hrtimer_interrupt+0x183/0x210
[ 1313.988406] [<c016c488>] ? tick_do_update_jiffies64+0x118/0x160
[ 1313.988410] [<c014cd8d>] do_softirq+0x3d/0x40
[ 1313.988414] [<c014cecd>] irq_exit+0x5d/0x70
[ 1313.988425] [<c011ddc7>] smp_apic_timer_interrupt+0x57/0x90
[ 1313.988433] [<c0103d71>] apic_timer_interrupt+0x31/0x40
[ 1313.988437] [<c0127365>] ? native_safe_halt+0x5/0x10
[ 1313.988441] [<c010a5b6>] default_idle+0x46/0xd0
[ 1313.988444] [<c010202c>] cpu_idle+0x8c/0xd0
[ 1313.988454] [<c0572498>] start_secondary+0xc6/0xc8
[ 1313.988457] ---[ end trace c5b28b21ada19e1d ]---
I also tried the 1.1.2-NAPI driver from Intel without success.
The 82575EB device works fine.
To get an impression of the performance improvement you
win from an directly assigned NIC, take a look at the
following numbers taken from sysstat in a real live
production system. Both tables show the same guest
on two subsequent days with similar every day workloads.
virtio-net
00:00:01 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
06:05:01 eth0 577,01 582,57 141,83 211,79 0,00 0,00 0,00
06:15:01 eth0 169,70 181,93 70,04 90,49 0,00 0,00 0,00
06:25:01 eth0 135,22 138,81 60,94 56,01 0,00 0,00 0,00
06:35:01 eth0 787,44 879,40 183,05 498,02 0,00 0,00 0,00
06:45:01 eth0 1430,22 1660,49 323,80 1163,14 0,00 0,00 0,00
06:55:01 eth0 1524,15 1730,98 400,70 1112,32 0,00 0,00 0,00
07:05:02 eth0 1300,95 1414,48 307,40 741,41 0,00 0,00 0,00
07:15:01 eth0 380,77 435,28 141,65 289,76 0,00 0,00 0,00
07:25:01 eth0 312,16 365,33 112,46 230,35 0,00 0,00 0,00
07:35:01 eth0 758,51 801,66 169,99 375,84 0,00 0,00 0,00
07:45:01 eth0 1685,90 1922,92 301,87 1408,62 0,00 0,00 0,00
07:55:02 eth0 2531,08 3205,48 579,60 3033,27 0,00 0,00 0,00
08:05:02 eth0 2011,90 2180,31 426,22 1041,64 0,00 0,00 0,00
08:15:01 eth0 1054,10 1252,39 267,30 648,87 0,00 0,00 0,00
08:25:02 eth0 613,17 761,34 170,96 551,79 0,00 0,00 0,00
08:35:02 eth0 858,47 921,50 205,33 440,02 0,00 0,00 0,00
08:45:02 eth0 1426,27 1465,96 336,28 635,85 0,00 0,00 0,00
08:55:02 eth0 1539,78 1600,87 361,98 716,52 0,00 0,00 0,00
09:05:01 eth0 998,43 1386,98 245,87 1337,09 0,00 0,00 0,00
09:15:01 eth0 893,36 1452,95 306,11 1630,68 0,00 0,00 0,00
09:25:01 eth0 594,01 426,42 463,71 120,65 0,00 0,00 0,00
09:35:01 eth0 842,04 911,00 254,95 517,71 0,00 0,00 0,00
09:45:01 eth0 1528,67 1839,70 288,08 1208,60 0,00 0,00 0,00
09:55:02 eth0 2059,99 3412,35 370,46 3451,00 0,00 0,00 0,00
82575EB igb
00:00:01 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
06:05:01 eth0 2008,58 2132,93 527,77 677,29 0,00 0,00 0,03
06:15:01 eth0 1802,28 1953,50 446,06 614,17 0,00 0,00 0,00
06:25:01 eth0 2734,59 3239,42 521,04 2049,75 0,00 0,00 0,03
06:35:01 eth0 1282,02 1712,26 386,02 1368,42 0,00 0,00 0,03
06:45:01 eth0 3402,61 3862,23 1017,19 2557,92 0,00 0,00 0,03
06:55:01 eth0 22594,46 24367,90 5194,55 13901,20 0,00 0,00 0,03
07:05:02 eth0 12398,94 13005,46 2660,65 6348,80 0,00 0,00 0,03
07:15:02 eth0 24914,17 26162,36 4857,22 11132,92 0,00 0,00 0,00
07:25:01 eth0 12499,14 15099,16 2602,93 11448,38 0,00 0,00 0,02
07:35:01 eth0 2944,58 3583,83 611,38 2354,26 0,00 0,00 0,03
07:45:02 eth0 2693,69 2985,04 652,37 1195,89 0,00 0,00 0,02
07:55:01 eth0 24266,21 26724,03 4175,56 14344,99 0,00 0,00 0,03
08:05:01 eth0 31024,78 33673,28 6005,85 18423,59 0,00 0,00 0,03
08:15:01 eth0 10117,74 12978,11 2494,69 11749,02 0,00 0,00 0,00
08:25:01 eth0 2952,17 3263,70 970,93 1934,32 0,00 0,00 0,03
08:35:01 eth0 5603,91 8992,43 986,02 9360,29 0,00 0,00 0,92
08:45:01 eth0 4192,03 5760,26 563,43 4769,12 0,00 0,00 0,10
08:55:02 eth0 21581,43 23074,14 4491,50 11611,92 0,00 0,00 0,10
09:05:02 eth0 26496,69 27805,50 5304,54 13161,67 0,00 0,00 0,10
09:15:01 eth0 16795,28 17719,70 4292,59 9279,21 0,00 0,00 0,45
09:25:01 eth0 9177,53 10760,43 4185,65 10342,94 0,00 0,00 0,20
09:35:01 eth0 6112,76 8195,99 1694,60 8275,08 0,00 0,00 0,10
09:45:01 eth0 7057,25 4243,00 5812,92 1392,42 0,00 0,00 0,08
09:55:01 eth0 24252,42 24980,30 5030,76 10703,52 0,00 0,00 0,10
^ permalink raw reply [flat|nested] 2+ messages in thread