PCI capabilities support for assigned devices

PCI capabilities support for assigned devices

[Sebastian Hetze]

Hi *,

in qemu-kvm/hw/device-assignment.c assigned_device_pci_cap_init()
appearently only PCI_CAP_ID_MSI and PCI_CAP_ID_MSIX are exposed
to the guest.

Linux Broadcom bnx2 and tg3 drivers expect PCI_CAP_ID_PM to be present.

Are there any plans to implement this and possibly other PCI capability
features for assigned devices?

If not, is there a list of network cards known to work with PCI
assignment in KVM?

Best Regards,

  Sebastian

Re: PCI capabilities support for assigned devices

[Sebastian Hetze]

On Wed, Mar 10, 2010 at 02:52:30PM +0100, Sebastian Hetze wrote:
> Hi *,
> 
> in qemu-kvm/hw/device-assignment.c assigned_device_pci_cap_init()
> appearently only PCI_CAP_ID_MSI and PCI_CAP_ID_MSIX are exposed
> to the guest.
> 
> Linux Broadcom bnx2 and tg3 drivers expect PCI_CAP_ID_PM to be present.
> 
> Are there any plans to implement this and possibly other PCI capability
> features for assigned devices?
> 
> If not, is there a list of network cards known to work with PCI
> assignment in KVM?
> 

Answering my own mail, I can contribute experience with direct
assignment of two network cards.
I was able to test Intel 82571EB (e1000e) and 82575EB (igb). Both drivers work
with IOMMU assigned devices in a SMP guest running 2.6.31-16-generic-pae
ubuntu kernel with 4 CPUs.

However, the e1000e driver hangs reproduceable on NFS load after 10 to 20 minutes

[ 1312.989127] 0000:00:05.0: eth0: Detected Tx Unit Hang:
[ 1312.989130] TDH <b8>
[ 1312.989131] TDT <f4>
[ 1312.989132] next_to_use <f4>
[ 1312.989133] next_to_clean <b5>
[ 1312.989134] buffer_info[next_to_clean]:
[ 1312.989135] time_stamp <3d3dc>
[ 1312.989136] next_to_watch <b8>
[ 1312.989137] jiffies <3dd3f>
[ 1312.989138] next_to_watch.status <0>
[ 1313.988199] ------------[ cut here ]------------
[ 1313.988237] WARNING: at /build/buildd/linux-2.6.31/net/sched/sch_generic.c:246 dev_watchdog+0x1f6/0x210()
[ 1313.988247] Hardware name:
[ 1313.988249] NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
[ 1313.988251] Modules linked in: ppdev lp parport autofs4 video output nfsd exportfs nfs lockd nfs_acl auth_rpcgss sunrpc e100 via_rhine 3c59x 8139too mii snd_ens1370 gameport snd_rawmidi snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr psmouse virtio_net i2c_piix4 serio_raw joydev virtio_pci virtio_ring virtio usbhid floppy fbcon tileblit font bitblit softcursor e1000e
[ 1313.988294] Pid: 0, comm: swapper Not tainted 2.6.31-16-generic-pae #53-Ubuntu
[ 1313.988297] Call Trace:
[ 1313.988323] [<c0146a1d>] warn_slowpath_common+0x6d/0xa0
[ 1313.988327] [<c04b6746>] ? dev_watchdog+0x1f6/0x210
[ 1313.988331] [<c04b6746>] ? dev_watchdog+0x1f6/0x210
[ 1313.988334] [<c0146a96>] warn_slowpath_fmt+0x26/0x30
[ 1313.988337] [<c04b6746>] dev_watchdog+0x1f6/0x210
[ 1313.988350] [<c0159efb>] ? insert_work+0x5b/0xa0
[ 1313.988361] [<c0128098>] ? default_spin_lock_flags+0x8/0x10
[ 1313.988375] [<c0576bda>] ? _spin_lock_irqsave+0x2a/0x40
[ 1313.988379] [<c015a291>] ? __queue_work+0x31/0x40
[ 1313.988383] [<c01520d7>] run_timer_softirq+0x117/0x200
[ 1313.988390] [<c016c045>] ? tick_dev_program_event+0x45/0xe0
[ 1313.988393] [<c04b6550>] ? dev_watchdog+0x0/0x210
[ 1313.988399] [<c014cc40>] __do_softirq+0x90/0x1a0
[ 1313.988402] [<c0161e83>] ? hrtimer_interrupt+0x183/0x210
[ 1313.988406] [<c016c488>] ? tick_do_update_jiffies64+0x118/0x160
[ 1313.988410] [<c014cd8d>] do_softirq+0x3d/0x40
[ 1313.988414] [<c014cecd>] irq_exit+0x5d/0x70
[ 1313.988425] [<c011ddc7>] smp_apic_timer_interrupt+0x57/0x90
[ 1313.988433] [<c0103d71>] apic_timer_interrupt+0x31/0x40
[ 1313.988437] [<c0127365>] ? native_safe_halt+0x5/0x10
[ 1313.988441] [<c010a5b6>] default_idle+0x46/0xd0
[ 1313.988444] [<c010202c>] cpu_idle+0x8c/0xd0
[ 1313.988454] [<c0572498>] start_secondary+0xc6/0xc8
[ 1313.988457] ---[ end trace c5b28b21ada19e1d ]---

I also tried the 1.1.2-NAPI driver from Intel without success.

The 82575EB device works fine.

To get an impression of the performance improvement you
win from an directly assigned NIC, take a look at the
following numbers taken from sysstat in a real live
production system. Both tables show the same guest
on two subsequent days with similar every day workloads.

virtio-net
00:00:01 IFACE rxpck/s  txpck/s  rxkB/s  txkB/s   rxcmp/s txcmp/s rxmcst/s
06:05:01 eth0 577,01    582,57   141,83  211,79   0,00    0,00    0,00
06:15:01 eth0 169,70    181,93   70,04   90,49    0,00    0,00    0,00
06:25:01 eth0 135,22    138,81   60,94   56,01    0,00    0,00    0,00
06:35:01 eth0 787,44    879,40   183,05  498,02   0,00    0,00    0,00
06:45:01 eth0 1430,22   1660,49  323,80  1163,14  0,00    0,00    0,00
06:55:01 eth0 1524,15   1730,98  400,70  1112,32  0,00    0,00    0,00
07:05:02 eth0 1300,95   1414,48  307,40  741,41   0,00    0,00    0,00
07:15:01 eth0 380,77    435,28   141,65  289,76   0,00    0,00    0,00
07:25:01 eth0 312,16    365,33   112,46  230,35   0,00    0,00    0,00
07:35:01 eth0 758,51    801,66   169,99  375,84   0,00    0,00    0,00
07:45:01 eth0 1685,90   1922,92  301,87  1408,62  0,00    0,00    0,00
07:55:02 eth0 2531,08   3205,48  579,60  3033,27  0,00    0,00    0,00
08:05:02 eth0 2011,90   2180,31  426,22  1041,64  0,00    0,00    0,00
08:15:01 eth0 1054,10   1252,39  267,30  648,87   0,00    0,00    0,00
08:25:02 eth0 613,17    761,34   170,96  551,79   0,00    0,00    0,00
08:35:02 eth0 858,47    921,50   205,33  440,02   0,00    0,00    0,00
08:45:02 eth0 1426,27   1465,96  336,28  635,85   0,00    0,00    0,00
08:55:02 eth0 1539,78   1600,87  361,98  716,52   0,00    0,00    0,00
09:05:01 eth0 998,43    1386,98  245,87  1337,09  0,00    0,00    0,00
09:15:01 eth0 893,36    1452,95  306,11  1630,68  0,00    0,00    0,00
09:25:01 eth0 594,01    426,42   463,71  120,65   0,00    0,00    0,00
09:35:01 eth0 842,04    911,00   254,95  517,71   0,00    0,00    0,00
09:45:01 eth0 1528,67   1839,70  288,08  1208,60  0,00    0,00    0,00
09:55:02 eth0 2059,99   3412,35  370,46  3451,00  0,00    0,00    0,00


82575EB igb
00:00:01 IFACE rxpck/s  txpck/s  rxkB/s  txkB/s   rxcmp/s txcmp/s rxmcst/s
06:05:01 eth0 2008,58   2132,93  527,77  677,29   0,00    0,00    0,03
06:15:01 eth0 1802,28   1953,50  446,06  614,17   0,00    0,00    0,00
06:25:01 eth0 2734,59   3239,42  521,04  2049,75  0,00    0,00    0,03
06:35:01 eth0 1282,02   1712,26  386,02  1368,42  0,00    0,00    0,03
06:45:01 eth0 3402,61   3862,23  1017,19 2557,92  0,00    0,00    0,03
06:55:01 eth0 22594,46  24367,90 5194,55 13901,20 0,00    0,00    0,03
07:05:02 eth0 12398,94  13005,46 2660,65 6348,80  0,00    0,00    0,03
07:15:02 eth0 24914,17  26162,36 4857,22 11132,92 0,00    0,00    0,00
07:25:01 eth0 12499,14  15099,16 2602,93 11448,38 0,00    0,00    0,02
07:35:01 eth0 2944,58   3583,83  611,38  2354,26  0,00    0,00    0,03
07:45:02 eth0 2693,69   2985,04  652,37  1195,89  0,00    0,00    0,02
07:55:01 eth0 24266,21  26724,03 4175,56 14344,99 0,00    0,00    0,03
08:05:01 eth0 31024,78  33673,28 6005,85 18423,59 0,00    0,00    0,03
08:15:01 eth0 10117,74  12978,11 2494,69 11749,02 0,00    0,00    0,00
08:25:01 eth0 2952,17   3263,70  970,93  1934,32  0,00    0,00    0,03
08:35:01 eth0 5603,91   8992,43  986,02  9360,29  0,00    0,00    0,92
08:45:01 eth0 4192,03   5760,26  563,43  4769,12  0,00    0,00    0,10
08:55:02 eth0 21581,43  23074,14 4491,50 11611,92 0,00    0,00    0,10
09:05:02 eth0 26496,69  27805,50 5304,54 13161,67 0,00    0,00    0,10
09:15:01 eth0 16795,28  17719,70 4292,59 9279,21  0,00    0,00    0,45
09:25:01 eth0 9177,53   10760,43 4185,65 10342,94 0,00    0,00    0,20
09:35:01 eth0 6112,76   8195,99  1694,60 8275,08  0,00    0,00    0,10
09:45:01 eth0 7057,25   4243,00  5812,92 1392,42  0,00    0,00    0,08
09:55:01 eth0 24252,42  24980,30 5030,76 10703,52 0,00    0,00    0,10