Re: Xen 4.3 PCI passthrough possible bug

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Tue, Feb 11, 2014 at 08:04:04AM -0500, Mike Neiderhauser wrote:
> So I went ahead to tested the setup I am trying to achieve using xen.  This
> setup basically requires two isolated machines that can be used for network
> testing.  On the hvm mentioned above, this testing fails due to something I
> cannot wrap my head around.  I believe it is still related to the PCI
> passthrough of a device and I believe it is related to the libxl error
> mentioned above. Can anyone shed some light on what is going on?  Is it a
> driver issue? (Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet)

That looks like this issue:

igb and bnx2: "NETDEV WATCHDOG: transmit queue timed out" when skb has huge linear buffer

(http://lkml.org/lkml/2014/1/30/358) ?
> 
> [79464.816085] ------------[ cut here ]------------
> [79464.816093] WARNING: at
> /build/buildd/linux-lts-raring-3.8.0/net/sched/sch_generic.c:254
> dev_watchdog+0x262/0x270()
> [79464.816094] Hardware name: HVM domU
> [79464.816096] NETDEV WATCHDOG: eth1 (bnx2): transmit queue 1 timed out
> [79464.816096] Modules linked in: esp6(F) ah6(F) xfrm6_mode_tunnel(F)
> xfrm_user(F) xfrm4_tunnel(F) tunnel4(F) ipcomp(F) xfrm_ipcomp(F) authenc(F)
> esp4(F) ah4(F) xfrm4_mode_tunnel(F) deflate(F) zlib_deflate(F) ctr(F)
> twofish_generic(F) twofish_avx_x86_64(F) twofish_x86_64_3way(F)
> twofish_x86_64(F) twofish_common(F) camellia_generic(F)
> camellia_aesni_avx_x86_64(F) camellia_x86_64(F) serpent_avx_x86_64(F)
> serpent_sse2_x86_64(F) glue_helper(F) serpent_generic(F)
> blowfish_generic(F) blowfish_x86_64(F) blowfish_common(F)
> cast5_avx_x86_64(F) cast5_generic(F) cast_common(F) des_generic(F) xcbc(F)
> rmd160(F) crypto_null(F) af_key(F) xfrm_algo(F) 8021q(F) garp(F) stp(F)
> llc(F) ipmi_msghandler(F) autofs4(F) ghash_clmulni_intel(F) aesni_intel(F)
> ablk_helper(F) cryptd(F) lrw(F) aes_x86_64(F) xts(F) gf128mul(F) cirrus(F)
> ttm(F) drm_kms_helper(F) nfsd(F) nfs_acl(F) drm(F) auth_rpcgss(F)
> sysimgblt(F) nfs(F) psmouse(F) sysfillrect(F) syscopyarea(F) fscache(F)
> lockd(F) microcode(F) joydev(F) xen_kbdfront(F) i2c_piix4(F) serio_raw(F)
> lp(F) mac_hid(F) sunrpc(F) parport(F) floppy(F) bnx2(F) [last unloaded:
> ipmi_msghandler]
> [79464.816139] Pid: 0, comm: swapper/1 Tainted: GF
>  3.8.0-29-generic #42~precise1-Ubuntu
> [79464.816140] Call Trace:
> [79464.816142]  <IRQ>  [<ffffffff81059b0f>] warn_slowpath_common+0x7f/0xc0
> [79464.816149]  [<ffffffff8135b9d4>] ? timerqueue_add+0x64/0xb0
> [79464.816151]  [<ffffffff81059c06>] warn_slowpath_fmt+0x46/0x50
> [79464.816154]  [<ffffffff81076794>] ? wake_up_worker+0x24/0x30
> [79464.816157]  [<ffffffff81602062>] dev_watchdog+0x262/0x270
> [79464.816160]  [<ffffffff810771f0>] ? __queue_work+0x2d0/0x2d0
> [79464.816161]  [<ffffffff81601e00>] ? pfifo_fast_dequeue+0xe0/0xe0
> [79464.816164]  [<ffffffff8106995b>] call_timer_fn+0x3b/0x150
> [79464.816167]  [<ffffffff8144f5a1>] ? add_interrupt_randomness+0x41/0x190
> [79464.816170]  [<ffffffff8106b427>] run_timer_softirq+0x267/0x2c0
> [79464.816173]  [<ffffffff810ee3c9>] ? handle_irq_event_percpu+0xa9/0x210
> [79464.816175]  [<ffffffff81601e00>] ? pfifo_fast_dequeue+0xe0/0xe0
> [79464.816177]  [<ffffffff81062620>] __do_softirq+0xc0/0x240
> [79464.816180]  [<ffffffff810b67cd>] ? tick_do_update_jiffies64+0x9d/0xd0
> [79464.816184]  [<ffffffff816fdd5c>] call_softirq+0x1c/0x30
> [79464.816188]  [<ffffffff81016775>] do_softirq+0x65/0xa0
> [79464.816189]  [<ffffffff810628fe>] irq_exit+0x8e/0xb0
> [79464.816193]  [<ffffffff8140a125>] xen_evtchn_do_upcall+0x35/0x50
> [79464.816195]  [<ffffffff816fdeed>] xen_hvm_callback_vector+0x6d/0x80
> [79464.816196]  <EOI>  [<ffffffff81084008>] ? hrtimer_start+0x18/0x20
> [79464.816201]  [<ffffffff81045136>] ? native_safe_halt+0x6/0x10
> [79464.816204]  [<ffffffff8101cc33>] default_idle+0x53/0x1f0
> [79464.816206]  [<ffffffff8101dad9>] cpu_idle+0xd9/0x120
> [79464.816209]  [<ffffffff816d10fe>] start_secondary+0xc3/0xc5
> [79464.816210] ---[ end trace 48cf6b13be16e0ae ]---
> [79464.816214] bnx2 0000:00:05.0 eth1: <--- start FTQ dump --->
> [79464.816220] bnx2 0000:00:05.0 eth1: RV2P_PFTQ_CTL 00010000
> [79464.816225] bnx2 0000:00:05.0 eth1: RV2P_TFTQ_CTL 00020000
> [79464.816258] bnx2 0000:00:05.0 eth1: RV2P_MFTQ_CTL 00004000
> [79464.816263] bnx2 0000:00:05.0 eth1: TBDR_FTQ_CTL 00004000
> [79464.816268] bnx2 0000:00:05.0 eth1: TDMA_FTQ_CTL 00010000
> [79464.816272] bnx2 0000:00:05.0 eth1: TXP_FTQ_CTL 00010000
> [79464.816276] bnx2 0000:00:05.0 eth1: TXP_FTQ_CTL 00010000
> [79464.816280] bnx2 0000:00:05.0 eth1: TPAT_FTQ_CTL 00010000
> [79464.816285] bnx2 0000:00:05.0 eth1: RXP_CFTQ_CTL 00008000
> [79464.816289] bnx2 0000:00:05.0 eth1: RXP_FTQ_CTL 00100000
> [79464.816293] bnx2 0000:00:05.0 eth1: COM_COMXQ_FTQ_CTL 00010000
> [79464.816297] bnx2 0000:00:05.0 eth1: COM_COMTQ_FTQ_CTL 00020000
> [79464.816302] bnx2 0000:00:05.0 eth1: COM_COMQ_FTQ_CTL 00010000
> [79464.816306] bnx2 0000:00:05.0 eth1: CP_CPQ_FTQ_CTL 00004000
> [79464.816308] bnx2 0000:00:05.0 eth1: CPU states:
> [79464.816321] bnx2 0000:00:05.0 eth1: 045000 mode b84c state 80001000
> evt_mask 500 pc 8001288 pc 8001288 instr 38640001
> [79464.816335] bnx2 0000:00:05.0 eth1: 085000 mode b84c state 80005000
> evt_mask 500 pc 8000a5c pc 8000a5c instr 10400016
> [79464.816349] bnx2 0000:00:05.0 eth1: 0c5000 mode b84c state 80000000
> evt_mask 500 pc 8004c14 pc 8004c14 instr 32050003
> [79464.816362] bnx2 0000:00:05.0 eth1: 105000 mode b8cc state 80000000
> evt_mask 500 pc 8000a9c pc 8000a94 instr 8c420020
> [79464.816375] bnx2 0000:00:05.0 eth1: 145000 mode b880 state 80000000
> evt_mask 500 pc 8009c00 pc 800d948 instr 30420040
> [79464.816389] bnx2 0000:00:05.0 eth1: 185000 mode b8cc state 80008000
> evt_mask 500 pc 8000c58 pc 8000c58 instr 1092823
> [79464.816392] bnx2 0000:00:05.0 eth1: <--- end FTQ dump --->
> [79464.816394] bnx2 0000:00:05.0 eth1: <--- start TBDC dump --->
> [79464.816399] bnx2 0000:00:05.0 eth1: TBDC free cnt: 32
> [79464.816401] bnx2 0000:00:05.0 eth1: LINE     CID  BIDX   CMD  VALIDS
> [79464.816410] bnx2 0000:00:05.0 eth1: 00    001000  00e8   00    [0]
> [79464.816420] bnx2 0000:00:05.0 eth1: 01    001000  00e8   00    [0]
> [79464.816429] bnx2 0000:00:05.0 eth1: 02    000800  afc8   00    [0]
> [79464.816438] bnx2 0000:00:05.0 eth1: 03    000800  afb8   00    [0]
> [79464.816447] bnx2 0000:00:05.0 eth1: 04    000800  afd8   00    [0]
> [79464.816456] bnx2 0000:00:05.0 eth1: 05    000800  afe0   00    [0]
> [79464.816465] bnx2 0000:00:05.0 eth1: 06    000800  afe8   00    [0]
> [79464.816474] bnx2 0000:00:05.0 eth1: 07    000800  afd0   00    [0]
> [79464.816485] bnx2 0000:00:05.0 eth1: 08    001000  3510   00    [0]
> [79464.816494] bnx2 0000:00:05.0 eth1: 09    000800  aec0   00    [0]
> [79464.816504] bnx2 0000:00:05.0 eth1: 0a    001000  3530   00    [0]
> [79464.816514] bnx2 0000:00:05.0 eth1: 0b    000800  aec8   00    [0]
> [79464.816523] bnx2 0000:00:05.0 eth1: 0c    000800  aed0   00    [0]
> [79464.816559] bnx2 0000:00:05.0 eth1: 0d    001000  34f8   00    [0]
> [79464.816570] bnx2 0000:00:05.0 eth1: 0e    001000  3500   00    [0]
> [79464.816580] bnx2 0000:00:05.0 eth1: 0f    001000  3518   00    [0]
> [79464.816590] bnx2 0000:00:05.0 eth1: 10    1fbc00  2fe8   7d    [0]
> [79464.816599] bnx2 0000:00:05.0 eth1: 11    1ab780  fff8   7d    [0]
> [79464.816608] bnx2 0000:00:05.0 eth1: 12    17ff00  b908   f7    [0]
> [79464.816618] bnx2 0000:00:05.0 eth1: 13    0cb700  ff40   d7    [0]
> [79464.816627] bnx2 0000:00:05.0 eth1: 14    177a80  efe0   03    [0]
> [79464.816637] bnx2 0000:00:05.0 eth1: 15    037d80  9f88   72    [0]
> [79464.816646] bnx2 0000:00:05.0 eth1: 16    1bae00  eef8   ce    [0]
> [79464.816657] bnx2 0000:00:05.0 eth1: 17    1bbc80  a7f8   df    [0]
> [79464.816666] bnx2 0000:00:05.0 eth1: 18    17e180  6aa8   e4    [0]
> [79464.816675] bnx2 0000:00:05.0 eth1: 19    07ff80  6e50   dd    [0]
> [79464.816683] bnx2 0000:00:05.0 eth1: 1a    1fda80  f790   6e    [0]
> [79464.816694] bnx2 0000:00:05.0 eth1: 1b    151580  d7b0   fc    [0]
> [79464.816703] bnx2 0000:00:05.0 eth1: 1c    1b9f80  cef8   6b    [0]
> [79464.816712] bnx2 0000:00:05.0 eth1: 1d    1ebf00  ffa8   df    [0]
> [79464.816723] bnx2 0000:00:05.0 eth1: 1e    1e7e00  ff78   ef    [0]
> [79464.816731] bnx2 0000:00:05.0 eth1: 1f    166e80  fbd8   aa    [0]
> [79464.816733] bnx2 0000:00:05.0 eth1: <--- end TBDC dump --->
> [79464.817101] bnx2 0000:00:05.0 eth1: DEBUG: intr_sem[0] PCI_CMD[00100406]
> [79464.817239] bnx2 0000:00:05.0 eth1: DEBUG: PCI_PM[19002008]
> PCI_MISC_CFG[92000088]
> [79464.817248] bnx2 0000:00:05.0 eth1: DEBUG: EMAC_TX_STATUS[00000008]
> EMAC_RX_STATUS[00000000]
> [79464.817252] bnx2 0000:00:05.0 eth1: DEBUG: RPM_MGMT_PKT_CTRL[40000088]
> [79464.817257] bnx2 0000:00:05.0 eth1: DEBUG:
> HC_STATS_INTERRUPT_STATUS[01fc0003]
> [79464.817261] bnx2 0000:00:05.0 eth1: DEBUG: PBA[00000003]
> [79464.817264] bnx2 0000:00:05.0 eth1: <--- start MCP states dump --->
> [79464.817270] bnx2 0000:00:05.0 eth1: DEBUG: MCP_STATE_P0[0003611e]
> MCP_STATE_P1[0003611e]
> [79464.817278] bnx2 0000:00:05.0 eth1: DEBUG: MCP mode[0000b880]
> state[80000000] evt_mask[00000500]
> [79464.817286] bnx2 0000:00:05.0 eth1: DEBUG: pc[08003ebc] pc[0800d994]
> instr[32020020]
> [79464.817288] bnx2 0000:00:05.0 eth1: DEBUG: shmem states:
> [79464.817296] bnx2 0000:00:05.0 eth1: DEBUG: drv_mb[01030027]
> fw_mb[00000027] link_status[0008506b]
> [79464.817301]  drv_pulse_mb[0000338a]
> [79464.817306] bnx2 0000:00:05.0 eth1: DEBUG: dev_info_signature[44564907]
> reset_type[01005254]
> [79464.817310]  condition[0003611e]
> [79464.817318] bnx2 0000:00:05.0 eth1: DEBUG: 000001c0: 01005254 42530083
> 0003611e 00000000
> [79464.817328] bnx2 0000:00:05.0 eth1: DEBUG: 000003cc: 00000000 00000000
> 00000000 00000000
> [79464.817357] bnx2 0000:00:05.0 eth1: DEBUG: 000003dc: 00000000 00000000
> 00000000 00000000
> [79464.817367] bnx2 0000:00:05.0 eth1: DEBUG: 000003ec: 00000000 00000000
> 00000000 00000000
> [79464.817372] bnx2 0000:00:05.0 eth1: DEBUG: 0x3fc[00000000]
> [79464.817374] bnx2 0000:00:05.0 eth1: <--- end MCP states dump --->
> [79464.872988] bnx2 0000:00:05.0 eth1: NIC Copper Link is Down
> 
> 
> 
> On Mon, Feb 10, 2014 at 1:47 PM, Konrad Rzeszutek Wilk <
> konrad.wilk@oracle.com> wrote:
> 
> > On Mon, Feb 10, 2014 at 01:29:22PM -0500, Mike Neiderhauser wrote:
> > > Thanks for the answers on the timeline.
> > >
> > > When I start the HVM with th broadcom adapter, I get this message back.
> > > Parsing config from /etc/xen/ubuntu-hvm-1.cfg
> > > libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't
> > > support reset from sysfs for PCI device 0000:05:00.0
> > > libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't
> > > support reset from sysfs for PCI device 0000:05:00.1
> > >
> > > However, the devices appear in the HVM.  Is this something that I should
> > be
> > > concerned about?
> >
> > No. Xen pciback does the reset automatically.
> >
> > Actually we might want to ditch that reporting in libxl, or maybe just
> > implement a stub function in xen-pciback so that libxl will be happy.
> >
> > >
> > >
> > >
> > > On Mon, Feb 10, 2014 at 12:11 PM, George Dunlap <dunlapg@umich.edu>
> > wrote:
> > >
> > > > On Sat, Feb 8, 2014 at 5:42 PM, Mike Neiderhauser
> > > > <mikeneiderhauser@gmail.com> wrote:
> > > > > Works like a charm.  I do not have physical access to the computer
> > this
> > > > > weekend to verify that the cards are isolated, but the HVM starts and
> > > > > appears to be working well.
> > > > >
> > > > > When do you think Xen 4.4 will be released.  The article I read
> > > > mentioned it
> > > > > will be released in 2014 (hinting towards the end of February).  I
> > also
> > > > read
> > > > > 'When it is ready.'
> > > > >
> > > > > Any timeline would be great.
> > > >
> > > > I'm afraid that's about all we can give. :-)  We've locked down
> > > > development for 2 months now and are working on finding and fixing
> > > > bugs.  If there are no more blocker bugs or other unforeseen delays,
> > > > it should be out by the end of February.  But there are necessarily
> > > > significant unknowns, so we can't make any promises.
> > > >
> > > >  -George
> > > >
> >