From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Fitzhardinge Subject: Re: xennet: skb rides the rocket messages in domU dmesg Date: Wed, 26 May 2010 15:39:00 -0700 Message-ID: <4BFDA304.3060803@goop.org> References: <4BFD90D6.2020107@xs4all.nl> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <4BFD90D6.2020107@xs4all.nl> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Mark Hurenkamp Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org On 05/26/2010 02:21 PM, Mark Hurenkamp wrote: > Hi, > > > On my home server i am running Xen-4.0.1-rc1 with a recent xen/next > kernel, Are you actually using the "xen/next" branch? I recommend you use xen/stable-2.6.32.x, since that's tracking all the other bugfixes going into Linux 2.6.32. > and a pvm domU with the same kernel, and 4 tuners passed through. > Because the mythtv backend domain would sometimes become unstable, i > decided > to split up my mythtv backend into 3 seperate virtual machines, one > master > backend with the database, and 2 slave backends with the tuners. > One of the slave backends has a cx23885 based dvb tuner card, the > other slave > backend runs 3 ivtv based tuners. > To keep consistency with old recording data, and since i would like to > have all > recordings in a single volume, i tried to use an nfs mount of the > recordings volume > from the dom0 to mount on all backends. This resulted in a very > unstable system, > to the point where my most important slave backend became unusable. Unstable how? > So i tried it the other way, have the slave backends each mount their own > recordings volume as a block device via xen, and for backwards > compatibility mount > the volume which holds the old recordings via nfs on the master backend. > > Now i see many "xennet: skb rides the rocket" messages appear in the > (pv) slave > backend which exports the recordings volume to the master backend. These > messages i did not see when there was only a single mythtv backend. > (both the dom0 as well as the mythtv domUs are ubuntu lucid server based) > Overall the system seems to perform ok, and the messages are not > causing the > system to become unusable or more unstable, so it is not a major issue. That appears to mean that you're getting single packets which are larger than 18 pages long (72k). I'm not quite sure how that's possible, since I thought the datagram limit is 64k.. Are you using nfs over udp or tcp? (I think tcp, from your stack trace.) Does turning of tso/gso with ethtool make a difference? J > > Note that both the master backend, and the slave backend which exports > the > volume, are paravirtualised domains. The slave backend has the following > xen config: > > kernel = '/boot/vmlinuz-2.6.32m5' > ramdisk = '/boot/initrd.img-2.6.32m5' > extra = 'root=/dev/xvda1 ro console=hvc0 noirqdebug iommu=soft > swiotlb=force' > maxmem = '1000' > memory = '500' > device_model='/usr/lib/xen/bin/qemu-dm' > serial='pty' > disk = [ > 'phy:/dev/vm/tilnes-lucid,hda,w', > 'phy:/dev/mythtv/recordings,hdb,w', > ] > boot='c' > name = 'tilnes' > vif = [ 'mac=aa:20:00:00:01:72, bridge=loc' ] > vfb = [ 'vnc=1,vnclisten=0.0.0.0,vncdisplay=5' ] > > usb=1 > usbdevice='tablet' > monitor=1 > pci = [ > '0000:08:02.0', > '0000:09:08.0', > '0000:09:09.0', > ] > vcpus=8 > > > The messagedump i see (this is only 1 example, my dmesg is full of > these): > > xennet: skb rides the rocket: 20 frags > Pid: 3237, comm: nfsd Tainted: G D 2.6.32m5 #9 > Call Trace: > [] xennet_start_xmit+0x75/0x678 [xen_netfront] > [] ? check_events+0x12/0x20 > [] ? rcu_read_unlock+0x0/0x1e > [] ? xen_restore_fl_direct_end+0x0/0x1 > [] ? lock_release+0x1e0/0x1ed > [] dev_hard_start_xmit+0x236/0x2e1 > [] sch_direct_xmit+0x68/0x16f > [] dev_queue_xmit+0x274/0x3de > [] ? dev_queue_xmit+0x164/0x3de > [] ? dst_output+0x0/0xd > [] ip_finish_output2+0x1df/0x222 > [] ip_finish_output+0x68/0x6a > [] ip_output+0x9c/0xa0 > [] ip_local_out+0x20/0x24 > [] ip_queue_xmit+0x309/0x37a > [] ? xen_force_evtchn_callback+0xd/0xf > [] ? check_events+0x12/0x20 > [] ? xen_force_evtchn_callback+0xd/0xf > [] ? check_events+0x12/0x20 > [] tcp_transmit_skb+0x648/0x686 > [] tcp_write_xmit+0x808/0x8f7 > [] ? check_events+0x12/0x20 > [] ? tcp_established_options+0x2e/0xa9 > [] __tcp_push_pending_frames+0x2a/0x58 > [] tcp_data_snd_check+0x24/0xea > [] tcp_rcv_established+0xdd/0x6d4 > [] ? check_events+0x12/0x20 > [] tcp_v4_do_rcv+0x1ba/0x375 > [] ? xen_restore_fl_direct_end+0x0/0x1 > [] ? tcp_v4_rcv+0x2b3/0x6b7 > [] tcp_v4_rcv+0x456/0x6b7 > [] ? ip_local_deliver_finish+0x0/0x235 > [] ip_local_deliver_finish+0x174/0x235 > [] ? ip_local_deliver_finish+0x44/0x235 > [] ip_local_deliver+0x72/0x7c > [] ip_rcv_finish+0x3cd/0x3fb > [] ip_rcv+0x2b9/0x2f9 > [] ? packet_rcv_spkt+0xd6/0xe1 > [] netif_receive_skb+0x445/0x46f > [] ? free_hot_page+0x3a/0x3f > [] xennet_poll+0xaf4/0xc7b [xen_netfront] > [] net_rx_action+0xab/0x1df > [] ? lock_release+0x1e0/0x1ed > [] __do_softirq+0xe0/0x1a2 > [] ? handle_level_irq+0xd1/0xda > [] ? __xen_evtchn_do_upcall+0x12e/0x163 > [] call_softirq+0x1c/0x30 > [] do_softirq+0x41/0x81 > [] irq_exit+0x36/0x78 > [] xen_evtchn_do_upcall+0x37/0x47 > [] xen_do_hypervisor_callback+0x1e/0x30 > [] ? __bio_add_page+0xee/0x212 > [] ? bio_alloc+0x10/0x1f > [] ? mpage_alloc+0x25/0x7d > [] ? bio_add_page+0x31/0x33 > [] ? do_mpage_readpage+0x3d3/0x488 > [] ? add_to_page_cache_locked+0xcc/0x108 > [] ? mpage_readpages+0xcb/0x10f > [] ? ext3_get_block+0x0/0xf9 > [] ? ext3_get_block+0x0/0xf9 > [] ? ext3_readpages+0x18/0x1a > [] ? __do_page_cache_readahead+0x140/0x1cd > [] ? check_events+0x12/0x20 > [] ? xen_restore_fl_direct_end+0x0/0x1 > [] ? ra_submit+0x1c/0x20 > [] ? ondemand_readahead+0x1de/0x1f1 > [] ? page_cache_sync_readahead+0x17/0x1c > [] ? __generic_file_splice_read+0xf0/0x41a > [] ? check_events+0x12/0x20 > [] ? rcu_read_unlock+0x0/0x1e > [] ? xen_restore_fl_direct_end+0x0/0x1 > [] ? lock_release+0x1e0/0x1ed > [] ? rcu_read_unlock+0x1c/0x1e > [] ? avc_has_perm_noaudit+0x3b5/0x3c7 > [] ? check_object+0x170/0x1a9 > [] ? xen_force_evtchn_callback+0xd/0xf > [] ? spd_release_page+0x0/0x14 > [] ? selinux_file_permission+0x57/0xae > [] ? generic_file_splice_read+0x44/0x72 > [] ? check_events+0x12/0x20 > [] ? do_splice_to+0x6c/0x79 > [] ? check_events+0x12/0x20 > [] ? splice_direct_to_actor+0xc2/0x1a1 > [] ? xen_restore_fl_direct_end+0x0/0x1 > [] ? nfsd_direct_splice_actor+0x0/0x12 [nfsd] > [] ? nfsd_vfs_read+0x276/0x385 [nfsd] > [] ? nfsd_read+0xa1/0xbf [nfsd] > [] ? svc_xprt_enqueue+0x22b/0x238 [sunrpc] > [] ? nfsd3_proc_read+0xe2/0x121 [nfsd] > [] ? cache_put+0x2d/0x2f [sunrpc] > [] ? nfsd_dispatch+0xec/0x1c7 [nfsd] > [] ? svc_process+0x436/0x637 [sunrpc] > [] ? exp_readlock+0x10/0x12 [nfsd] > [] ? nfsd+0xf3/0x13e [nfsd] > [] ? nfsd+0x0/0x13e [nfsd] > [] ? kthread+0x7a/0x82 > [] ? child_rip+0xa/0x20 > [] ? int_ret_from_sys_call+0x7/0x1b > [] ? retint_restore_args+0x5/0x6 > [] ? child_rip+0x0/0x20 > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >