Re: PCI passthrough issue

[Jean Baptiste Favre <xen-devel@jbfavre.org>]

Hello,
Sorry for not answering earlier.

I made a test with OpenWRT, applying patches bellow.
DomU boots, can connect, but nothing changed.
I still have incoming packets dropped without any console log.

Looking deeper, I can see ARP request coming in on my GW, replies coming
out.
On the domU, arp command output is:
# arp
IP address       HW type     Flags       HW address            Mask
Device
10.0.0.1         0x1         0x0         00:00:00:00:00:00     *        eth0

So, now I've an Layer 2 issue as well :(

I'm still trying to make my Debian kernel working so that I can test on
Debian too.

Regards,
JB


>From Konrad:
diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 7d85a38..37c0631 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -2331,7 +2331,7 @@ static struct sk_buff *receive_copy(struct
sky2_port *sky2,
 	if (likely(skb)) {
 		pci_dma_sync_single_for_cpu(sky2->hw->pdev, re->data_addr,
 					    length, PCI_DMA_FROMDEVICE);
-		skb_copy_from_linear_data(re->skb, skb->data, length);
+		skb_copy_from_linear_data(skb, re->skb->data, length);
 		skb->ip_summed = re->skb->ip_summed;
 		skb->csum = re->skb->csum;
 		pci_dma_sync_single_for_device(sky2->hw->pdev, re->data_addr,


And from Ian:
diff --git a/drivers/net/sky2.c b/drivers/net/sky2.c
index 7d85a38..645d9e9 100644
--- a/drivers/net/sky2.c
+++ b/drivers/net/sky2.c
@@ -2339,6 +2339,7 @@ static struct sk_buff *receive_copy(struct
sky2_port *sky2,
 		re->skb->ip_summed = CHECKSUM_NONE;
 		skb_put(skb, length);
 	}
+	WARN(skb == NULL, "sky2: receive_copy failed netdev_alloc_skb_ip_align
length %u\n", length);
 	return skb;
 }

@@ -2386,10 +2387,15 @@ static struct sk_buff *receive_new(struct
sky2_port *sky2,

 	nre.skb = sky2_rx_alloc(sky2);
 	if (unlikely(!nre.skb))
+	{
+		WARN(1, "sky2: receive_new failed sky2_rx_alloc\n");
 		goto nobuf;
-
+	}
 	if (sky2_rx_map_skb(sky2->hw->pdev, &nre, hdr_space))
+	{
+		WARN(1, "sky2: receive_new failed sky2_rx_map_skb\n");
 		goto nomap;
+	}

 	skb = re->skb;
 	sky2_rx_unmap_skb(sky2->hw->pdev, re);
diff --git a/net/core/dev.c b/net/core/dev.c
index 54277df..9c99612 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1516,6 +1516,7 @@ int dev_forward_skb(struct net_device *dev, struct
sk_buff *skb)

 	if (unlikely(!(dev->flags & IFF_UP) ||
 		     (skb->len > (dev->mtu + dev->hard_header_len + VLAN_HLEN)))) {
+		printk(KERN_CRIT "dev_forward_skb dropping skb\n");
 		atomic_long_inc(&dev->rx_dropped);
 		kfree_skb(skb);
 		return NET_RX_DROP;
@@ -2700,6 +2701,7 @@ enqueue:

 	local_irq_restore(flags);

+	printk(KERN_CRIT "enqueue_to_backlog dropping skb\n");
 	atomic_long_inc(&skb->dev->rx_dropped);
 	kfree_skb(skb);
 	return NET_RX_DROP;
@@ -3125,6 +3127,7 @@ ncls:
 	if (pt_prev) {
 		ret = pt_prev->func(skb, skb->dev, pt_prev, orig_dev);
 	} else {
+		printk(KERN_CRIT "__netif_receive_skb dropping skb proto %#x\n", type);
 		atomic_long_inc(&skb->dev->rx_dropped);
 		kfree_skb(skb);
 		/* Jamal, now you will not able to escape explaining



Le 02/02/2011 18:42, Konrad Rzeszutek Wilk a écrit :
> On Wed, Feb 02, 2011 at 11:24:51AM +0100, Jean Baptiste Favre wrote:
>> Hello Ian,
>>
>> My domU config file:
>>
>> # cat /cluster/xen/xps-106.cfg
>> kernel       = '/cluster/kernels/vmlinuz-2.6.37-trunk-686-bigmem'
>> ramdisk      = '/cluster/kernels/initrd.img-2.6.37-trunk-686-bigmem'
>> builder      = 'linux'
>> memory       = '398'
>> vcpus        = '1'
>> cpus         = '2'
>> localtime    = 0
>> serial       = 'pty'
>> boot         = 'cdn'
>> disk         = [ 'drbd:xps-106,xvda,w' ]
>> on_poweroff  = 'destroy'
>> on_reboot    = 'restart'
>> on_crash     = 'restart'
>> extra = "root=/dev/mapper/xps--106-root ro iommu=soft swiotlb=force
>> console=hvc0 xencons=tty"
>> pci = [ '04:00.0' ]
>> name         = 'xps-106'
>> hostname     = 'xps-106.clichy.jbfavre.org'
>>
>>
>> Le 02/02/2011 10:27, Ian Campbell a écrit :
>>> On Tue, 2011-02-01 at 22:04 +0000, Jean Baptiste Favre wrote:
>>>> Le 01/02/2011 17:23, Ian Campbell a écrit :
>>>
>>>>> I assume you are not seeing "rx mapping error" in your domU dmesg? Did
>>>>> you post a full guest console log at some point? Comparing the logs for
>>>>> the 256MB, 398MB and 512MB guest RAM case might be useful.
>>>> No sure I've ever posted that logs. But I can redo my tests :)
>>>
>>> yes, please do that.
>> Please find attached both console startup logs with 256M and 512M:
>> 256M_domU_console_logs.txt
>> 512M_domU_console_logs.txt
>>
>> For 512M, I saw some kernel CallTrace I can not explain. There are not
>> present with 256M.
>>
>> For 398M memory, I can't even start domU :
>> # xm create /cluster/xen/xps-106.cfg -c
>> Using config file "/cluster/xen/xps-106.cfg".
>> [215739.007871] pciback 0000:04:00.0: device has been assigned to
>> another domain! Over-writting the ownership, but beware.
>> Started domain xps-106 (id=23)
>> (XEN) mm.c:798:d23 Non-privileged (23) attempt to map I/O space 00000000
>> (XEN) mm.c:4644:d23 ptwr_emulate: could not get_page_from_l1e()
>>
>> As I told you, I'm still using Debian 2.6.37 kernel because I've some
>> problem to compile 2.6.32.27 from Jeremy's git repository.
>> I hope I can get it compiled today so I'll be able to test with that
>> kernel as well.
> 
> So I've tried this on my Abit IP-35 box which has a 
> 
> 04:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller (rev 13)
>         Subsystem: ABIT Computer Corp. Device 1085
>         Flags: bus master, fast devsel, latency 0, IRQ 29
>         Memory at fdefc000 (64-bit, non-prefetchable) [size=16K]
>         I/O ports at be00 [size=256]
>         Expansion ROM at <ignored> [disabled]
>         Capabilities: [48] Power Management version 3
>         Capabilities: [50] Vital Product Data
>         Capabilities: [5c] MSI: Enable+ Count=1/1 Maskable- 64bit+
>         Capabilities: [e0] Express Legacy Endpoint, MSI 00
>         Capabilities: [100] Advanced Error Reporting
>         Kernel driver in use: sky2
>         Kernel modules: sky2
> 
> And when I launch this guest with a 32-bit DomU:
> kernel="/mnt/lab/latest/vmlinuz"
> ramdisk="/mnt/lab/latest/initramfs.cpio.gz"
> extra="console=hvc0 debug iommu=soft"
> memory=320
> vcpus=1
> cpu='2'
> on_crash="preserve"
> #vif = [ 'bridge=switch' ]
> pci = ["04:00.0"]
> vfb = [ 'vnc=1, vnclisten=0.0.0.0,vncunused=1']
> 
> And played around with the 'extra' to add 'swiotlb=force'.
> 
> The moment I had 'swiotlb=force' I could not get any DHCP
> address from the NIC. If I did not have 'swiotlb=force' it would
> work just fine (can ping any size, etc, this is with 320MB)
> 
> 
> For fun, I upped the memory (320->720) and kept 'swiotlb=force' in effect.
> Same effect: can't do DHCP.
> 
> I look to have a different issue than you, which is that whenever I use
> swiotlb=force, things go haywire.
> 
> Fyi, this is what DomU tells me:
> 
> 12:22:41 # 9 :~/ 
>> dmesg |grep Memor
> [    0.000000] Memory: 145640k/335872k available (3731k kernel code, 189784k reserved, 1565k data, 436k init, 0k highmem)
> 
> 12:22:56 # 10 :~/ 
>> uname -a
> Linux (none) 2.6.38-rc2-00028-gf2a2d8b #2 SMP Wed Feb 2 12:10:25 EST 2011 i686 i686 i386 GNU/Linux
> rnet driver.
> [    0.921406] udevd (1126): /proc/1126/oom_adj is deprecated, please use /proc/1126/oom_score_adj instead.
> [    0.984886] sky2: driver version 1.28
> [    0.995595] sky2 0000:04:00.0: BAR 0: set to [mem 0xfdefc000-0xfdefffff 64bit] (PCI address [0xfdefc000-0xfdefffff])
> [    0.995662] sky2 0000:04:00.0: BAR 2: set to [io  0xbe00-0xbeff] (PCI address [0xbe00-0xbeff])
> [    0.995697] sky2 0000:04:00.0: enabling device (0000 -> 0003)
> [    0.996440] sky2 0000:04:00.0: Xen PCI mapped GSI18 to IRQ27
> 
> (This is the #master branch from git://git.kernel.org/pub/scm/linux/kernel/git/konrad/xen.git)
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>