pvops netconsole

pvops netconsole

[Brad Plant]

[-- Attachment #1.1: Type: text/plain, Size: 723 bytes --]

Hi,

I tried asking this question on xen-users, but didn't get a response so I'm now asking on xen-devel instead....

Is netconsole supposed to work under pvops? When I try to enable it I get the messages below. I'm wondering if netconsole just doesn't work at all with pvops or whether it's dependent on the NIC used by the dom0.

Initialising Xen virtual ethernet driver.
blkfront: xvda1: barriers enabled
netconsole: local port 11111
netconsole: local IP 10.20.40.2
netconsole: interface eth1
netconsole: remote port 9514
netconsole: remote IP 10.20.50.2
netconsole: remote ethernet address 00:16:3e:00:50:02
netconsole: eth1 doesn't support polling, aborting.
netconsole: cleaning up

Cheers,

Brad

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pvops netconsole

[Konrad Rzeszutek Wilk]

On Fri, Feb 05, 2010 at 11:15:07AM +1100, Brad Plant wrote:
> Hi,
> 
> I tried asking this question on xen-users, but didn't get a response so I'm now asking on xen-devel instead....
> 
> Is netconsole supposed to work under pvops? When I try to enable it I get the messages below. I'm wondering if netconsole just doesn't work at all with pvops or whether it's dependent on the NIC used by the dom0.

Hmmm, I _think_ I had it working at some point, but this was with dom0,
not with domUs, which is what you are looking to be doing?

> Initialising Xen virtual ethernet driver.

Re: pvops netconsole

[Brad Plant]

[-- Attachment #1.1: Type: text/plain, Size: 530 bytes --]

On Fri, 5 Feb 2010 11:34:30 -0500
Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> > Is netconsole supposed to work under pvops? When I try to enable it I get the messages below. I'm wondering if netconsole just doesn't work at all with pvops or whether it's dependent on the NIC used by the dom0.
> 
> Hmmm, I _think_ I had it working at some point, but this was with dom0,
> not with domUs, which is what you are looking to be doing?

That's correct - I'm trying to use netconsole on a domU.

Cheers,

Brad

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pvops netconsole

[Jeremy Fitzhardinge]

On 02/04/2010 04:15 PM, Brad Plant wrote:
> Initialising Xen virtual ethernet driver.
> blkfront: xvda1: barriers enabled
> netconsole: local port 11111
> netconsole: local IP 10.20.40.2
> netconsole: interface eth1
> netconsole: remote port 9514
> netconsole: remote IP 10.20.50.2
> netconsole: remote ethernet address 00:16:3e:00:50:02
> netconsole: eth1 doesn't support polling, aborting.
> netconsole: cleaning up
>    

The xen-netfront device doesn't support the polling interface that 
netconsole relies on.  It probably wouldn't be too hard to add that 
capability if you want to.  It's independent of the backend and/or the 
actual underlying hardware device.

     J

Re: pvops netconsole

[Brad Plant]

[-- Attachment #1.1: Type: text/plain, Size: 877 bytes --]

On Fri, 12 Feb 2010 15:09:12 -0800
Jeremy Fitzhardinge <jeremy@goop.org> wrote:

> On 02/04/2010 04:15 PM, Brad Plant wrote:
> > Initialising Xen virtual ethernet driver.
> > blkfront: xvda1: barriers enabled
> > netconsole: local port 11111
> > netconsole: local IP 10.20.40.2
> > netconsole: interface eth1
> > netconsole: remote port 9514
> > netconsole: remote IP 10.20.50.2
> > netconsole: remote ethernet address 00:16:3e:00:50:02
> > netconsole: eth1 doesn't support polling, aborting.
> > netconsole: cleaning up
> >    
> 
> The xen-netfront device doesn't support the polling interface that 
> netconsole relies on.  It probably wouldn't be too hard to add that 
> capability if you want to.  It's independent of the backend and/or the 
> actual underlying hardware device.

I like the sound of that. I'll give it a crack :-)

Cheers,

Brad

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pvops netconsole

[Brad Plant]

[-- Attachment #1.1.1: Type: text/plain, Size: 1674 bytes --]

On Fri, 12 Feb 2010 15:09:12 -0800
Jeremy Fitzhardinge <jeremy@goop.org> wrote:

> On 02/04/2010 04:15 PM, Brad Plant wrote:
> > Initialising Xen virtual ethernet driver.
> > blkfront: xvda1: barriers enabled
> > netconsole: local port 11111
> > netconsole: local IP 10.20.40.2
> > netconsole: interface eth1
> > netconsole: remote port 9514
> > netconsole: remote IP 10.20.50.2
> > netconsole: remote ethernet address 00:16:3e:00:50:02
> > netconsole: eth1 doesn't support polling, aborting.
> > netconsole: cleaning up
> >    
> 
> The xen-netfront device doesn't support the polling interface that 
> netconsole relies on.  It probably wouldn't be too hard to add that 
> capability if you want to.  It's independent of the backend and/or the 
> actual underlying hardware device.

Ok, I've come up with something that "works". Patch against 2.6.32.7 is attached. While it works, I've got no idea if what I've done is the "right way". I used virtio_net.c as a guide.

I've got some issues that I'm stuck on and would like some guidance:

1. When the first message is sent over the net console, the WARN_ONCE() at net/core/netpoll.c:327 triggers. I'm not sure what to do about this.

2. When either netconsole is setup or the first message is sent, messages already logged to the main console get logged again. Maybe this will go away when the above is fixed?

3. Initially netconsole would not initialise because the interface didn't yet have a mac address as setup_netfront hadn't yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES && CONFIG_XEN. Is this allowed and sane?

Cheers,

Brad

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.1.2: xen-netconsole.patch --]
[-- Type: text/x-patch, Size: 1693 bytes --]

diff -urpNX linux-2.6.32.7/Documentation/dontdiff linux-2.6.32.7/drivers/net/netconsole.c linux-2.6.32.7-xen-netconsole/drivers/net/netconsole.c
--- linux-2.6.32.7/drivers/net/netconsole.c	2009-12-03 14:51:21.000000000 +1100
+++ linux-2.6.32.7-xen-netconsole/drivers/net/netconsole.c	2010-02-14 12:37:00.000000000 +1100
@@ -816,5 +816,13 @@ static void __exit cleanup_netconsole(vo
 	}
 }
 
+#ifndef CONFIG_MODULES
+#ifdef CONFIG_XEN
+late_initcall(init_netconsole);
+#else
 module_init(init_netconsole);
+#endif
+#else
+module_init(init_netconsole);
+#endif
 module_exit(cleanup_netconsole);
diff -urpNX linux-2.6.32.7/Documentation/dontdiff linux-2.6.32.7/drivers/net/xen-netfront.c linux-2.6.32.7-xen-netconsole/drivers/net/xen-netfront.c
--- linux-2.6.32.7/drivers/net/xen-netfront.c	2009-12-03 14:51:21.000000000 +1100
+++ linux-2.6.32.7-xen-netconsole/drivers/net/xen-netfront.c	2010-02-14 12:58:22.000000000 +1100
@@ -989,6 +989,15 @@ err:
 	return work_done;
 }
 
+#ifdef CONFIG_NET_POLL_CONTROLLER
+static void xennet_netpoll(struct net_device *dev)
+{
+	struct netfront_info *np = netdev_priv(dev);
+
+	napi_schedule(&np->napi);
+}
+#endif
+
 static int xennet_change_mtu(struct net_device *dev, int mtu)
 {
 	int max = xennet_can_sg(dev) ? 65535 - ETH_HLEN : ETH_DATA_LEN;
@@ -1113,6 +1122,9 @@ static const struct net_device_ops xenne
 	.ndo_change_mtu	     = xennet_change_mtu,
 	.ndo_set_mac_address = eth_mac_addr,
 	.ndo_validate_addr   = eth_validate_addr,
+#ifdef CONFIG_NET_POLL_CONTROLLER
+	.ndo_poll_controller = xennet_netpoll,
+#endif
 };
 
 static struct net_device * __devinit xennet_create_dev(struct xenbus_device *dev)

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pvops netconsole

[Jeremy Fitzhardinge]

On 02/13/2010 06:28 PM, Brad Plant wrote:
> On Fri, 12 Feb 2010 15:09:12 -0800
> Jeremy Fitzhardinge<jeremy@goop.org>  wrote:
>
>    
>> On 02/04/2010 04:15 PM, Brad Plant wrote:
>>      
>>> Initialising Xen virtual ethernet driver.
>>> blkfront: xvda1: barriers enabled
>>> netconsole: local port 11111
>>> netconsole: local IP 10.20.40.2
>>> netconsole: interface eth1
>>> netconsole: remote port 9514
>>> netconsole: remote IP 10.20.50.2
>>> netconsole: remote ethernet address 00:16:3e:00:50:02
>>> netconsole: eth1 doesn't support polling, aborting.
>>> netconsole: cleaning up
>>>
>>>        
>> The xen-netfront device doesn't support the polling interface that
>> netconsole relies on.  It probably wouldn't be too hard to add that
>> capability if you want to.  It's independent of the backend and/or the
>> actual underlying hardware device.
>>      
> Ok, I've come up with something that "works". Patch against 2.6.32.7 is attached. While it works, I've got no idea if what I've done is the "right way". I used virtio_net.c as a guide.
>
> I've got some issues that I'm stuck on and would like some guidance:
>
> 1. When the first message is sent over the net console, the WARN_ONCE() at net/core/netpoll.c:327 triggers. I'm not sure what to do about this.
>
>    

What's the backtrace in the warning?

I had a quick look at e1000's and it calls the interrupt handler; the 
netfront equivalent would be to call xennet_interrupt() with interrupts 
disabled.  I'm not sure virtio_net is a good model because it doesn't 
seem to have any interrupt-related stuff in it; I guess that happens 
elsewhere in virtio.

> 2. When either netconsole is setup or the first message is sent, messages already logged to the main console get logged again. Maybe this will go away when the above is fixed?
>    

Probably not.  It sounds like a feature.

> 3. Initially netconsole would not initialise because the interface didn't yet have a mac address as setup_netfront hadn't yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES&&  CONFIG_XEN. Is this allowed and sane?
>    

Probably not.  In practice every kernel will have modules and Xen 
enabled, and it will change netconsole for all users.  I guess we 
probably need to find some way to init the MAC address earlier, or 
trigger netconsole init once the mac address is known.
> Cheers,
>
> Brad
>    

Re: pvops netconsole

[Brad Plant]

[-- Attachment #1.1.1: Type: text/plain, Size: 4001 bytes --]

On Sat, 13 Feb 2010 19:56:51 -0800
Jeremy Fitzhardinge <jeremy@goop.org> wrote:

> > 1. When the first message is sent over the net console, the WARN_ONCE() at net/core/netpoll.c:327 triggers. I'm not sure what to do about this.
> 
> What's the backtrace in the warning?

------------[ cut here ]------------
WARNING: at net/core/netpoll.c:329 netpoll_send_skb+0x145/0x1db()
netpoll_send_skb(): eth1 enabled interrupts in poll (xennet_start_xmit+0x0/0x6bb)
Pid: 1, comm: swapper Not tainted 2.6.32.7 #36
Call Trace:
 [<ffffffff81039100>] warn_slowpath_common+0x72/0x8a
 [<ffffffff81039165>] warn_slowpath_fmt+0x3c/0x3e
 [<ffffffff8100dd7f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8124e2a0>] ? xennet_start_xmit+0x0/0x6bb
 [<ffffffff81294e48>] netpoll_send_skb+0x145/0x1db
 [<ffffffff813604d4>] ? _spin_unlock_irqrestore+0x19/0x1c
 [<ffffffff812950dc>] netpoll_send_udp+0x1fe/0x20e
 [<ffffffff8124004d>] ? firmware_loading_store+0x120/0x1b5
 [<ffffffff8124f9d6>] write_msg+0x89/0xca
 [<ffffffff8124004d>] ? firmware_loading_store+0x120/0x1b5
 [<ffffffff81039281>] __call_console_drivers+0x67/0x79
 [<ffffffff810392ec>] _call_console_drivers+0x59/0x5d
 [<ffffffff81039613>] release_console_sem+0x120/0x1c7
 [<ffffffff81039e73>] register_console+0x22e/0x2ae
 [<ffffffff81503380>] init_netconsole+0x15d/0x1c8
 [<ffffffff81049033>] ? schedule_delayed_work+0x16/0x1b
 [<ffffffff81503223>] ? init_netconsole+0x0/0x1c8
 [<ffffffff8100a05f>] do_one_initcall+0x59/0x154
 [<ffffffff814e56a9>] kernel_init+0x152/0x1a8
 [<ffffffff810112aa>] child_rip+0xa/0x20
 [<ffffffff810104a1>] ? int_ret_from_sys_call+0x7/0x1b
 [<ffffffff81010c5d>] ? retint_restore_args+0x5/0x6
 [<ffffffff810112a0>] ? child_rip+0x0/0x20
---[ end trace 0f898deb8e1e2914 ]---

 
> I had a quick look at e1000's and it calls the interrupt handler; the 
> netfront equivalent would be to call xennet_interrupt() with interrupts 
> disabled.  I'm not sure virtio_net is a good model because it doesn't 
> seem to have any interrupt-related stuff in it; I guess that happens 
> elsewhere in virtio.

Actually, I tried something like this first because I first looked at pcnet32.c and e100.c:

disable_irq(dev->irq);
xennet_interrupt(dev->irq, dev);
enable_irq(dev->irq);

The same WARN_ONCE() was triggered.


> > 2. When either netconsole is setup or the first message is sent, messages already logged to the main console get logged again. Maybe this will go away when the above is fixed?
> 
> Probably not.  It sounds like a feature.

Ok. This one was the least of my worries anyway.


> > 3. Initially netconsole would not initialise because the interface didn't yet have a mac address as setup_netfront hadn't yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES&&  CONFIG_XEN. Is this allowed and sane?
>
> Probably not.  In practice every kernel will have modules and Xen 
> enabled, and it will change netconsole for all users.  I guess we 
> probably need to find some way to init the MAC address earlier, or 
> trigger netconsole init once the mac address is known.

I figured it was dodgy. How would you init the MAC address earlier though? I've observed the xen devices being initialised at slightly different times with respect to other devices which suggests to me that they (and possibly all devices) are being initialised in separate threads. It would mean setting the MAC address before device initialisation yes?

Not sure how correct this is either, but another solution might be to set the MAC address to say 00:00:00:00:00:01 when the netdev structure is first created. This lets netconsole initialise since the MAC address isn't all zeros or all 0xFF. By the time any net console logs are transmitted, setup_netfront will have already been called and the correct MAC address set so no packets will actually be sent with the above MAC address. I have attached a patch that does this.

Cheers,

Brad

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.1.2: xen-netconsole2.patch --]
[-- Type: text/x-patch, Size: 2528 bytes --]

diff -urpNX linux-2.6.32.7/Documentation/dontdiff linux-2.6.32.7/arch/x86/boot/compressed/piggy.S linux-2.6.32.7-xen-netconsole/arch/x86/boot/compressed/piggy.S
--- linux-2.6.32.7/arch/x86/boot/compressed/piggy.S	2010-02-14 09:21:54.000000000 +1100
+++ linux-2.6.32.7-xen-netconsole/arch/x86/boot/compressed/piggy.S	2010-02-14 15:46:52.000000000 +1100
@@ -1,6 +1,6 @@
 .section ".rodata.compressed","a",@progbits
 .globl z_input_len
-z_input_len = 2406261
+z_input_len = 2406344
 .globl z_output_len
 z_output_len = 11827328
 .globl z_extract_offset
diff -urpNX linux-2.6.32.7/Documentation/dontdiff linux-2.6.32.7/arch/x86/boot/zoffset.h linux-2.6.32.7-xen-netconsole/arch/x86/boot/zoffset.h
--- linux-2.6.32.7/arch/x86/boot/zoffset.h	2010-02-14 09:21:54.000000000 +1100
+++ linux-2.6.32.7-xen-netconsole/arch/x86/boot/zoffset.h	2010-02-14 15:46:52.000000000 +1100
@@ -3,5 +3,5 @@
 #define ZO_startup_32 0x0000000000000000
 #define ZO_z_extract_offset 0x0000000000905000
 #define ZO_z_extract_offset_negative 0xffffffffff6fb000
-#define ZO_z_input_len 0x000000000024b775
+#define ZO_z_input_len 0x000000000024b7c8
 #define ZO_z_output_len 0x0000000000b47880
diff -urpNX linux-2.6.32.7/Documentation/dontdiff linux-2.6.32.7/drivers/net/xen-netfront.c linux-2.6.32.7-xen-netconsole/drivers/net/xen-netfront.c
--- linux-2.6.32.7/drivers/net/xen-netfront.c	2009-12-03 14:51:21.000000000 +1100
+++ linux-2.6.32.7-xen-netconsole/drivers/net/xen-netfront.c	2010-02-14 15:51:51.000000000 +1100
@@ -989,6 +989,15 @@ err:
 	return work_done;
 }
 
+#ifdef CONFIG_NET_POLL_CONTROLLER
+static void xennet_netpoll(struct net_device *dev)
+{
+	struct netfront_info *np = netdev_priv(dev);
+
+	napi_schedule(&np->napi);
+}
+#endif
+
 static int xennet_change_mtu(struct net_device *dev, int mtu)
 {
 	int max = xennet_can_sg(dev) ? 65535 - ETH_HLEN : ETH_DATA_LEN;
@@ -1113,6 +1122,9 @@ static const struct net_device_ops xenne
 	.ndo_change_mtu	     = xennet_change_mtu,
 	.ndo_set_mac_address = eth_mac_addr,
 	.ndo_validate_addr   = eth_validate_addr,
+#ifdef CONFIG_NET_POLL_CONTROLLER
+	.ndo_poll_controller = xennet_netpoll,
+#endif
 };
 
 static struct net_device * __devinit xennet_create_dev(struct xenbus_device *dev)
@@ -1181,6 +1193,9 @@ static struct net_device * __devinit xen
 
 	np->netdev = netdev;
 
+	/* Set MAC to temporary (non-zero) value so that netconsole initialises */
+	netdev->dev_addr[ETH_ALEN - 1] = 1;
+
 	netif_carrier_off(netdev);
 
 	return netdev;

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pvops netconsole

[Jeremy Fitzhardinge]

On 02/13/2010 08:56 PM, Brad Plant wrote:
> Actually, I tried something like this first because I first looked at pcnet32.c and e100.c:
>
> disable_irq(dev->irq);
> xennet_interrupt(dev->irq, dev);
> enable_irq(dev->irq);
>
> The same WARN_ONCE() was triggered.
>    

Given that netconsole works with e1000 (I presume; haven't tested it 
lately), I guess it needs a close look to see what differs from the 
netfront case.

>>> 3. Initially netconsole would not initialise because the interface didn't yet have a mac address as setup_netfront hadn't yet been called. I changed the module_init() in netconsole.c to late_initcall() if !CONFIG_MODULES&&   CONFIG_XEN. Is this allowed and sane?
>>>        
>> Probably not.  In practice every kernel will have modules and Xen
>> enabled, and it will change netconsole for all users.  I guess we
>> probably need to find some way to init the MAC address earlier, or
>> trigger netconsole init once the mac address is known.
>>      
> I figured it was dodgy. How would you init the MAC address earlier though? I've observed the xen devices being initialised at slightly different times with respect to other devices which suggests to me that they (and possibly all devices) are being initialised in separate threads. It would mean setting the MAC address before device initialisation yes?
>
> Not sure how correct this is either, but another solution might be to set the MAC address to say 00:00:00:00:00:01 when the netdev structure is first created. This lets netconsole initialise since the MAC address isn't all zeros or all 0xFF. By the time any net console logs are transmitted, setup_netfront will have already been called and the correct MAC address set so no packets will actually be sent with the above MAC address. I have attached a patch that does this.
>    

That seems a bit sketchy.

netfront is like a lot of network devices; it needs to query some 
external source to find its own mac address.  Mostly this means querying 
some eeprom over i2c or something, but in our case it means talking to 
xenstore to find the configured mac address.  I think the problem here 
is that this happens asynchronously, rather than synchronously in the 
net device's init routine.  One fix might be to synchronously wait for 
the mac address to be set before returning from the init function.

     J